Proper prior and proper posterior

Definition taught in Bayesian foundations, part of Bayesian statistics

Proper prior and proper posterior

Type Definition
Taught in Bayesian foundations
Part Bayesian statistics
Properties 1
Worked examples 2
Questions using it 4

A prior is proper if it is a probability distribution, so its total mass is one. A posterior is proper if the product of likelihood and prior has a finite, nonzero normalising constant. An improper prior may sometimes produce a proper posterior, but this must be checked rather than assumed.

Properties

A proper approximation to the flat prior

Main article: A proper approximation to the flat prior

The honest bounded version of the improper kernel p(\mu)\propto1 is

p_C(\mu)=\frac1{2C}\mathbf{1}\{-C\leq\mu\leq C\}.

Taking, for example, C=10^4 makes this density nearly constant over an ordinary data range while keeping it proper. It also exposes what “very diffuse” means. For any fixed 0<M<C,

\mathbb{P}_C(|\mu|\leq M)=\int_{-M}^{M}\frac1{2C}\,\mathrm{d}\mu=\frac MC \longrightarrow0 \qquad(C\longrightarrow\infty).

Thus the limiting sequence puts essentially all of its probability beyond every fixed finite range; it does not converge to a uniform probability distribution on \mathbb{R}. Human height makes the point concrete: no observed person is three metres tall, yet a sufficiently broad uniform prior assigns almost no mass to the entire range of heights that have actually been encountered.

Examples

Question 12. Improper prior for a normal sample

Question page: Question 12. Improper prior for a normal sample

The prior and the sample are

f(\mu,\sigma^{2})\propto 1/\sigma^{2}, \qquad x_1,\ldots,x_n\mid\mu,\sigma^{2}\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^{2}),

with n\geq 2.

(i) Derive the joint posterior kernel of (\mu,\sigma^{2}). [2 marks]

(ii) Show that \sum_{i=1}^{n}(x_i-\mu)^{2}=\sum_{i=1}^{n}(x_i-\bar x)^{2}+n(\mu-\bar x)^{2}. [2 marks]

(iii) Show that \mu\mid\sigma^{2},{\boldsymbol{x}} is normal, and give its mean and variance. [2 marks]

(iv) Show that \sigma^{2}\mid\mu,{\boldsymbol{x}} is inverse-gamma, and give its shape and rate. [2 marks]

(v) Describe how to sample from the joint posterior. [4 marks]

(vi) Describe 90\% posterior credible intervals for \mu and for \sigma^{2}. [2 marks]

Definitions. Write v=\sigma^2, \bar x=n^{-1}\sum_i x_i and S=\sum_i(x_i-\bar x)^2, the unnormalised sum of squares. The prior p(\mu,v)\propto1/v is a density kernel relative to d\mu\,dv, and \operatorname{InvGamma}(a,b) has kernel v^{-a-1}e^{-b/v}. Take the observed sample to be nonconstant (S>0), as required for a proper posterior with this improper prior.

(i) The likelihood contributes (\sigma^{2})^{-n/2}\exp\bigl(-\sum(x_i-\mu)^{2}/(2\sigma^{2})\bigr), and the prior contributes \sigma^{-2}. For \sigma^{2}>0,

f(\mu,\sigma^{2}\mid{\boldsymbol{x}}) \propto (\sigma^{2})^{-n/2-1} \exp\Bigl(-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(x_i-\mu)^{2}\Bigr).

(ii)

\sum_{i=1}^{n}(x_i-\mu)^{2} = \sum_{i=1}^{n}(x_i-\bar x+\bar x-\mu)^{2} = \sum_{i=1}^{n}(x_i-\bar x)^{2}+n(\mu-\bar x)^{2},

because \sum(x_i-\bar x)=0.

(iii) Let S=\sum(x_i-\bar x)^{2}. By (ii), the kernel in \mu is \exp\bigl(-n(\mu-\bar x)^{2}/(2\sigma^{2})\bigr). Thus

\mu\mid\sigma^{2},{\boldsymbol{x}}\sim \operatorname{N}(\bar x,\,\sigma^{2}/n).

(iv) The kernel in \sigma^{2} is

(\sigma^{2})^{-(n/2)-1} \exp\Bigl(-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(x_i-\mu)^{2}\Bigr).

\boxed{\sigma^{2}\mid\mu,{\boldsymbol{x}} \sim \operatorname{InvGamma}\Bigl(\frac n2,\;\frac12\sum_{i=1}^{n}(x_i-\mu)^{2}\Bigr).}

(v) The two full conditionals give a valid Gibbs sampler, but direct independent draws are available. Integrate the joint kernel over \mu. By (ii), the normal integral contributes \sqrt{2\pi v/n}, so

\begin{aligned} p(v\mid{\boldsymbol{x}}) &\propto v^{-n/2-1}e^{-S/(2v)} \int_{-\infty}^{\infty}e^{-n(\mu-\bar x)^2/(2v)}\,d\mu\\ &\propto v^{-(n-1)/2-1}e^{-S/(2v)}. \end{aligned}

Thus v\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}((n-1)/2,S/2) when S>0. For each independent posterior pair, draw v from this marginal and then \mu\mid v,{\boldsymbol{x}}\sim\operatorname{N}(\bar x,v/n). This samples p(\mu,v\mid{\boldsymbol{x}})=p(v\mid{\boldsymbol{x}})p(\mu\mid v,{\boldsymbol{x}}) exactly, with no burn-in. The pairs are independent across draws, though \mu and v within a pair are dependent. The marginal shape (n-1)/2 differs from the full-conditional shape n/2 because integrating over \mu contributes v^{1/2}.

If using Gibbs instead, use the two full conditionals:

These are dependent Markov-chain draws, so convergence must be assessed. Setting \mu=\bar x in the variance full conditional would not give the marginal variance posterior: it would still have shape n/2 and would miss the integration factor above.

(vi) Let F_{\mathrm{IG}} be the CDF of \operatorname{InvGamma}((n-1)/2,S/2). A central 90\% credible interval for v=\sigma^2 is [F_{\mathrm{IG}}^{-1}(0.05),F_{\mathrm{IG}}^{-1}(0.95)]. Integrating v out gives \mu\mid{\boldsymbol{x}} a Student-t distribution with n-1 degrees of freedom, centre \bar x and scale \sqrt{S/[n(n-1)]}. Hence its central 90\% interval is

\boxed{\bar x\ \pm\ t_{0.95,n-1}\sqrt{\frac{S}{n(n-1)}}.}

Empirical 5\% and 95\% quantiles of the direct draws are an alternative approximation to these intervals.

Q43. Unknown Normal mean and variance: conditional versus marginal

Question page: Q43. Unknown Normal mean and variance: conditional versus marginal

[14 marks]

Suppose X_1,\ldots,X_n\mid\mu,v \overset{\mathrm{iid}}{\sim}N(\mu,v), \qquad p(\mu,v)\propto\frac1v, where \mu\in\mathbb R, v>0, n\ge2, and S=\sum_{i=1}^n(x_i-\bar x)^2>0. Use the inverse-Gamma convention f(v)\propto v^{-a-1}e^{-b/v}.

(a) Establish the identity \sum_{i=1}^n(x_i-\mu)^2=S+n(\mu-\bar x)^2. [2 marks]

(b) Derive the full conditional distributions \mu\mid v,{\boldsymbol{x}} and v\mid\mu,{\boldsymbol{x}}. [4 marks]

(c) Integrate out \mu to derive p(v\mid{\boldsymbol{x}}). Explain why it differs from p(v\mid\mu,{\boldsymbol{x}}). [4 marks]

(d) Describe a Gibbs sampler using part (b). A student instead samples v from its marginal posterior, then \mu conditional on that draw, repeating with fresh random numbers. Is this also a valid way to sample the joint posterior? Are its resulting pairs independent? [4 marks]

Definitions. Write v=\sigma^2, \bar x=n^{-1}\sum_i x_i, and S=\sum_i(x_i-\bar x)^2, the unnormalised sum of squares. We use N(m,v) with variance v and \operatorname{InvGamma}(a,b) with kernel v^{-a-1}e^{-b/v} on v>0. A full conditional fixes the other parameter; a marginal posterior integrates it out.

Assumptions. The Normal observations are conditionally iid, with prior p(\mu,v)\propto1/v relative to d\mu\,dv. Because this prior is improper, require n\ge2 and S>0 for the posterior below to be proper.

(a) Sum-of-squares identity. Write x_i-\mu=(x_i-\bar x)+(\bar x-\mu) and expand: \begin{aligned} \sum_{i=1}^n(x_i-\mu)^2 &=\sum_{i=1}^n(x_i-\bar x)^2 +2(\bar x-\mu)\sum_{i=1}^n(x_i-\bar x) +n(\bar x-\mu)^2\\ &=\boxed{S+n(\mu-\bar x)^2}, \end{aligned} because \sum_i(x_i-\bar x)=0.

(b) Full conditional distributions. The likelihood and joint posterior kernels are L(\mu,v;{\boldsymbol{x}})\propto v^{-n/2}\exp\left\{-\frac{S+n(\mu-\bar x)^2}{2v}\right\}, p(\mu,v\mid{\boldsymbol{x}})\propto v^{-n/2-1}\exp\left\{-\frac{S+n(\mu-\bar x)^2}{2v}\right\}, \qquad \mu\in\mathbb R,\ v>0. For fixed v, retain only factors involving \mu: p(\mu\mid v,{\boldsymbol{x}})\propto \exp\left\{-\frac{n(\mu-\bar x)^2}{2v}\right\}. Recognising a Normal kernel, \boxed{\mu\mid v,{\boldsymbol{x}}\sim N\!\left(\bar x,\frac vn\right).} For fixed \mu, retain all the v-dependent factors: p(v\mid\mu,{\boldsymbol{x}})\propto v^{-n/2-1}\exp\left\{-\frac{S+n(\mu-\bar x)^2}{2v}\right\}. Under the stated inverse-Gamma convention, \boxed{ v\mid\mu,{\boldsymbol{x}}\sim \operatorname{InvGamma}\!\left(\frac n2,\frac{S+n(\mu-\bar x)^2}{2}\right). } Its support is v>0. In this conditional distribution, \mu is held fixed; it has not been replaced by \bar x or integrated out.

(c) Marginal posterior for v. Integrate the joint posterior kernel over all \mu\in\mathbb R: \begin{aligned} p(v\mid{\boldsymbol{x}}) &\propto v^{-n/2-1}e^{-S/(2v)} \int_{-\infty}^{\infty} \exp\left\{-\frac{n(\mu-\bar x)^2}{2v}\right\}\,d\mu\\ &=v^{-n/2-1}e^{-S/(2v)}\sqrt{\frac{2\pi v}{n}}\\ &\propto v^{-(n-1)/2-1}e^{-S/(2v)}. \end{aligned} Therefore \boxed{v\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}\!\left(\frac{n-1}{2},\frac S2\right), \qquad v>0.} The assumptions n\ge2 and S>0 make both parameters positive, so the posterior is proper despite the improper prior.

The marginal distribution accounts for uncertainty in \mu by integrating over it. The integral contributes a factor v^{1/2}, changing the shape from n/2 to (n-1)/2. Even setting \mu=\bar x in the full conditional does not produce the marginal posterior, because it misses this factor.

(d) Gibbs sampling and direct joint sampling. Start from any \mu^{(0)}\in\mathbb R and v^{(0)}>0. At iteration t=1,\ldots,T, draw \mu^{(t)}\sim N\!\left(\bar x,\frac{v^{(t-1)}}n\right), v^{(t)}\sim\operatorname{InvGamma}\!\left( \frac n2,\frac{S+n(\mu^{(t)}-\bar x)^2}{2}\right). Each update uses the latest available value of the other parameter. The resulting Markov chain has the joint posterior as its equilibrium distribution; successive pairs are generally dependent, and convergence from the chosen starting value must be assessed.

The student’s alternative is also valid. Independently for each repetition, draw v^{(t)}\sim\operatorname{InvGamma}\!\left(\frac{n-1}{2},\frac S2\right), \qquad \mu^{(t)}\mid v^{(t)}\sim N\!\left(\bar x,\frac{v^{(t)}}n\right). This samples the factorisation p(\mu,v\mid{\boldsymbol{x}})=p(v\mid{\boldsymbol{x}})p(\mu\mid v,{\boldsymbol{x}}) directly. With fresh independent random numbers for each repetition, the pairs are iid joint-posterior draws, with no burn-in required. The two components within a pair are not independent: the conditional variance of \mu depends on v.

Used in

See also

Bayesian credible interval · Bayes’ rule · Posterior point summaries · Credible-region constructions · Posterior summaries from draws · R’s distribution interface · Prior and posterior predictive distributions · Predictive-variance decomposition

Glossary and index