Question 12. Improper prior for a normal sample
The prior and the sample are
f(\mu,\sigma^{2})\propto 1/\sigma^{2}, \qquad x_1,\ldots,x_n\mid\mu,\sigma^{2}\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^{2}),
with n\geq 2.
(i) Derive the joint posterior kernel of (\mu,\sigma^{2}). [2 marks]
(ii) Show that \sum_{i=1}^{n}(x_i-\mu)^{2}=\sum_{i=1}^{n}(x_i-\bar x)^{2}+n(\mu-\bar x)^{2}. [2 marks]
(iii) Show that \mu\mid\sigma^{2},{\boldsymbol{x}} is normal, and give its mean and variance. [2 marks]
(iv) Show that \sigma^{2}\mid\mu,{\boldsymbol{x}} is inverse-gamma, and give its shape and rate. [2 marks]
(v) Describe how to sample from the joint posterior. [4 marks]
(vi) Describe 90\% posterior credible intervals for \mu and for \sigma^{2}. [2 marks]
Definitions. Write v=\sigma^2, \bar x=n^{-1}\sum_i x_i and S=\sum_i(x_i-\bar x)^2, the unnormalised sum of squares. The prior p(\mu,v)\propto1/v is a density kernel relative to d\mu\,dv, and \operatorname{InvGamma}(a,b) has kernel v^{-a-1}e^{-b/v}. Take the observed sample to be nonconstant (S>0), as required for a proper posterior with this improper prior.
(i) The likelihood contributes (\sigma^{2})^{-n/2}\exp\bigl(-\sum(x_i-\mu)^{2}/(2\sigma^{2})\bigr), and the prior contributes \sigma^{-2}. For \sigma^{2}>0,
f(\mu,\sigma^{2}\mid{\boldsymbol{x}}) \propto (\sigma^{2})^{-n/2-1} \exp\Bigl(-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(x_i-\mu)^{2}\Bigr).
(ii)
\sum_{i=1}^{n}(x_i-\mu)^{2} = \sum_{i=1}^{n}(x_i-\bar x+\bar x-\mu)^{2} = \sum_{i=1}^{n}(x_i-\bar x)^{2}+n(\mu-\bar x)^{2},
because \sum(x_i-\bar x)=0.
(iii) Let S=\sum(x_i-\bar x)^{2}. By (ii), the kernel in \mu is \exp\bigl(-n(\mu-\bar x)^{2}/(2\sigma^{2})\bigr). Thus
\mu\mid\sigma^{2},{\boldsymbol{x}}\sim \operatorname{N}(\bar x,\,\sigma^{2}/n).
(iv) The kernel in \sigma^{2} is
(\sigma^{2})^{-(n/2)-1} \exp\Bigl(-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(x_i-\mu)^{2}\Bigr).
\boxed{\sigma^{2}\mid\mu,{\boldsymbol{x}} \sim \operatorname{InvGamma}\Bigl(\frac n2,\;\frac12\sum_{i=1}^{n}(x_i-\mu)^{2}\Bigr).}
(v) The two full conditionals give a valid Gibbs sampler, but direct independent draws are available. Integrate the joint kernel over \mu. By (ii), the normal integral contributes \sqrt{2\pi v/n}, so
\begin{aligned} p(v\mid{\boldsymbol{x}}) &\propto v^{-n/2-1}e^{-S/(2v)} \int_{-\infty}^{\infty}e^{-n(\mu-\bar x)^2/(2v)}\,d\mu\\ &\propto v^{-(n-1)/2-1}e^{-S/(2v)}. \end{aligned}
Thus v\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}((n-1)/2,S/2) when S>0. For each independent posterior pair, draw v from this marginal and then \mu\mid v,{\boldsymbol{x}}\sim\operatorname{N}(\bar x,v/n). This samples p(\mu,v\mid{\boldsymbol{x}})=p(v\mid{\boldsymbol{x}})p(\mu\mid v,{\boldsymbol{x}}) exactly, with no burn-in. The pairs are independent across draws, though \mu and v within a pair are dependent. The marginal shape (n-1)/2 differs from the full-conditional shape n/2 because integrating over \mu contributes v^{1/2}.
If using Gibbs instead, use the two full conditionals:
These are dependent Markov-chain draws, so convergence must be assessed. Setting \mu=\bar x in the variance full conditional would not give the marginal variance posterior: it would still have shape n/2 and would miss the integration factor above.
(vi) Let F_{\mathrm{IG}} be the CDF of \operatorname{InvGamma}((n-1)/2,S/2). A central 90\% credible interval for v=\sigma^2 is [F_{\mathrm{IG}}^{-1}(0.05),F_{\mathrm{IG}}^{-1}(0.95)]. Integrating v out gives \mu\mid{\boldsymbol{x}} a Student-t distribution with n-1 degrees of freedom, centre \bar x and scale \sqrt{S/[n(n-1)]}. Hence its central 90\% interval is
\boxed{\bar x\ \pm\ t_{0.95,n-1}\sqrt{\frac{S}{n(n-1)}}.}
Empirical 5\% and 95\% quantiles of the direct draws are an alternative approximation to these intervals.