Q43. Unknown Normal mean and variance: conditional versus marginal
[14 marks]
Suppose X_1,\ldots,X_n\mid\mu,v \overset{\mathrm{iid}}{\sim}N(\mu,v), \qquad p(\mu,v)\propto\frac1v, where \mu\in\mathbb R, v>0, n\ge2, and S=\sum_{i=1}^n(x_i-\bar x)^2>0. Use the inverse-Gamma convention f(v)\propto v^{-a-1}e^{-b/v}.
(a) Establish the identity \sum_{i=1}^n(x_i-\mu)^2=S+n(\mu-\bar x)^2. [2 marks]
(b) Derive the full conditional distributions \mu\mid v,{\boldsymbol{x}} and v\mid\mu,{\boldsymbol{x}}. [4 marks]
(c) Integrate out \mu to derive p(v\mid{\boldsymbol{x}}). Explain why it differs from p(v\mid\mu,{\boldsymbol{x}}). [4 marks]
(d) Describe a Gibbs sampler using part (b). A student instead samples v from its marginal posterior, then \mu conditional on that draw, repeating with fresh random numbers. Is this also a valid way to sample the joint posterior? Are its resulting pairs independent? [4 marks]
Definitions. Write v=\sigma^2, \bar x=n^{-1}\sum_i x_i, and S=\sum_i(x_i-\bar x)^2, the unnormalised sum of squares. We use N(m,v) with variance v and \operatorname{InvGamma}(a,b) with kernel v^{-a-1}e^{-b/v} on v>0. A full conditional fixes the other parameter; a marginal posterior integrates it out.
Assumptions. The Normal observations are conditionally iid, with prior p(\mu,v)\propto1/v relative to d\mu\,dv. Because this prior is improper, require n\ge2 and S>0 for the posterior below to be proper.
(a) Sum-of-squares identity. Write x_i-\mu=(x_i-\bar x)+(\bar x-\mu) and expand: \begin{aligned} \sum_{i=1}^n(x_i-\mu)^2 &=\sum_{i=1}^n(x_i-\bar x)^2 +2(\bar x-\mu)\sum_{i=1}^n(x_i-\bar x) +n(\bar x-\mu)^2\\ &=\boxed{S+n(\mu-\bar x)^2}, \end{aligned} because \sum_i(x_i-\bar x)=0.
(b) Full conditional distributions. The likelihood and joint posterior kernels are L(\mu,v;{\boldsymbol{x}})\propto v^{-n/2}\exp\left\{-\frac{S+n(\mu-\bar x)^2}{2v}\right\}, p(\mu,v\mid{\boldsymbol{x}})\propto v^{-n/2-1}\exp\left\{-\frac{S+n(\mu-\bar x)^2}{2v}\right\}, \qquad \mu\in\mathbb R,\ v>0. For fixed v, retain only factors involving \mu: p(\mu\mid v,{\boldsymbol{x}})\propto \exp\left\{-\frac{n(\mu-\bar x)^2}{2v}\right\}. Recognising a Normal kernel, \boxed{\mu\mid v,{\boldsymbol{x}}\sim N\!\left(\bar x,\frac vn\right).} For fixed \mu, retain all the v-dependent factors: p(v\mid\mu,{\boldsymbol{x}})\propto v^{-n/2-1}\exp\left\{-\frac{S+n(\mu-\bar x)^2}{2v}\right\}. Under the stated inverse-Gamma convention, \boxed{ v\mid\mu,{\boldsymbol{x}}\sim \operatorname{InvGamma}\!\left(\frac n2,\frac{S+n(\mu-\bar x)^2}{2}\right). } Its support is v>0. In this conditional distribution, \mu is held fixed; it has not been replaced by \bar x or integrated out.
(c) Marginal posterior for v. Integrate the joint posterior kernel over all \mu\in\mathbb R: \begin{aligned} p(v\mid{\boldsymbol{x}}) &\propto v^{-n/2-1}e^{-S/(2v)} \int_{-\infty}^{\infty} \exp\left\{-\frac{n(\mu-\bar x)^2}{2v}\right\}\,d\mu\\ &=v^{-n/2-1}e^{-S/(2v)}\sqrt{\frac{2\pi v}{n}}\\ &\propto v^{-(n-1)/2-1}e^{-S/(2v)}. \end{aligned} Therefore \boxed{v\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}\!\left(\frac{n-1}{2},\frac S2\right), \qquad v>0.} The assumptions n\ge2 and S>0 make both parameters positive, so the posterior is proper despite the improper prior.
The marginal distribution accounts for uncertainty in \mu by integrating over it. The integral contributes a factor v^{1/2}, changing the shape from n/2 to (n-1)/2. Even setting \mu=\bar x in the full conditional does not produce the marginal posterior, because it misses this factor.
(d) Gibbs sampling and direct joint sampling. Start from any \mu^{(0)}\in\mathbb R and v^{(0)}>0. At iteration t=1,\ldots,T, draw \mu^{(t)}\sim N\!\left(\bar x,\frac{v^{(t-1)}}n\right), v^{(t)}\sim\operatorname{InvGamma}\!\left( \frac n2,\frac{S+n(\mu^{(t)}-\bar x)^2}{2}\right). Each update uses the latest available value of the other parameter. The resulting Markov chain has the joint posterior as its equilibrium distribution; successive pairs are generally dependent, and convergence from the chosen starting value must be assessed.
The student’s alternative is also valid. Independently for each repetition, draw v^{(t)}\sim\operatorname{InvGamma}\!\left(\frac{n-1}{2},\frac S2\right), \qquad \mu^{(t)}\mid v^{(t)}\sim N\!\left(\bar x,\frac{v^{(t)}}n\right). This samples the factorisation p(\mu,v\mid{\boldsymbol{x}})=p(v\mid{\boldsymbol{x}})p(\mu\mid v,{\boldsymbol{x}}) directly. With fresh independent random numbers for each repetition, the pairs are iid joint-posterior draws, with no burn-in required. The two components within a pair are not independent: the conditional variance of \mu depends on v.