Question 5. Normal–inverse-\chi^2 regression
Suppose
y\mid\beta,\sigma^2,X\sim \operatorname{N}_n(X\beta,\sigma^2 I_n),
with prior
\beta\mid\sigma^2\sim \operatorname{N}_p(\beta_0,\sigma^2\Sigma_0), \qquad \sigma^2\sim\operatorname{Inv\text{-}\chi^2}(\mu_0,\sigma_0^2).
Here X is known, \Sigma_0 is positive definite, and \mu_0,\sigma_0^2>0. The inverse-\chi^2 convention is
\frac{\mu_0\sigma_0^2}{\sigma^2}\sim\chi^2_{\mu_0}.
(a) State the inverse-\chi^2 prior kernel and its equivalent inverse-gamma parameters.
(b) State the updated parameters \Sigma_n and \beta_n.
(c) State the updated parameters \mu_n and \sigma_n^2.
(d) Express the joint posterior through a conditional posterior for \beta and a marginal posterior for \sigma^2.
(e) Express the updated scale using a residual sum of squares and a prior penalty. Explain why it is positive.
(f) Is \Sigma_n the conditional covariance matrix of \beta? Is \sigma_n^2 the posterior mean of \sigma^2? Give the correct expressions and the posterior mode of \sigma^2.
(g) State the full conditional posterior of \sigma^2\mid\beta,y,X. How does it differ from its marginal posterior?
(h) Without consulting the updates, complete the square to reconstruct them. Explain why the marginal degrees of freedom are \mu_0+n.
Definitions. There are n responses and p regression coefficients; X is an n\times p known design matrix. Write v=\sigma^2. The matrix \Sigma_0 scales the conditional prior covariance v\Sigma_0. In this question \mu_0 denotes prior degrees of freedom, not a normal mean. Our \operatorname{Inv\text{-}\chi^2}(\nu,s^2) convention means \nu s^2/v\sim\chi^2_\nu, or equivalently v\sim\operatorname{InvGamma}(\nu/2,\nu s^2/2).
(a) The transformation \mu_0\sigma_0^2/\sigma^2\sim\chi^2_{\mu_0} has kernel
\pi(\sigma^2) \propto (\sigma^2)^{-\mu_0/2-1} \exp\Bigl(-\frac{\mu_0\sigma_0^2}{2\sigma^2}\Bigr), \qquad\sigma^2>0.
Matching (\sigma^2)^{-a-1}\exp(-b/\sigma^2) gives a=\mu_0/2 and b=\mu_0\sigma_0^2/2.
For completeness, if U=\mu_0\sigma_0^2/v, then |dU/dv|=\mu_0\sigma_0^2/v^2. Multiplying the \chi^2_{\mu_0} density at U by this Jacobian gives the displayed inverse-gamma kernel. The Jacobian supplies the power v^{-2} and must be retained.
\boxed{ \sigma^2\sim\operatorname{InvGamma}\Bigl(\frac{\mu_0}{2},\frac{\mu_0\sigma_0^2}{2}\Bigr). }
(b) Since \Sigma_0^{-1} is positive definite and X^\top X is positive semidefinite, \Sigma_n^{-1} is positive definite. Precision addition gives
\boxed{\Sigma_n^{-1}=X^\top X+\Sigma_0^{-1},} \qquad \boxed{\beta_n=\Sigma_n(X^\top y+\Sigma_0^{-1}\beta_0).}
The data and prior precision contributions add, and their combined precision determines the posterior centre.
At fixed v, the likelihood contributes X^\top X/v to the precision of \beta, and the prior contributes \Sigma_0^{-1}/v. Their sum is \Sigma_n^{-1}/v. A proper positive-definite prior therefore gives a proper conditional update even if X is not of full column rank; an unregularised least-squares inverse (X^\top X)^{-1} is not needed.
(c)
\boxed{\mu_n=\mu_0+n,} \qquad \boxed{ \sigma_n^2 = \frac{ \mu_0\sigma_0^2+y^\top y+\beta_0^\top\Sigma_0^{-1}\beta_0 -\beta_n^\top\Sigma_n^{-1}\beta_n }{\mu_n}. }
(d)
\boxed{\beta\mid\sigma^2,y,X\sim \operatorname{N}_p(\beta_n,\sigma^2\Sigma_n),} \qquad \boxed{\sigma^2\mid y,X\sim\operatorname{Inv\text{-}\chi^2}(\mu_n,\sigma_n^2).}
Together these specify a normal–inverse-\chi^2 joint posterior.
This is a conditional–marginal factorisation. To draw an independent posterior pair, draw v from the marginal inverse-\chi^2 distribution and then \beta from its normal distribution conditional on that v. The components within a pair are dependent through the conditional covariance, even though different pairs can be generated independently.
(e) Equivalently,
\mu_n\sigma_n^2 = \mu_0\sigma_0^2 +(y-X\beta_n)^\top(y-X\beta_n) +(\beta_n-\beta_0)^\top\Sigma_0^{-1}(\beta_n-\beta_0).
Both quadratic forms are nonnegative and \mu_0\sigma_0^2>0, so \sigma_n^2>0. The algebra is completed in (h).
(f) No. The conditional covariance is \sigma^2\Sigma_n, not \Sigma_n:
\boxed{\operatorname{Var}(\beta\mid\sigma^2,y,X)=\sigma^2\Sigma_n.}
The parameter \sigma_n^2 is a scale. By the inverse-gamma equivalence in (a),
\boxed{ \mathbb{E}(\sigma^2\mid y,X) = \frac{\mu_n\sigma_n^2}{\mu_n-2}, \qquad\mu_n>2, }
and
\boxed{ \operatorname{mode}(\sigma^2\mid y,X) = \frac{\mu_n\sigma_n^2}{\mu_n+2}. }
(g)
\boxed{ \sigma^2\mid\beta,y,X \sim \operatorname{Inv\text{-}\chi^2}\!\left( \mu_n+p,\; \frac{\mu_n\sigma_n^2+(\beta-\beta_n)^\top\Sigma_n^{-1}(\beta-\beta_n)}{\mu_n+p} \right). }
The full conditional fixes \beta. The marginal posterior integrates \beta out, which removes p from the degrees of freedom.
The difference comes from the normal density’s factor v^{-p/2}. It remains when \beta is fixed and therefore adds p/2 to the inverse-gamma shape. Integrating over \beta instead contributes a factor v^{p/2}, cancelling it. Substituting \beta=\beta_n in the full conditional would leave the wrong degrees of freedom.
(h) Expand the likelihood quadratic and the prior quadratic:
\begin{aligned} &(y-X\beta)^\top(y-X\beta) +(\beta-\beta_0)^\top\Sigma_0^{-1}(\beta-\beta_0)\\ &\quad= \beta^\top(X^\top X+\Sigma_0^{-1})\beta -2\beta^\top(X^\top y+\Sigma_0^{-1}\beta_0) +y^\top y+\beta_0^\top\Sigma_0^{-1}\beta_0\\ &\quad= (\beta-\beta_n)^\top\Sigma_n^{-1}(\beta-\beta_n) +y^\top y+\beta_0^\top\Sigma_0^{-1}\beta_0 -\beta_n^\top\Sigma_n^{-1}\beta_n. \end{aligned}
Adding \mu_0\sigma_0^2, the terms free of \beta are the numerator of \mu_n\sigma_n^2 in (c), with \mu_n=\mu_0+n. Thus
\boxed{ Q(\beta) = (\beta-\beta_n)^\top\Sigma_n^{-1}(\beta-\beta_n) +\mu_n\sigma_n^2. }
The joint posterior kernel is
(\sigma^2)^{-(\mu_0+n+p)/2-1} \exp\Bigl(-\frac{Q(\beta)}{2\sigma^2}\Bigr).
The integral over \beta of the normal kernel equals (2\pi\sigma^2)^{p/2}|\Sigma_n|^{1/2}. The remaining power of \sigma^2 is
-\frac{\mu_0+n+p}{2}-1+\frac p2 = -\frac{\mu_0+n}{2}-1.
\therefore \text{the marginal inverse-}\chi^2\text{ degrees of freedom are }\mu_0+n.