Question 4. Jointly conjugate normal–inverse-gamma

Topic: Conjugate analysis for normal models · more questions on this topic
‹ Question 3 Question 7 ›

Let X_i\mid\mu,\sigma^2\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2), with the jointly conjugate prior

\mu\mid\sigma^2\sim \operatorname{N}(\mu_0,\sigma^2/\kappa_0), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0),

where \kappa_0,\alpha_0,\beta_0>0. The inverse-gamma kernel is (\sigma^2)^{-\alpha_0-1}\exp(-\beta_0/\sigma^2).

(a) How does the jointly conjugate prior for (\mu,\sigma^2) differ from independent normal and inverse-gamma priors?

(b) What remains in the same family after updating, and what is the marginal posterior distribution of \mu?

(c) What has to be checked before an inverse-gamma moment, or a software default, is used?

(d) State the four updated posterior hyperparameters \kappa_n, \mu_n, \alpha_n, and \beta_n.

(e) State the joint posterior family and the conditional posterior of \mu\mid\sigma^2,{\boldsymbol{x}}.

(f) State the marginal posterior of \sigma^2.

(g) State the marginal posterior of \mu, including its degrees of freedom, location, and scale.

(h) Explain why integrating out \sigma^2 produces a Student-t distribution rather than a normal distribution.

(i) Interpret \kappa_0 and the weighted-average form of the posterior mean.

(j) Explain how this model differs from the independent-prior model in Question 3.

Worked answer (click to show)

Definitions. Let v=\sigma^2, S=\sum_i(x_i-\bar x)^2, and \kappa_0>0. The normal–inverse-gamma family is defined by the factorisation p(\mu,v)=p(v)p(\mu\mid v), with v\sim\operatorname{InvGamma}(\alpha_0,\beta_0) and \mu\mid v\sim\operatorname{N}(\mu_0,v/\kappa_0). A Student-t location–scale distribution t_\nu(m,s) means m+sT with T\sim t_\nu; s is a scale.

(a) The jointly conjugate prior scales the prior variance of the mean by the unknown variance:

\mu\mid\sigma^2\sim \operatorname{N}(\mu_0,\sigma^2/\kappa_0), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0).

The independent prior uses a fixed \sigma_0^2, not \sigma^2/\kappa_0.

Thus this prior already makes the uncertainty in \mu depend on v: larger observation variance brings larger conditional prior variance for the mean. Conjugacy concerns this dependent joint family. It does not mean that updating preserves independence.

(b) The joint posterior stays normal–inverse-gamma. The marginal posterior of \mu is a Student-t distribution.

(c) Match the density in use,

f(\sigma^2)\propto(\sigma^2)^{-\alpha-1}\exp(-\beta/\sigma^2),

before a moment or a software default is used. Packages do not all take (\alpha,\beta) in this shape–rate order.

(d) Let S=\sum_{i=1}^n(x_i-\bar x)^2. The updated hyperparameters are

\begin{aligned} \kappa_n&=\kappa_0+n,\\ \mu_n&=\frac{\kappa_0\mu_0+n\bar x}{\kappa_0+n},\\ \alpha_n&=\alpha_0+\frac n2,\\ \beta_n &= \beta_0+\frac12 S+\frac{\kappa_0 n}{2(\kappa_0+n)}(\bar x-\mu_0)^2. \end{aligned}

The update follows by combining the two quadratic penalties. The useful identity is

n(\mu-\bar x)^2+\kappa_0(\mu-\mu_0)^2 =\kappa_n(\mu-\mu_n)^2 +\frac{\kappa_0n}{\kappa_n}(\bar x-\mu_0)^2.

The last term records disagreement between the prior centre and the sample mean. This disagreement adds to the posterior scale for v, alongside the within-sample scatter S. Ignoring it would understate uncertainty when the prior centre and data differ.

(e) Multiplying the likelihood by the joint prior and using that identity gives

p(\mu,v\mid{\boldsymbol{x}})\propto v^{-(\alpha_n+3/2)} \exp\left\{-\frac{\beta_n+\kappa_n(\mu-\mu_n)^2/2}{v}\right\}.

This factors into a normal density in \mu\mid v and an inverse-gamma density in v, so the joint posterior remains normal–inverse-gamma. Conditionally,

\mu\mid\sigma^2,{\boldsymbol{x}} \sim \operatorname{N}\bigl(\mu_n,\sigma^2/\kappa_n\bigr).

(f) Integrating out \mu leaves the inverse-gamma factor,

\sigma^2\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}(\alpha_n,\beta_n).

The integral of the conditional normal density is one. Equivalently, integrating its unnormalised exponential kernel contributes \sqrt{2\pi v/\kappa_n}, cancelling the extra factor v^{-1/2} in the joint kernel. This is why the marginal shape is \alpha_n.

(g) The marginal posterior is Student-t, with 2\alpha_n degrees of freedom, location \mu_n, and scale \sqrt{\beta_n/(\alpha_n\kappa_n)}:

\mu\mid{\boldsymbol{x}} \sim t_{2\alpha_n}\!\left(\mu_n,\sqrt{\frac{\beta_n}{\alpha_n\kappa_n}}\right).

The second argument is a scale, not a standard deviation.

When \alpha_n>1, its variance is \beta_n/[\kappa_n(\alpha_n-1)], which is larger than the squared scale. The marginal mean equals \mu_n when 2\alpha_n>1; when this moment does not exist, \mu_n is still the distribution’s location and symmetry centre.

(h) Given \sigma^2, \mu is normal with variance proportional to \sigma^2. Mixing that normal over the inverse-gamma posterior of \sigma^2 gives the Student-t kernel

\left(1+\frac{\kappa_n(\mu-\mu_n)^2}{2\beta_n}\right)^{-(2\alpha_n+1)/2}.

Uncertainty in \sigma^2 produces heavier tails than a normal distribution with the same location and scale.

Indeed, the integral over v is proportional to [\beta_n+\kappa_n(\mu-\mu_n)^2/2]^{-(\alpha_n+1/2)}. Matching this with [1+(\mu-m)^2/(\nu s^2)]^{-(\nu+1)/2} gives \nu=2\alpha_n and s^2=\beta_n/(\alpha_n\kappa_n). A plug-in normal posterior at an estimated variance would not reproduce these tails.

(i) The weight \kappa_0 is the prior equivalent sample size: \mu_0 is weighted as \kappa_0 observations. Thus

\mu_n=\frac{\kappa_0\mu_0+n\bar x}{\kappa_0+n}

is the sample-size-weighted average of the prior location and the sample mean. The prior location is also its marginal mean when that mean exists.

(j) In the prior variance of \mu is a fixed \sigma_0^2, and the joint posterior does not stay normal–inverse-gamma. Here the prior variance is \sigma^2/\kappa_0, so the joint posterior stays in that family. Both marginals are then available in closed form: inverse-gamma for \sigma^2, and Student-t for \mu.

The distinction is visible in the conditional update: in the independent-prior model the relative weights depend on the current v, whereas here both precision contributions share the factor 1/v. That factor cancels from \mu_n, leaving fixed weights n and \kappa_0.