Normal–inverse-Gamma prior

Definition taught in Bayesian linear regression, part of Bayesian statistics

Normal–inverse-Gamma prior

Type	Definition
Taught in	Bayesian linear regression
Part	Bayesian statistics
Properties	4
Worked examples	2

Full joint conjugacy is obtained from the scaled prior {\boldsymbol{\beta}}\mid\sigma^2 \sim\operatorname{N}_p({\boldsymbol{m}}_0,\sigma^2{\color{blue}\boldsymbol{V}}_0), \qquad \sigma^2\sim\operatorname{InvGamma}(a_0,b_0).

Properties

Normal–inverse-Gamma conjugacy

Main article: Normal–inverse-Gamma conjugacy

{\boldsymbol{\beta}}\mid\sigma^2,{\boldsymbol{y}} \sim\operatorname{N}_p({\boldsymbol{m}}_n,\sigma^2{\color{blue}\boldsymbol{V}}_n), \qquad \sigma^2\mid{\boldsymbol{y}}\sim\operatorname{InvGamma}(a_n,b_n).

Conditional coefficient posterior

Main article: Conditional coefficient posterior

\boxed{ {\boldsymbol{\beta}}\mid\sigma^2,{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim\operatorname{N}_p\!\left( \widehat{\boldsymbol{\beta}}, \sigma^2({\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}})^{-1}\right).} To check the normalising factor, set {\color{blue}\boldsymbol{C}}=({\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}})^{-1}. Then

|\sigma^2{\color{blue}\boldsymbol{C}}|^{-1/2} =(\sigma^2)^{-p/2}|{\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}}|^{1/2},

and (\sigma^2{\color{blue}\boldsymbol{C}})^{-1} =\sigma^{-2}{\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}}, exactly matching Equation (18) in Bayesian linear regression.

Marginal variance posterior

Main article: Marginal variance posterior

\boxed{ \sigma^2\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim\operatorname{InvGamma}\!\left( \frac{n-p}{2},\frac{\operatorname{SSE}}2\right).} The p/2 from the normalising constant is subtracted from n/2; it is not added. Equivalently, \frac{\operatorname{SSE}}{\sigma^2}\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim\chi^2_{n-p}. If n>p+2, the posterior mean exists and is

\mathbb{E}(\sigma^2\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}}) =\frac{\operatorname{SSE}}{n-p-2} =\frac{n-p}{n-p-2}s^2.

The posterior mode is \operatorname{SSE}/(n-p+2). Thus the unbiased estimator s^2 is generally neither the posterior mean nor the posterior mode; calling any one quantity “the estimate” implicitly chooses a loss function.

Marginal Student posterior

Main article: Marginal Student posterior

\boxed{ {\boldsymbol{\beta}}\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim t_{n-p}\!\left( \widehat{\boldsymbol{\beta}}, s^2({\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}})^{-1}\right),} where the displayed matrix is the multivariate t scale matrix. In particular, \beta_j\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim t_{n-p}\!\left(\widehat\beta_j,s\sqrt{c_{jj}}\right), so an equal-tailed 100(1-\alpha)\% credible interval is \widehat\beta_j\ \pm\ t_{1-\alpha/2,n-p}s\sqrt{c_{jj}}. Its endpoints coincide with the usual classical interval under this reference prior. The interpretations do not coincide: the Bayesian statement concerns posterior uncertainty about \beta_j for the observed data, whereas the frequentist statement concerns repeated-sample coverage by a random interval.

Examples

Question 4. Jointly conjugate normal–inverse-gamma

Question page: Question 4. Jointly conjugate normal–inverse-gamma

Let X_i\mid\mu,\sigma^2\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2), with the jointly conjugate prior

\mu\mid\sigma^2\sim \operatorname{N}(\mu_0,\sigma^2/\kappa_0), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0),

where \kappa_0,\alpha_0,\beta_0>0. The inverse-gamma kernel is (\sigma^2)^{-\alpha_0-1}\exp(-\beta_0/\sigma^2).

(a) How does the jointly conjugate prior for (\mu,\sigma^2) differ from independent normal and inverse-gamma priors?

(b) What remains in the same family after updating, and what is the marginal posterior distribution of \mu?

(c) What has to be checked before an inverse-gamma moment, or a software default, is used?

(d) State the four updated posterior hyperparameters \kappa_n, \mu_n, \alpha_n, and \beta_n.

(e) State the joint posterior family and the conditional posterior of \mu\mid\sigma^2,{\boldsymbol{x}}.

(f) State the marginal posterior of \sigma^2.

(g) State the marginal posterior of \mu, including its degrees of freedom, location, and scale.

(h) Explain why integrating out \sigma^2 produces a Student-t distribution rather than a normal distribution.

(i) Interpret \kappa_0 and the weighted-average form of the posterior mean.

(j) Explain how this model differs from the independent-prior model in Question 3.

Worked answer (click to show)

Definitions. Let v=\sigma^2, S=\sum_i(x_i-\bar x)^2, and \kappa_0>0. The normal–inverse-gamma family is defined by the factorisation p(\mu,v)=p(v)p(\mu\mid v), with v\sim\operatorname{InvGamma}(\alpha_0,\beta_0) and \mu\mid v\sim\operatorname{N}(\mu_0,v/\kappa_0). A Student-t location–scale distribution t_\nu(m,s) means m+sT with T\sim t_\nu; s is a scale.

(a) The jointly conjugate prior scales the prior variance of the mean by the unknown variance:

\mu\mid\sigma^2\sim \operatorname{N}(\mu_0,\sigma^2/\kappa_0), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0).

The independent prior uses a fixed \sigma_0^2, not \sigma^2/\kappa_0.

Thus this prior already makes the uncertainty in \mu depend on v: larger observation variance brings larger conditional prior variance for the mean. Conjugacy concerns this dependent joint family. It does not mean that updating preserves independence.

(b) The joint posterior stays normal–inverse-gamma. The marginal posterior of \mu is a Student-t distribution.

(c) Match the density in use,

f(\sigma^2)\propto(\sigma^2)^{-\alpha-1}\exp(-\beta/\sigma^2),

before a moment or a software default is used. Packages do not all take (\alpha,\beta) in this shape–rate order.

(d) Let S=\sum_{i=1}^n(x_i-\bar x)^2. The updated hyperparameters are

\begin{aligned} \kappa_n&=\kappa_0+n,\\ \mu_n&=\frac{\kappa_0\mu_0+n\bar x}{\kappa_0+n},\\ \alpha_n&=\alpha_0+\frac n2,\\ \beta_n &= \beta_0+\frac12 S+\frac{\kappa_0 n}{2(\kappa_0+n)}(\bar x-\mu_0)^2. \end{aligned}

The update follows by combining the two quadratic penalties. The useful identity is

n(\mu-\bar x)^2+\kappa_0(\mu-\mu_0)^2 =\kappa_n(\mu-\mu_n)^2 +\frac{\kappa_0n}{\kappa_n}(\bar x-\mu_0)^2.

The last term records disagreement between the prior centre and the sample mean. This disagreement adds to the posterior scale for v, alongside the within-sample scatter S. Ignoring it would understate uncertainty when the prior centre and data differ.

(e) Multiplying the likelihood by the joint prior and using that identity gives

p(\mu,v\mid{\boldsymbol{x}})\propto v^{-(\alpha_n+3/2)} \exp\left\{-\frac{\beta_n+\kappa_n(\mu-\mu_n)^2/2}{v}\right\}.

This factors into a normal density in \mu\mid v and an inverse-gamma density in v, so the joint posterior remains normal–inverse-gamma. Conditionally,

\mu\mid\sigma^2,{\boldsymbol{x}} \sim \operatorname{N}\bigl(\mu_n,\sigma^2/\kappa_n\bigr).

(f) Integrating out \mu leaves the inverse-gamma factor,

\sigma^2\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}(\alpha_n,\beta_n).

The integral of the conditional normal density is one. Equivalently, integrating its unnormalised exponential kernel contributes \sqrt{2\pi v/\kappa_n}, cancelling the extra factor v^{-1/2} in the joint kernel. This is why the marginal shape is \alpha_n.

(g) The marginal posterior is Student-t, with 2\alpha_n degrees of freedom, location \mu_n, and scale \sqrt{\beta_n/(\alpha_n\kappa_n)}:

\mu\mid{\boldsymbol{x}} \sim t_{2\alpha_n}\!\left(\mu_n,\sqrt{\frac{\beta_n}{\alpha_n\kappa_n}}\right).

The second argument is a scale, not a standard deviation.

When \alpha_n>1, its variance is \beta_n/[\kappa_n(\alpha_n-1)], which is larger than the squared scale. The marginal mean equals \mu_n when 2\alpha_n>1; when this moment does not exist, \mu_n is still the distribution’s location and symmetry centre.

(h) Given \sigma^2, \mu is normal with variance proportional to \sigma^2. Mixing that normal over the inverse-gamma posterior of \sigma^2 gives the Student-t kernel

\left(1+\frac{\kappa_n(\mu-\mu_n)^2}{2\beta_n}\right)^{-(2\alpha_n+1)/2}.

Uncertainty in \sigma^2 produces heavier tails than a normal distribution with the same location and scale.

Indeed, the integral over v is proportional to [\beta_n+\kappa_n(\mu-\mu_n)^2/2]^{-(\alpha_n+1/2)}. Matching this with [1+(\mu-m)^2/(\nu s^2)]^{-(\nu+1)/2} gives \nu=2\alpha_n and s^2=\beta_n/(\alpha_n\kappa_n). A plug-in normal posterior at an estimated variance would not reproduce these tails.

(i) The weight \kappa_0 is the prior equivalent sample size: \mu_0 is weighted as \kappa_0 observations. Thus

\mu_n=\frac{\kappa_0\mu_0+n\bar x}{\kappa_0+n}

is the sample-size-weighted average of the prior location and the sample mean. The prior location is also its marginal mean when that mean exists.

(j) In the prior variance of \mu is a fixed \sigma_0^2, and the joint posterior does not stay normal–inverse-gamma. Here the prior variance is \sigma^2/\kappa_0, so the joint posterior stays in that family. Both marginals are then available in closed form: inverse-gamma for \sigma^2, and Student-t for \mu.

The distinction is visible in the conditional update: in the independent-prior model the relative weights depend on the current v, whereas here both precision contributions share the factor 1/v. That factor cancels from \mu_n, leaving fixed weights n and \kappa_0.

Question 5. Normal–inverse-\chi^2 regression

Question page: Question 5. Normal–inverse-\chi^2 regression

Suppose

y\mid\beta,\sigma^2,X\sim \operatorname{N}_n(X\beta,\sigma^2 I_n),

with prior

\beta\mid\sigma^2\sim \operatorname{N}_p(\beta_0,\sigma^2\Sigma_0), \qquad \sigma^2\sim\operatorname{Inv\text{-}\chi^2}(\mu_0,\sigma_0^2).

Here X is known, \Sigma_0 is positive definite, and \mu_0,\sigma_0^2>0. The inverse-\chi^2 convention is

\frac{\mu_0\sigma_0^2}{\sigma^2}\sim\chi^2_{\mu_0}.

(a) State the inverse-\chi^2 prior kernel and its equivalent inverse-gamma parameters.

(b) State the updated parameters \Sigma_n and \beta_n.

(c) State the updated parameters \mu_n and \sigma_n^2.

(d) Express the joint posterior through a conditional posterior for \beta and a marginal posterior for \sigma^2.

(e) Express the updated scale using a residual sum of squares and a prior penalty. Explain why it is positive.

(f) Is \Sigma_n the conditional covariance matrix of \beta? Is \sigma_n^2 the posterior mean of \sigma^2? Give the correct expressions and the posterior mode of \sigma^2.

(g) State the full conditional posterior of \sigma^2\mid\beta,y,X. How does it differ from its marginal posterior?

(h) Without consulting the updates, complete the square to reconstruct them. Explain why the marginal degrees of freedom are \mu_0+n.

Worked answer (click to show)

Definitions. There are n responses and p regression coefficients; X is an n\times p known design matrix. Write v=\sigma^2. The matrix \Sigma_0 scales the conditional prior covariance v\Sigma_0. In this question \mu_0 denotes prior degrees of freedom, not a normal mean. Our \operatorname{Inv\text{-}\chi^2}(\nu,s^2) convention means \nu s^2/v\sim\chi^2_\nu, or equivalently v\sim\operatorname{InvGamma}(\nu/2,\nu s^2/2).

(a) The transformation \mu_0\sigma_0^2/\sigma^2\sim\chi^2_{\mu_0} has kernel

\pi(\sigma^2) \propto (\sigma^2)^{-\mu_0/2-1} \exp\Bigl(-\frac{\mu_0\sigma_0^2}{2\sigma^2}\Bigr), \qquad\sigma^2>0.

Matching (\sigma^2)^{-a-1}\exp(-b/\sigma^2) gives a=\mu_0/2 and b=\mu_0\sigma_0^2/2.

For completeness, if U=\mu_0\sigma_0^2/v, then |dU/dv|=\mu_0\sigma_0^2/v^2. Multiplying the \chi^2_{\mu_0} density at U by this Jacobian gives the displayed inverse-gamma kernel. The Jacobian supplies the power v^{-2} and must be retained.

\boxed{ \sigma^2\sim\operatorname{InvGamma}\Bigl(\frac{\mu_0}{2},\frac{\mu_0\sigma_0^2}{2}\Bigr). }

(b) Since \Sigma_0^{-1} is positive definite and X^\top X is positive semidefinite, \Sigma_n^{-1} is positive definite. Precision addition gives

\boxed{\Sigma_n^{-1}=X^\top X+\Sigma_0^{-1},} \qquad \boxed{\beta_n=\Sigma_n(X^\top y+\Sigma_0^{-1}\beta_0).}

The data and prior precision contributions add, and their combined precision determines the posterior centre.

At fixed v, the likelihood contributes X^\top X/v to the precision of \beta, and the prior contributes \Sigma_0^{-1}/v. Their sum is \Sigma_n^{-1}/v. A proper positive-definite prior therefore gives a proper conditional update even if X is not of full column rank; an unregularised least-squares inverse (X^\top X)^{-1} is not needed.

(c)

\boxed{\mu_n=\mu_0+n,} \qquad \boxed{ \sigma_n^2 = \frac{ \mu_0\sigma_0^2+y^\top y+\beta_0^\top\Sigma_0^{-1}\beta_0 -\beta_n^\top\Sigma_n^{-1}\beta_n }{\mu_n}. }

(d)

\boxed{\beta\mid\sigma^2,y,X\sim \operatorname{N}_p(\beta_n,\sigma^2\Sigma_n),} \qquad \boxed{\sigma^2\mid y,X\sim\operatorname{Inv\text{-}\chi^2}(\mu_n,\sigma_n^2).}

Together these specify a normal–inverse-\chi^2 joint posterior.

This is a conditional–marginal factorisation. To draw an independent posterior pair, draw v from the marginal inverse-\chi^2 distribution and then \beta from its normal distribution conditional on that v. The components within a pair are dependent through the conditional covariance, even though different pairs can be generated independently.

(e) Equivalently,

\mu_n\sigma_n^2 = \mu_0\sigma_0^2 +(y-X\beta_n)^\top(y-X\beta_n) +(\beta_n-\beta_0)^\top\Sigma_0^{-1}(\beta_n-\beta_0).

Both quadratic forms are nonnegative and \mu_0\sigma_0^2>0, so \sigma_n^2>0. The algebra is completed in (h).

(f) No. The conditional covariance is \sigma^2\Sigma_n, not \Sigma_n:

\boxed{\operatorname{Var}(\beta\mid\sigma^2,y,X)=\sigma^2\Sigma_n.}

The parameter \sigma_n^2 is a scale. By the inverse-gamma equivalence in (a),

\boxed{ \mathbb{E}(\sigma^2\mid y,X) = \frac{\mu_n\sigma_n^2}{\mu_n-2}, \qquad\mu_n>2, }

and

\boxed{ \operatorname{mode}(\sigma^2\mid y,X) = \frac{\mu_n\sigma_n^2}{\mu_n+2}. }

(g)

\boxed{ \sigma^2\mid\beta,y,X \sim \operatorname{Inv\text{-}\chi^2}\!\left( \mu_n+p,\; \frac{\mu_n\sigma_n^2+(\beta-\beta_n)^\top\Sigma_n^{-1}(\beta-\beta_n)}{\mu_n+p} \right). }

The full conditional fixes \beta. The marginal posterior integrates \beta out, which removes p from the degrees of freedom.

The difference comes from the normal density’s factor v^{-p/2}. It remains when \beta is fixed and therefore adds p/2 to the inverse-gamma shape. Integrating over \beta instead contributes a factor v^{p/2}, cancelling it. Substituting \beta=\beta_n in the full conditional would leave the wrong degrees of freedom.

(h) Expand the likelihood quadratic and the prior quadratic:

\begin{aligned} &(y-X\beta)^\top(y-X\beta) +(\beta-\beta_0)^\top\Sigma_0^{-1}(\beta-\beta_0)\\ &\quad= \beta^\top(X^\top X+\Sigma_0^{-1})\beta -2\beta^\top(X^\top y+\Sigma_0^{-1}\beta_0) +y^\top y+\beta_0^\top\Sigma_0^{-1}\beta_0\\ &\quad= (\beta-\beta_n)^\top\Sigma_n^{-1}(\beta-\beta_n) +y^\top y+\beta_0^\top\Sigma_0^{-1}\beta_0 -\beta_n^\top\Sigma_n^{-1}\beta_n. \end{aligned}

Adding \mu_0\sigma_0^2, the terms free of \beta are the numerator of \mu_n\sigma_n^2 in (c), with \mu_n=\mu_0+n. Thus

\boxed{ Q(\beta) = (\beta-\beta_n)^\top\Sigma_n^{-1}(\beta-\beta_n) +\mu_n\sigma_n^2. }

The joint posterior kernel is

(\sigma^2)^{-(\mu_0+n+p)/2-1} \exp\Bigl(-\frac{Q(\beta)}{2\sigma^2}\Bigr).

The integral over \beta of the normal kernel equals (2\pi\sigma^2)^{p/2}|\Sigma_n|^{1/2}. The remaining power of \sigma^2 is

-\frac{\mu_0+n+p}{2}-1+\frac p2 = -\frac{\mu_0+n}{2}-1.

\therefore \text{the marginal inverse-}\chi^2\text{ degrees of freedom are }\mu_0+n.

Used in

• Taught in Bayesian linear regression, where the derivation and examples are.
• All 8 questions on Bayesian linear regression

See also

Generic exponential-family conjugate update · Gaussian linear model · Error and residual · Ordinary least squares · Sampling distribution of the OLS estimator · Studentised coefficient pivot · Least-squares decomposition · Conditional coefficient posterior

Glossary and index