Inverse-Gamma distribution

Definition taught in Conjugate analysis for normal models, part of Bayesian statistics · Also on the distribution sheet

Inverse-Gamma distribution

Notation x\sim\operatorname{InvGamma}(\alpha,\beta)
Parameters \alpha,\beta>0
Support x>0
Density p(x)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}x^{-\alpha-1}e^{-\beta/x}
Mean \mathbb{E}(x)=\beta/(\alpha-1) for \alpha>1
Mode \beta/(\alpha+1)
Variance \operatorname{Var}(x)=\beta^{2}/((\alpha-1)^{2}(\alpha-2)) for \alpha>2
Conjugate prior for X_i\mid\sigma^2\sim\operatorname{N}(\mu,\sigma^2), giving \operatorname{InvGamma}(\alpha+n/2,\beta+S_\mu/2)
Taught in Conjugate analysis for normal models
Part Bayesian statistics
Properties 3
Worked examples 2
Questions using it 19

The mode is \beta/(\alpha+1), which is not the variance.

From the distribution sheet; mode and conjugate pairs from the formula lookup.

In the shape–rate convention,

\sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0),

means that \sigma^2>0 has density

p(\sigma^2) =\frac{\beta_0^{\alpha_0}}{\Gamma(\alpha_0)} (\sigma^2)^{-\alpha_0-1} \exp\!\left(-\frac{\beta_0}{\sigma^2}\right).

Properties

Normal–inverse-Gamma variance update

Main article: Normal–inverse-Gamma variance update

\boxed{ \sigma^2\mid{\boldsymbol{x}} \sim\operatorname{InvGamma}\!\left( \alpha_0+\frac n2, \beta_0+\frac12S_\mu \right)}.

Normal–inverse-Gamma conjugacy

Main article: Normal–inverse-Gamma conjugacy

{\boldsymbol{\beta}}\mid\sigma^2,{\boldsymbol{y}} \sim\operatorname{N}_p({\boldsymbol{m}}_n,\sigma^2{\color{blue}\boldsymbol{V}}_n), \qquad \sigma^2\mid{\boldsymbol{y}}\sim\operatorname{InvGamma}(a_n,b_n).

Marginal variance posterior

Main article: Marginal variance posterior

\boxed{ \sigma^2\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim\operatorname{InvGamma}\!\left( \frac{n-p}{2},\frac{\operatorname{SSE}}2\right).} The p/2 from the normalising constant is subtracted from n/2; it is not added. Equivalently, \frac{\operatorname{SSE}}{\sigma^2}\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim\chi^2_{n-p}. If n>p+2, the posterior mean exists and is

\mathbb{E}(\sigma^2\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}}) =\frac{\operatorname{SSE}}{n-p-2} =\frac{n-p}{n-p-2}s^2.

The posterior mode is \operatorname{SSE}/(n-p+2). Thus the unbiased estimator s^2 is generally neither the posterior mean nor the posterior mode; calling any one quantity “the estimate” implicitly chooses a loss function.

Examples

Question 2. Normal variance, known mean

Question page: Question 2. Normal variance, known mean

Let

X_i\mid\sigma^2\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2),

with \mu known and

\sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0),

in the shape–rate parameterisation

f(\sigma^2)\propto(\sigma^2)^{-\alpha_0-1}\exp(-\beta_0/\sigma^2).

(a) Write the posterior distribution of \sigma^2, and state what is added to each parameter.

(b) State the posterior mean and mode of \sigma^2, including the conditions under which each exists.

(c) Explain why that mean and mode generally differ.

(d) Explain why the variance update uses squared deviations from the known mean \mu, rather than from \bar x.

Definitions. Put v=\sigma^2 and Q_\mu=\sum_i(x_i-\mu)^2, with \mu known. Our \operatorname{InvGamma}(a,b) convention has density b^a v^{-a-1}e^{-b/v}/\Gamma(a) for v>0. Equivalently, 1/v\sim\operatorname{Gamma}(a,b) in shape–rate form. Specifying the kernel avoids ambiguity about the second parameter.

(a) Independence gives the likelihood kernel L(v)\propto v^{-n/2}e^{-Q_\mu/(2v)}. Multiplication by the prior gives

p(v\mid{\boldsymbol{x}})\propto v^{-n/2}e^{-Q_\mu/(2v)}v^{-\alpha_0-1}e^{-\beta_0/v} =v^{-(\alpha_0+n/2)-1}e^{-(\beta_0+Q_\mu/2)/v}.

Let \alpha_n=\alpha_0+n/2 and \beta_n=\beta_0+Q_\mu/2. Recognising this kernel,

\boxed{ \sigma^2\mid{\boldsymbol{x}} \sim \operatorname{InvGamma}(\alpha_n,\beta_n). }

Each observation adds 1/2 to the shape. Half the sum of squared deviations from the known mean is added to the rate.

(b) For this shape–rate kernel, and for \beta_n>0,

\mathbb{E}(\sigma^2\mid{\boldsymbol{x}})=\frac{\beta_n}{\alpha_n-1} \quad\text{when }\alpha_n>1, \qquad \operatorname{mode}(\sigma^2\mid{\boldsymbol{x}})=\frac{\beta_n}{\alpha_n+1} \quad\text{when }\alpha_n>0.

The mode solves (-\alpha_n-1)/v+\beta_n/v^2=0. The mean is the gamma integral of v\cdot v^{-\alpha_n-1}e^{-\beta_n/v}, which converges only for \alpha_n>1.

More explicitly, substituting u=\beta_n/v into the expectation gives \mathbb{E}(v\mid{\boldsymbol{x}})=\beta_n\Gamma(\alpha_n-1)/\Gamma(\alpha_n)=\beta_n/(\alpha_n-1). When 0<\alpha_n\le1, the posterior is still a proper distribution, but its right tail is too heavy to have a finite mean. Posterior propriety and existence of a posterior moment are separate requirements.

(c) For \alpha_n>1,

\frac{\beta_n}{\alpha_n-1}-\frac{\beta_n}{\alpha_n+1} = \frac{2\beta_n}{(\alpha_n-1)(\alpha_n+1)}>0.

The inverse-gamma density is right-skewed, so the long right tail pulls the mean above the mode.

(d) Under the sampling model, \operatorname{Var}(X_i\mid\sigma^2)=\mathbb{E}[(X_i-\mu)^2\mid\sigma^2]. The sufficient statistic for \sigma^2, with \mu known, is \sum(x_i-\mu)^2. Since

\sum_{i=1}^n(x_i-\bar x)^2 = \sum_{i=1}^n(x_i-\mu)^2-n(\bar x-\mu)^2,

centring at \bar x omits the term n(\bar x-\mu)^2.

When the mean is known, departures of the sample mean from that value are evidence about the observation variance too. For example, a tightly clustered sample far from the known mean has small \sum_i(x_i-\bar x)^2 but large Q_\mu. Replacing Q_\mu by the centred sum of squares would discard that evidence and change the likelihood.

Question 14. Inverse-gamma waiting time

Question page: Question 14. Inverse-gamma waiting time

The waiting time until the mth animal at location i has density

f(x_i\mid\theta) = \frac{1}{\Gamma(m)\,\theta^{m}}x_i^{m-1}e^{-x_i/\theta}, \quad x_i>0,

so \theta>0 is a scale. The prior is

f(\theta) = \frac{\beta_0^{\alpha_0}}{\Gamma(\alpha_0)}\theta^{-\alpha_0-1}e^{-\beta_0/\theta}, \quad\alpha_0>0,\ \beta_0>0.

The exam states that the maximum likelihood estimator is \hat\theta=\bar x/m.

(i) State the prior mean. [1 mark]

(ii) Find and identify the posterior distribution of \theta. [5 marks]

(iii) State the posterior mean, and show that it is a weighted sum of the prior mean and \hat\theta. [5 marks]

(iv) What happens to the posterior mean as m\to\infty? [2 marks]

(v) What happens if \alpha_0,\beta_0\to 0? [2 marks]

(vi) Complete: “The effect of the prior hyperparameters \alpha_0 and \beta_0 on the posterior mean of \theta is like…” [2 marks]

(vii) Describe a posterior 100(1-\alpha)\% credible interval for \theta. [5 marks]

Definitions. Here m is a known gamma shape and \theta is its unknown scale: the sampling kernel in \theta is \theta^{-m}\exp(-x/\theta). This differs from a gamma rate parameterisation. Write T=\sum_i x_i and use \operatorname{InvGamma}(\alpha_0,\beta_0) with kernel \theta^{-\alpha_0-1}e^{-\beta_0/\theta}. The estimator \hat\theta=T/(nm) has the same units as \theta.

The sampling density is gamma with shape m and scale \theta, so \mathbb{E}(X_i\mid\theta)=m\theta and the supplied estimator \hat\theta=\bar x/m is the maximum likelihood estimator.

(i) For \alpha_0>1,

\mathbb{E}(\theta)=\beta_0/(\alpha_0-1).

(ii) The likelihood contributes

\theta^{-nm}\exp\bigl(-(\textstyle\sum_i x_i)/\theta\bigr).

The prior contributes

\theta^{-\alpha_0-1}\exp(-\beta_0/\theta).

Hence

\boxed{\theta\mid{\boldsymbol{x}} \sim\operatorname{InvGamma}(\alpha_0+nm,\;\beta_0+\textstyle\sum_i x_i).}

(iii) For \alpha_0+nm>1,

\mathbb{E}(\theta\mid{\boldsymbol{x}}) = \frac{\beta_0+\sum x_i}{\alpha_0+nm-1} = \frac{(\alpha_0-1)\,\mathbb{E}(\theta)+nm\,\hat\theta}{(\alpha_0-1)+nm}.

Writing the mean explicitly as a convex combination gives

\mathbb{E}(\theta\mid{\boldsymbol{x}}) =\underbrace{\frac{\alpha_0-1}{\alpha_0+nm-1}}_{w_0}\mathbb{E}(\theta) +\underbrace{\frac{nm}{\alpha_0+nm-1}}_{w_n}\hat\theta.

For \alpha_0>1, both weights are nonnegative and w_0+w_n=1. The data contribution is nm, the total number of captured animals across the n locations.

(iv) With n and the prior fixed, m\to\infty makes w_0\to0 and w_n\to1. If \hat\theta=\bar x/m remains bounded, as it does in probability under the stated sampling model at a fixed true scale, the posterior mean approaches \hat\theta.

(v) Taking \alpha_0,\beta_0\to0 in the posterior expression gives \sum_i x_i/(nm-1)=\{nm/(nm-1)\}\hat\theta, provided nm>1. It is close to, but not exactly, the MLE. This is a formal weak-prior limit: the proper inverse-gamma prior’s mean does not exist along the path once \alpha_0\leq1, so the weighted-average formula involving that mean cannot be used to take this limit.

(vi) When \alpha_0>1, the hyperparameters act like prior information that pulls the MLE toward the prior mean \beta_0/(\alpha_0-1), with weight w_0. In the posterior-mean ratio, \alpha_0-1 plays a pseudo-count role beside the observed nm, and \beta_0 plays a prior total-waiting-time role beside \sum_i x_i. These need not represent literal extra animals.

(vii) Let a=\alpha_0+nm and b=\beta_0+\sum x_i. The central credible interval is the \alpha/2 and 1-\alpha/2 quantiles of \operatorname{InvGamma}(a,b).

Equivalently, let q_u be the u-quantile of \operatorname{Gamma}(a,b) in shape–rate form. Since 1/\theta has this gamma posterior, the interval is [1/q_{1-\alpha/2},1/q_{\alpha/2}]. The reversal of endpoints follows because taking reciprocals is decreasing. Conditional on the data and model, the interval contains posterior probability 1-\alpha; it describes uncertainty about the common scale, not the waiting time of a future animal.

Used in

See also

Three easy confusions · Normal–Normal model · A proper approximation to the flat prior · Normal–Normal conjugacy · Precision weighting · Normal–inverse-Gamma variance update · Full conditionals under independent priors

Glossary and index