Normal–inverse-Gamma variance update

Proposition taught in Conjugate analysis for normal models, part of Bayesian statistics

Normal–inverse-Gamma variance update

Type Proposition
Taught in Conjugate analysis for normal models
Part Bayesian statistics
Worked examples 2

\boxed{ \sigma^2\mid{\boldsymbol{x}} \sim\operatorname{InvGamma}\!\left( \alpha_0+\frac n2, \beta_0+\frac12S_\mu \right)}.

Examples

Question 2. Normal variance, known mean

Question page: Question 2. Normal variance, known mean

Let

X_i\mid\sigma^2\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2),

with \mu known and

\sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0),

in the shape–rate parameterisation

f(\sigma^2)\propto(\sigma^2)^{-\alpha_0-1}\exp(-\beta_0/\sigma^2).

(a) Write the posterior distribution of \sigma^2, and state what is added to each parameter.

(b) State the posterior mean and mode of \sigma^2, including the conditions under which each exists.

(c) Explain why that mean and mode generally differ.

(d) Explain why the variance update uses squared deviations from the known mean \mu, rather than from \bar x.

Definitions. Put v=\sigma^2 and Q_\mu=\sum_i(x_i-\mu)^2, with \mu known. Our \operatorname{InvGamma}(a,b) convention has density b^a v^{-a-1}e^{-b/v}/\Gamma(a) for v>0. Equivalently, 1/v\sim\operatorname{Gamma}(a,b) in shape–rate form. Specifying the kernel avoids ambiguity about the second parameter.

(a) Independence gives the likelihood kernel L(v)\propto v^{-n/2}e^{-Q_\mu/(2v)}. Multiplication by the prior gives

p(v\mid{\boldsymbol{x}})\propto v^{-n/2}e^{-Q_\mu/(2v)}v^{-\alpha_0-1}e^{-\beta_0/v} =v^{-(\alpha_0+n/2)-1}e^{-(\beta_0+Q_\mu/2)/v}.

Let \alpha_n=\alpha_0+n/2 and \beta_n=\beta_0+Q_\mu/2. Recognising this kernel,

\boxed{ \sigma^2\mid{\boldsymbol{x}} \sim \operatorname{InvGamma}(\alpha_n,\beta_n). }

Each observation adds 1/2 to the shape. Half the sum of squared deviations from the known mean is added to the rate.

(b) For this shape–rate kernel, and for \beta_n>0,

\mathbb{E}(\sigma^2\mid{\boldsymbol{x}})=\frac{\beta_n}{\alpha_n-1} \quad\text{when }\alpha_n>1, \qquad \operatorname{mode}(\sigma^2\mid{\boldsymbol{x}})=\frac{\beta_n}{\alpha_n+1} \quad\text{when }\alpha_n>0.

The mode solves (-\alpha_n-1)/v+\beta_n/v^2=0. The mean is the gamma integral of v\cdot v^{-\alpha_n-1}e^{-\beta_n/v}, which converges only for \alpha_n>1.

More explicitly, substituting u=\beta_n/v into the expectation gives \mathbb{E}(v\mid{\boldsymbol{x}})=\beta_n\Gamma(\alpha_n-1)/\Gamma(\alpha_n)=\beta_n/(\alpha_n-1). When 0<\alpha_n\le1, the posterior is still a proper distribution, but its right tail is too heavy to have a finite mean. Posterior propriety and existence of a posterior moment are separate requirements.

(c) For \alpha_n>1,

\frac{\beta_n}{\alpha_n-1}-\frac{\beta_n}{\alpha_n+1} = \frac{2\beta_n}{(\alpha_n-1)(\alpha_n+1)}>0.

The inverse-gamma density is right-skewed, so the long right tail pulls the mean above the mode.

(d) Under the sampling model, \operatorname{Var}(X_i\mid\sigma^2)=\mathbb{E}[(X_i-\mu)^2\mid\sigma^2]. The sufficient statistic for \sigma^2, with \mu known, is \sum(x_i-\mu)^2. Since

\sum_{i=1}^n(x_i-\bar x)^2 = \sum_{i=1}^n(x_i-\mu)^2-n(\bar x-\mu)^2,

centring at \bar x omits the term n(\bar x-\mu)^2.

When the mean is known, departures of the sample mean from that value are evidence about the observation variance too. For example, a tightly clustered sample far from the known mean has small \sum_i(x_i-\bar x)^2 but large Q_\mu. Replacing Q_\mu by the centred sum of squares would discard that evidence and change the likelihood.

Q28. Lognormal variance and conjugate inference

Question page: Q28. Lognormal variance and conjugate inference

[20 marks]

The size X of a certain type of particle can be modelled by a lognormal distribution with density f(x\mid\mu,\sigma^2) =\frac1{x\sigma\sqrt{2\pi}} \exp\!\left\{-\frac{(\log x-\mu)^2}{2\sigma^2}\right\}, \qquad x>0. Note that if X\sim\operatorname{logN}(\mu,\sigma^2), then Y=\log X\sim N(\mu,\sigma^2). Assume that \mu is known and \sigma^2>0 is unknown. For the posterior calculations, let X_1,\ldots,X_n be conditionally iid observations from this model.

(i) [4 marks] Show that this lognormal model belongs to the exponential family.

(ii) [4 marks] Find the form of a conjugate prior for \sigma^2 and identify the distribution.

(iii) [5 marks] Find the posterior distribution of \sigma^2 given the observed data {\boldsymbol{x}}=(x_1,\ldots,x_n).

(iv) [4 marks] State, without derivation, the posterior mean and posterior mode. Explain their behaviour as n\to\infty, with the prior fixed.

(v) [3 marks] Briefly describe how you would construct a 90% credible interval for \sigma^2.

Taking logs converts this to a Normal model with known mean and unknown variance.

Definitions. Write v=\sigma^2 and S=\sum_i(\log x_i-\mu)^2. The log observations have Normal mean \mu and variance v; \mu is known. An \operatorname{InvGamma}(a,b) density has kernel v^{-a-1}e^{-b/v} on v>0. Throughout, Gamma distributions of the precision 1/v use shape–rate parameters. The positive observations are conditionally iid; prior propriety and moment conditions are stated below.

(i) Exponential-family form. The density can be written as f(x\mid v) =\frac1{x\sqrt{2\pi}} \frac{\exp\!\left\{-\dfrac1v\, \dfrac{(\log x-\mu)^2}{2}\right\}}{\sqrt v}, \qquad x>0,\quad v>0. Comparing with the form f(x\mid v)=b(x)\frac{\exp\{c(v)T(x)\}}{a(v)}, we identify \boxed{ \begin{gathered} b(x)=\frac1{x\sqrt{2\pi}},\qquad T(x)=\frac{(\log x-\mu)^2}{2},\\ c(v)=-\frac1v,\qquad a(v)=\sqrt v. \end{gathered}} Here \mu is known, so T(x) involves no unknown parameter. The support x>0 also does not depend on v. This establishes the required one-parameter exponential-family representation.

(ii) Conjugate prior. For this representation, the general conjugate prior has kernel \pi(v)\propto\frac{\exp\{\eta c(v)\}}{a(v)^\kappa} =v^{-\kappa/2}\exp\!\left(-\frac\eta v\right),\qquad v>0. Compare this with the inverse-Gamma kernel \pi(v)\propto v^{-a_0-1}e^{-b_0/v}. Matching the power and exponential term gives a_0=\frac\kappa2-1,\qquad b_0=\eta. Thus a convenient parameterisation of the conjugate prior is \boxed{\sigma^2\sim\operatorname{InvGamma}(a_0,b_0),\qquad a_0,b_0>0,} with normalised density \pi(v)=\frac{b_0^{a_0}}{\Gamma(a_0)} v^{-a_0-1}e^{-b_0/v},\qquad v>0. In the general conjugacy parameters, propriety requires \eta>0 and \kappa>2. The prior is on the variance \sigma^2, not on the standard deviation \sigma.

(iii) Posterior distribution. Define the sum of squared deviations on the log scale, S=\sum_{i=1}^{n}(\log x_i-\mu)^2. The likelihood is \begin{aligned} L(v;{\boldsymbol{x}}) &=\prod_{i=1}^{n}\frac1{x_i\sqrt{2\pi v}} \exp\!\left\{-\frac{(\log x_i-\mu)^2}{2v}\right\}\\ &=(2\pi)^{-n/2}\left(\prod_{i=1}^{n}\frac1{x_i}\right) v^{-n/2}\exp\!\left(-\frac{S}{2v}\right)\\ &\propto v^{-n/2}\exp\!\left(-\frac{S}{2v}\right). \end{aligned} The factors omitted in the last line do not depend on v. Multiplying by the inverse-Gamma prior, \begin{aligned} \pi(v\mid{\boldsymbol{x}}) &\propto v^{-n/2}e^{-S/(2v)}v^{-a_0-1}e^{-b_0/v}\\ &=v^{-(a_0+n/2)-1} \exp\!\left\{-\frac{b_0+S/2}{v}\right\}. \end{aligned} Let A_n=a_0+\frac n2,\qquad B_n=b_0+\frac S2. Recognising the inverse-Gamma kernel gives \boxed{\sigma^2\mid{\boldsymbol{x}}\sim\operatorname{InvGamma}(A_n,B_n),\qquad\sigma^2>0.} Its normalised density is \pi(v\mid{\boldsymbol{x}})=\frac{B_n^{A_n}}{\Gamma(A_n)} v^{-A_n-1}e^{-B_n/v},\qquad v>0.

(iv) Posterior mean, mode, and large-sample behaviour. The posterior summaries are \boxed{\mathbb{E}(\sigma^2\mid{\boldsymbol{x}})=\frac{B_n}{A_n-1} =\frac{b_0+S/2}{a_0+n/2-1},\qquad A_n>1,} \boxed{\widehat{\sigma^2}_{\mathrm{MAP}}=\frac{B_n}{A_n+1} =\frac{b_0+S/2}{a_0+n/2+1}.} If A_n\leq1, the posterior mean is not finite; the mode exists for all A_n,B_n>0.

For fixed prior parameters and data generated at true variance \sigma_0^2, the law of large numbers gives \frac{S_n}{n}=\frac1n\sum_{i=1}^{n}(\log X_i-\mu)^2 \longrightarrow\sigma_0^2. Both posterior summaries approach S_n/n and hence \sigma_0^2: the fixed prior’s influence vanishes. Since \mu is known, S_n/n is the usual likelihood estimate; there is no need to estimate \mu or replace it by the sample log-mean.

(v) A 90% credible interval for the variance. One valid choice is an equal-tailed interval. Let F_{\mathrm{IG}} be the CDF of the posterior \operatorname{InvGamma}(A_n,B_n). Choose its 5% and 95% quantiles: \boxed{\left[F_{\mathrm{IG}}^{-1}(0.05), F_{\mathrm{IG}}^{-1}(0.95)\right].} This leaves posterior probability 0.05 below the interval and 0.05 above it, so the interval contains 0.90 posterior probability.

Equivalently, use the reciprocal precision W=\frac1{\sigma^2},\qquad W\mid{\boldsymbol{x}}\sim\operatorname{Gamma}(A_n,B_n) \quad\text{in shape--rate form}. If q_{0.05} and q_{0.95} are the corresponding Gamma quantiles, then \mathbb{P}(q_{0.05}\leq W\leq q_{0.95}\mid{\boldsymbol{x}})=0.90. Taking reciprocals reverses the inequalities, giving \boxed{\left[\frac1{q_{0.95}},\frac1{q_{0.05}}\right]} as the same 90% credible interval for \sigma^2. The quantiles can be computed numerically, or approximated by the empirical 5% and 95% quantiles of a large sample drawn from the inverse-Gamma posterior.

This interval is for the unknown variance, not for a new particle size. It is an equal-tailed interval; it need not be the shortest or highest-posterior-density interval.

Key result. Take logs, recognise a normal model with known mean, and update a_0\mapsto a_0+n/2 and b_0\mapsto b_0+\tfrac12\sum_i(\log x_i-\mu)^2.

Used in

See also

Normal–Normal model · A proper approximation to the flat prior · Normal–Normal conjugacy · Precision weighting · Inverse-Gamma distribution · Full conditionals under independent priors

Glossary and index