Normal–Normal model

Definition taught in Conjugate analysis for normal models, part of Bayesian statistics

Normal–Normal model

Type Definition
Taught in Conjugate analysis for normal models
Part Bayesian statistics
Properties 2
Worked examples 2

Assume

X_1,\ldots,X_n\mid\mu \stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2),

where \sigma^2 is known, and use the prior

\mu\sim\operatorname{N}(\mu_0,\sigma_0^2).

Properties

Normal–Normal conjugacy

Main article: Normal–Normal conjugacy

\boxed{ \mu\mid{\boldsymbol{x}}\sim\operatorname{N}(\mu_1,\sigma_1^2)}, with \begin{aligned} \sigma_1^2 &=\frac1A =\left(\frac n{\sigma^2}+\frac1{\sigma_0^2}\right)^{-1} =\frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2}, \\ \mu_1 &=\frac BA =\frac{\sum_i x_i/\sigma^2+\mu_0/\sigma_0^2} {n/\sigma^2+1/\sigma_0^2}\\ &=\frac{n\overline x\,\sigma_0^2+\mu_0\sigma^2} {n\sigma_0^2+\sigma^2}. \end{aligned}

Precision weighting

Main article: Precision weighting

Writing

w=\frac{\sigma^2}{n\sigma_0^2+\sigma^2}

turns Equation (2) in Conjugate analysis for normal models into

\mu_1=(1-w)\overline x+w\mu_0.

The posterior mean is therefore a precision-weighted average. Equivalently, if \kappa_0=\sigma^2/\sigma_0^2, then

\mu_1=\frac{n\overline x+\kappa_0\mu_0}{n+\kappa_0}.

Thus the prior behaves like \kappa_0 observations whose mean is \mu_0. In the special case \sigma_0^2=\sigma^2, it behaves like one extra observation at \mu_0.

Examples

Question 1. Normal mean, known variance

Question page: Question 1. Normal mean, known variance

Let

X_i\mid\mu\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2),

with \sigma^2 known and \mu\sim \operatorname{N}(\mu_0,\sigma_0^2).

(a) Using only the precision-addition rule and the precision-weighted mean, write the two updated parameters of the posterior for \mu.

(b) Define the prior equivalent sample size, and state what it counts.

(c) Explain why the posterior mean is a precision-weighted average of the prior mean and the sample mean.

(d) State and interpret the limiting posterior mean and variance as n\to\infty, with the prior parameters fixed and \bar X_n\to\mu^{\ast}.

(e) State the posterior predictive distribution of one new observation X_{\mathrm{new}}, conditionally independent of the sample given \mu.

(f) Explain why the posterior predictive variance is larger than the posterior variance of \mu.

Definitions. Write \bar x=n^{-1}\sum_i x_i. A normal distribution \operatorname{N}(m,v) has mean m and variance v; its precision is 1/v. Here \sigma^2 is the known observation variance, \sigma_0^2 is the prior variance of the unknown mean, and \sigma_1^2 will be its posterior variance. These are variances of different objects.

(a) The data precision of n observations is n/\sigma^2, and the prior precision is 1/\sigma_0^2. Precision addition gives

\frac1{\sigma_1^2} = \frac n{\sigma^2}+\frac1{\sigma_0^2}.

The posterior mean is the precision-weighted average of \bar x and \mu_0:

\mu_1 = \frac{(n/\sigma^2)\bar x+(1/\sigma_0^2)\mu_0}{n/\sigma^2+1/\sigma_0^2} = \frac{n\bar x\,\sigma_0^2+\mu_0\sigma^2}{n\sigma_0^2+\sigma^2}.

\boxed{\mu\mid x\sim \operatorname{N}(\mu_1,\sigma_1^2).}

(b)

\kappa_0=\frac{\sigma^2}{\sigma_0^2}.

It counts how many observations of variance \sigma^2 have the same precision as the prior, since 1/\sigma_0^2=\kappa_0/\sigma^2.

Consequently, \mu_1=(n\bar x+\kappa_0\mu_0)/(n+\kappa_0) and \sigma_1^2=\sigma^2/(n+\kappa_0). The equivalent sample size measures information about the common mean. It need not be an integer or represent actual previous observations.

(c) Up to an additive constant free of \mu, the log posterior is

-\frac{n}{2\sigma^2}(\bar x-\mu)^2-\frac{1}{2\sigma_0^2}(\mu-\mu_0)^2.

Its derivative is (n/\sigma^2)(\bar x-\mu)-(1/\sigma_0^2)(\mu-\mu_0). Setting the derivative equal to zero balances the two pulls, so

\mu_1 = \frac{(n/\sigma^2)\bar x+(1/\sigma_0^2)\mu_0}{n/\sigma^2+1/\sigma_0^2}.

The second derivative is -n/\sigma^2-1/\sigma_0^2<0, so this stationary point is the unique posterior mode. The posterior is normal, hence its mode equals its mean. This derives the weighting rule: moving away from a more precise source incurs a larger quadratic penalty. The two weights are nonnegative and sum to one, so the posterior mean lies between \bar x and \mu_0.

(d) Let w_n=(n/\sigma^2)/(n/\sigma^2+1/\sigma_0^2). Then w_n\to 1 and

\mu_1=w_n\bar X_n+(1-w_n)\mu_0\to\mu^{\ast}, \qquad \sigma_1^2=\Bigl(\frac n{\sigma^2}+\frac1{\sigma_0^2}\Bigr)^{-1}\to 0.

With the prior held fixed, the data precision dominates. The posterior concentrates at the limiting sample mean.

This conclusion uses the stated convergence \bar X_n\to\mu^\ast. Under the iid normal model with true mean \mu^\ast, that convergence follows from the law of large numbers. A fixed prior contributes a fixed amount of precision, while the likelihood contributes an amount growing linearly with n.

(e) Given \mu, X_{\mathrm{new}} is independent of the sample and X_{\mathrm{new}}\mid\mu\sim \operatorname{N}(\mu,\sigma^2). Since \mu\mid{\boldsymbol{x}}\sim \operatorname{N}(\mu_1,\sigma_1^2),

\boxed{X_{\mathrm{new}}\mid{\boldsymbol{x}}\sim \operatorname{N}(\mu_1,\sigma^2+\sigma_1^2).}

To see the distributional result, write X_{\mathrm{new}}=\mu+\varepsilon_{\mathrm{new}}, where \varepsilon_{\mathrm{new}}\sim\operatorname{N}(0,\sigma^2) is independent of the posterior draw of \mu. Conditional on the observed sample, this is a sum of two independent normal variables. Merely replacing \mu by \mu_1 would omit uncertainty about the unknown mean.

(f) By the law of total variance,

\operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) = \mathbb{E}[\sigma^2\mid{\boldsymbol{x}}]+\operatorname{Var}(\mu\mid{\boldsymbol{x}}) = \sigma^2+\sigma_1^2.

The term \sigma_1^2 is uncertainty about \mu. The term \sigma^2 is the variance of a new observation even if \mu is known. Both are positive, so the predictive variance exceeds the posterior variance of \mu.

Q7. Normal estimation and prediction

Question page: Q7. Normal estimation and prediction

Suppose that, conditionally on \mu, X_1,\ldots,X_n\mid\mu \overset{\mathrm{iid}}{\sim}\mathcal{N}(\mu,\sigma^2), where \sigma^2>0 is known. Use the prior \mu\sim\mathcal{N}(\mu_0,\tau^2), where \tau^2>0 is known.

(a) By completing the square in \mu, derive its posterior distribution. Denote the posterior mean and variance by m and v. [4 marks]

(b) Let X_{\mathrm{new}} be a further observation, independent of the observed sample given \mu. Derive its posterior predictive distribution. Explain the two sources of its variance. [4 marks]

(c) As n\to\infty, with \sigma^2 and the prior fixed, what happens to the posterior variance of \mu and the predictive variance of X_{\mathrm{new}}? Explain why they behave differently. [2 marks]

Definitions. The notation N(m,v) uses mean m and variance v; precision is 1/v. The data have known variance \sigma^2>0, and the mean prior is N(\mu_0,\tau^2) with \tau^2>0. Write \bar x=n^{-1}\sum_i x_i. Posterior prediction averages a new observation’s sampling distribution over the parameter posterior; that observation is independent of the original sample conditional on \mu.

(a) Posterior distribution of \mu

Start from independence and the specified prior: L(\mu;{\boldsymbol{x}})=(2\pi\sigma^2)^{-n/2} \exp\left[-\frac{\sum_i(x_i-\mu)^2}{2\sigma^2}\right], \qquad p(\mu)\propto\exp\left[-\frac{(\mu-\mu_0)^2}{2\tau^2}\right]. The identity \sum_i(x_i-\mu)^2=\sum_i(x_i-\bar x)^2+n(\mu-\bar x)^2 follows by expanding around \bar x; the cross term is zero because \sum_i(x_i-\bar x)=0. Bayes’ theorem multiplies these two kernels. Ignoring terms that do not depend on \mu, \begin{aligned} p(\mu\mid{\boldsymbol{x}}) &\propto \exp\left[ -\frac{n}{2\sigma^2}(\mu-\bar{x})^2 -\frac{1}{2\tau^2}(\mu-\mu_0)^2 \right]\\ &\propto \exp\left[ -\frac12 \left\{ \left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)\mu^2 -2\left( \frac{n\bar{x}}{\sigma^2}+\frac{\mu_0}{\tau^2} \right)\mu \right\} \right]. \end{aligned} Define v=\left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1}, \qquad m=v\left( \frac{n\bar{x}}{\sigma^2}+\frac{\mu_0}{\tau^2} \right). Completing the square gives p(\mu\mid{\boldsymbol{x}}) \propto \exp\left[-\frac{(\mu-m)^2}{2v}\right], so \boxed{\mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m,v)}. Equivalently, v=\frac{\sigma^2\tau^2}{n\tau^2+\sigma^2}, \qquad m=\frac{n\tau^2\bar{x}+\sigma^2\mu_0} {n\tau^2+\sigma^2}.

(b) Posterior predictive distribution

Conditional on \mu, X_{\mathrm{new}}=\mu+\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\sigma^2), where \varepsilon is independent of \mu\mid{\boldsymbol{x}}. Since \mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m,v), \boxed{ X_{\mathrm{new}}\mid{\boldsymbol{x}} \sim\mathcal{N}(m,\sigma^2+v) }. The predictive variance can also be obtained from \begin{aligned} \operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) &= \mathbb{E}\!\left[ \operatorname{Var}(X_{\mathrm{new}}\mid\mu,{\boldsymbol{x}}) \mid{\boldsymbol{x}} \right]\\ &\quad+ \operatorname{Var}\!\left( \mathbb{E}[X_{\mathrm{new}}\mid\mu,{\boldsymbol{x}}] \mid{\boldsymbol{x}} \right)\\ &=\sigma^2+v. \end{aligned} Here \sigma^2 is future-observation noise and v is uncertainty about the unknown mean.

(c) Limiting variances

As n\to\infty, v=\left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1} \longrightarrow0, whereas \operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) =\sigma^2+v \longrightarrow\sigma^2. Increasing n removes uncertainty about \mu, but it does not remove the irreducible sampling variation in a new observation.

Used in

See also

A proper approximation to the flat prior · Normal–Normal conjugacy · Precision weighting · Inverse-Gamma distribution · Normal–inverse-Gamma variance update · Full conditionals under independent priors

Glossary and index