Precision weighting
Remark taught in Conjugate analysis for normal models, part of Bayesian statistics
Precision weighting
| Type | Remark |
| Taught in | Conjugate analysis for normal models |
| Part | Bayesian statistics |
| Worked examples | 1 |
| Questions using it | 1 |
Writing
w=\frac{\sigma^2}{n\sigma_0^2+\sigma^2}
turns Equation (2) in Conjugate analysis for normal models into
\mu_1=(1-w)\overline x+w\mu_0.
The posterior mean is therefore a precision-weighted average. Equivalently, if \kappa_0=\sigma^2/\sigma_0^2, then
\mu_1=\frac{n\overline x+\kappa_0\mu_0}{n+\kappa_0}.
Thus the prior behaves like \kappa_0 observations whose mean is \mu_0. In the special case \sigma_0^2=\sigma^2, it behaves like one extra observation at \mu_0.
Examples
Question 1. Normal mean, known variance
Question page: Question 1. Normal mean, known variance
Let
X_i\mid\mu\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2),
with \sigma^2 known and \mu\sim \operatorname{N}(\mu_0,\sigma_0^2).
(a) Using only the precision-addition rule and the precision-weighted mean, write the two updated parameters of the posterior for \mu.
(b) Define the prior equivalent sample size, and state what it counts.
(c) Explain why the posterior mean is a precision-weighted average of the prior mean and the sample mean.
(d) State and interpret the limiting posterior mean and variance as n\to\infty, with the prior parameters fixed and \bar X_n\to\mu^{\ast}.
(e) State the posterior predictive distribution of one new observation X_{\mathrm{new}}, conditionally independent of the sample given \mu.
(f) Explain why the posterior predictive variance is larger than the posterior variance of \mu.
Definitions. Write \bar x=n^{-1}\sum_i x_i. A normal distribution \operatorname{N}(m,v) has mean m and variance v; its precision is 1/v. Here \sigma^2 is the known observation variance, \sigma_0^2 is the prior variance of the unknown mean, and \sigma_1^2 will be its posterior variance. These are variances of different objects.
(a) The data precision of n observations is n/\sigma^2, and the prior precision is 1/\sigma_0^2. Precision addition gives
\frac1{\sigma_1^2} = \frac n{\sigma^2}+\frac1{\sigma_0^2}.
The posterior mean is the precision-weighted average of \bar x and \mu_0:
\mu_1 = \frac{(n/\sigma^2)\bar x+(1/\sigma_0^2)\mu_0}{n/\sigma^2+1/\sigma_0^2} = \frac{n\bar x\,\sigma_0^2+\mu_0\sigma^2}{n\sigma_0^2+\sigma^2}.
\boxed{\mu\mid x\sim \operatorname{N}(\mu_1,\sigma_1^2).}
(b)
\kappa_0=\frac{\sigma^2}{\sigma_0^2}.
It counts how many observations of variance \sigma^2 have the same precision as the prior, since 1/\sigma_0^2=\kappa_0/\sigma^2.
Consequently, \mu_1=(n\bar x+\kappa_0\mu_0)/(n+\kappa_0) and \sigma_1^2=\sigma^2/(n+\kappa_0). The equivalent sample size measures information about the common mean. It need not be an integer or represent actual previous observations.
(c) Up to an additive constant free of \mu, the log posterior is
-\frac{n}{2\sigma^2}(\bar x-\mu)^2-\frac{1}{2\sigma_0^2}(\mu-\mu_0)^2.
Its derivative is (n/\sigma^2)(\bar x-\mu)-(1/\sigma_0^2)(\mu-\mu_0). Setting the derivative equal to zero balances the two pulls, so
\mu_1 = \frac{(n/\sigma^2)\bar x+(1/\sigma_0^2)\mu_0}{n/\sigma^2+1/\sigma_0^2}.
The second derivative is -n/\sigma^2-1/\sigma_0^2<0, so this stationary point is the unique posterior mode. The posterior is normal, hence its mode equals its mean. This derives the weighting rule: moving away from a more precise source incurs a larger quadratic penalty. The two weights are nonnegative and sum to one, so the posterior mean lies between \bar x and \mu_0.
(d) Let w_n=(n/\sigma^2)/(n/\sigma^2+1/\sigma_0^2). Then w_n\to 1 and
\mu_1=w_n\bar X_n+(1-w_n)\mu_0\to\mu^{\ast}, \qquad \sigma_1^2=\Bigl(\frac n{\sigma^2}+\frac1{\sigma_0^2}\Bigr)^{-1}\to 0.
With the prior held fixed, the data precision dominates. The posterior concentrates at the limiting sample mean.
This conclusion uses the stated convergence \bar X_n\to\mu^\ast. Under the iid normal model with true mean \mu^\ast, that convergence follows from the law of large numbers. A fixed prior contributes a fixed amount of precision, while the likelihood contributes an amount growing linearly with n.
(e) Given \mu, X_{\mathrm{new}} is independent of the sample and X_{\mathrm{new}}\mid\mu\sim \operatorname{N}(\mu,\sigma^2). Since \mu\mid{\boldsymbol{x}}\sim \operatorname{N}(\mu_1,\sigma_1^2),
\boxed{X_{\mathrm{new}}\mid{\boldsymbol{x}}\sim \operatorname{N}(\mu_1,\sigma^2+\sigma_1^2).}
To see the distributional result, write X_{\mathrm{new}}=\mu+\varepsilon_{\mathrm{new}}, where \varepsilon_{\mathrm{new}}\sim\operatorname{N}(0,\sigma^2) is independent of the posterior draw of \mu. Conditional on the observed sample, this is a sum of two independent normal variables. Merely replacing \mu by \mu_1 would omit uncertainty about the unknown mean.
(f) By the law of total variance,
\operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) = \mathbb{E}[\sigma^2\mid{\boldsymbol{x}}]+\operatorname{Var}(\mu\mid{\boldsymbol{x}}) = \sigma^2+\sigma_1^2.
The term \sigma_1^2 is uncertainty about \mu. The term \sigma^2 is the variance of a new observation even if \mu is known. Both are positive, so the predictive variance exceeds the posterior variance of \mu.
Used in
- Taught in Conjugate analysis for normal models, where the derivation and examples are.
- Questions that use it (1) — Question 1
- All 19 questions on Conjugate analysis for normal models
See also
Normal–Normal model · A proper approximation to the flat prior · Normal–Normal conjugacy · Inverse-Gamma distribution · Normal–inverse-Gamma variance update · Full conditionals under independent priors