Q7. Normal estimation and prediction

Topic: Conjugate analysis for normal models · more questions on this topic
‹ Normal–inverse-\chi^2 conjugacy Q20 ›

Suppose that, conditionally on \mu, X_1,\ldots,X_n\mid\mu \overset{\mathrm{iid}}{\sim}\mathcal{N}(\mu,\sigma^2), where \sigma^2>0 is known. Use the prior \mu\sim\mathcal{N}(\mu_0,\tau^2), where \tau^2>0 is known.

(a) By completing the square in \mu, derive its posterior distribution. Denote the posterior mean and variance by m and v. [4 marks]

(b) Let X_{\mathrm{new}} be a further observation, independent of the observed sample given \mu. Derive its posterior predictive distribution. Explain the two sources of its variance. [4 marks]

(c) As n\to\infty, with \sigma^2 and the prior fixed, what happens to the posterior variance of \mu and the predictive variance of X_{\mathrm{new}}? Explain why they behave differently. [2 marks]

Worked answer (click to show)

Definitions. The notation N(m,v) uses mean m and variance v; precision is 1/v. The data have known variance \sigma^2>0, and the mean prior is N(\mu_0,\tau^2) with \tau^2>0. Write \bar x=n^{-1}\sum_i x_i. Posterior prediction averages a new observation’s sampling distribution over the parameter posterior; that observation is independent of the original sample conditional on \mu.

(a) Posterior distribution of \mu

Start from independence and the specified prior: L(\mu;{\boldsymbol{x}})=(2\pi\sigma^2)^{-n/2} \exp\left[-\frac{\sum_i(x_i-\mu)^2}{2\sigma^2}\right], \qquad p(\mu)\propto\exp\left[-\frac{(\mu-\mu_0)^2}{2\tau^2}\right]. The identity \sum_i(x_i-\mu)^2=\sum_i(x_i-\bar x)^2+n(\mu-\bar x)^2 follows by expanding around \bar x; the cross term is zero because \sum_i(x_i-\bar x)=0. Bayes’ theorem multiplies these two kernels. Ignoring terms that do not depend on \mu, \begin{aligned} p(\mu\mid{\boldsymbol{x}}) &\propto \exp\left[ -\frac{n}{2\sigma^2}(\mu-\bar{x})^2 -\frac{1}{2\tau^2}(\mu-\mu_0)^2 \right]\\ &\propto \exp\left[ -\frac12 \left\{ \left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)\mu^2 -2\left( \frac{n\bar{x}}{\sigma^2}+\frac{\mu_0}{\tau^2} \right)\mu \right\} \right]. \end{aligned} Define v=\left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1}, \qquad m=v\left( \frac{n\bar{x}}{\sigma^2}+\frac{\mu_0}{\tau^2} \right). Completing the square gives p(\mu\mid{\boldsymbol{x}}) \propto \exp\left[-\frac{(\mu-m)^2}{2v}\right], so \boxed{\mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m,v)}. Equivalently, v=\frac{\sigma^2\tau^2}{n\tau^2+\sigma^2}, \qquad m=\frac{n\tau^2\bar{x}+\sigma^2\mu_0} {n\tau^2+\sigma^2}.

(b) Posterior predictive distribution

Conditional on \mu, X_{\mathrm{new}}=\mu+\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\sigma^2), where \varepsilon is independent of \mu\mid{\boldsymbol{x}}. Since \mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m,v), \boxed{ X_{\mathrm{new}}\mid{\boldsymbol{x}} \sim\mathcal{N}(m,\sigma^2+v) }. The predictive variance can also be obtained from \begin{aligned} \operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) &= \mathbb{E}\!\left[ \operatorname{Var}(X_{\mathrm{new}}\mid\mu,{\boldsymbol{x}}) \mid{\boldsymbol{x}} \right]\\ &\quad+ \operatorname{Var}\!\left( \mathbb{E}[X_{\mathrm{new}}\mid\mu,{\boldsymbol{x}}] \mid{\boldsymbol{x}} \right)\\ &=\sigma^2+v. \end{aligned} Here \sigma^2 is future-observation noise and v is uncertainty about the unknown mean.

(c) Limiting variances

As n\to\infty, v=\left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1} \longrightarrow0, whereas \operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) =\sigma^2+v \longrightarrow\sigma^2. Increasing n removes uncertainty about \mu, but it does not remove the irreducible sampling variation in a new observation.