Prior and posterior predictive distributions

Definition taught in Bayesian foundations, part of Bayesian statistics

Prior and posterior predictive distributions

Type Definition
Taught in Bayesian foundations
Part Bayesian statistics
Properties 3
Worked examples 2
Questions using it 6

Before data are observed, a new observation \widetilde x is generated by first drawing \theta from its prior and then drawing \widetilde x from the sampling model. Marginalising \theta gives the prior predictive density: \begin{align*} p(\widetilde x) &=\int p(\widetilde x,\theta)\,\mathrm{d}\theta\\ &=\int p(\widetilde x\mid\theta)p(\theta)\,\mathrm{d}\theta. \end{align*}

After observing x, the analogous calculation uses the posterior: \begin{align*} p(\widetilde x\mid x) &=\int p(\widetilde x,\theta\mid x)\,\mathrm{d}\theta\\ &=\int p(\widetilde x\mid\theta,x)p(\theta\mid x)\,\mathrm{d}\theta\\ &=\int p(\widetilde x\mid\theta)p(\theta\mid x)\,\mathrm{d}\theta. \end{align*} The last equality uses conditional independence of future and observed data given \theta. Prediction therefore averages the sampling model over parameter uncertainty instead of plugging in a single estimate.

The prior predictive distribution asks what data the complete model regarded as plausible before seeing the observations. It is therefore useful for study design and for detecting a prior that places most of its mass on scientifically absurd data. The posterior predictive distribution asks what future data are plausible after learning from x. It is used for forecasting and for posterior predictive model checks. Although both are mixtures of the sampling distribution, they mix with different distributions for \theta: p(\theta) before the data and p(\theta\mid x) after the data.

Properties

Predictive-variance decomposition

Main article: Predictive-variance decomposition

Under the conditional-independence assumption above, \boxed{ \operatorname{Var}(\widetilde X\mid x) =\underbrace{\mathbb{E}\{v(\theta)\mid x\}}_{\textnormal{future-observation noise}} +\underbrace{\operatorname{Var}\{m(\theta)\mid x\}}_{\textnormal{parameter uncertainty}}.}

Beta–Binomial posterior prediction

Main article: Beta–Binomial posterior prediction

If \theta\mid x\sim\operatorname{Beta}(a,b) and, conditionally on \theta, \widetilde X\sim\operatorname{Bin}(m,\theta), then

\mathbb{P}(\widetilde X=k\mid x) =\binom mk\frac{B(k+a,m-k+b)}{B(a,b)}, \qquad k=0,\ldots,m.

For one future Bernoulli observation, \mathbb{P}(\widetilde Y=1\mid x)=a/(a+b).

Posterior prediction under the reference prior

Main article: Posterior prediction under the reference prior

\boxed{ \widetilde Y\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}},\widetilde{\boldsymbol{x}} \sim t_{n-p}\!\left( \widetilde{\boldsymbol{x}}^\top\widehat{\boldsymbol{\beta}}, s\sqrt{1+h(\widetilde{\boldsymbol{x}})}\right).} For the unknown mean response itself, there is no new observational error: \widetilde{\boldsymbol{x}}^\top{\boldsymbol{\beta}}\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}} \sim t_{n-p}\!\left( \widetilde{\boldsymbol{x}}^\top\widehat{\boldsymbol{\beta}}, s\sqrt{h(\widetilde{\boldsymbol{x}})}\right).

Examples

Q7. Normal estimation and prediction

Question page: Q7. Normal estimation and prediction

Suppose that, conditionally on \mu, X_1,\ldots,X_n\mid\mu \overset{\mathrm{iid}}{\sim}\mathcal{N}(\mu,\sigma^2), where \sigma^2>0 is known. Use the prior \mu\sim\mathcal{N}(\mu_0,\tau^2), where \tau^2>0 is known.

(a) By completing the square in \mu, derive its posterior distribution. Denote the posterior mean and variance by m and v. [4 marks]

(b) Let X_{\mathrm{new}} be a further observation, independent of the observed sample given \mu. Derive its posterior predictive distribution. Explain the two sources of its variance. [4 marks]

(c) As n\to\infty, with \sigma^2 and the prior fixed, what happens to the posterior variance of \mu and the predictive variance of X_{\mathrm{new}}? Explain why they behave differently. [2 marks]

Definitions. The notation N(m,v) uses mean m and variance v; precision is 1/v. The data have known variance \sigma^2>0, and the mean prior is N(\mu_0,\tau^2) with \tau^2>0. Write \bar x=n^{-1}\sum_i x_i. Posterior prediction averages a new observation’s sampling distribution over the parameter posterior; that observation is independent of the original sample conditional on \mu.

(a) Posterior distribution of \mu

Start from independence and the specified prior: L(\mu;{\boldsymbol{x}})=(2\pi\sigma^2)^{-n/2} \exp\left[-\frac{\sum_i(x_i-\mu)^2}{2\sigma^2}\right], \qquad p(\mu)\propto\exp\left[-\frac{(\mu-\mu_0)^2}{2\tau^2}\right]. The identity \sum_i(x_i-\mu)^2=\sum_i(x_i-\bar x)^2+n(\mu-\bar x)^2 follows by expanding around \bar x; the cross term is zero because \sum_i(x_i-\bar x)=0. Bayes’ theorem multiplies these two kernels. Ignoring terms that do not depend on \mu, \begin{aligned} p(\mu\mid{\boldsymbol{x}}) &\propto \exp\left[ -\frac{n}{2\sigma^2}(\mu-\bar{x})^2 -\frac{1}{2\tau^2}(\mu-\mu_0)^2 \right]\\ &\propto \exp\left[ -\frac12 \left\{ \left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)\mu^2 -2\left( \frac{n\bar{x}}{\sigma^2}+\frac{\mu_0}{\tau^2} \right)\mu \right\} \right]. \end{aligned} Define v=\left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1}, \qquad m=v\left( \frac{n\bar{x}}{\sigma^2}+\frac{\mu_0}{\tau^2} \right). Completing the square gives p(\mu\mid{\boldsymbol{x}}) \propto \exp\left[-\frac{(\mu-m)^2}{2v}\right], so \boxed{\mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m,v)}. Equivalently, v=\frac{\sigma^2\tau^2}{n\tau^2+\sigma^2}, \qquad m=\frac{n\tau^2\bar{x}+\sigma^2\mu_0} {n\tau^2+\sigma^2}.

(b) Posterior predictive distribution

Conditional on \mu, X_{\mathrm{new}}=\mu+\varepsilon, \qquad \varepsilon\sim\mathcal{N}(0,\sigma^2), where \varepsilon is independent of \mu\mid{\boldsymbol{x}}. Since \mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m,v), \boxed{ X_{\mathrm{new}}\mid{\boldsymbol{x}} \sim\mathcal{N}(m,\sigma^2+v) }. The predictive variance can also be obtained from \begin{aligned} \operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) &= \mathbb{E}\!\left[ \operatorname{Var}(X_{\mathrm{new}}\mid\mu,{\boldsymbol{x}}) \mid{\boldsymbol{x}} \right]\\ &\quad+ \operatorname{Var}\!\left( \mathbb{E}[X_{\mathrm{new}}\mid\mu,{\boldsymbol{x}}] \mid{\boldsymbol{x}} \right)\\ &=\sigma^2+v. \end{aligned} Here \sigma^2 is future-observation noise and v is uncertainty about the unknown mean.

(c) Limiting variances

As n\to\infty, v=\left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1} \longrightarrow0, whereas \operatorname{Var}(X_{\mathrm{new}}\mid{\boldsymbol{x}}) =\sigma^2+v \longrightarrow\sigma^2. Increasing n removes uncertainty about \mu, but it does not remove the irreducible sampling variation in a new observation.

Q20. Estimation and prediction

Question page: Q20. Estimation and prediction

[4 marks]

Suppose measurements are conditionally independent with X_i\mid\mu\sim\mathcal{N}(\mu,\sigma^2), where \sigma^2>0 is known, and a proper Normal prior is used for \mu.

Explain the difference between the posterior distribution of \mu and the posterior predictive distribution of one new measurement. As the number of observations becomes very large, would you expect uncertainty about both quantities to approach zero? Explain your reasoning.

Definitions. \mu is the population mean and \widetilde X is a new observation. Write \mu\mid{\boldsymbol{x}}\sim N(m_n,v_n), where v_n is the posterior variance. The known observation variance is \sigma^2>0; conditional on \mu, the new Normal observation is independent of the original sample. A predictive distribution averages over parameter uncertainty as well as new-observation noise.

The posterior distribution p(\mu\mid{\boldsymbol{x}}) describes remaining uncertainty about the unknown population mean \mu after observing the data. The posterior predictive distribution p(\widetilde x\mid{\boldsymbol{x}}) =\int p(\widetilde x\mid\mu) p(\mu\mid{\boldsymbol{x}})\,d\mu describes uncertainty about a future observation \widetilde X. It includes both uncertainty about \mu and the new observation’s sampling variation around \mu.

If \mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m_n,v_n), then \boxed{ \widetilde X\mid{\boldsymbol{x}} \sim\mathcal{N}(m_n,\sigma^2+v_n). } As n\to\infty, under the model and standard regularity conditions, v_n\longrightarrow0. Hence posterior uncertainty about \mu approaches zero. By contrast, \operatorname{Var}(\widetilde X\mid{\boldsymbol{x}}) =\underbrace{\sigma^2}_{\text{new-observation noise}} +\underbrace{v_n}_{\text{parameter uncertainty}} \longrightarrow\sigma^2. Predictive uncertainty therefore does not vanish: even if \mu were known exactly, a future measurement would still vary randomly around it.

Used in

See also

Bayesian credible interval · Bayes’ rule · Proper prior and proper posterior · Posterior point summaries · Credible-region constructions · Posterior summaries from draws · R’s distribution interface · Predictive-variance decomposition

Glossary and index