Predictive-variance decomposition
Proposition taught in Bayesian foundations, part of Bayesian statistics
Predictive-variance decomposition
| Type | Proposition |
| Taught in | Bayesian foundations |
| Part | Bayesian statistics |
| Worked examples | 1 |
Under the conditional-independence assumption above, \boxed{ \operatorname{Var}(\widetilde X\mid x) =\underbrace{\mathbb{E}\{v(\theta)\mid x\}}_{\textnormal{future-observation noise}} +\underbrace{\operatorname{Var}\{m(\theta)\mid x\}}_{\textnormal{parameter uncertainty}}.}
Examples
Q20. Estimation and prediction
Question page: Q20. Estimation and prediction
[4 marks]
Suppose measurements are conditionally independent with X_i\mid\mu\sim\mathcal{N}(\mu,\sigma^2), where \sigma^2>0 is known, and a proper Normal prior is used for \mu.
Explain the difference between the posterior distribution of \mu and the posterior predictive distribution of one new measurement. As the number of observations becomes very large, would you expect uncertainty about both quantities to approach zero? Explain your reasoning.
Definitions. \mu is the population mean and \widetilde X is a new observation. Write \mu\mid{\boldsymbol{x}}\sim N(m_n,v_n), where v_n is the posterior variance. The known observation variance is \sigma^2>0; conditional on \mu, the new Normal observation is independent of the original sample. A predictive distribution averages over parameter uncertainty as well as new-observation noise.
The posterior distribution p(\mu\mid{\boldsymbol{x}}) describes remaining uncertainty about the unknown population mean \mu after observing the data. The posterior predictive distribution p(\widetilde x\mid{\boldsymbol{x}}) =\int p(\widetilde x\mid\mu) p(\mu\mid{\boldsymbol{x}})\,d\mu describes uncertainty about a future observation \widetilde X. It includes both uncertainty about \mu and the new observation’s sampling variation around \mu.
If \mu\mid{\boldsymbol{x}}\sim\mathcal{N}(m_n,v_n), then \boxed{ \widetilde X\mid{\boldsymbol{x}} \sim\mathcal{N}(m_n,\sigma^2+v_n). } As n\to\infty, under the model and standard regularity conditions, v_n\longrightarrow0. Hence posterior uncertainty about \mu approaches zero. By contrast, \operatorname{Var}(\widetilde X\mid{\boldsymbol{x}}) =\underbrace{\sigma^2}_{\text{new-observation noise}} +\underbrace{v_n}_{\text{parameter uncertainty}} \longrightarrow\sigma^2. Predictive uncertainty therefore does not vanish: even if \mu were known exactly, a future measurement would still vary randomly around it.
Used in
- Taught in Bayesian foundations, where the derivation and examples are.
- All 15 questions on Bayesian foundations
See also
Bayesian credible interval · Bayes’ rule · Proper prior and proper posterior · Posterior point summaries · Credible-region constructions · Posterior summaries from draws · R’s distribution interface · Prior and posterior predictive distributions