Posterior point summaries

Definition taught in Bayesian foundations, part of Bayesian statistics

Posterior point summaries

Type Definition
Taught in Bayesian foundations
Part Bayesian statistics
Properties 1
Worked examples 2
Questions using it 9

For a scalar parameter, common point summaries include

\mathbb{E}(\theta\mid x)=\int\theta p(\theta\mid x)\,\mathrm{d}\theta, \qquad \widehat\theta_{\mathrm{MAP}} =\operatorname*{arg\,max}_{\theta}p(\theta\mid x),

the posterior median, and posterior quantiles. These summaries need not agree for a skewed distribution.

Properties

Posterior summaries from draws

Main article: Posterior summaries from draws

Let \theta^{(1)},\ldots,\theta^{(L)} be posterior draws and let \theta_{(1)}\leq\cdots\leq\theta_{(L)} be their order statistics. Then \begin{align*} \widehat{\mathbb{E}}(\theta\mid x) &=\frac1L\sum_{l=1}^L\theta^{(l)},\\ \widehat{\operatorname{Var}}(\theta\mid x) &=\frac1{L-1}\sum_{l=1}^L \left(\theta^{(l)}-\overline\theta\right)^2,\\ \widehat{\mathbb{P}}(\theta<c\mid x) &=\frac1L\sum_{l=1}^L\mathbf{1}\{\theta^{(l)}<c\}. \end{align*} The posterior median and equal-tailed interval are estimated by the corresponding empirical quantiles. With L=1000, the median can be estimated by the average of \theta_{(500)} and \theta_{(501)}, and a direct 95\% interval approximation by [\theta_{(25)},\theta_{(975)}]. The precise indices vary slightly with a software package’s quantile convention.

Examples

Posterior mean under squared-error loss

Question page: Posterior mean under squared-error loss

For observed data {\boldsymbol{x}}, consider the posterior mean squared error of a point estimate d=d({\boldsymbol{x}}),

R(d\mid{\boldsymbol{x}})=\mathbb{E}\bigl[(d-\theta)^2\mid{\boldsymbol{x}}\bigr].

Show, without omitting the quadratic expansion or the minimisation step, that the posterior mean d=\mathbb{E}(\theta\mid{\boldsymbol{x}}) is the minimum-mean-square-error estimator of \theta.

Question. For observed data {\boldsymbol{x}}, consider the posterior mean squared error of a point estimate d=d({\boldsymbol{x}}),

R(d\mid{\boldsymbol{x}})=\mathbb{E}\bigl[(d-\theta)^2\mid{\boldsymbol{x}}\bigr].

Show, without omitting the quadratic expansion or the minimisation step, that the posterior mean d=\mathbb{E}(\theta\mid{\boldsymbol{x}}) is the minimum-mean-square-error estimator of \theta.

Solution. Once {\boldsymbol{x}} has been observed, d=d({\boldsymbol{x}}) is a fixed number and only \theta is random under the posterior. Therefore \begin{align*} R(d\mid{\boldsymbol{x}}) &=\mathbb{E}\bigl[(d-\theta)^2\mid{\boldsymbol{x}}\bigr]\\ &=\mathbb{E}\bigl[d^2-2d\theta+\theta^2\mid{\boldsymbol{x}}\bigr]\\ &=d^2-2d\mathbb{E}(\theta\mid{\boldsymbol{x}})+\mathbb{E}(\theta^2\mid{\boldsymbol{x}}). \end{align*} Put m=\mathbb{E}(\theta\mid{\boldsymbol{x}}). Adding and subtracting m^2 gives \begin{align*} R(d\mid{\boldsymbol{x}}) &=d^2-2dm+m^2+\mathbb{E}(\theta^2\mid{\boldsymbol{x}})-m^2\\ &=(d-m)^2+\operatorname{Var}(\theta\mid{\boldsymbol{x}}). \end{align*} The posterior variance does not depend on d, and (d-m)^2\geq0 with equality exactly when d=m. Equivalently,

\frac{\partial R}{\partial d}=2d-2m, \qquad \frac{\partial^2 R}{\partial d^2}=2>0.

Thus the unique minimiser is

\boxed{d_{\mathrm{MMSE}}({\boldsymbol{x}})=\mathbb{E}(\theta\mid{\boldsymbol{x}})}.

C7. MAP and squared-error loss

Question page: C7. MAP and squared-error loss

How does a MAP estimate differ from a Bayes estimate under squared-error loss?

Definitions/notation. The MAP maximises p(\theta\mid x), whereas a Bayes estimate minimises the posterior expected loss for a specified loss function. Under squared-error loss and a finite posterior second moment, \mathbb{E}[(\theta-d)^2\mid x]=\operatorname{Var}(\theta\mid x)+(\mathbb{E}[\theta\mid x]-d)^2, so the Bayes estimate is the posterior mean, not generally the mode.

Used in

See also

Marginal variance posterior · Two pseudo-count conventions · Bayesian credible interval · Bayes’ rule · Proper prior and proper posterior · Credible-region constructions · Posterior summaries from draws · R’s distribution interface

Glossary and index