Posterior mean

Topic of Bayesian foundations, part of Bayesian statistics

Posterior mean

Type Topic
Taught in Bayesian foundations
Part Bayesian statistics
Questions using it 33

The mean \mathbb{E}(\theta\mid y) of the posterior distribution. It is the Bayes estimate under squared-error loss, and in conjugate models it is a weighted average of the prior mean and a data summary.

In Bayesian foundations

From Bayesian foundations

Definition 1 (Posterior point summaries) For a scalar parameter, common point summaries include

\mathbb{E}(\theta\mid x)=\int\theta p(\theta\mid x)\,\mathrm{d}\theta, \qquad \widehat\theta_{\mathrm{MAP}} =\operatorname*{arg\,max}_{\theta}p(\theta\mid x),

the posterior median, and posterior quantiles. These summaries need not agree for a skewed distribution.

Article: Posterior point summaries →

Definition 2 (Credible-region constructions) Three interval constructions are worth distinguishing. An equal-tailed 95\% credible interval (l,u) satisfies

\mathbb{P}(\theta<l\mid x)=0.025, \qquad \mathbb{P}(\theta>u\mid x)=0.025.

The minimum-width interval solves

\min_{l<u}(u-l) \quad\text{subject to}\quad \int_l^u p(\theta\mid x)\,\mathrm{d}\theta=0.95.

A highest-posterior-density set chooses a level c so that

\mathcal H_c=\{\theta:p(\theta\mid x)\geq c\}, \qquad \int_{\mathcal H_c}p(\theta\mid x)\,\mathrm{d}\theta=0.95.

Every included point then has density at least as large as every excluded point. For a continuous unimodal posterior this gives the minimum-width interval; for a multimodal posterior the highest-density set can be disjoint. For a symmetric unimodal density, the mean, median, and mode coincide and the common interval constructions often agree.

Article: Credible-region constructions →

Figure 1: Two 95\% summaries of the same skewed \operatorname{Beta}(3,9) posterior. Equal tails force 2.5\% of the probability to each side. The minimum-width interval shifts toward the high-density part of the posterior; because this density is continuous and unimodal, it is also the HPD set and its two endpoints have equal density.

The MAP is a useful point, but a point alone discards the uncertainty that the posterior was introduced to represent. A posterior sample converts most summaries into ordinary sample calculations.

Proposition 1 (Posterior summaries from draws) Let \theta^{(1)},\ldots,\theta^{(L)} be posterior draws and let \theta_{(1)}\leq\cdots\leq\theta_{(L)} be their order statistics. Then \begin{align*} \widehat{\mathbb{E}}(\theta\mid x) &=\frac1L\sum_{l=1}^L\theta^{(l)},\\ \widehat{\operatorname{Var}}(\theta\mid x) &=\frac1{L-1}\sum_{l=1}^L \left(\theta^{(l)}-\overline\theta\right)^2,\\ \widehat{\mathbb{P}}(\theta<c\mid x) &=\frac1L\sum_{l=1}^L\mathbf{1}\{\theta^{(l)}<c\}. \end{align*} The posterior median and equal-tailed interval are estimated by the corresponding empirical quantiles. With L=1000, the median can be estimated by the average of \theta_{(500)} and \theta_{(501)}, and a direct 95\% interval approximation by [\theta_{(25)},\theta_{(975)}]. The precise indices vary slightly with a software package’s quantile convention.

Article: Posterior summaries from draws →

The mode is the exception. It is not an order statistic and, for continuous draws, looking for the most frequently repeated value is generally useless. One should maximise the posterior density or log-posterior, or deliberately construct and optimise a density estimate if the density itself is unavailable.

Remark 1 (R’s distribution interface). For a common distribution, R prefixes the distribution name by d for density, p for cumulative probability, q for quantile, and r for random generation. These are numerical calculations to run once the posterior distribution has been derived. For the \operatorname{Beta}(3,9) posterior in Figure, the following commands compute its median, a posterior probability, an equal-tailed interval, and Monte Carlo versions of the same summaries.

Article: R’s distribution interface →

# theta | x ~ Beta(3, 9)
qbeta(0.5, shape1 = 3, shape2 = 9)       # posterior median
pbeta(0.5, shape1 = 3, shape2 = 9)       # P(theta <= 0.5 | x)
qbeta(c(0.025, 0.975), 3, 9)             # equal-tailed interval

set.seed(3500)
theta <- rbeta(1000, shape1 = 3, shape2 = 9)
c(mean = mean(theta),
  median = median(theta),
  Pr_below_half = mean(theta < 0.5))
quantile(theta, c(0.025, 0.975))

The first three lines use the known Beta posterior directly and are preferable when the required function is available. The simulated calculation is the version that survives when the posterior is accessible only through a sampler. Increasing the number of draws reduces Monte Carlo error; it does not repair a sampler that targets the wrong distribution.

Used in

See also

Marginal variance posterior · Precision · Precision weighting · Two pseudo-count conventions · Warm-up is not a convergence proof · Bayesian credible interval · Bayes’ rule · Proper prior and proper posterior

Glossary and index