Bayes factors and posterior odds
Section of Bayesian hypotheses and model comparison, part of Bayesian statistics · Also in Model selection for stochastic models
Bayes factors and posterior odds
| Type | Section |
| Taught in | Bayesian hypotheses and model comparison · Model selection for stochastic models |
| Part | Bayesian statistics · Statistical learning |
In Bayesian hypotheses and model comparison
Theorem 1 (Posterior-odds factorisation) For two models M_1 and M_2, Bayes’ rule gives
\mathbb{P}(M_j\mid x) =\frac{p(x\mid M_j)\mathbb{P}(M_j)}{p(x)}.
Taking the ratio cancels the common evidence p(x): \begin{aligned} \frac{\mathbb{P}(M_1\mid x)}{\mathbb{P}(M_2\mid x)} &=\frac{p(x\mid M_1)\mathbb{P}(M_1)/p(x)} {p(x\mid M_2)\mathbb{P}(M_2)/p(x)}\\ &=\underbrace{\frac{p(x\mid M_1)}{p(x\mid M_2)}}_{ \operatorname{BF}_{12}} \underbrace{\frac{\mathbb{P}(M_1)}{\mathbb{P}(M_2)}}_{ \text{prior odds}}. \end{aligned}
Definition 1 (Marginal likelihood and Bayes factor) The marginal likelihood in model M_j is p(x\mid M_j) =\int p(x\mid\theta_j,M_j) p(\theta_j\mid M_j)\,\mathrm{d}\theta_j. It averages the likelihood over the parameter prior; it does not evaluate the likelihood at a fitted maximum. The Bayes factor comparing M_1 with M_2 is \operatorname{BF}_{12}=p(x\mid M_1)/p(x\mid M_2).
A Bayes factor compares whole models, not merely their best-fitting parameter values. The marginal likelihood averages over each model’s prior parameter space, so diffuse priors can spread support across many poor explanations and change the comparison. Equal prior model probabilities are a transparent modelling choice, not a mathematical requirement. Model probabilities, parameter priors, and the set of models under consideration must all be stated before a numerical ratio can be interpreted responsibly.
Remark 1 (A hypothesis region is not a separate model). Two constructions must not be confused:
If H_0 and H_1 are regions of one parameter space, compute one posterior and integrate it over the regions.
If M_1 and M_2 have different dimensions or parameterisations, each model needs its own parameter prior and marginal likelihood. A continuous prior on the larger space assigns probability zero to a lower-dimensional surface.
The fruit example is the first construction. Independence in the advertising table is the second.
In Model selection for stochastic models
Consider two candidate models, M_1 and M_2. Within each, specify a proper prior on its parameters, and specify prior probabilities for the models. Bayes’ theorem then gives probabilities conditional on this candidate list and these priors. A candidate can be used in the comparison without being literally true. The resulting probabilities leave out every model omitted from the list (Kass and Raftery, 1995).
\mathbb{P}(M_k\mid X_{\mathrm{all}}) =\frac{\mathbb{P}(X_{\mathrm{all}}\mid M_k)\mathbb{P}(M_k)} {\mathbb{P}(X_{\mathrm{all}}\mid M_1)\mathbb{P}(M_1)+\mathbb{P}(X_{\mathrm{all}}\mid M_2)\mathbb{P}(M_2)}, \qquad k=1,2.
The denominator does not depend on which model is being scored, so the comparison lives in the numerator. The posterior odds of M_1 against M_2 are
\frac{\mathbb{P}(M_1\mid X_{\mathrm{all}})}{\mathbb{P}(M_2\mid X_{\mathrm{all}})} =\frac{\mathbb{P}(X_{\mathrm{all}}\mid M_1)\mathbb{P}(M_1)}{\mathbb{P}(X_{\mathrm{all}}\mid M_2)\mathbb{P}(M_2)},
and the Bayes factor for model 1 against model 2 is the ratio of marginal likelihoods,
B_{12}=\frac{\mathbb{P}(X_{\mathrm{all}}\mid M_1)}{\mathbb{P}(X_{\mathrm{all}}\mid M_2)}.
On the Bayes factor page the same objects are written for a sample x and a parameter \theta_j. The marginal likelihood in model M_j is the integral of the likelihood against the parameter prior,
p(x\mid M_j) =\int p(x\mid\theta_j,M_j)\,p(\theta_j\mid M_j)\,d\theta_j,
and the Bayes factor comparing M_1 with M_2 is \mathrm{BF}_{12}=p(x\mid M_1)/p(x\mid M_2). It averages the likelihood over the parameter prior. It does not evaluate the likelihood at a fitted maximum. Posterior odds equal the Bayes factor times the prior odds. The same relation holds for any finite list of models.
The marginal likelihood sometimes has a closed form. More often it is approximated, by Markov chain Monte Carlo or by the Laplace expansion below.
The main difficulty is that models with different component counts have different parameter spaces. Comparing the heights of two likelihood peaks is a different calculation. The integral is taken inside each model’s own parameter space, and the resulting marginal likelihoods are what get compared. That integration accounts for how much parameter space fits the observations well.
For either model the marginal likelihood of model k is
\mathbb{P}(X_{\mathrm{all}}\mid M_k) =\int_{\Theta_k}\mathbb{P}(X_{\mathrm{all}}\mid\Psi_k,M_k)\,\mathbb{P}(\Psi_k\mid M_k)\,d\Psi_k.
Think of the maximised likelihood as a peak height and the marginal likelihood as a prior-weighted volume. A tall, very narrow peak need not win against a lower, broader one. For continuous data the probability notation in these displays denotes densities of the observed measurements. The probabilities of the discrete model labels remain ordinary probabilities.
Used in
See also
Marginal likelihood and Bayes factor · Bayesian information criterion · Posterior hypothesis probability · Bayes action for two hypotheses · Posterior-odds factorisation · A hypothesis region is not a separate model · Probability statements and decisions · Poisson imperfect-fruit example