Marginal likelihood and Bayes factor

Definition taught in Bayesian hypotheses and model comparison, part of Bayesian statistics

Marginal likelihood and Bayes factor

Type Definition
Taught in Bayesian hypotheses and model comparison
Part Bayesian statistics
Properties 2
Worked examples 2
Questions using it 13

The marginal likelihood in model M_j is p(x\mid M_j) =\int p(x\mid\theta_j,M_j) p(\theta_j\mid M_j)\,\mathrm{d}\theta_j. It averages the likelihood over the parameter prior; it does not evaluate the likelihood at a fitted maximum. The Bayes factor comparing M_1 with M_2 is \operatorname{BF}_{12}=p(x\mid M_1)/p(x\mid M_2).

Properties

Posterior-odds factorisation

Main article: Posterior-odds factorisation

For two models M_1 and M_2, Bayes’ rule gives

\mathbb{P}(M_j\mid x) =\frac{p(x\mid M_j)\mathbb{P}(M_j)}{p(x)}.

Taking the ratio cancels the common evidence p(x): \begin{aligned} \frac{\mathbb{P}(M_1\mid x)}{\mathbb{P}(M_2\mid x)} &=\frac{p(x\mid M_1)\mathbb{P}(M_1)/p(x)} {p(x\mid M_2)\mathbb{P}(M_2)/p(x)}\\ &=\underbrace{\frac{p(x\mid M_1)}{p(x\mid M_2)}}_{ \operatorname{BF}_{12}} \underbrace{\frac{\mathbb{P}(M_1)}{\mathbb{P}(M_2)}}_{ \text{prior odds}}. \end{aligned}

A hypothesis region is not a separate model

Main article: A hypothesis region is not a separate model

Two constructions must not be confused:

  1. If H_0 and H_1 are regions of one parameter space, compute one posterior and integrate it over the regions.

  2. If M_1 and M_2 have different dimensions or parameterisations, each model needs its own parameter prior and marginal likelihood. A continuous prior on the larger space assigns probability zero to a lower-dimensional surface.

The fruit example is the first construction. Independence in the advertising table is the second.

Examples

Q11. Bayes factors and posterior odds

Question page: Q11. Bayes factors and posterior odds

Suppose X\mid\theta\sim\operatorname{Bin}(n,\theta). Consider M_0:\theta=\frac12, \qquad M_1:\theta\sim\operatorname{Unif}(0,1). Under M_1, the stated distribution is the prior for \theta. Write m_j(x)=p(x\mid M_j).

(a) Derive m_0(x) and m_1(x). Retain the constants in the binomial probability mass function. [4 marks]

(b) Obtain B_{10}=\frac{m_1(x)}{m_0(x)}. Explain how it relates prior model odds to posterior model odds. [3 marks]

(c) Suppose n=3, x=3, and \mathbb{P}(M_1)=1/5. Calculate B_{10} and \mathbb{P}(M_1\mid x). Explain how the data can favour M_1 while M_0 still has the larger posterior probability. [3 marks]

Definitions. M_0 fixes \theta=1/2; M_1 assigns \theta\sim\operatorname{Beta}(1,1). Write m_j(x)=\mathbb P(X=x\mid M_j) for the marginal likelihood and B_{10}=m_1(x)/m_0(x) for the Bayes factor favouring M_1. The Beta integral is B(a,b)=\int_0^1u^{a-1}(1-u)^{b-1}\,du. Prior probabilities of the two models are distinct from their priors for \theta.

(a) Marginal likelihoods

Under M_0, the marginal probability of observing X=x is m_0(x) = \mathbb{P}(X=x\mid M_0) = \binom{n}{x}\left(\frac12\right)^x \left(\frac12\right)^{n-x} = \boxed{\binom{n}{x}2^{-n}}.

Under M_1, integrate the likelihood with respect to the uniform prior: \begin{aligned} m_1(x) &= \mathbb{P}(X=x\mid M_1)\\ &= \binom{n}{x} \int_0^1\theta^x(1-\theta)^{n-x}\,d\theta\\ &= \binom{n}{x}B(x+1,n-x+1)\\ &= \binom{n}{x} \frac{x!(n-x)!}{(n+1)!}\\ &= \boxed{\frac{1}{n+1}}. \end{aligned}

(b) Bayes factor

The Bayes factor in favour of M_1 over M_0 is \boxed{ B_{10} = \frac{m_1(x)}{m_0(x)} = \frac{2^n}{(n+1)\binom{n}{x}}. } Values greater than one favour M_1, while values less than one favour M_0.

(c) Posterior model odds

Posterior odds equal the Bayes factor multiplied by prior odds: \boxed{ \frac{\mathbb{P}(M_1\mid x)}{\mathbb{P}(M_0\mid x)} = B_{10} \frac{\mathbb{P}(M_1)}{\mathbb{P}(M_0)}. }

(d) Numerical comparison

For n=3 and x=3, m_0(3)=\frac18, \qquad m_1(3)=\frac14, \qquad B_{10}=2. Thus the observed data are twice as probable under M_1 as under M_0, so the data favour M_1.

If \mathbb{P}(M_1)=\frac15, \qquad \mathbb{P}(M_0)=\frac45, then the prior odds for M_1 are 1/4. Hence \frac{\mathbb{P}(M_1\mid x)}{\mathbb{P}(M_0\mid x)} =2\left(\frac14\right) =\frac12. Therefore \boxed{ \mathbb{P}(M_1\mid x)=\frac13, \qquad \mathbb{P}(M_0\mid x)=\frac23. } The data favour M_1, but the prior favoured M_0 by four to one. The evidence from this small sample is not strong enough to overturn that prior preference, so M_0 remains more probable a posteriori.

Question 15. Battery lifetimes and a Bayes factor

Question page: Question 15. Battery lifetimes and a Bayes factor

Researchers are attempting to determine the distribution of lifetimes of a type of battery. They have measured the lifetimes (years) of n of these batteries, recorded as x_1,\ldots,x_n, which we assume were the result of a random sample and we will assume that the observations could be considered independent and identically distributed. Two models were proposed:

M_1\colon x\sim\operatorname{Exp}(\theta) \qquad\text{and}\qquad M_2\colon x\sim\operatorname{Weibull}(5,\theta).

This Weibull density is given by

f(x)=5\theta x^{4}e^{-\theta x^{5}}, \qquad\text{for }x>0.

Consider the prior distribution \theta\sim\operatorname{Exp}(1).

(a) Find the marginal likelihood under model M_1. [3 marks]

(b) Find the marginal likelihood under model M_2. [3 marks]

(c) Obtain an expression for the Bayes factor comparing M_1 to M_2. [2 marks]

Definitions. Under M_1, \theta is an exponential rate. Under M_2, its meaning is fixed by the supplied Weibull density 5\theta x^4e^{-\theta x^5}, rather than by a software scale convention. In both models the prior density is e^{-\theta} on \theta>0. Write S_1=\sum_i x_i, S_5=\sum_i x_i^5 and P_4=\prod_i x_i^4. The marginal likelihood averages a sampling density over the specified parameter prior.

(a) Let S_1=\sum_{i=1}^{n}x_i. Independence gives, for \theta>0,

p({\boldsymbol{x}}\mid\theta,M_1) = \prod_{i=1}^{n}\theta e^{-\theta x_i} = \theta^{n}e^{-\theta S_1}.

Since S_1+1>0,

p({\boldsymbol{x}}\mid M_1) = \int_0^{\infty}\theta^{n}e^{-\theta(S_1+1)}\,d\theta.

Put u=(S_1+1)\theta. Then d\theta=du/(S_1+1) and u runs over (0,\infty), so

\begin{aligned} p({\boldsymbol{x}}\mid M_1) &= (S_1+1)^{-(n+1)}\int_0^{\infty}u^{n}e^{-u}\,du = \frac{\Gamma(n+1)}{(S_1+1)^{n+1}}. \end{aligned}

Here n is a positive integer, so \Gamma(n+1)=n!.

\boxed{p({\boldsymbol{x}}\mid M_1) = \dfrac{n!}{\bigl(1+\sum_{i=1}^{n}x_i\bigr)^{n+1}}.}

(b) Let P_4=\prod_{i=1}^{n}x_i^{4} and S_5=\sum_{i=1}^{n}x_i^{5}. The given density and independence give

p({\boldsymbol{x}}\mid\theta,M_2) = \prod_{i=1}^{n}5\theta x_i^{4}e^{-\theta x_i^{5}} = 5^{n}\theta^{n}P_4 e^{-\theta S_5}, \qquad\theta>0.

Thus

p({\boldsymbol{x}}\mid M_2) = 5^{n}P_4\int_0^{\infty}\theta^{n}e^{-(S_5+1)\theta}\,d\theta.

Now S_5+1>0. The same substitution as in (a) gives \int_0^{\infty}\theta^{n}e^{-(S_5+1)\theta}\,d\theta=n!/(S_5+1)^{n+1}. The factor 5^{n}P_4 does not depend on \theta, but it belongs to the normalised sampling density, so it remains in the marginal likelihood.

\boxed{p({\boldsymbol{x}}\mid M_2) = \dfrac{5^{n}\,n!\prod_{i=1}^{n}x_i^{4}}{\bigl(1+\sum_{i=1}^{n}x_i^{5}\bigr)^{n+1}}.}

(c) By (a) and (b),

\begin{aligned} \operatorname{BF}_{12} &= \frac{p({\boldsymbol{x}}\mid M_1)}{p({\boldsymbol{x}}\mid M_2)} = \frac{n!/(1+S_1)^{n+1}}{5^{n}n!\,P_4/(1+S_5)^{n+1}} = \frac{1}{5^{n}P_4} \Biggl(\frac{1+S_5}{1+S_1}\Biggr)^{n+1}. \end{aligned}

\boxed{\operatorname{BF}_{12} = \dfrac{1}{5^{n}\prod_{i=1}^{n}x_i^{4}} \Biggl( \dfrac{1+\sum_{i=1}^{n}x_i^{5}}{1+\sum_{i=1}^{n}x_i} \Biggr)^{n+1}.}

Thus \operatorname{BF}_{12}>1 favours M_1, and \operatorname{BF}_{12}<1 favours M_2.

The transformation U=X^5 clarifies the second integration: under M_2, U\mid\theta\sim\operatorname{Exp}(\theta). However, the observed data are the x_i, so the transformation Jacobian \prod_i5x_i^4=5^nP_4 remains in their likelihood. Dropping it would compare densities expressed for different observed variables. In posterior inference within M_2 it is constant in \theta and can be omitted; in this model comparison it cannot.

The prior in each model must also be normalised. Using a density proportional to one on the positive line in place of \operatorname{Exp}(1) would introduce an unspecified multiplicative constant into each marginal likelihood. The stated proper priors avoid that ambiguity. Posterior model odds are obtained only after multiplying BF_{12} by the prior model odds.

Used in

See also

A hypothesis region is not a separate model · Posterior hypothesis probability · Bayes action for two hypotheses · Posterior-odds factorisation

Glossary and index