Bayes’ rule
Theorem taught in Bayesian foundations, part of Bayesian statistics
Bayes’ rule
| Type | Theorem |
| Taught in | Bayesian foundations |
| Part | Bayesian statistics |
| Worked examples | 1 |
| Questions using it | 8 |
For events A and B with \mathbb{P}(B)>0,
\mathbb{P}(A\mid B)=\frac{\mathbb{P}(B\mid A)\mathbb{P}(A)}{\mathbb{P}(B)}.
The density version follows from two factorizations of a joint density:
p(x,\theta)=p(x\mid\theta)p(\theta) =p(\theta\mid x)p(x).
Equating the right-hand sides and dividing by p(x) gives p(\theta\mid x) =\frac{p(x\mid\theta)p(\theta)}{p(x)}.
Examples
Q15. Prior beliefs and conflicting data
Question page: Q15. Prior beliefs and conflicting data
[3 marks]
A researcher models independent observations as Bernoulli with unknown success probability \theta. The researcher chooses a prior that is uniform on (0,1/2).
Of 100 observed trials, 80 are successes. The researcher argues that these data should make the posterior probability of \theta>1/2 large.
Is this reasoning correct? Explain what happens and whether collecting more observations can overcome the restriction imposed by this prior.
Definitions. Let L(\theta;{\boldsymbol{x}})=p({\boldsymbol{x}}\mid\theta) and let m({\boldsymbol{x}})=\int L(\theta;{\boldsymbol{x}})p(\theta)\,d\theta be the marginal likelihood. Bayes’ update is p(\theta\mid{\boldsymbol{x}})=L(\theta;{\boldsymbol{x}})p(\theta)/m({\boldsymbol{x}}), provided 0<m({\boldsymbol{x}})<\infty. The prior here excludes the entire region \theta>1/2, not merely a single value.
The reasoning is incorrect. The prior density is zero for every \theta>1/2. Bayes’ theorem gives p(\theta\mid{\boldsymbol{x}}) \propto p({\boldsymbol{x}}\mid\theta)p(\theta), so p(\theta\mid{\boldsymbol{x}})=0 \qquad\text{whenever }\theta>1/2. Hence \boxed{ \mathbb{P}(\theta>1/2\mid{\boldsymbol{x}})=0. } This remains true regardless of how many observations are collected. The likelihood can redistribute posterior mass only among parameter values to which the prior assigns positive density; it cannot create support in a region excluded by the prior.
The observed proportion 80/100=0.8 instead reveals serious conflict between the data and the assumed model–prior combination. The researcher should reconsider the restricted prior, or investigate whether the Bernoulli model is appropriate. With the restriction left unchanged, additional data cannot make the posterior probability above 1/2 positive.
Used in
- Taught in Bayesian foundations, where the derivation and examples are.
- Also used in Random experiments and probability · Probability spaces and events · Statistical inference · Bayesian hypotheses and model comparison · Hierarchical Bayes · Bayesian linear regression · Reference formulas and problem-solving workflow
- Questions that use it (8) — A one received on a noisy channel (answer) · A channel with unequal inputs (answer) · A geometric observation and a uniform prior (answer) · Sufficiency and the posterior · Q2 (answer) · Q7 (answer) · Q15 (answer) · Question 3 (answer)
- All 15 questions on Bayesian foundations
See also
Joint probability density · Posterior-odds factorisation · Bayesian credible interval · Proper prior and proper posterior · Posterior point summaries · Credible-region constructions · Posterior summaries from draws · R’s distribution interface