Q6. Beta-Binomial conjugacy and prior influence

Topic: The Beta–Binomial model · more questions on this topic
‹ A logit-normal prior for a coin Q17 ›

Suppose X\mid\theta\sim\operatorname{Bin}(n,\theta), \qquad \theta\sim\operatorname{Beta}(a,b), where n\geq1, a,b>0, and the observed value of X is x.

(a) Derive the posterior density of \theta. Identify its distribution, parameters, and support. [3 marks]

(b) Express the posterior mean as a weighted average of the prior mean and x/n. Explain how the weight on the prior changes as n increases, with a and b fixed. [3 marks]

(c) Another statistician instead uses a prior uniform on (0,1/2). If x/n=4/5, can the posterior assign positive probability to \theta>1/2? Explain why or why not. [2 marks]

(d) Under squared-error loss, explain why the posterior mean is the Bayes estimate. Is it necessarily the Bayes estimate under a different loss? [2 marks]

Worked answer (click to show)

Definitions. For a,b>0, a \operatorname{Beta}(a,b) density is \theta^{a-1}(1-\theta)^{b-1}/B(a,b) on (0,1), where B(a,b)=\int_0^1u^{a-1}(1-u)^{b-1}\,du. Here x is a Binomial success count out of known n. A Bayes estimate is an action d minimising posterior expected loss. Part (c) substitutes a restricted prior; part (d) returns to the Beta prior from (a).

(a) Posterior distribution

The likelihood and prior kernels are p(x\mid\theta)\propto\theta^x(1-\theta)^{n-x}, \qquad p(\theta)\propto\theta^{a-1}(1-\theta)^{b-1}. Therefore, \begin{aligned} p(\theta\mid x) &\propto p(x\mid\theta)p(\theta)\\ &\propto \theta^{a+x-1}(1-\theta)^{b+n-x-1}, \qquad 0<\theta<1. \end{aligned} Thus \boxed{ \theta\mid x\sim\operatorname{Beta}(a+x,b+n-x) }. The normalised posterior density is p(\theta\mid x) =\frac{\theta^{a+x-1}(1-\theta)^{b+n-x-1}} {B(a+x,b+n-x)}, \qquad 0<\theta<1.

(b) Posterior mean and prior influence

The posterior mean is \mathbb{E}(\theta\mid x)=\frac{a+x}{a+b+n}. Since the prior mean is \mathbb{E}(\theta)=a/(a+b), \begin{aligned} \mathbb{E}(\theta\mid x) &= \frac{a+b}{a+b+n}\frac{a}{a+b} +\frac{n}{a+b+n}\frac{x}{n}\\ &= \boxed{ \frac{a+b}{a+b+n}\mathbb{E}(\theta) +\frac{n}{a+b+n}\frac{x}{n} }. \end{aligned} The two weights sum to one. With a and b fixed, the prior weight (a+b)/(a+b+n) decreases to zero as n\to\infty, while the data weight increases to one.

(c) Restricted prior support

No. A prior uniform on (0,1/2) has density zero for \theta>1/2. Because p(\theta\mid x)\propto p(x\mid\theta)p(\theta), the posterior is also zero wherever the prior is zero. Hence \boxed{\mathbb{P}(\theta>1/2\mid x)=0}, even when x/n=4/5. The likelihood can reweight values inside the prior support, but it cannot create posterior mass outside that support.

(d) Bayes estimate under squared-error loss

Write m=\mathbb{E}(\theta\mid x) and expand \theta-d=(\theta-m)+(m-d). The cross term has posterior expectation 2(m-d)\mathbb{E}(\theta-m\mid x)=0. Consequently, for an estimate d, the posterior expected squared loss is \begin{aligned} \mathbb{E}\bigl[(\theta-d)^2\mid x\bigr] &= \operatorname{Var}(\theta\mid x) +\bigl\{\mathbb{E}(\theta\mid x)-d\bigr\}^2. \end{aligned} The variance term does not depend on d, so this expression is minimised by \boxed{ \widehat{\theta}_{\mathrm{Bayes}} =\mathbb{E}(\theta\mid x) =\frac{a+x}{a+b+n}. } The Bayes estimate depends on the loss function: for example, absolute-error loss gives a posterior median rather than necessarily the posterior mean.