Q26. General Beta–Binomial Gibbs sampler

Topic: Markov chains, Gibbs sampling, and Metropolis–Hastings · more questions on this topic
‹ Q25 Q50 ›

[15 marks]

Consider the joint density/mass function, given up to proportionality, f(x,\theta)\propto \binom{n}{x}\theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1}, for x=0,1,\ldots,n, 0\leq\theta\leq1, with n a known positive integer and \alpha,\beta>0 known.

Normalisation clarification: the expression in the screenshot is treated as a kernel, since a constant is needed to make its total mass one.

(i) [4 marks] Find the conditional distribution of X given \theta and name the distribution. It is sufficient to work out the kernel, without specifying the normalising constant.

(ii) [4 marks] Find the conditional distribution of \theta given X=x and name the distribution. As in part (i), you may omit the normalising constant.

(iii) [7 marks] Using parts (i) and (ii), or otherwise, describe how you would construct a Gibbs sampler to sample from f(x,\theta).

Worked answer (click to show)

Definitions. A \operatorname{Binomial}(n,p) mass function is \binom nx p^x(1-p)^{n-x} for x=0,\ldots,n. A \operatorname{Beta}(a,b) density has kernel \theta^{a-1}(1-\theta)^{b-1} on (0,1) and normalising integral B(a,b). Here n is a known positive integer and \alpha,\beta>0. A full conditional treats the other variable as fixed; (t) indexes complete Gibbs sweeps.

(i) Conditional distribution of X\mid\theta. For fixed 0<\theta<1, factor the joint kernel as f(x,\theta)\propto \underbrace{\theta^{\alpha-1}(1-\theta)^{\beta-1}}_{\text{constant in }x} \binom nx\theta^x(1-\theta)^{n-x}. The first factor can be dropped when finding the conditional distribution of X, but \binom nx must remain because it depends on x. Thus f(x\mid\theta)\propto\binom nx\theta^x(1-\theta)^{n-x}, \qquad x=0,1,\ldots,n. This is the Binomial kernel, so \boxed{X\mid\theta\sim\operatorname{Binomial}(n,\theta).} Indeed, the displayed Binomial probabilities already sum to one by the binomial theorem. At the endpoints the corresponding conditional is degenerate at 0 or n; these endpoints have zero probability under the continuous Beta marginal.

(ii) Conditional distribution of \theta\mid X=x. Now fix x. The binomial coefficient is constant with respect to \theta, giving f(\theta\mid x)\propto \theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1},\qquad 0<\theta<1. Matching the exponents to a Beta density yields \boxed{\theta\mid X=x\sim \operatorname{Beta}(\alpha+x,\beta+n-x).} Both shape parameters are positive because \alpha,\beta>0 and 0\leq x\leq n.

Normalisation check. Summing the joint kernel over x gives \begin{aligned} \sum_{x=0}^{n}\binom nx \theta^{x+\alpha-1}(1-\theta)^{n-x+\beta-1} &=\theta^{\alpha-1}(1-\theta)^{\beta-1} \sum_{x=0}^{n}\binom nx\theta^x(1-\theta)^{n-x}\\ &=\theta^{\alpha-1}(1-\theta)^{\beta-1}. \end{aligned} Integrating this over \theta gives B(\alpha,\beta). The normalised joint density/mass function is therefore the supplied expression divided by B(\alpha,\beta). This missing constant does not affect either conditional distribution.

(iii) Gibbs sampler. Choose x^{(0)}\in\{0,\ldots,n\} and 0<\theta^{(0)}<1. For t=1,\ldots,T:

1. Draw X^{(t)}\sim\operatorname{Binomial}(n,\theta^{(t-1)}).

2. Using the new count, draw \theta^{(t)}\sim \operatorname{Beta}(\alpha+X^{(t)},\beta+n-X^{(t)}).

3. Record (X^{(t)},\theta^{(t)}) and repeat.

The second update must use X^{(t)}, not the old value X^{(t-1)}. Each update samples exactly from a full conditional, so every draw is accepted and there is no Metropolis–Hastings acceptance step.

To see why the target is invariant, suppose a current pair has joint distribution f. Updating X conditional on its current \theta gives the joint distribution f(\theta)f(x\mid\theta)=f(x,\theta) again. Updating \theta conditional on that newly drawn X likewise preserves the same joint distribution. Thus the full sweep has f as its equilibrium distribution.

Run the chain for sufficient warm-up, assess convergence, and retain subsequent pairs for inference. The retained pairs are generally dependent; their empirical averages approximate joint expectations as the run length increases under the usual ergodicity conditions.

Key result. Hold one variable fixed, retain every factor involving the other, and then alternate exact Binomial and Beta draws using the most recently updated value.