Q14. Marginal and conditional distributions

Topic: Markov chains, Gibbs sampling, and Metropolis–Hastings · more questions on this topic
‹ Q10 Q18 ›

[4 marks]

Suppose two unknown parameters have joint posterior density p(\theta_1,\theta_2\mid{\boldsymbol{x}}). Explain the distinction between the marginal posterior p(\theta_1\mid{\boldsymbol{x}}) and the full conditional posterior p(\theta_1\mid\theta_2,{\boldsymbol{x}}). In your answer, state what is being conditioned on and describe how each density can be obtained from the joint posterior.

Worked answer (click to show)

Definitions. Let {\boldsymbol{x}} be the observed data and p(\theta_1,\theta_2\mid{\boldsymbol{x}}) a proper joint posterior. A marginal integrates out an unreported coordinate; a full conditional holds every other coordinate fixed. Conditional-density ratios below are used only where p(\theta_2\mid{\boldsymbol{x}})>0. The symbol u is a dummy integration variable; whether \theta_2 is integrated out or held fixed distinguishes the two calculations.

The marginal posterior p(\theta_1\mid{\boldsymbol{x}}) conditions only on the observed data. It describes uncertainty about \theta_1 while averaging over all possible values of \theta_2. It is obtained by integrating \theta_2 out of the joint posterior: \boxed{ p(\theta_1\mid{\boldsymbol{x}}) =\int p(\theta_1,\theta_2\mid{\boldsymbol{x}})\,d\theta_2. }

The full conditional posterior p(\theta_1\mid\theta_2,{\boldsymbol{x}}) conditions on both the data and a specified value of \theta_2. By the definition of conditional density, \begin{aligned} p(\theta_1\mid\theta_2,{\boldsymbol{x}}) &=\frac{p(\theta_1,\theta_2\mid{\boldsymbol{x}})} {p(\theta_2\mid{\boldsymbol{x}})}\\ &=\boxed{ \frac{p(\theta_1,\theta_2\mid{\boldsymbol{x}})} {\int p(u,\theta_2\mid{\boldsymbol{x}})\,du} }. \end{aligned} Equivalently, when viewed as a function of \theta_1, p(\theta_1\mid\theta_2,{\boldsymbol{x}}) \propto p(\theta_1,\theta_2\mid{\boldsymbol{x}}), because the normalising denominator is constant with respect to \theta_1. Thus marginalisation averages over the other parameter, whereas conditioning holds it fixed. Full conditionals are the distributions used for Gibbs updates.