Full conditional distribution

Topic of Markov chains, Gibbs sampling, and Metropolis–Hastings, part of Bayesian statistics · Also in Conjugate analysis for normal models

Full conditional distribution

Type Topic
Taught in Markov chains, Gibbs sampling, and Metropolis–Hastings · Conjugate analysis for normal models
Part Bayesian statistics
Questions using it 23

The distribution of one parameter given the data and all the other parameters, \pi(x_j\mid x_{-j}). A Gibbs sampler draws from each full conditional in turn. Up to a constant it is the joint density read as a function of x_j alone, which is how it is found.

In Markov chains, Gibbs sampling, and Metropolis–Hastings

From Markov chains, Gibbs sampling, and Metropolis–Hastings

Suppose the target is a joint density \pi(x_1,\ldots,x_p) that is difficult to sample from directly, while every full conditional is available. One Gibbs sweep updates coordinates in order: \begin{align*} X_1^{(t+1)}&\sim \pi(x_1\mid X_2^{(t)},\ldots,X_p^{(t)}),\\ X_2^{(t+1)}&\sim \pi(x_2\mid X_1^{(t+1)},X_3^{(t)},\ldots,X_p^{(t)}),\\ &\ \vdots\\ X_p^{(t+1)}&\sim \pi(x_p\mid X_1^{(t+1)},\ldots,X_{p-1}^{(t+1)}). \end{align*} Each update uses the newest available coordinate values. Under standard ergodicity conditions, X^{(t)} converges in distribution to \pi. For a full conditional, the other coordinates are held fixed. Therefore

\pi(x_i\mid x_{-i}) =\frac{\pi(x_i,x_{-i})}{\pi(x_{-i})} \propto\pi(x_i,x_{-i}) \quad\text{as a function of }x_i.

In Conjugate analysis for normal models

From Conjugate analysis for normal models

Consider the independent prior

\mu\sim\operatorname{N}(\mu_0,\sigma_0^2), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0), \qquad \mu\perp\sigma^2.

This prior has two coordinates, so a request to software for its mean should return two numbers, not four. When \alpha_0>1,

\mathbb{E}\!\begin{pmatrix}\mu\\ \sigma^2\end{pmatrix} =\begin{pmatrix}\mu_0\\ \beta_0/(\alpha_0-1)\end{pmatrix}.

If the required second moments exist, the covariance matrix is diagonal under the prior because \mu and \sigma^2 are independent. This statement does not survive the update: the joint posterior in Equation (4) in Conjugate analysis for normal models couples them.

With both parameters unknown, the joint posterior is \begin{aligned} p(\mu,\sigma^2\mid{\boldsymbol{x}}) &\propto p({\boldsymbol{x}}\mid\mu,\sigma^2) p(\mu)p(\sigma^2)\\ &\propto (\sigma^2)^{-n/2} \exp\!\left[-\frac1{2\sigma^2} \sum_{i=1}^n(x_i-\mu)^2\right]\\ &\qquad\times \exp\!\left[-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right] (\sigma^2)^{-\alpha_0-1} \exp\!\left(-\frac{\beta_0}{\sigma^2}\right)\\ &\propto (\sigma^2)^{-(\alpha_0+n/2)-1} \exp\!\left[-\frac{\beta_0+\frac12\sum_i(x_i-\mu)^2} {\sigma^2} -\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right]. \end{aligned} This joint density is not a product of independent posterior marginals. It is also not the normal–inverse-Gamma family in which the prior variance of \mu is a multiple of \sigma^2. Here that prior variance stays \sigma_0^2, independent of \sigma^2. Nevertheless, both full conditional distributions are standard.

Proposition 1 (Full conditionals under independent priors) Holding \sigma^2 fixed, the calculation from Equation (1) in Conjugate analysis for normal models applies verbatim: \boxed{ \mu\mid\sigma^2,{\boldsymbol{x}} \sim\operatorname{N}\!\left( \frac{n\overline x\,\sigma_0^2+\mu_0\sigma^2} {n\sigma_0^2+\sigma^2}, \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2} \right)}. Holding \mu fixed, the calculation from Equation (3) in Conjugate analysis for normal models applies: \boxed{ \sigma^2\mid\mu,{\boldsymbol{x}} \sim\operatorname{InvGamma}\!\left( \alpha_0+\frac n2, \beta_0+\frac12\sum_{i=1}^n(x_i-\mu)^2 \right)}.

Article: Full conditionals under independent priors →

These two conditionals lead directly to the Gibbs sampler in the next chapter.

Used in

See also

Gibbs sampling · Systematic-scan Gibbs sampler · Joint probability density · Markov chain · Stationarity and detailed balance · Detailed balance implies stationarity · Warm-up is not a convergence proof · Metropolis–Hastings update

Glossary and index