Full conditionals under independent priors

Proposition taught in Conjugate analysis for normal models, part of Bayesian statistics

Full conditionals under independent priors

Type Proposition
Taught in Conjugate analysis for normal models
Part Bayesian statistics
Worked examples 3

Holding \sigma^2 fixed, the calculation from Equation (1) in Conjugate analysis for normal models applies verbatim: \boxed{ \mu\mid\sigma^2,{\boldsymbol{x}} \sim\operatorname{N}\!\left( \frac{n\overline x\,\sigma_0^2+\mu_0\sigma^2} {n\sigma_0^2+\sigma^2}, \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2} \right)}. Holding \mu fixed, the calculation from Equation (3) in Conjugate analysis for normal models applies: \boxed{ \sigma^2\mid\mu,{\boldsymbol{x}} \sim\operatorname{InvGamma}\!\left( \alpha_0+\frac n2, \beta_0+\frac12\sum_{i=1}^n(x_i-\mu)^2 \right)}.

Examples

Normal mean and variance

Worked example from Markov chains, Gibbs sampling, and Metropolis–Hastings

Question. For the joint normal-model posterior in Equation (4) in Conjugate analysis for normal models, describe a systematic Gibbs sampler and state how retained draws estimate the posterior mean of \mu.

Solution. For the joint posterior in Equation (4) in Conjugate analysis for normal models, alternate \begin{align*} \mu^{(t+1)}&\sim p(\mu\mid\sigma^{2(t)},{\boldsymbol{x}}) \quad\text{using (5)},\\ \sigma^{2(t+1)}&\sim p(\sigma^2\mid\mu^{(t+1)},{\boldsymbol{x}}) \quad\text{using (6)}. \end{align*} After discarding a warm-up segment, Monte Carlo estimates use the retained draws. For example,

\widehat{\mathbb{E}}(\mu\mid{\boldsymbol{x}}) =\frac1L\sum_{t=B+1}^{B+L}\mu^{(t)},

where B is the number of discarded iterations. Because the draws are dependent, L correlated draws contain less information than L independent draws. Autocorrelation, trace plots, multiple chains, and effective sample size are therefore part of a serious analysis.

Q49. A Gibbs full conditional under independent priors

Question page: Q49. A Gibbs full conditional under independent priors

[6 marks]

Suppose X_i\mid\mu,\tau\overset{\mathrm{iid}}{\sim} N\!\left(\mu,\frac1\tau\right),\qquad i=1,\ldots,n, where \mu\in\mathbb R and \tau>0 are unknown.

Assign independent priors \mu\sim N\!\left(\mu_0,\frac1{\kappa_0}\right), \qquad \tau\sim\operatorname{Gamma}(a_0,b_0), where all hyperparameters are known and \kappa_0,a_0,b_0>0. The Gamma distribution is parametrised by shape and rate.

(a) Derive the full conditional distribution \tau\mid\mu,{\boldsymbol{x}}. Show its kernel and identify the distribution and parameters. [4 marks]

(b) Does this full conditional depend explicitly on \mu_0 or \kappa_0? Explain why or why not. [2 marks]

Definitions. Precision is \tau=1/\sigma^2. N(m,v) uses variance v, and \operatorname{Gamma}(a,b) uses shape a and rate b, with kernel \tau^{a-1}e^{-b\tau} on \tau>0. The full conditional for \tau holds \mu and the data fixed, so only factors that vary with \tau remain.

Assumptions. The priors are independent: \mu\sim N(\mu_0,1/\kappa_0) and \tau\sim\operatorname{Gamma}(a_0,b_0), with \kappa_0,a_0,b_0>0. In particular, the prior variance of \mu does not depend on \tau; this distinguishes the question from a Normal–Gamma conditional prior.

(a) Full conditional for \tau. The Normal density with precision \tau is f(x_i\mid\mu,\tau) =\left(\frac\tau{2\pi}\right)^{1/2} \exp\left\{-\frac\tau2(x_i-\mu)^2\right\}. Therefore the likelihood kernel is L(\mu,\tau;{\boldsymbol{x}})\propto \tau^{n/2}\exp\left\{-\frac\tau2\sum_{i=1}^n(x_i-\mu)^2\right\}. The independent priors give the joint posterior kernel \begin{aligned} p(\mu,\tau\mid{\boldsymbol{x}}) &\propto \tau^{n/2}\exp\left\{-\frac\tau2\sum_i(x_i-\mu)^2\right\}\\ &\quad\times\exp\left\{-\frac{\kappa_0}{2}(\mu-\mu_0)^2\right\} \tau^{a_0-1}e^{-b_0\tau}. \end{aligned} For fixed \mu, its prior factor is constant with respect to \tau. Dropping that factor and collecting powers and exponents gives \boxed{ p(\tau\mid\mu,{\boldsymbol{x}})\propto \tau^{a_0+n/2-1} \exp\left[-\left\{b_0+\frac12\sum_i(x_i-\mu)^2\right\}\tau\right], \quad\tau>0. } Hence \boxed{ \tau\mid\mu,{\boldsymbol{x}}\sim\operatorname{Gamma}\!\left( a_0+\frac n2,\ b_0+\frac12\sum_{i=1}^n(x_i-\mu)^2\right) \quad\text{(shape--rate)}. }

(b) Dependence on \mu_0 and \kappa_0. No: the full conditional has no explicit dependence on these two hyperparameters. Under the independent priors, p(\mu,\tau)=p(\mu)p(\tau), and once \mu is conditioned on, p(\mu) is a positive constant in the conditional kernel for \tau. It cancels on normalisation.

This does not mean that the prior for \mu is irrelevant to all inference about \tau. It affects the posterior distribution of \mu, and thus can affect the marginal posterior of \tau after \mu is integrated out.

Key result. Do not silently replace the independent prior with \mu\mid\tau\sim N(\mu_0,1/(\kappa_0\tau)). That is a different model whose prior contributes additional \tau-dependent factors to the full conditional.

Question 3. Normal mean and variance

Question page: Question 3. Normal mean and variance

Let X_i\mid\mu,\sigma^2\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2), with independent priors

\mu\sim \operatorname{N}(\mu_0,\sigma_0^2), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0).

(a) Does the joint posterior have this same product form?

(b) Write the full conditional distributions of \mu and of \sigma^2.

(c) Which updating algorithm uses these two distributions as its steps?

(d) Explain why independent priors for \mu and \sigma^2 do not generally give independent posteriors.

(e) Distinguish a full conditional posterior from a marginal posterior.

(f) Explain why a standard full conditional distribution does not imply that the corresponding marginal posterior belongs to the same family.

(g) Explain how a Gibbs sampler uses these full conditional distributions.

Definitions. Write v=\sigma^2, S=\sum_i(x_i-\bar x)^2, and use \operatorname{InvGamma}(a,b) with kernel v^{-a-1}e^{-b/v}. Prior independence means p(\mu,v)=p(\mu)p(v). A full conditional fixes the other unknown parameter; a marginal posterior averages over its posterior uncertainty.

(a) No. After the data are observed, \mu and \sigma^2 are dependent, so the joint posterior is not an independent normal times an independent inverse-gamma.

Bayes’ theorem gives the joint kernel

p(\mu,v\mid{\boldsymbol{x}})\propto v^{-(\alpha_0+n/2)-1} \exp\left\{-\frac{\beta_0+\{S+n(\mu-\bar x)^2\}/2}{v} -\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right\}.

The term (\mu-\bar x)^2/v couples the two unknowns. The same discrepancy between \mu and the data is more plausible when the observation variance is larger.

(b) With the other parameter held fixed, the updates are the univariate conjugate posteriors:

\begin{aligned} \mu\mid\sigma^2,x &\sim N\Biggl( \frac{n\bar x\,\sigma_0^2+\mu_0\sigma^2}{n\sigma_0^2+\sigma^2},\; \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2} \Biggr),\\[4pt] \sigma^2\mid\mu,x &\sim \operatorname{InvGamma}\Bigl( \alpha_0+\frac n2,\; \beta_0+\frac12\sum_{i=1}^n(x_i-\mu)^2 \Bigr). \end{aligned}

The normal variance equals \sigma_1^2 from the known-variance update, because

\Bigl(\frac n{\sigma^2}+\frac1{\sigma_0^2}\Bigr)^{-1} = \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2}.

(c) A Gibbs sampler uses these two distributions. Choose v^{(0)}>0. At sweep t, draw \mu^{(t)} from the normal full conditional using v^{(t-1)}, then draw v^{(t)} from the inverse-gamma full conditional using \mu^{(t)}. Record the completed pair. The second draw uses the newly updated mean, rather than its value from the previous sweep.

For compact notation in the algorithm, define

\begin{aligned} m(v)&=\frac{n\bar x/v+\mu_0/\sigma_0^2}{n/v+1/\sigma_0^2}, &V(v)&=\left(\frac n v+\frac1{\sigma_0^2}\right)^{-1},\\ b(\mu)&=\beta_0+\frac12\sum_i(x_i-\mu)^2. \end{aligned}

Here B is the number of initial sweeps discarded as warm-up, not a guarantee of convergence. Assess mixing from multiple valid starting values, trace plots and autocorrelation, and account for dependence when computing Monte Carlo error. An exact full-conditional draw needs no Metropolis acceptance step.

(d) The prior density factors as p(\mu)\,p(\sigma^2). The likelihood does not. The factor \sigma^{-2}\sum_{i=1}^n(x_i-\mu)^2 depends on both parameters, so

p(\mu,\sigma^2\mid{\boldsymbol{x}})

does not factor into a function of \mu alone times a function of \sigma^2 alone.

(e) A full conditional treats the other parameter as known:

p(\mu\mid\sigma^2,{\boldsymbol{x}}) \quad\text{and}\quad p(\sigma^2\mid\mu,{\boldsymbol{x}}).

The marginal posteriors integrate that parameter out,

p(\mu\mid{\boldsymbol{x}}) = \int p(\mu\mid\sigma^2,{\boldsymbol{x}})\,p(\sigma^2\mid{\boldsymbol{x}})\,d\sigma^2, \qquad p(\sigma^2\mid{\boldsymbol{x}}) = \int p(\sigma^2\mid\mu,{\boldsymbol{x}})\,p(\mu\mid{\boldsymbol{x}})\,d\mu.

(f) The full conditional p(\mu\mid\sigma^2,{\boldsymbol{x}}) is normal at each fixed \sigma^2. The marginal p(\mu\mid{\boldsymbol{x}}) mixes those normals over the posterior distribution of \sigma^2; a mixture need not belong to the component family. Here, integrating the joint kernel over v gives

p(\mu\mid{\boldsymbol{x}})\propto \exp\left\{-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right\} \left\{\beta_0+\frac{S+n(\mu-\bar x)^2}{2}\right\}^{-(\alpha_0+n/2)}.

This is generally neither a normal nor a Student-t kernel. Likewise, the marginal variance posterior is a mixture of inverse-gamma full conditionals and is not generally inverse-gamma. Substituting a point estimate for the other unknown does not perform the required integration.

(g) Each Gibbs step draws from one full conditional, using the most recent value of the conditioned parameter. The stationary distribution of the pair is the joint posterior p(\mu,\sigma^2\mid{\boldsymbol{x}}). A marginal posterior is represented by the simulated values of one coordinate, not by a single full-conditional distribution.

After suitable convergence assessment and warm-up, averages of functions of the retained pairs approximate joint-posterior expectations. Keeping only the \mu coordinate approximates its marginal posterior automatically: uncertainty in v has been propagated through the chain. Successive draws are generally correlated, so Monte Carlo standard errors must account for that dependence.

Used in

See also

Normal–Normal model · A proper approximation to the flat prior · Normal–Normal conjugacy · Precision weighting · Inverse-Gamma distribution · Normal–inverse-Gamma variance update

Glossary and index