Question 3. Normal mean and variance

Topic: Conjugate analysis for normal models · more questions on this topic
‹ Question 2 Question 4 ›

Let X_i\mid\mu,\sigma^2\stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2), with independent priors

\mu\sim \operatorname{N}(\mu_0,\sigma_0^2), \qquad \sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0).

(a) Does the joint posterior have this same product form?

(b) Write the full conditional distributions of \mu and of \sigma^2.

(c) Which updating algorithm uses these two distributions as its steps?

(d) Explain why independent priors for \mu and \sigma^2 do not generally give independent posteriors.

(e) Distinguish a full conditional posterior from a marginal posterior.

(f) Explain why a standard full conditional distribution does not imply that the corresponding marginal posterior belongs to the same family.

(g) Explain how a Gibbs sampler uses these full conditional distributions.

Worked answer (click to show)

Definitions. Write v=\sigma^2, S=\sum_i(x_i-\bar x)^2, and use \operatorname{InvGamma}(a,b) with kernel v^{-a-1}e^{-b/v}. Prior independence means p(\mu,v)=p(\mu)p(v). A full conditional fixes the other unknown parameter; a marginal posterior averages over its posterior uncertainty.

(a) No. After the data are observed, \mu and \sigma^2 are dependent, so the joint posterior is not an independent normal times an independent inverse-gamma.

Bayes’ theorem gives the joint kernel

p(\mu,v\mid{\boldsymbol{x}})\propto v^{-(\alpha_0+n/2)-1} \exp\left\{-\frac{\beta_0+\{S+n(\mu-\bar x)^2\}/2}{v} -\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right\}.

The term (\mu-\bar x)^2/v couples the two unknowns. The same discrepancy between \mu and the data is more plausible when the observation variance is larger.

(b) With the other parameter held fixed, the updates are the univariate conjugate posteriors:

\begin{aligned} \mu\mid\sigma^2,x &\sim N\Biggl( \frac{n\bar x\,\sigma_0^2+\mu_0\sigma^2}{n\sigma_0^2+\sigma^2},\; \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2} \Biggr),\\[4pt] \sigma^2\mid\mu,x &\sim \operatorname{InvGamma}\Bigl( \alpha_0+\frac n2,\; \beta_0+\frac12\sum_{i=1}^n(x_i-\mu)^2 \Bigr). \end{aligned}

The normal variance equals \sigma_1^2 from the known-variance update, because

\Bigl(\frac n{\sigma^2}+\frac1{\sigma_0^2}\Bigr)^{-1} = \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2}.

(c) A Gibbs sampler uses these two distributions. Choose v^{(0)}>0. At sweep t, draw \mu^{(t)} from the normal full conditional using v^{(t-1)}, then draw v^{(t)} from the inverse-gamma full conditional using \mu^{(t)}. Record the completed pair. The second draw uses the newly updated mean, rather than its value from the previous sweep.

For compact notation in the algorithm, define

\begin{aligned} m(v)&=\frac{n\bar x/v+\mu_0/\sigma_0^2}{n/v+1/\sigma_0^2}, &V(v)&=\left(\frac n v+\frac1{\sigma_0^2}\right)^{-1},\\ b(\mu)&=\beta_0+\frac12\sum_i(x_i-\mu)^2. \end{aligned}

Here B is the number of initial sweeps discarded as warm-up, not a guarantee of convergence. Assess mixing from multiple valid starting values, trace plots and autocorrelation, and account for dependence when computing Monte Carlo error. An exact full-conditional draw needs no Metropolis acceptance step.

(d) The prior density factors as p(\mu)\,p(\sigma^2). The likelihood does not. The factor \sigma^{-2}\sum_{i=1}^n(x_i-\mu)^2 depends on both parameters, so

p(\mu,\sigma^2\mid{\boldsymbol{x}})

does not factor into a function of \mu alone times a function of \sigma^2 alone.

(e) A full conditional treats the other parameter as known:

p(\mu\mid\sigma^2,{\boldsymbol{x}}) \quad\text{and}\quad p(\sigma^2\mid\mu,{\boldsymbol{x}}).

The marginal posteriors integrate that parameter out,

p(\mu\mid{\boldsymbol{x}}) = \int p(\mu\mid\sigma^2,{\boldsymbol{x}})\,p(\sigma^2\mid{\boldsymbol{x}})\,d\sigma^2, \qquad p(\sigma^2\mid{\boldsymbol{x}}) = \int p(\sigma^2\mid\mu,{\boldsymbol{x}})\,p(\mu\mid{\boldsymbol{x}})\,d\mu.

(f) The full conditional p(\mu\mid\sigma^2,{\boldsymbol{x}}) is normal at each fixed \sigma^2. The marginal p(\mu\mid{\boldsymbol{x}}) mixes those normals over the posterior distribution of \sigma^2; a mixture need not belong to the component family. Here, integrating the joint kernel over v gives

p(\mu\mid{\boldsymbol{x}})\propto \exp\left\{-\frac{(\mu-\mu_0)^2}{2\sigma_0^2}\right\} \left\{\beta_0+\frac{S+n(\mu-\bar x)^2}{2}\right\}^{-(\alpha_0+n/2)}.

This is generally neither a normal nor a Student-t kernel. Likewise, the marginal variance posterior is a mixture of inverse-gamma full conditionals and is not generally inverse-gamma. Substituting a point estimate for the other unknown does not perform the required integration.

(g) Each Gibbs step draws from one full conditional, using the most recent value of the conditioned parameter. The stationary distribution of the pair is the joint posterior p(\mu,\sigma^2\mid{\boldsymbol{x}}). A marginal posterior is represented by the simulated values of one coordinate, not by a single full-conditional distribution.

After suitable convergence assessment and warm-up, averages of functions of the retained pairs approximate joint-posterior expectations. Keeping only the \mu coordinate approximates its marginal posterior automatically: uncertainty in v has been propagated through the chain. Successive draws are generally correlated, so Monte Carlo standard errors must account for that dependence.