Random-walk Metropolis for a correlated normal
Topic: Markov chains, Gibbs sampling, and Metropolis–Hastings · more questions on this topic
‹ Metropolis–Hastings for the logit-normal posterior An independent Gamma proposal ›
For the bivariate-normal target of the bivariate-normal Gibbs exercise, describe a random-walk Metropolis–Hastings sampler using increments
{\boldsymbol{\varepsilon}}\sim\operatorname{N}_2({\boldsymbol{0}},{\color{blue}\boldsymbol{\Psi}}), \qquad {\color{blue}\boldsymbol{\Psi}}= \begin{bmatrix}a^2&0\\0&b^2\end{bmatrix}.
Give the proposal, the acceptance probability in explicit quadratic form and its equivalent norm/distance reading, and the accept–reject update. Discuss what a and b control.
Question. Let the target be the bivariate normal distribution
{\boldsymbol{Z}}\sim\operatorname{N}_2\!\left( \begin{bmatrix}0\\0\end{bmatrix}, \begin{bmatrix}1&\rho\\ \rho&1\end{bmatrix} \right),\qquad |\rho|<1.
Describe a random-walk Metropolis–Hastings sampler using increments
{\boldsymbol{\varepsilon}}\sim\operatorname{N}_2({\boldsymbol{0}},{\color{blue}\boldsymbol{\Psi}}), \qquad {\color{blue}\boldsymbol{\Psi}}= \begin{bmatrix}a^2&0\\0&b^2\end{bmatrix}.
Give the proposal, the acceptance probability in explicit quadratic form and its equivalent norm/distance reading, and the accept–reject update. Discuss what a and b control.
Solution. Let {\boldsymbol{z}}_t=(x_t,y_t)^{\mathsf T}. Generate an increment
{\boldsymbol{\varepsilon}}_t\sim \operatorname{N}_2({\boldsymbol{0}},{\color{blue}\boldsymbol{\Psi}}), \qquad {\boldsymbol{z}}^\ast={\boldsymbol{z}}_t+{\boldsymbol{\varepsilon}}_t.
The proposal density satisfies q({\boldsymbol{z}}^\ast\mid{\boldsymbol{z}}_t) =q({\boldsymbol{z}}_t\mid{\boldsymbol{z}}^\ast), so the Hastings correction cancels. Since
{\color{blue}\boldsymbol{\Sigma}}^{-1} =\frac1{1-\rho^2} \begin{bmatrix}1&-\rho\\-\rho&1\end{bmatrix},
For a generic state {\boldsymbol{z}}=(x,y)^{\mathsf T}, define \begin{align*} Q(x,y) &=\frac{x^2-2\rho xy+y^2}{1-\rho^2}\\ &={\boldsymbol{z}}^\top{\color{blue}\boldsymbol{\Sigma}}^{-1}{\boldsymbol{z}}\\ &=\lVert{\boldsymbol{z}}\rVert_{{\color{blue}\boldsymbol{\Sigma}}^{-1}}^2. \end{align*} The same scalar is therefore the explicit coordinate quadratic, the matrix quadratic form, and squared Mahalanobis distance from the origin. Its level sets are the target’s elliptical density contours. The target ratio is \begin{align*} \frac{\pi({\boldsymbol{z}}^\ast)}{\pi({\boldsymbol{z}}_t)} &=\frac{\exp\{-Q(x^\ast,y^\ast)/2\}} {\exp\{-Q(x_t,y_t)/2\}}\\ &=\exp\!\left[-\frac12 \{Q(x^\ast,y^\ast)-Q(x_t,y_t)\}\right]\\ &=\exp\!\left[-\frac12\left\{ \lVert{\boldsymbol{z}}^\ast\rVert_{{\color{blue}\boldsymbol{\Sigma}}^{-1}}^2 -\lVert{\boldsymbol{z}}_t\rVert_{{\color{blue}\boldsymbol{\Sigma}}^{-1}}^2 \right\}\right]. \end{align*} Thus
a({\boldsymbol{z}}_t,{\boldsymbol{z}}^\ast) =\min\!\left(1, \exp\!\left[-\frac12 \{Q(x^\ast,y^\ast)-Q(x_t,y_t)\}\right] \right).
Draw U\sim\operatorname{Unif}(0,1); set {\boldsymbol{z}}_{t+1}={\boldsymbol{z}}^\ast if U<a, and otherwise repeat the current state. The constants a and b are the proposal standard deviations in the two coordinate directions. If they are too small, acceptance is high but travel is slow; if too large, rejections dominate. For |\rho| near one, a diagonal proposal also ignores the target’s rotated geometry, so a proposal covariance aligned with {\color{blue}\boldsymbol{\Sigma}} can be much more efficient.