The bivariate conditional

Topic: Covariance and the multivariate normal · more questions on this topic
‹ Normal marginals without joint normality A Schur-complement check ›

Specialise the conditional normal theorem to p=2. Show that

X_1\mid(X_2=x_2) \sim\operatorname{N}\Bigl(\mu_1+\rho\frac{\sigma_1}{\sigma_2}(x_2-\mu_2),\;\sigma_1^2(1-\rho^2)\Bigr),

and explain why the regression line of X_1 on X_2 is not the same line as the one obtained by solving the regression of X_2 on X_1 for x_2.

Worked answer (click to show)

Keep the prediction direction fixed while comparing lines. Predicting the first variable from the second, and solving the reverse prediction equation, are different operations.

With A=\{1\} and B=\{2\}, \Sigma_{AB}=\rho\sigma_1\sigma_2, \Sigma_{BB}=\sigma_2^2 and \Sigma_{AB}\Sigma_{BB}^{-1}=\rho\sigma_1/\sigma_2. The conditional mean is \mu_1+\rho(\sigma_1/\sigma_2)(x_2-\mu_2) and the conditional variance is

\sigma_1^2-(\rho\sigma_1\sigma_2)^2/\sigma_2^2=\sigma_1^2(1-\rho^2).

The regression of X_1 on X_2 predicts x_1 from x_2 and has slope \rho\sigma_1/\sigma_2 in the (x_2,x_1) plane. The regression of X_2 on X_1 is x_2=\mu_2+\rho(\sigma_2/\sigma_1)(x_1-\mu_1). Solving it for x_1 gives

x_1=\mu_1+\frac{\sigma_1}{\rho\sigma_2}(x_2-\mu_2),

with slope \sigma_1/(\rho\sigma_2) in the same plane. For \rho\neq 0 the ratio of the two slopes is \rho^2, so the regression lines agree only in the degenerate limit \rho^2=1. When \rho=0 they are the horizontal and vertical lines through the mean. A principal axis may coincide with one of these lines when the covariance is diagonal, and there is no unique principal axis when the covariance is proportional to the identity. Each line answers a different conditional question. The least-squares line and the conditional mean of an unconstrained bivariate normal fitted to the same pairs coincide. The second line here is the inverse regression, solved for the other coordinate.