Conditioning a normal vector
Section of Covariance and the multivariate normal, part of Statistical learning
Conditioning a normal vector
| Type | Section |
| Taught in | Covariance and the multivariate normal |
| Part | Statistical learning |
Split \mathbf{X} into \mathbf{X}_A, containing the first q coordinates (0<q<p), and \mathbf{X}_B, containing the last p-q. Partition the parameters in the same way:
\mathbf{X}=\begin{pmatrix}\mathbf{X}_A\\ \mathbf{X}_B\end{pmatrix}, \qquad \boldsymbol{\mu}=\begin{pmatrix}\boldsymbol{\mu}_A\\ \boldsymbol{\mu}_B\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}\Sigma_{AA}&\Sigma_{AB}\\ \Sigma_{AB}^\top&\Sigma_{BB}\end{pmatrix}.
The blocks have shapes q\times q, q\times(p-q) and (p-q)\times(p-q). The selection argument gives
\mathbf{X}_A\sim\operatorname{N}_q(\boldsymbol{\mu}_A,\Sigma_{AA}), \qquad \mathbf{X}_B\sim\operatorname{N}_{p-q}(\boldsymbol{\mu}_B,\Sigma_{BB}).
Inside the normal family, \mathbf{X}_A and \mathbf{X}_B are independent if and only if \Sigma_{AB}=0. Zero covariance does not give independence for a general pair of random vectors. The sign example is a pair with zero covariance that is not independent, and it is not jointly normal.
The marginal of a block is the corresponding mean and covariance block. Conditioning needs a correction, because the observed coordinates carry information about the others. The correction comes from dividing the joint density by the marginal density and completing a square. The covariance that remains is the Schur complement. Positive definiteness of \Sigma makes that complement positive definite as well.
The target is the density of \mathbf{X}_A given \mathbf{X}_B=\mathbf{x}_B. By the general ratio,
f_{\mathbf{X}_A\mid\mathbf{X}_B}=\frac{f_{\mathbf{X}_A\mathbf{X}_B}}{f_{\mathbf{X}_B}},
where the denominator is the normal density
f_{\mathbf{X}_B}(\mathbf{x}_B)=\frac{1}{(2\pi)^{(p-q)/2}|\Sigma_{BB}|^{1/2}} \exp\Bigl\{-\tfrac12(\mathbf{x}_B-\boldsymbol{\mu}_B)^\top\Sigma_{BB}^{-1}(\mathbf{x}_B-\boldsymbol{\mu}_B)\Bigr\}.
The numerator needs \Sigma^{-1} and |\Sigma| in the blocks. Both are written with the Schur complement of \Sigma_{BB} in \Sigma,
\Sigma_{A\mid B}=\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top.
Block inverse and determinant. The inverse in the same blocks is
\Sigma^{-1}= \begin{pmatrix} \Sigma_{A\mid B}^{-1} & -\Sigma_{A\mid B}^{-1}\Sigma_{AB}\Sigma_{BB}^{-1}\\ -\Sigma_{BB}^{-1}\Sigma_{AB}^\top\Sigma_{A\mid B}^{-1} & \Sigma_{BB}^{-1}+\Sigma_{BB}^{-1}\Sigma_{AB}^\top\Sigma_{A\mid B}^{-1}\Sigma_{AB}\Sigma_{BB}^{-1} \end{pmatrix},
and the determinant factorises as
|\Sigma|=|\Sigma_{BB}|\,|\Sigma_{A\mid B}|.
Normalising constants. The ratio of the constants in the joint density and the marginal density is
\frac{(2\pi)^{(p-q)/2}|\Sigma_{BB}|^{1/2}}{(2\pi)^{p/2}|\Sigma_{BB}|^{1/2}|\Sigma_{A\mid B}|^{1/2}} =\frac{1}{(2\pi)^{q/2}|\Sigma_{A\mid B}|^{1/2}}.
The exponent. Write \mathbf{a}=\mathbf{x}_A-\boldsymbol{\mu}_A and \mathbf{b}=\mathbf{x}_B-\boldsymbol{\mu}_B. The exponent of the ratio is -\tfrac12 E, where E is the joint quadratic form minus the marginal quadratic form. Multiply the block inverse through (\mathbf{a}^\top,\mathbf{b}^\top):
\begin{aligned} E &=\mathbf{a}^\top\Sigma_{A\mid B}^{-1}\mathbf{a} -\mathbf{a}^\top\Sigma_{A\mid B}^{-1}\Sigma_{AB}\Sigma_{BB}^{-1}\mathbf{b} -\mathbf{b}^\top\Sigma_{BB}^{-1}\Sigma_{AB}^\top\Sigma_{A\mid B}^{-1}\mathbf{a}\\ &\qquad+\mathbf{b}^\top\bigl(\Sigma_{BB}^{-1}+\Sigma_{BB}^{-1}\Sigma_{AB}^\top\Sigma_{A\mid B}^{-1}\Sigma_{AB}\Sigma_{BB}^{-1}\bigr)\mathbf{b} -\mathbf{b}^\top\Sigma_{BB}^{-1}\mathbf{b}. \end{aligned}
The two copies of \mathbf{b}^\top\Sigma_{BB}^{-1}\mathbf{b} cancel. The two cross terms are transposes of each other and both are scalars, so they are equal. Therefore
E=\mathbf{a}^\top\Sigma_{A\mid B}^{-1}\mathbf{a} -2\mathbf{a}^\top\Sigma_{A\mid B}^{-1}\Sigma_{AB}\Sigma_{BB}^{-1}\mathbf{b} +\mathbf{b}^\top\Sigma_{BB}^{-1}\Sigma_{AB}^\top\Sigma_{A\mid B}^{-1}\Sigma_{AB}\Sigma_{BB}^{-1}\mathbf{b}.
Completing the square. Define
\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B)=\boldsymbol{\mu}_A+\Sigma_{AB}\Sigma_{BB}^{-1}(\mathbf{x}_B-\boldsymbol{\mu}_B).
Then \mathbf{a}-\Sigma_{AB}\Sigma_{BB}^{-1}\mathbf{b}=\mathbf{x}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B). Expand
\bigl(\mathbf{a}-\Sigma_{AB}\Sigma_{BB}^{-1}\mathbf{b}\bigr)^\top \Sigma_{A\mid B}^{-1} \bigl(\mathbf{a}-\Sigma_{AB}\Sigma_{BB}^{-1}\mathbf{b}\bigr).
The expansion is exactly the three terms in E. Hence
E=\bigl(\mathbf{x}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B)\bigr)^\top \Sigma_{A\mid B}^{-1} \bigl(\mathbf{x}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B)\bigr).
Combine this quadratic with the normalising constant. The conditional density is
f_{\mathbf{X}_A\mid\mathbf{X}_B=\mathbf{x}_B}(\mathbf{x}_A) =\frac{1}{(2\pi)^{q/2}|\Sigma_{A\mid B}|^{1/2}} \exp\Bigl\{-\tfrac12 \bigl(\mathbf{x}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B)\bigr)^\top \Sigma_{A\mid B}^{-1} \bigl(\mathbf{x}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B)\bigr) \Bigr\}.
Theorem 1 (Conditional distribution of a multivariate normal) If \mathbf{X}=(\mathbf{X}_A,\mathbf{X}_B)\sim\operatorname{N}_p(\boldsymbol{\mu},\Sigma) is partitioned as above, then
\mathbf{X}_A\mid(\mathbf{X}_B=\mathbf{x}_B) \sim\operatorname{N}_q\bigl(\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B),\,\Sigma_{A\mid B}\bigr),
with
\boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B)=\boldsymbol{\mu}_A+\Sigma_{AB}\Sigma_{BB}^{-1}(\mathbf{x}_B-\boldsymbol{\mu}_B)
and
\Sigma_{A\mid B}=\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top.
Article: Conditional distribution of a multivariate normal →
The conditional mean is affine in \mathbf{x}_B. It starts at \boldsymbol{\mu}_A and adjusts by the observed deviation \mathbf{x}_B-\boldsymbol{\mu}_B. The conditional covariance does not depend on the observed value. The product \Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top is positive semi-definite, so conditioning reduces covariance in the matrix sense \Sigma_{A\mid B}\preceq\Sigma_{AA}: the difference \Sigma_{AA}-\Sigma_{A\mid B} has nonnegative eigenvalues. This is a statement about the Gaussian model with its parameters fixed. The narrowing ends of an ellipse are a property of a joint contour. They do not say that the conditional variance gets smaller at extreme predictor values. A contour and a conditional density are different objects. A slice of a bivariate normal ellipse, taken at a fixed x_B, is a univariate normal density whose mean has moved and whose variance is the smaller Schur complement.
The conditional mean is the best predictor under squared-error loss whenever the conditional expectation exists, including outside the Gaussian family. Joint normality supplies this particular affine formula and the constant conditional covariance. Replacing a missing value by its conditional mean suppresses residual variability. The filled-in number is not an observed measurement.
Residuals, and a second derivation
Main article: Residuals, and a second derivation
Define the population residual
\mathbf{X}_{A\cdot B}=\mathbf{X}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{X}_B).
This is the part of \mathbf{X}_A left after its population conditional mean has been removed. Under joint normality it is independent of \mathbf{X}_B. That conclusion belongs to the population residual. It does not transfer automatically to residuals from a model fitted to the same sample. Ordinary least squares gives the sample orthogonality \mathbf{X}^\top\mathbf{e}=0, and orthogonality is not independence. For a random design and homoscedastic Gaussian errors, the residual vector given the design has covariance \sigma^2(I-H), which depends on the design through its projection matrix. A population residual and a fitted residual need separate arguments.
The two properties of \mathbf{X}_{A\cdot B} follow from the linear transformation rules. Centre it:
\mathbf{X}_{A\cdot B}-\mathbb{E}(\mathbf{X}_{A\cdot B}) =\mathbf{X}_A-\boldsymbol{\mu}_A-\Sigma_{AB}\Sigma_{BB}^{-1}(\mathbf{X}_B-\boldsymbol{\mu}_B) =M(\mathbf{X}-\boldsymbol{\mu}),
with M=(I_q,\ -\Sigma_{AB}\Sigma_{BB}^{-1}). Then
\begin{aligned} \operatorname{Cov}(\mathbf{X}_{A\cdot B},\mathbf{X}_{A\cdot B}^\top) &=M\Sigma M^\top\\ &=\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top -\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top +\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{BB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top\\ &=\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top =\Sigma_{A\mid B}, \end{aligned}
because the last term equals \Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top and cancels one of the two cross terms. The cross-covariance with the conditioning block is
\operatorname{Cov}(\mathbf{X}_{A\cdot B},\mathbf{X}_B^\top)=\Sigma_{AB}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{BB}=0.
The pair (\mathbf{X}_{A\cdot B},\mathbf{X}_B) is a linear image of a normal vector, and the cross-covariance is zero, so the residual is independent of the conditioning variables. This is a second proof of the conditional theorem. Write
\mathbf{X}_A=\boldsymbol{\mu}_{A\mid B}(\mathbf{X}_B)+\mathbf{X}_{A\cdot B}
with \mathbf{X}_{A\cdot B}\sim\operatorname{N}_q(\mathbf{0},\Sigma_{A\mid B}) independent of \mathbf{X}_B. Given \mathbf{X}_B=\mathbf{x}_B, the first term is the constant \boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B) and the second keeps its distribution.
The bivariate regression lines
Main article: Bivariate regression lines
Specialise the conditional theorem to p=2, with A=\{1\} and B=\{2\}. Then \Sigma_{AB}=\rho\sigma_1\sigma_2 and \Sigma_{BB}=\sigma_2^2, so
\Sigma_{AB}\Sigma_{BB}^{-1}=\rho\frac{\sigma_1}{\sigma_2}
and
X_1\mid(X_2=x_2) \sim\operatorname{N}\Bigl(\mu_1+\rho\frac{\sigma_1}{\sigma_2}(x_2-\mu_2),\;\sigma_1^2(1-\rho^2)\Bigr).
The regression of X_1 on X_2 predicts x_1 from x_2 and has slope \rho\sigma_1/\sigma_2 in the (x_2,x_1) plane. The regression of X_2 on X_1 is
x_2=\mu_2+\rho\frac{\sigma_2}{\sigma_1}(x_1-\mu_1).
Solving that equation for x_1 gives slope \sigma_1/(\rho\sigma_2) in the same plane. For \rho\neq 0 the ratio of the two slopes is \rho^2, so the lines agree only in the limit \rho^2=1. When \rho=0 they are the horizontal and vertical lines through the mean. A principal axis may coincide with one of these lines when the covariance is diagonal, and there is no unique principal axis when \Sigma is a multiple of the identity. Each line answers a different question.
Partial correlation
Main article: Partial correlation
When X_{q+1},\ldots,X_p are held fixed, the entries of \Sigma_{A\mid B} are the partial variances and covariances. Write \sigma_{jk\mid q+1,\ldots,p} for the (j,k) entry, 1\le j,k\le q.
Definition 1 (Partial correlation) The partial correlation of coordinates j and k given the remaining coordinates is
\rho_{jk\mid q+1,\ldots,p} =\frac{\sigma_{jk\mid q+1,\ldots,p}}{\sqrt{\sigma_{jj\mid q+1,\ldots,p}\,\sigma_{kk\mid q+1,\ldots,p}}}.
If D_{A\mid B}=\operatorname{diag}(\Sigma_{A\mid B}), the matrix of partial correlations is
P_{A\mid B}=D_{A\mid B}^{-1/2}\,\Sigma_{A\mid B}\,D_{A\mid B}^{-1/2},
where the power -1/2 takes the reciprocal square root of each diagonal entry.
Inside the Gaussian model, partial correlation is the association between the remaining coordinates after conditioning on the others. Equivalently, it is the correlation of their population linear-regression residuals. Outside the Gaussian family, a residual partial correlation remains defined when the moments and the regressions exist, and it need not equal the conditional correlation at every conditioning value. Standardising a covariance matrix, as cov2cor does, produces a correlation matrix. The standardisation does not create the assumptions behind a conditional reading.
Example 1 (A trivariate conditional distribution) Let \mathbf{X}=(X_1,X_2,X_3)^\top\sim\operatorname{N}_3(\boldsymbol{\mu},\Sigma) with
\boldsymbol{\mu}=\begin{pmatrix}1\\2\\3\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}4&2&1\\2&3&0.5\\1&0.5&2\end{pmatrix}.
Question. Find the distribution of (X_1,X_2) given X_3=x_3, and the partial correlation of X_1 and X_2 given X_3.
Solution. Take A=\{1,2\} and B=\{3\}. Then \Sigma_{AB}=(1,0.5)^\top and \Sigma_{BB}=2, so
\Sigma_{AB}\Sigma_{BB}^{-1}=(0.5,0.25)^\top
and
\boldsymbol{\mu}_{A\mid B}(x_3) =\begin{pmatrix}1\\2\end{pmatrix}+\begin{pmatrix}0.5\\0.25\end{pmatrix}(x_3-3) =\begin{pmatrix}1+0.5(x_3-3)\\2+0.25(x_3-3)\end{pmatrix}.
The Schur complement subtracts the rank-one update (0.5,0.25)^\top(1,\ 0.5):
\Sigma_{A\mid B} =\begin{pmatrix}4&2\\2&3\end{pmatrix} -\begin{pmatrix}0.5&0.25\\0.25&0.125\end{pmatrix} =\begin{pmatrix}3.5&1.75\\1.75&2.875\end{pmatrix}.
Thus (X_1,X_2)\mid(X_3=x_3) is bivariate normal with that mean and that covariance. The partial correlation is
\rho_{12\mid 3}=\frac{1.75}{\sqrt{3.5\times 2.875}}=\frac{1.75}{\sqrt{10.0625}}=0.5517.
The marginal correlation is
\rho_{12}=\frac{2}{\sqrt{4\times 3}}=\frac{2}{\sqrt{12}}=0.5774.
The association that remains after conditioning on X_3 is weaker in this model. The comparison is a statistical adjustment. It is not evidence that X_3 causes either of the other coordinates.
The conditional-mean line and least squares
Main article: Conditional-mean line and least squares
Least squares of a response Y on a predictor X minimises \sum_i(y_i-a-bx_i)^2 and treats the observed x_i as fixed. The unconstrained bivariate normal, fitted to the same complete pairs, estimates (\widehat\mu_X,\widehat\mu_Y) and a covariance with one divisor in every entry, whether that divisor is n or n-1. Its conditional mean of Y given X=x is
\widehat\mu_Y+\frac{s_{YX}}{s_{XX}}(x-\widehat\mu_X).
The slope s_{YX}/s_{XX} and the intercept \widehat\mu_Y-(s_{YX}/s_{XX})\widehat\mu_X are exactly the least-squares slope and intercept. The Gaussian fit and the least-squares fit draw the same conditional-mean line. They do not estimate the same object. Least squares does not model the marginal distribution of X, and it does not by itself supply the conditional variance s_{YY}-s_{YX}^2/s_{XX}. The major axis of the fitted ellipse is a different line, and so is the line obtained by regressing X on Y and solving for Y. The laboratory checks the agreement of the first two lines on Iris setosa.
Used in
See also
Joint distribution and density · Mean vector and covariance matrix · Correlation · Multivariate normal distribution · Squared Mahalanobis distance · Linear images of a normal vector · Conditional distribution of a multivariate normal · Partial correlation