Residuals, and a second derivation
Section of Covariance and the multivariate normal, part of Statistical learning
Residuals, and a second derivation
| Type | Section |
| Taught in | Covariance and the multivariate normal |
| Part | Statistical learning |
Define the population residual
\mathbf{X}_{A\cdot B}=\mathbf{X}_A-\boldsymbol{\mu}_{A\mid B}(\mathbf{X}_B).
This is the part of \mathbf{X}_A left after its population conditional mean has been removed. Under joint normality it is independent of \mathbf{X}_B. That conclusion belongs to the population residual. It does not transfer automatically to residuals from a model fitted to the same sample. Ordinary least squares gives the sample orthogonality \mathbf{X}^\top\mathbf{e}=0, and orthogonality is not independence. For a random design and homoscedastic Gaussian errors, the residual vector given the design has covariance \sigma^2(I-H), which depends on the design through its projection matrix. A population residual and a fitted residual need separate arguments.
The two properties of \mathbf{X}_{A\cdot B} follow from the linear transformation rules. Centre it:
\mathbf{X}_{A\cdot B}-\mathbb{E}(\mathbf{X}_{A\cdot B}) =\mathbf{X}_A-\boldsymbol{\mu}_A-\Sigma_{AB}\Sigma_{BB}^{-1}(\mathbf{X}_B-\boldsymbol{\mu}_B) =M(\mathbf{X}-\boldsymbol{\mu}),
with M=(I_q,\ -\Sigma_{AB}\Sigma_{BB}^{-1}). Then
\begin{aligned} \operatorname{Cov}(\mathbf{X}_{A\cdot B},\mathbf{X}_{A\cdot B}^\top) &=M\Sigma M^\top\\ &=\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top -\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top +\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{BB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top\\ &=\Sigma_{AA}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top =\Sigma_{A\mid B}, \end{aligned}
because the last term equals \Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{AB}^\top and cancels one of the two cross terms. The cross-covariance with the conditioning block is
\operatorname{Cov}(\mathbf{X}_{A\cdot B},\mathbf{X}_B^\top)=\Sigma_{AB}-\Sigma_{AB}\Sigma_{BB}^{-1}\Sigma_{BB}=0.
The pair (\mathbf{X}_{A\cdot B},\mathbf{X}_B) is a linear image of a normal vector, and the cross-covariance is zero, so the residual is independent of the conditioning variables. This is a second proof of the conditional theorem. Write
\mathbf{X}_A=\boldsymbol{\mu}_{A\mid B}(\mathbf{X}_B)+\mathbf{X}_{A\cdot B}
with \mathbf{X}_{A\cdot B}\sim\operatorname{N}_q(\mathbf{0},\Sigma_{A\mid B}) independent of \mathbf{X}_B. Given \mathbf{X}_B=\mathbf{x}_B, the first term is the constant \boldsymbol{\mu}_{A\mid B}(\mathbf{x}_B) and the second keeps its distribution.
Used in
See also
Joint distribution and density · Mean vector and covariance matrix · Correlation · Multivariate normal distribution · Squared Mahalanobis distance · Linear images of a normal vector · Conditional distribution of a multivariate normal · Partial correlation