Mean, covariance, and correlation
Section of Covariance and the multivariate normal, part of Statistical learning
Mean, covariance, and correlation
| Type | Section |
| Taught in | Covariance and the multivariate normal |
| Part | Statistical learning |
A full joint density is a large object. The first summary asks where each coordinate is centred, and which deviations tend to occur together. Throughout this section the second moments are finite, so the covariance matrix exists.
Definition 1 (Mean vector and covariance matrix) The expected value of \mathbf{X} is the vector of coordinate expectations,
\mathbb{E}(\mathbf{X})=\boldsymbol{\mu}=\bigl(\mathbb{E} X_1,\ldots,\mathbb{E} X_p\bigr)^\top.
The covariance of coordinates j and k is
\sigma_{jk}=\operatorname{Cov}(X_j,X_k)=\mathbb{E}\bigl[(X_j-\mathbb{E} X_j)(X_k-\mathbb{E} X_k)\bigr]=\mathbb{E}(X_j X_k)-\mathbb{E}(X_j)\mathbb{E}(X_k).
The covariance matrix is the p\times p matrix of these entries,
\Sigma=\operatorname{Var}(\mathbf{X})=\operatorname{Cov}(\mathbf{X},\mathbf{X}^\top)=\mathbb{E}\bigl[(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^\top\bigr].
The diagonal entry \sigma_{jj}=\sigma_j^2 is the variance of X_j, and \sigma_j is its standard deviation.
The matrix \Sigma is real and symmetric, and it is positive semi-definite. For any fixed \mathbf{z}\in\mathbb{R}^p the scalar \mathbf{z}^\top\mathbf{X} has variance
\operatorname{Var}(\mathbf{z}^\top\mathbf{X})=\mathbb{E}\bigl[\mathbf{z}^\top(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^\top\mathbf{z}\bigr]=\mathbf{z}^\top\Sigma\mathbf{z},
and a variance cannot be negative, so \mathbf{z}^\top\Sigma\mathbf{z}\ge 0. A real symmetric matrix has an orthonormal basis of eigenvectors: \Sigma=\Gamma\Lambda\Gamma^\top with \Gamma^{-1}=\Gamma^\top and \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p). If \Sigma\boldsymbol{\gamma}=\lambda\boldsymbol{\gamma} and \boldsymbol{\gamma}^\top\boldsymbol{\gamma}=1, then \lambda=\boldsymbol{\gamma}^\top\Sigma\boldsymbol{\gamma}=\operatorname{Var}(\boldsymbol{\gamma}^\top\mathbf{X})\ge 0. In the eigen-coordinates u_j=\boldsymbol{\gamma}_j^\top\mathbf{z},
\mathbf{z}^\top\Sigma\mathbf{z}=\sum_{j=1}^p\lambda_j u_j^2,
so nonnegative eigenvalues give a nonnegative quadratic form as well. Positive semi-definiteness is that property.
The same inequality gives |\sigma_{jk}|\le\sqrt{\sigma_{jj}\sigma_{kk}}. Apply \mathbf{z}^\top\Sigma\mathbf{z}\ge 0 to the 2\times 2 principal submatrix with entries \sigma_{jj}, \sigma_{jk}, \sigma_{kk}. Its determinant is nonnegative:
\sigma_{jj}\sigma_{kk}-\sigma_{jk}^2\ge 0.
Symmetry of a proposed covariance is necessary and is not enough: every linear combination must have a nonnegative variance. Sylvester’s criterion turns the quadratic-form condition into determinants. Strict positive definiteness requires every leading principal minor to be positive. Positive semi-definiteness requires every principal minor, not only the leading ones, to be nonnegative. A matrix can pass a leading-minor check aimed at the wrong one of these two conclusions and still fail to be a covariance.
Translations do not change covariances. For constants c,d\in\mathbb{R},
\operatorname{Cov}(X_j+c,X_k+d)=\operatorname{Cov}(X_j,X_k).
Scale changes pass through,
\operatorname{Cov}(c X_j,d X_k)=cd\,\operatorname{Cov}(X_j,X_k).
For a fixed vector \mathbf{a}=(a_1,\ldots,a_p)^\top,
\operatorname{Var}(\mathbf{a}^\top\mathbf{X})=\sum_{j=1}^p\sum_{k=1}^p a_j a_k\sigma_{jk}=\mathbf{a}^\top\Sigma\mathbf{a},
and for a second fixed vector \mathbf{b},
\operatorname{Cov}(\mathbf{a}^\top\mathbf{X},\mathbf{b}^\top\mathbf{X})=\sum_{j=1}^p\sum_{k=1}^p a_j b_k\sigma_{jk}=\mathbf{a}^\top\Sigma\mathbf{b}.
More generally, let A be r\times p and B be s\times p, and set \mathbf{Y}=A\mathbf{X}, \mathbf{Z}=B\mathbf{X}. Then \mathbb{E}\mathbf{Y}=A\boldsymbol{\mu}, and
\begin{aligned} \operatorname{Cov}(\mathbf{Y},\mathbf{Y}^\top)&=\mathbb{E}\bigl[A(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^\top A^\top\bigr]=A\Sigma A^\top,\\ \operatorname{Cov}(\mathbf{Z},\mathbf{Z}^\top)&=B\Sigma B^\top,\\ \operatorname{Cov}(\mathbf{Y},\mathbf{Z}^\top)&=\mathbb{E}\bigl[A(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^\top B^\top\bigr]=A\Sigma B^\top. \end{aligned}
The matrices move outside the expectation because they are fixed. The product A\Sigma B^\top is r\times s, which is the shape of the cross-covariance of an r-vector and an s-vector. The scalar formulas are the case of one-row matrices.
Definition 2 (Correlation) With positive finite variances, the Pearson correlation is the scale-free covariance
\rho_{jk}=\operatorname{corr}(X_j,X_k)=\frac{\operatorname{Cov}(X_j,X_k)}{\sqrt{\operatorname{Var}(X_j)\operatorname{Var}(X_k)}}=\frac{\sigma_{jk}}{\sigma_j\sigma_k}\in[-1,1].
The correlation matrix is P=(\rho_{jk}), with ones on the diagonal. If D=\operatorname{diag}(\sigma_1,\ldots,\sigma_p), then
P=D^{-1}\Sigma D^{-1}, \qquad \Sigma=DPD.
Independence of X_j and X_k implies \rho_{jk}=0. The converse is false, and the example below is the counterexample. If X_k=c X_j+d then \rho_{jk}=1 for c>0 and \rho_{jk}=-1 for c<0. If a variance is zero, the correlation is undefined.
Correlation is the slope of the best linear predictor in standard-deviation units. Let X and Y have means \mu_X,\mu_Y, variances \sigma_X^2,\sigma_Y^2>0 and correlation \rho. For a predictor a+bX,
\mathbb{E}(Y-a-bX)^2=\operatorname{Var}(Y-bX)+(\mu_Y-a-b\mu_X)^2,
because adding a constant does not change a variance. The second term is zero at a=\mu_Y-b\mu_X. The first is
\operatorname{Var}(Y-bX)=\sigma_Y^2-2b\operatorname{Cov}(X,Y)+b^2\sigma_X^2.
Differentiating in b and setting the derivative to zero gives -2\operatorname{Cov}(X,Y)+2b\sigma_X^2=0, so
b=\frac{\operatorname{Cov}(X,Y)}{\sigma_X^2}=\rho\frac{\sigma_Y}{\sigma_X}, \qquad a=\mu_Y-b\mu_X.
Substitute this b. The minimal value is
\sigma_Y^2-2\rho^2\sigma_Y^2+\rho^2\sigma_Y^2=\sigma_Y^2(1-\rho^2).
So \rho is the slope after both variables are measured in standard deviations, and 1-\rho^2 is the fraction of \operatorname{Var}(Y) left by this linear prediction. On the original axes the slope also depends on the units. A perfect positive correlation is an exact increasing affine relationship, and that relationship need not have slope one.
Example 1 (Zero covariance with a deterministic relationship) Let X\sim\operatorname{N}(0,1) and Y=X^2.
Question. Show that \operatorname{Cov}(X,Y)=0, and that X and Y are dependent.
Solution. The covariance is
\operatorname{Cov}(X,X^2)=\mathbb{E}(X^3)-\mathbb{E}(X)\mathbb{E}(X^2)=0,
because \mathbb{E}(X)=0 and the third moment of a standard normal variable is zero: the density is even and x^3 is odd, so the integral of x^3\varphi(x) over \mathbb{R} vanishes. The pair is dependent because Y is a function of X. Concretely, \mathbb{P}(|X|>1)>0, while
\mathbb{P}(|X|>1\mid Y\le 1)=0,
since Y\le 1 forces |X|\le 1. Independence would have made the conditional probability equal the marginal. Zero covariance rules out a nonzero linear association. It leaves nonlinear dependence untouched.
Used in
See also
Joint distribution and density · Mean vector and covariance matrix · Correlation · Multivariate normal distribution · Squared Mahalanobis distance · Linear images of a normal vector · Conditional distribution of a multivariate normal · Partial correlation