Eigenvalues and eigenvectors

Topic of Principal component analysis, part of Statistical learning · Also in Covariance and the multivariate normal

Eigenvalues and eigenvectors

Type Topic
Taught in Principal component analysis · Covariance and the multivariate normal
Part Statistical learning
Questions using it 14

For a symmetric matrix S, S\mathbf{a}=\lambda\mathbf{a} with \mathbf{a}\ne\mathbf{0}. The eigenvalues are real and the eigenvectors can be chosen orthonormal. For a covariance matrix the eigenvectors are the principal component directions and the eigenvalues the variances along them.

In Principal component analysis

From Principal component analysis

The stationary points are the eigenvectors of S, and the multipliers are the eigenvalues. Substitute a stationary point back into the objective:

\mathbf{a}_1^\top S\mathbf{a}_1=\lambda\,\mathbf{a}_1^\top\mathbf{a}_1=\lambda.

In Covariance and the multivariate normal

From Covariance and the multivariate normal

An ellipse drawn through a data cloud describes the spread of observations. A confidence region for the mean describes uncertainty about the centre. The second region contracts as more independent observations are collected. The population cloud does not. The contours below hold \boldsymbol{\mu} and \Sigma fixed.

Fix c>0 and consider \{\mathbf{x}:f(\mathbf{x})=c\}. If c exceeds the maximum of f, the set is empty. Otherwise divide f(\mathbf{x})=c by the leading constant and take logarithms:

-\frac12(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}) =\log\bigl(c\,(2\pi)^{p/2}|\Sigma|^{1/2}\bigr),

which is

(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})=C

for a constant C determined by c. For C>0 this is an ellipsoid centred at \boldsymbol{\mu}. In the bivariate case the level sets are ellipses. Correlation rotates and stretches them: the contour for \rho=0 is axis-aligned, the contour for \rho=0.75 leans up, and the contour for \rho=-0.75 leans down.

To find the axes, use \Sigma=\Gamma\Lambda\Gamma^\top with orthonormal columns \boldsymbol{\gamma}_1,\ldots,\boldsymbol{\gamma}_p and \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p). Then \Sigma^{-1}=\Gamma\Lambda^{-1}\Gamma^\top. In the rotated coordinates \mathbf{u}=\Gamma^\top(\mathbf{x}-\boldsymbol{\mu}),

(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})=\mathbf{u}^\top\Lambda^{-1}\mathbf{u}=\sum_{j=1}^p\frac{u_j^2}{\lambda_j}=C.

This is an ellipsoid whose jth semi-axis points along \boldsymbol{\gamma}_j and has length \sqrt{C\lambda_j}. Those directions are the principal component directions derived on Principal component analysis.

Definition 1 (Squared Mahalanobis distance) The squared Mahalanobis distance of \mathbf{X} from its mean is

(\mathbf{X}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{X}-\boldsymbol{\mu}).

Article: Squared Mahalanobis distance →

If \mathbf{X}\sim\operatorname{N}_p(\boldsymbol{\mu},\Sigma), set \mathbf{Z}=\Sigma^{-1/2}(\mathbf{X}-\boldsymbol{\mu}), where \Sigma^{-1/2}=\Gamma\Lambda^{-1/2}\Gamma^\top is the symmetric inverse square root. The linear-image formula for covariances gives

\operatorname{Var}(\mathbf{Z})=\Sigma^{-1/2}\Sigma\Sigma^{-1/2}=I.

The next proposition says \mathbf{Z} is normal, so its coordinates are independent standard normals. The squared Mahalanobis distance is therefore

\mathbf{Z}^\top\mathbf{Z}=\sum_{j=1}^p Z_j^2\sim\chi^2_p,

the chi-squared distribution with p degrees of freedom. A density contour with C=\chi^2_{p;\alpha}, the upper \alpha point of that law, encloses probability 1-\alpha. The same upper-tail convention, with the upper probability in the first subscript, is used for the t and F critical values on Estimation and inference for the multivariate normal.

Used in

See also

Principal component directions and scores · Principal component directions · Singular value decomposition · Mean vector and covariance matrix · Variance · Kernel density estimator · Scree plot · Scaling

Glossary and index