Singular value decomposition

Section of Principal component analysis, part of Statistical learning

Type	Section
Taught in	Principal component analysis
Part	Statistical learning

Forming S=\mathbf{X}^\top\mathbf{X}/(n-1) and then computing its eigenvectors repeats information that the singular value decomposition of the centred data matrix already contains. The decomposition is also the stable computation, including when n<p. Centre each column and write \mathbf{X} for the centred n\times p matrix, so that

S=\frac{1}{n-1}\mathbf{X}^\top\mathbf{X}

agrees with the sample covariance already used. The singular value decomposition writes

\mathbf{X}=UEV^\top.

Use a compact decomposition with r=\operatorname{rank}(\mathbf{X}): U is n\times r, E is the r\times r diagonal matrix of positive singular values e_j, and V is p\times r. Both U^\top U and V^\top V equal I_r. This works whether or not n exceeds p. Substituting gives

S=\frac{1}{n-1}V E U^\top U E V^\top=\frac{1}{n-1}V E^2 V^\top.

Thus the nonzero eigenvalues of S are e_j^2/(n-1), and their directions are the columns of V. If r<p, complete these columns to an orthonormal basis of the zero-variance directions when a full p-dimensional basis is wanted. The nonzero score columns are

\mathbf{X}V=UEV^\top V=UE.

This is also the computation used by prcomp: it applies the singular value decomposition directly to the centred, and optionally scaled, data, and it uses divisor n-1 for the reported variances. The signs of directions may differ between implementations. Compare the axes or the reconstructions, rather than treating a sign reversal as a disagreement.

In a full decomposition the square matrices U and V are orthogonal. In a compact decomposition their columns are orthonormal, while UU^\top and VV^\top are projections onto their column spaces. The two factors do not have interchangeable dimensions. The distinction is unavoidable when there are more variables than observations.

Whitening takes one more step: divide each nonzero principal-component score by its standard deviation. For positive definite S this gives identity sample covariance. If S is singular, only the r positive-variance directions can be whitened this way. Division by a zero eigenvalue is undefined. Identity covariance makes Gaussian contours spherical. It does not make an arbitrary distribution Gaussian or spherically symmetric.

A principal direction and a prediction of a response answer different questions. If the goal is to predict a response, the object to report is that prediction. An axis of the explanatory cloud summarises the explanatory variables, and the variance-maximising theorem is computed from those variables alone. The least-squares line minimises squared error in a named response. The principal axis maximises variance among the explanatory coordinates. The two coincide only in special cases, such as an uncorrelated cloud whose larger variance is horizontal.

The fragment below reads the singular-value route from a prcomp object: squared singular values scaled by n-1, directions in the rotation matrix, and scores in the coordinates.

pc <- prcomp(x, center = TRUE, scale. = FALSE)  # SVD of the centred data
pc$sdev^2                                        # eigenvalues e_j^2/(n-1)
pc$rotation                                      # columns are the directions a_j
head(pc$x)                                       # the scores X V = U E
summary(pc)$importance["Proportion of Variance", ]

Used in

Taught in Principal component analysis
Also used in High-dimensional analysis
All 7 questions on Principal component analysis

Used in

See also