Choosing the number of principal components

Section of Principal component analysis, part of Statistical learning

Type	Section
Taught in	Principal component analysis
Part	Statistical learning

The proportion of variance explained by component j is \lambda_j/\sum_k\lambda_k, and the proportion explained by the first k components is the corresponding partial sum. A common rule keeps the smallest k that reaches a stated fraction of the total, for example 90\%, 95\%, or 99\%. The fraction is a choice about residual squared error. The rule is computed from the explanatory covariance alone.

A component can explain little predictor variance and still be highly predictive of the response. The exercise on a low-variance predictor constructs exactly that situation. Variance explained is a reconstruction criterion, not a general measure of predictive usefulness. If prediction is the goal, choose the number of components using validation inside the modelling pipeline. Estimate all preprocessing on the training folds, and leave the final test set for assessment.

Remark 1 (Scaling). Changing metres to millimetres multiplies a coordinate by 1000 and its variance by 10^6. Covariance principal components can therefore change even though the physical observations have not changed. Standardising each nonconstant coordinate to unit variance gives correlation principal components. This chooses equal weight per standardised coordinate, which may or may not suit the scientific question. It is a modelling choice. Record the choice, and reuse the training means and scales for new observations.

Article: Scaling →

Example 1 (Principal components of the twelve paired measurements) Return to the twelve pairs (x_1,x_2) of the Hotelling example, with

S=\begin{pmatrix}5.9097&3.9964\\3.9964&3.2402\end{pmatrix}.

Question. Find the principal component directions, the variance each one carries, and the proportion explained by the first. Repeat the calculation on the correlation matrix.

Solution. The characteristic equation is

\lambda^2-\operatorname{tr}(S)\lambda+|S|=\lambda^2-9.1499\lambda+3.1778=0.

The roots are \lambda_1=8.788 and \lambda_2=0.362. For \lambda_1, (S-\lambda_1 I)\mathbf{a}=\mathbf{0} gives

(5.9097-8.788)a_1+3.9964 a_2=0,

so a_2=0.7202\,a_1. The unit eigenvector is

\mathbf{a}_1=(0.8114,\ 0.5845)^\top, \qquad \mathbf{a}_2=(-0.5845,\ 0.8114)^\top,

the second being orthogonal to the first. The first component is

Y_1=0.8114 X_1+0.5845 X_2,

roughly a weighted average of the two measurements. Its variance is 8.788, which is

8.788/9.150=96.05\%

of the total variance \operatorname{tr}(S)=9.150, while

Y_2=-0.5845 X_1+0.8114 X_2

contributes only 3.95\%. Direct computation of the scores confirms that the sample variances of Y_1 and Y_2 equal 8.788 and 0.362, and that their sample covariance is zero up to rounding. The sample mean \bar{\mathbf{x}}=(0.87,-0.58)^\top points mostly along the second direction, with absolute cosine 0.94, where the data hardly vary. That geometry is why the T^2 test rejects while neither marginal t test does.

If the variables are standardised first, the covariance matrix becomes the correlation matrix

R=\begin{pmatrix}1&0.9133\\0.9133&1\end{pmatrix},

with eigenvalues 1+r=1.913 and 1-r=0.087 and directions exactly (1,1)^\top/\sqrt{2} and (1,-1)^\top/\sqrt{2}. The first component now explains 1.913/2=95.7\%. The directions differ from the unscaled analysis because the two variables have different variances, 5.91 and 3.24.

How many components, on a correlation with no response

Main article: How many components, on a correlation with no response

An eight-dimensional correlation with no response attached asks only how fast the cumulative proportion grows. Take the AR(1) correlation R_{ij}=0.6^{|i-j|} for i,j=1,\ldots,8. Its eigenvalues and cumulative proportions are

lambda <- eigen(0.6^abs(outer(1:8, 1:8, "-")), symmetric = TRUE)$values
data.frame(
  Component = 1:8,
  Eigenvalue = round(lambda, 3),
  Cumulative_percent = round(100 * cumsum(lambda) / sum(lambda), 1)
)

Component  Eigenvalue  Cumulative percent
        1       3.151                39.4
        2       1.832                62.3
        3       1.035                75.2
        4       0.644                83.3
        5       0.448                88.9
        6       0.344                93.2
        7       0.287                96.8
        8       0.259               100.0

A rule that keeps 90\% of the variance keeps six components, because the cumulative proportion first exceeds 90 at component 6 (88.9 then 93.2). A rule that keeps 75\% keeps three. Keeping every eigenvalue above the average, here 1, also keeps three, since the trace of a correlation matrix equals its dimension and the average eigenvalue is 1. None of these cutoffs is best for every purpose, and none of them sees a response. Each of them answers the reconstruction problem, with a chosen fraction of \operatorname{tr}(S).

A scree plot offers a visual heuristic: look for a point after which additional eigenvalues contribute relatively little. The cumulative percentages on this correlation are about 39, 62, 75, 83, 89, 93, 97 and 100. A bend can be ambiguous or absent, and it does not validate a predictive model. For prediction, select the number of components using a validation set or an inner cross-validation. Keep the final test set untouched. Using the same observations both to choose the number of components and to assess the fit leaves no untouched assessment sample.

Used in

Taught in Principal component analysis
All 7 questions on Principal component analysis

How many components, on a correlation with no response

Used in

See also