How many components, on a correlation with no response

Section of Principal component analysis, part of Statistical learning

Type	Section
Taught in	Principal component analysis
Part	Statistical learning

An eight-dimensional correlation with no response attached asks only how fast the cumulative proportion grows. Take the AR(1) correlation R_{ij}=0.6^{|i-j|} for i,j=1,\ldots,8. Its eigenvalues and cumulative proportions are

lambda <- eigen(0.6^abs(outer(1:8, 1:8, "-")), symmetric = TRUE)$values
data.frame(
  Component = 1:8,
  Eigenvalue = round(lambda, 3),
  Cumulative_percent = round(100 * cumsum(lambda) / sum(lambda), 1)
)

Component  Eigenvalue  Cumulative percent
        1       3.151                39.4
        2       1.832                62.3
        3       1.035                75.2
        4       0.644                83.3
        5       0.448                88.9
        6       0.344                93.2
        7       0.287                96.8
        8       0.259               100.0

A rule that keeps 90\% of the variance keeps six components, because the cumulative proportion first exceeds 90 at component 6 (88.9 then 93.2). A rule that keeps 75\% keeps three. Keeping every eigenvalue above the average, here 1, also keeps three, since the trace of a correlation matrix equals its dimension and the average eigenvalue is 1. None of these cutoffs is best for every purpose, and none of them sees a response. Each of them answers the reconstruction problem, with a chosen fraction of \operatorname{tr}(S).

A scree plot offers a visual heuristic: look for a point after which additional eigenvalues contribute relatively little. The cumulative percentages on this correlation are about 39, 62, 75, 83, 89, 93, 97 and 100. A bend can be ambiguous or absent, and it does not validate a predictive model. For prediction, select the number of components using a validation set or an inner cross-validation. Keep the final test set untouched. Using the same observations both to choose the number of components and to assess the fit leaves no untouched assessment sample.

Used in

Taught in Principal component analysis
All 7 questions on Principal component analysis

Used in

See also