Principal components when p is large
Section of High-dimensional analysis, part of Statistical learning
Principal components when p is large
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
Standard principal components can be used when p>n, with the rank bound already in hand. Since S is symmetric, all its eigenvalues are real. Its rank is at most n-1, so the number of nonzero eigenvalues is at most n-1, and the number of principal components is limited to \min(n-1,p), which equals n-1 when p>n. Vectors in a p-dimensional space, n of them, span at most an n-dimensional subspace, and the analysis loses one degree of freedom through the sample mean.
Later principal components can separate groups when the first does not. The first component is the direction of greatest variance. It is not automatically the scientific direction. The leading components often follow the bulk of the variation, and a later component may be the one aligned with a group difference.
For p\gg n, forming a full p\times p covariance can waste memory and arithmetic. Write the compact singular-value decomposition X_c=U_r E_r V_r^\top, where r=\operatorname{rank}(X_c)\le n-1 and all diagonal entries e_i of E_r are positive. The nonzero eigenvalues of X_c X_c^\top are e_i^2=(n-1)\lambda_i. Recover the corresponding loading as a_i=X_c^\top u_i/e_i. Division by a zero singular value is not a way to recover an additional component. An economy decomposition avoids explicitly constructing either a large covariance or a zero-variance basis.
The (n-1)-dimensional basis can be an extreme summary of a small sample. The main purpose is to reduce dimensionality, often well below \min(n-1,p), which reduces the room to overfit. Other methods may require this before they can be applied. To fit the covariance matrices, quadratic discrimination requires the dimension of the data to be less than the sample size of the smallest class, while linear discrimination estimates a pooled covariance and so uses all the data. Sparse principal-component methods are among the extensions built for this regime.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from