Rank of the sample covariance
Section of High-dimensional analysis, part of Statistical learning
Rank of the sample covariance
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
For a single data set with p>n-1 the sample covariance matrix is singular, hence not invertible. The n\times p data matrix X_{\mathrm{all}} has rank at most \min(n,p). With A=\sum_{i=1}^{n}(x_i-\bar x)(x_i-\bar x)^\top and S=\frac{1}{n-1}A, the matrices A and S have the same rank. Each observation can contribute one dimension, and centring spends one of them: the rows of the centred matrix sum to zero.
Theorem 1 (Rank bound) \operatorname{rank}(S)\le\min(n-1,p). Hence S is invertible only if p\le n-1 and the centred data have rank p.
Let X_c be the centred data matrix with rows (x_i-\bar x)^\top, so that A=X_c^\top X_c and \operatorname{rank}(A)=\operatorname{rank}(X_c). The rows of X_c sum to zero, \sum_i(x_i-\bar x)=0, so they are linearly dependent and span a subspace of dimension at most n-1. Also \operatorname{rank}(X_c)\le p.
The key dependence is 1^\top X_c=0^\top: each column of the centred matrix lies in the (n-1)-dimensional subspace of \mathbb{R}^n perpendicular to 1. Equivalently, its rows sum to zero. Centring each of g nonempty groups gives the pooled within-group rank bound n-g. For n=100 and g=2, the full sample can have at most 99 positive principal variances, while a pooled covariance has rank at most 98. Neither upper bound guarantees full rank. In cross-validation, calculate these bounds using the training fold.
Example 1 (A singular covariance) Question. Three observations are recorded on five variables. What fails if this sample covariance is dropped into a Mahalanobis distance, Hotelling’s T^2, or an unregularised Gaussian discriminant rule?
Solution. S has rank at most two. At least three eigenvalues are zero. Those procedures need S^{-1}, which does not exist. The Gaussian likelihood itself is still defined for a candidate positive-definite population covariance. The problem is that its unrestricted covariance maximum-likelihood estimate is not a positive-definite matrix. For quadratic discrimination, n_g-1\ge p is necessary in each class. For linear discrimination with G nonempty classes, n-G\ge p is necessary for an invertible pooled estimate. Collinearity can make either estimate singular even when those inequalities hold.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Regression when p > n · Where high-dimensional columns come from · Questions to ask of high-dimensional data