Regression when p > n
Section of High-dimensional analysis, part of Statistical learning
Regression when p > n
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
The same rank obstruction appears in the normal equations. The least-squares coefficient for a linear model with an intercept is
\hat\beta=(X^\top X)^{-1}X^\top y,
where y is the vector of continuous responses and X is the n\times(p+1) matrix of explanatory variables augmented by a column of ones. The product is X^\top y. Dropping the transpose leaves Xy, which is undefined: X is n\times(p+1) and y is n\times 1. The same transpose belongs on the ridge formula below. The closed form is the one derived for linear regression when the columns are linearly independent.
The inverse formula requires full column rank, not merely a favourable count of rows and columns. When rank fails, least squares still projects y onto the column space of X, giving a unique fitted vector. The coefficient representation is non-unique: adding any v with Xv=0 leaves the fitted values unchanged. Exact interpolation occurs when y lies in that column space. Full row rank is sufficient. Having p\ge n alone is not sufficient.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Where high-dimensional columns come from · Questions to ask of high-dimensional data