Multicollinearity
Topic of High-dimensional analysis, part of Statistical learning
Multicollinearity
| Type | Topic |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
| Questions using it | 2 |
Strong linear dependence among predictors. It makes \mathbf{X}^\top\mathbf{X} nearly singular, so least-squares coefficients become unstable with large standard errors; when p>n the matrix is exactly singular. Ridge regression was introduced for it.
In High-dimensional analysis
Here \lambda controls the amount of shrinkage. It can be chosen manually or by optimising a measure of predictive accuracy such as cross-validation. Two layers of cross-validation are needed to both select such a parameter and estimate the accuracy of the resulting model. Ridge regression was introduced to improve regression for non-orthogonal predictors, those with substantial multicollinearity, where the slope estimates are unstable, with large confidence intervals and high sensitivity to individual observations. This is also seen as near-singularity of X^\top X. With centred y and standardised X, so that the intercept is \bar y,
\hat\beta_R=(X^\top X+\lambda I)^{-1}X^\top y.
Used in
- Taught in High-dimensional analysis
- Also used in Mixtures and the EM algorithm
- Questions that use it (2) — Euclidean clusters and mixture components (answer) · Ridge makes the matrix invertible
- All 9 questions on High-dimensional analysis
See also
Standard error · Ridge regression · Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance