Subset selection and shrinkage
Section of High-dimensional analysis, part of Statistical learning
Subset selection and shrinkage
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
One way of limiting overfitting is to penalise the magnitude of the parameters. Another, as with the Bayesian information criterion, is to penalise the number of parameters. Another way to choose a subset begins with a method of assessing predictive accuracy: the cross-validation PRESS statistic for a continuous response, or classical cross-validation for a nominal response. One then searches over all possible combinations of the available variables. With p explanatory variables the number of subsets is 2^p. For p=30 this is more than 10^9, and for p=100 about 10^{30}.
Greedy searches make a manageable sequence of comparisons. Forward selection adds one variable at a time. Backward selection removes one from an initially fitted model. That reduces the search, and it does not guarantee the best subset. A better training fit does not settle the prediction question. For nested least-squares models, training R^2 cannot decrease as variables are added. Selection therefore needs a criterion that accounts for complexity or evaluates predictions on observations excluded from the relevant fitting and selection steps.
Best subset selection fixes the number of selected variables at some value k<p and searches over this smaller domain. It can be viewed as a hard-constraint version of the information criterion, or as a method that uses an L^0 penalty, which counts the nonzero elements:
\lVert\beta\rVert_{L^0}=\#\{j:\beta_j\neq 0\}.
This is not formally a norm. Absolute homogeneity fails in general: \lVert a\beta\rVert_{L^0}\neq |a|\,\lVert\beta\rVert_{L^0} for most scalars a and vectors \beta.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from