Extensions of the lasso
Section of High-dimensional analysis, part of Statistical learning
Extensions of the lasso
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
The lasso can also be used with a generalised linear model, one with g(\mu(x))=x^\top\beta where \mu(x)=\mathbb{E}(Y\mid X=x), g is monotone, and Y has a distribution from the exponential family. In particular it applies to logistic regression: minimise the negative log-likelihood plus \lambda\sum_j|\beta_j|, with \lambda chosen by cross-validation of the binomial deviance.
The lasso is a shrinkage method. An L^1 penalty constrains the overall coefficient size, but with correlated predictors an individual coefficient need not be smaller in magnitude than its least-squares counterpart, so the estimator is biased. One question is whether this bias can be reduced, at least as n grows.
Assume a lasso is fit with \lambda chosen by cross-validation. This typically removes many variables. Refitting an unpenalised model on the same data the lasso selected is biased. A sample the lasso did not see is the clean check. Only about half the data is then used for each of selection and fitting.
Another option, proposed by Zou (2006), is the adaptive lasso
(\hat\beta_0,\hat\beta) =\arg\min_{\beta_0,\beta} \Biggl\{ \sum_{i=1}^{n}\Biggl(y_i-\beta_0-\sum_{j=1}^{p}\beta_j x_{ij}\Biggr)^2 +\lambda\sum_{j=1}^{p} w_j|\beta_j| \Biggr\}.
The displayed version uses w_j=1/|\tilde\beta_j| from a pilot fit. The general adaptive lasso permits powers |\tilde\beta_j|^{-\gamma} with \gamma>0. Large pilot coefficients receive less shrinkage. A zero pilot requires an explicit rule, such as exclusion or a stated stabilisation. Zou (2006, Theorem 2) gives oracle properties under assumptions on the design, a sufficiently accurate pilot, and the rate of the penalty sequence. These are not guarantees for any cross-validated penalty or any marginal-regression pilot, particularly with correlated predictors and growing dimension. Selection consistency and coefficient consistency must be checked under the stated asymptotic regime.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from