Choosing the lasso penalty
Section of High-dimensional analysis, part of Statistical learning
Choosing the lasso penalty
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
The penalty determines a trade-off, so it must be chosen for an objective. Prediction-oriented tuning can use cross-validation with the entire fitting procedure repeated within each fold. The broader family \lambda\sum_j|\beta_j|^q connects several familiar penalties: q=2 gives ridge, q=1 gives the lasso, and counting nonzero coefficients gives the L^0 formulation. The mathematical appeal of q=1 is concrete: it retains convexity while introducing a corner at zero. For 0<q<1, the penalty is nonconvex. These geometric facts do not by themselves select a penalty value (Tibshirani, 1996; Frank and Friedman, 1993).
A path algorithm computes fits over a range of penalties. Each fixed-\lambda least-squares lasso problem is convex. Under the usual nondegeneracy conditions, coefficients vary piecewise linearly with \lambda, with changes when variables enter or leave the active set. Calling the path itself convex is a different claim. Efficient path algorithms exploit the piecewise-linear structure. Their workload still depends on the design and the number of changes.
Read a coefficient path as a sequence of fitted models, not a ranking of biological importance. A coefficient can change sign, reach zero, or later re-enter. A seeded sparse regression produces one such path, together with a ten-fold cross-validation curve and its reported standard errors. The curve selects a prediction trade-off. It does not certify support recovery.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from