Why the L¹ penalty gives zeros
Section of High-dimensional analysis, part of Statistical learning
Why the L¹ penalty gives zeros
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
The ellipses of a least-squares cost are centred at the unpenalised estimate. The constraint sets are |\beta_1|+|\beta_2|\le t and \beta_1^2+\beta_2^2\le t^2. The penalised estimate is the point where the lowest contour still touches the constraint. The L^1 set is a diamond. A smooth ellipse tends to touch it at a corner, and a corner sets one coefficient to zero. The L^2 set is a disk. The ellipse meets the disk at a smooth point, and both coefficients stay nonzero. Sparsity depends on the penalty, the loss, and the data. High dimension alone does not force the right variables to be the ones selected.
The diamond does not force the optimum to a corner on every data set. It makes corners available over a range of loss contours, which is the mechanism for exact zeros. Ridge can have a zero coefficient through cancellation or symmetry. Its smooth penalty does not create a thresholding region around zero.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from