Soft-thresholding for orthonormal predictors
Section of High-dimensional analysis, part of Statistical learning
Soft-thresholding for orthonormal predictors
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
When X^\top X=I, the penalty separates across coordinates. For ridge, the derivative of (\beta_j-\hat\beta_j^{\mathrm{LS}})^2+\lambda\beta_j^2 is zero at
\hat\beta_j=\frac{\hat\beta_j^{\mathrm{LS}}}{1+\lambda}.
Every coefficient shrinks by the same factor. For the lasso, minimise (\beta_j-b)^2+\lambda|\beta_j| with b=\hat\beta_j^{\mathrm{LS}}. For \beta_j>0 the derivative is 2(\beta_j-b)+\lambda, zero at \beta_j=b-\lambda/2, which is positive only if b>\lambda/2. The negative side is symmetric. If |b|\le\lambda/2 the minimum is at 0. Hence
\hat\beta_j^{\mathrm{lasso}}=\operatorname{sign}(b)\bigl(|b|-\lambda/2\bigr)_+,
the soft-thresholding operator. Coefficients smaller than \lambda/2 in magnitude are set to zero, and the others are shrunk towards zero by \lambda/2. The factor \lambda/2, rather than \lambda, is the derivative of a square that was not written with a leading \tfrac12.
Best-subset selection with a penalty \lambda on \lVert\beta\rVert_{L^0} gives hard thresholding: keep b unchanged if |b|>\sqrt{\lambda} and set it to 0 otherwise. At |b|=\sqrt{\lambda}, both 0 and b minimise the criterion (\beta-b)^2+\lambda\,1_{\beta\neq 0}. The displayed rule chooses zero.
For example, with least-squares estimates (3,-1.5,0.8,-0.3,0.1) and \lambda=1: ridge gives (1.5,-0.75,0.4,-0.15,0.05), all nonzero; the lasso gives (2.5,-1.0,0.3,0,0); hard thresholding gives (3,-1.5,0,0,0). The soft-threshold also shows the lasso’s bias: even a coefficient as large as 3 is shrunk by \lambda/2.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from