Soft threshold

Topic: High-dimensional analysis · more questions on this topic
‹ A step-up subtlety A tempting wrong turn ›

Orthonormal predictors have least-squares estimates 2.4, -0.9, 0.5, and -0.1. Compute the ridge estimates for \lambda=2, and the lasso estimates for \lambda=2. Which variables are selected by the lasso?

Worked answer (click to show)

The objective is the raw sum of squares plus \lambda times the penalty, so the lasso threshold is \lambda/2. Ridge with \lambda=2 divides by 1+\lambda=3:

(2.4,\,-0.9,\,0.5,\,-0.1)/3=(0.80,\,-0.30,\,0.167,\,-0.033),

all nonzero. The lasso threshold is \lambda/2=1, so

\operatorname{sign}(b)\bigl(|b|-1\bigr)_+=(1.4,\,0,\,0,\,0).

Only the first variable is selected. Its coefficient is shrunk from 2.4 to 1.4.