Least squares, ridge, and the lasso objective
Section of High-dimensional analysis, part of Statistical learning
Least squares, ridge, and the lasso objective
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
For standard linear regression use Y=\beta_0+X^\top\beta+\epsilon with \beta=(\beta_1,\ldots,\beta_p)^\top the slopes, \beta_0 the constant, and \epsilon\sim\operatorname{N}(0,\sigma^2). The intercept and the error variance are present in every model of interest. They are not shrunk or removed. The maximum-likelihood solution is the least-squares solution
(\hat\beta_0,\hat\beta) =\arg\min_{\beta_0,\beta} \sum_{i=1}^{n}\Biggl(y_i-\beta_0-\sum_{j=1}^{p}\beta_j x_{ij}\Biggr)^2,
which has the inverse-based closed form above when the augmented design has full column rank.
Ridge regression, also called weight decay or Tikhonov regularisation (Hoerl and Kennard, 1970), is a long-standing shrinkage method. Standardise the predictors first: subtract each sample mean and divide by the sample standard deviation. Then add the sum of squared regression weights, an L^2 penalty. Do not run one \lambda on variables that live on different scales. Divide a variable by 1000 and its coefficient has to grow by 1000 to give the same fit. The same \lambda is then a different penalty, because a large coefficient dominates the sum.
(\hat\beta_0,\hat\beta) =\arg\min_{\beta_0,\beta} \Biggl\{ \sum_{i=1}^{n}\Biggl(y_i-\beta_0-\sum_{j=1}^{p}\beta_j x_{ij}\Biggr)^2 +\lambda\sum_{j=1}^{p}\beta_j^2 \Biggr\}.
Here \lambda controls the amount of shrinkage. It can be chosen manually or by optimising a measure of predictive accuracy such as cross-validation. Two layers of cross-validation are needed to both select such a parameter and estimate the accuracy of the resulting model. Ridge regression was introduced to improve regression for non-orthogonal predictors, those with substantial multicollinearity, where the slope estimates are unstable, with large confidence intervals and high sensitivity to individual observations. This is also seen as near-singularity of X^\top X. With centred y and standardised X, so that the intercept is \bar y,
\hat\beta_R=(X^\top X+\lambda I)^{-1}X^\top y.
Adding \lambda I reduces the condition number of the matrix to be inverted. If p>n then X^\top X is rank-deficient, so not invertible. It is positive semi-definite, so at least one eigenvalue is zero and the others are nonnegative. Adding \lambda I raises every eigenvalue by \lambda>0, making the matrix positive definite and invertible, and hence allowing a unique solution even when p>n.
Tibshirani (1996), building in part on Donoho and Johnstone (1994) and Breiman (1995), replaces the L^2 penalty by an L^1 penalty. The resulting estimator is the lasso, the least absolute shrinkage and selection operator:
(\hat\beta_0,\hat\beta) =\arg\min_{\beta_0,\beta} \Biggl\{ \sum_{i=1}^{n}\Biggl(y_i-\beta_0-\sum_{j=1}^{p}\beta_j x_{ij}\Biggr)^2 +\lambda\sum_{j=1}^{p}|\beta_j| \Biggr\}.
This is the Lagrangian equivalent of minimising the same sum of squares subject to \sum_j|\beta_j|\le t, with a corresponding constraint radius t. The correspondence need not be one-to-one: flat portions of the solution path and inactive constraints can give multiple parameter values for the same fit. The method aims to shrink some parameter values and set others to exactly zero, achieving both shrinkage and variable selection. With standardised data the solution for \hat\beta_0 is \bar y, and one then only needs to choose \beta_1,\ldots,\beta_p.
On the Bayesian page, a Laplace prior on the slopes has a posterior mode at this same L^1-penalised least-squares fit. The derivation here is the objective and the coordinate-wise solution, not the prior.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from