Sparsity

Topic of High-dimensional analysis, part of Statistical learning · Also in High-dimensional analysis

Sparsity

Type Topic
Taught in High-dimensional analysis · High-dimensional analysis
Part Statistical learning
Questions using it 1

A model is sparse when only a few of its parameters are non-zero. Sparsity is what makes estimation possible when p>n; the lasso and the graphical lasso exploit it by setting coefficients exactly to zero.

In High-dimensional analysis

From High-dimensional analysis

The ellipses of a least-squares cost are centred at the unpenalised estimate. The constraint sets are |\beta_1|+|\beta_2|\le t and \beta_1^2+\beta_2^2\le t^2. The penalised estimate is the point where the lowest contour still touches the constraint. The L^1 set is a diamond. A smooth ellipse tends to touch it at a corner, and a corner sets one coefficient to zero. The L^2 set is a disk. The ellipse meets the disk at a smooth point, and both coefficients stay nonzero. Sparsity depends on the penalty, the loss, and the data. High dimension alone does not force the right variables to be the ones selected.

The diamond does not force the optimum to a corner on every data set. It makes corners available over a range of loss contours, which is the mechanism for exact zeros. Ridge can have a zero coefficient through cancellation or symmetry. Its smooth penalty does not create a thresholding region around zero.

In High-dimensional analysis, continued

From High-dimensional analysis

Iris has only four predictors, so it is a poor example of a genuinely high-dimensional problem. Instead simulate 100 training observations and 150 predictors, of which five have nonzero coefficients. Independent test observations separate prediction error from support recovery. Cross-validation chooses \lambda inside the training sample.

The implementation normalises its loss differently from the raw sum-of-squares objective above. Compare fitted procedures and the definition of the penalty, not bare numerical values of \lambda across conventions.

set.seed(18006)
n <- 100; p <- 150
beta <- c(3, -2, 1.5, 1, -1, rep(0, p - 5))
X <- matrix(rnorm(n * p), n, p)
y <- drop(X %*% beta + rnorm(n))
Xnew <- matrix(rnorm(500 * p), 500, p)
ynew <- drop(Xnew %*% beta + rnorm(500))
fold <- sample(rep(1:10, length.out = n))
lasso <- glmnet::cv.glmnet(X, y, alpha = 1, foldid = fold)
pred <- drop(predict(lasso, Xnew, s = "lambda.1se"))
b <- as.numeric(coef(lasso, s = "lambda.1se"))[-1]
selected <- which(abs(b) > 1e-8)
c(test_MSE = mean((pred - ynew)^2),
  selected = length(selected),
  true_positives = sum(selected <= 5),
  false_positives = sum(selected > 5))
#>         test_MSE         selected  true_positives false_positives
#>            1.534            6.000           5.000           1.000

The coefficient pattern is known because the simulation specified it. The reported test mean squared error is 1.534. Six coefficients are selected, five of them among the five true signals and one a false inclusion. Prediction error, false inclusions, and missed signals are different outcomes. A single simulated data set illustrates the distinction. It does not establish a universal selection guarantee, and it is not a false-discovery calculation: the lasso was not asked to control an expected proportion of false inclusions.

Key idea

High dimension makes rank and selection assumptions consequential. Least-squares fitted values still exist, while coefficient identification can fail. Ridge supplies a unique penalised coefficient vector. The lasso can produce sparse fits. Benjamini–Hochberg controls an expected false-discovery proportion under its calibration and dependence assumptions. None of these facts says that the variables selected in one data set are exactly the true ones.

Used in

See also

Lasso · Graphical lasso · Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance

Glossary and index