A sparse simulation where the truth is known
Section of High-dimensional analysis, part of Statistical learning
A sparse simulation where the truth is known
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
Iris has only four predictors, so it is a poor example of a genuinely high-dimensional problem. Instead simulate 100 training observations and 150 predictors, of which five have nonzero coefficients. Independent test observations separate prediction error from support recovery. Cross-validation chooses \lambda inside the training sample.
The implementation normalises its loss differently from the raw sum-of-squares objective above. Compare fitted procedures and the definition of the penalty, not bare numerical values of \lambda across conventions.
set.seed(18006)
n <- 100; p <- 150
beta <- c(3, -2, 1.5, 1, -1, rep(0, p - 5))
X <- matrix(rnorm(n * p), n, p)
y <- drop(X %*% beta + rnorm(n))
Xnew <- matrix(rnorm(500 * p), 500, p)
ynew <- drop(Xnew %*% beta + rnorm(500))
fold <- sample(rep(1:10, length.out = n))
lasso <- glmnet::cv.glmnet(X, y, alpha = 1, foldid = fold)
pred <- drop(predict(lasso, Xnew, s = "lambda.1se"))
b <- as.numeric(coef(lasso, s = "lambda.1se"))[-1]
selected <- which(abs(b) > 1e-8)
c(test_MSE = mean((pred - ynew)^2),
selected = length(selected),
true_positives = sum(selected <= 5),
false_positives = sum(selected > 5))
#> test_MSE selected true_positives false_positives
#> 1.534 6.000 5.000 1.000The coefficient pattern is known because the simulation specified it. The reported test mean squared error is 1.534. Six coefficients are selected, five of them among the five true signals and one a false inclusion. Prediction error, false inclusions, and missed signals are different outcomes. A single simulated data set illustrates the distinction. It does not establish a universal selection guarantee, and it is not a false-discovery calculation: the lasso was not asked to control an expected proportion of false inclusions.
High dimension makes rank and selection assumptions consequential. Least-squares fitted values still exist, while coefficient identification can fail. Ridge supplies a unique penalised coefficient vector. The lasso can produce sparse fits. Benjamini–Hochberg controls an expected false-discovery proportion under its calibration and dependence assumptions. None of these facts says that the variables selected in one data set are exactly the true ones.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from