Selection in many dimensions

Topic: High-dimensional analysis · more questions on this topic
‹ Principal components before linear discrimination

You have 200 observations and 1000 predictors. In a simulation, only the first five coefficients in a linear conditional-mean model are nonzero.

  1. Explain why a cross-validated lasso need not select exactly those five variables, even if it predicts well.

  2. You also test 1000 separate marginal regressions. Explain the Benjamini–Hochberg procedure at target false-discovery rate 0.1, define the error criterion, and distinguish marginal association from a nonzero coefficient in the joint model.

  1. Cross-validation commonly targets prediction error, not exact support recovery. Correlated predictors can substitute for one another, and finite-sample noise can add or omit variables. Penalisation also biases coefficient estimates. A good predictive loss is not proof of recovering the generating coefficients.

  2. Sort valid p-values p_{(1)}\le\cdots\le p_{(1000)} and take

k=\max\{i:p_{(i)}\le 0.1\,i/1000\},

rejecting the first k, or none if the set is empty. The false discovery rate is \mathbb{E}\bigl[V/\max(R,1)\bigr], the expected fraction of false rejections among discoveries. It is not a guarantee on each realised sample. The usual control requires appropriate dependence conditions, such as independence or a suitable positive dependence. Arbitrary dependence needs the harmonic correction. A predictor can be marginally associated through correlation with other predictors even when its joint coefficient is zero, so the two procedures test different targets.