A tempting wrong turn

Topic: High-dimensional analysis · more questions on this topic
‹ Soft threshold When ordinary linear discrimination is not defined ›

A researcher screens 20{,}000 genes, picks the 20 with the smallest p-values, fits a logistic regression on these using all 62 samples, and reports the cross-validated error from this model as 4\%. Explain why this estimate is optimistic and how to correct the procedure.

Worked answer (click to show)

The 20 genes were chosen by looking at the labels of all 62 samples. The cross-validation is then done only on the modelling step, with the selection already informed by the held-out samples in each fold. The selected genes are, by construction, the ones that happen to separate these samples. With 20{,}000 candidates, some genes will separate the classes well by chance alone. The held-out observations were therefore not independent of the model, and the 4\% error is optimistic. It may be near 0 even for pure noise.

The correction is to put the entire pipeline inside the cross-validation. In each fold, compute the p-values, or fit the lasso, using the training folds only, select the genes, fit the classifier, and evaluate on the held-out fold. Choosing the number of genes or \lambda also needs an inner loop.