A tempting wrong turn

Topic: Estimating error rates · more questions on this topic
‹ Costs of cross-validation Separate tuning from evaluation ›

A student tries 50 values of a tuning parameter, picks the one with the lowest 10-fold cross-validation error (8.0\%), and reports 8.0\% as the expected error on new data. Explain why this is biased and how nested cross-validation corrects it.

Worked answer (click to show)

Selection favours candidates with favourable estimation noise. The minimum inner cross-validation score is not an independent estimate of the chosen procedure’s prediction error, so the reported 8.0\% can be optimistic. Its bias relative to a particular full-sample target also involves training-size effects, so there is no guaranteed numerical ordering on every split. Outer cross-validation repeats selection using only each outer training set, then evaluates the selected fit on the omitted fold. It estimates performance at that outer training size.