Separate tuning from evaluation
A student tries 20 classifiers on one validation set and reports the highest validation accuracy as the accuracy expected on new observations.
Explain the problem with this reported value.
Describe a training, validation, and test workflow that selects a classifier and estimates its performance. Include preprocessing and any final refitting.
Explain what changes if cross-validation replaces the single validation split.
Validation results vary because of sampling noise. Selecting the largest value favours candidates with favourable noise as well as genuinely good performance. Reporting that selected value creates optimistic selection bias. It need not exceed test accuracy in every realised split.
Fit preprocessing and candidate models on training data. Apply the fitted transforms to the validation data and choose settings using a stated criterion. Freeze those choices. Optionally refit the chosen procedure, including preprocessing, on the training data plus the validation data. Evaluate that fixed fitted procedure once on the independent test set, and state the metric and the sample size. Further choices based on the test results would use up its independence.
Each fold has its own fitted preprocessing and model. Cross-validation can tune, and the best cross-validation score is still selected. Use an external test set, or an outer cross-validation loop, for evaluation. An outer loop must repeat all inner choices.