What an accuracy table does and does not say
An untouched test set gives the following confusion matrix for a tuned classifier. Rows are true classes and columns are predicted classes.
| true / predicted | class 1 | class 2 |
|---|---|---|
| class 1 | 18 | 2 |
| class 2 | 8 | 32 |
The chosen model had validation accuracy 0.93.
Calculate the overall and both class-specific test error rates.
Explain why the validation accuracy may exceed test accuracy, without assuming that the data-generating distribution changed.
Describe a valid cross-validation design for choosing both scaling and k in a nearest-neighbour classifier.
A colleague compares 50 further settings using this same test set. Explain how the status of that set changes.
Would classifying everyone as class 2 be a useful baseline? Calculate its overall accuracy on this set and explain the limitation of that metric.
The overall error is (2+8)/60=1/6=0.1667. The class-1 error is 2/20=0.10. The class-2 error is 8/40=0.20. The class denominators are the row totals.
Validation-driven selection favours favourable noise, and the independent test sample has different noise. Sampling variation alone can also cause a difference, so a single gap is not a proof of selection bias or of a change in the data-generating distribution.
Keep an outer test split, or outer cross-validation folds, separate from selection. For every candidate pipeline in the inner cross-validation, fit its scaling, and any feature selection, on the inner training fold. Predict that fold’s validation observations from its training neighbours. Select the candidate using the aggregated inner loss, then refit on the outer training data and evaluate externally.
Once the former test set is used to choose settings, it is part of model selection. The best score on it is selected and is no longer an independent final assessment. Use genuinely fresh test data, or an assessment design that repeats the selection within outer training folds. Renaming the reused data does not repair the problem.
Always predicting class 2 gives 40/60=2/3 accuracy and is a useful no-feature baseline. It misses every class-1 observation. Overall accuracy hides that failure, so report class-specific errors and consider the losses and the prevalence that the deployment actually uses.