Tuning a nearest-neighbour classifier
Topic: Nearest neighbours · more questions on this topic
‹ Normalisation and nearest neighbours What an accuracy table does and does not say ›
A dataset contains labelled measurements of two plant species. Predictor variables use different units. There are enough observations for separate training, validation, and test sets. You want to select k and estimate the accuracy of the resulting nearest-neighbour procedure.
Explain what normalising the predictors means here and why it is desirable. Specify which data determine the transformation.
Describe how all three sets should be used, including candidate choices, selection, optional refitting, and final evaluation.
Across many independent repetitions, selected validation accuracy tends to exceed test accuracy. Explain the mechanism.
Would a validation set also be useful for a decision tree? Give a reason and a concrete example.
Different units can dominate squared distances. Subtract each training mean and divide by its training standard deviation, then apply the same transform to held-out points. This makes differences comparable in standard-deviation units. It does not guarantee equal predictive relevance, and it does not make the marginals Gaussian.
Prespecify plausible candidate values of k and a metric. Fit the preprocessing on the training data. For each k, predict the validation observations from the training neighbours. Select the best k by the validation criterion, with a stated tie rule. Optionally refit the preprocessing and the neighbour reference set on the training data plus the validation data, with k fixed. Estimate accuracy on the untouched test set, without making new choices from its results. State whether the report is overall or class-specific accuracy. A later refit on all the data is possible for deployment. The test score evaluated the preceding procedure and is not a new independent test of that final refit.
The best observed validation result benefits from favourable validation noise as well as from true performance. Selection therefore induces optimism. Independent test outcomes do not share that same selection event. This is a repeated-sampling tendency, not a guaranteed ordering on every split.
A validation set is useful for a tree. Choose a pruning penalty, a depth limit, or a leaf-size restriction from training-fitted candidates, using validation loss. A fully grown tree can isolate training noise. Preserve a separate final evaluation.