Normalisation and nearest neighbours

Topic: Nearest neighbours · more questions on this topic
‹ A tempting wrong turn Tuning a nearest-neighbour classifier ›

A nearest-neighbour classifier uses height in millimetres and a proportion between 0 and 1. An analyst proposes standardising the predictors before selecting k.

1. Explain why their relative scales matter and state one suitable transformation.

2. Describe how to choose k using a validation set without contaminating a final test set.

3. Explain why standardisation does not guarantee that the two predictors become equally informative.

Worked answer (click to show)

1. Euclidean distances sum squared differences in the chosen units, so height can dominate solely through its scale. Fit z_j=(x_j-\bar x_{j,\mathrm{train}})/s_{j,\mathrm{train}} on the training data and reuse those constants. A constant predictor must be removed or handled separately.

2. For a prespecified set of k values, predict validation labels using training neighbours and the training-fitted scaling. Choose k from the validation errors, optionally refit on the training data plus the validation data, and assess the frozen rule on the untouched test set. No test observation is a neighbour during fitting or tuning.

3. Equalising numerical variance does not create signal, remove correlation, or assign an optimal metric. A pure-noise predictor still affects distance after standardisation. Normalisation here is not a transformation of every marginal into a normal distribution.