Laboratory: what a distance metric changes

Section of Nearest neighbours, part of Statistical learning

Laboratory: what a distance metric changes

Type Section
Taught in Nearest neighbours
Part Statistical learning

The comparison holds k fixed at 7, chosen in advance, so the test set is not used to retune the neighbourhood. One classifier uses the raw predictors. The other standardises from the training flowers only and applies that same shift and scale to the test flowers. Any change in the predictions is then a change in the metric. iris_scaled stores that training-only standardisation.

raw_pred <- class::knn(train[, 1:4], test[, 1:4], train$Species, k = 7)
std_pred <- class::knn(iris_scaled$train, iris_scaled$test,
                        train$Species, k = 7)
c(raw_test_error = mean(raw_pred != test$Species),
  scaled_test_error = mean(std_pred != test$Species),
  changed_predictions = sum(raw_pred != std_pred))
#>     raw_test_error scaled_test_error changed_predictions
#>            0.06667           0.06667                   2
d <- sqrt(rowSums(sweep(iris_scaled$train, 2,
                         iris_scaled$test[1, ])^2))
data.frame(Species = train$Species[order(d)[1:7]],
           Distance = round(sort(d)[1:7], 3))
#>   Species Distance
#>   setosa     0.225
#>   setosa     0.444
#>   setosa     0.503
#>   setosa     0.554
#>   setosa     0.600
#>   setosa     0.652
#>   setosa     0.906

Both versions have test error 0.06667, and two predictions change. Similar accuracy can sit on different neighbours, which is why the laboratory also prints the seven nearest training flowers to the first held-out flower under the standardised metric. All seven are setosa, at standardised distances 0.225, 0.444, 0.503, 0.554, 0.600, 0.652, and 0.906. Standardisation gives the four measurements a common marginal scale. It does not make them equally relevant, it does not remove correlation, and it does not produce an optimal metric. An irrelevant predictor, scaled to unit variance, still moves the distances.

Key idea

The rule replaces a coefficient fit by stored observations, a metric, and a neighbourhood size. That simplicity does not remove model selection. Choose preprocessing and k within training data, and inspect whether the resulting neighbours are meaningful. Consistency describes an infinite-data limit, not the adequacy of a particular sample.

Used in

See also

Rank distance · -nearest neighbours · Bayes consistency of -nearest neighbours · Distance and scale in nearest neighbours · Ties and the choice of k · Consistency and the factor of two · Cover–Hart bound

Glossary and index