Laboratory: tune neighbours without leaking the test set

Section of Estimating error rates, part of Statistical learning

Laboratory: tune neighbours without leaking the test set

Type Section
Taught in Estimating error rates
Part Statistical learning

The test flowers stay out of the choice of k. Inside the 105 training flowers, five stratified folds choose k, and the scaling is refit on each fold’s training piece so that the held-out fold does not help to set the means and standard deviations. The objects train, test, and iris_scaled are that split: scale_train standardises its second argument with the means and standard deviations of its first, and iris_scaled was built from the full training table alone. The final test evaluation uses the chosen k and is not allowed to change it. The first minimum is the tie rule, so a later k with the same cross-validation error is not selected.

fold <- stratified_folds(train$Species, k = 5, seed = 9006)
ks <- seq(1, 21, by = 2)
cv_error <- vapply(ks, function(k) {
  pred <- character(nrow(train))
  for (f in 1:5) {
    a <- fold != f; b <- !a
    zz <- scale_train(train[a, 1:4], train[b, 1:4])
    pred[b] <- as.character(class::knn(
      zz$train, zz$test, train$Species[a], k = k))
  }
  mean(pred != train$Species)
}, numeric(1))
chosen_k <- ks[which.min(cv_error)] # first minimum is the fixed tie rule
data.frame(k = ks, CV_error = cv_error)
#>      k CV_error
#> 1    1  0.03810
#> 2    3  0.02857
#> 3    5  0.03810
#> 4    7  0.04762
#> 5    9  0.02857
#> 6   11  0.04762
#> 7   13  0.05714
#> 8   15  0.03810
#> 9   17  0.03810
#> 10  19  0.04762
#> 11  21  0.05714
final_pred <- class::knn(iris_scaled$train, iris_scaled$test,
                          train$Species, k = chosen_k)
c(chosen_k = chosen_k, test_error = mean(final_pred != test$Species))
#> chosen_k test_error
#>  3.00000    0.08889

The smallest cross-validation error in the table is 0.02857, attained at k=3 and again at k=9. The first-minimum rule returns k=3. That 0.02857 is an optimistic estimate of the selected procedure, because the same folds chose k. That is nested cross-validation in miniature: selection and assessment have to use different cases. The untouched test flowers supply the separate assessment, and the test error at the frozen k=3 is 0.08889. The same held-out flowers are reused, on later pages, to illustrate one method after another. Reusing them that way does not make a competition in which the best observed test score can be declared the winner.

Key idea

Estimate the error rate of the classifier you intend to use, on data that played no part in building or selecting it. When data are scarce, cross-validation lets every observation serve in both roles, but never at the same time, and any model selection must be repeated inside each training split.

Used in

See also

Fixed-rule error · Confusion matrix · Apparent error · Leave-one-out cross-validation · -fold cross-validation · Bootstrap error and the correction · Jackknife bias and standard error · A held-out test set

Glossary and index