Laboratory: prune using training cross-validation

Section of Classification trees, part of Statistical learning

Type	Section
Taught in	Classification trees
Part	Statistical learning

The laboratory grows a large tree on the training flowers and then uses rpart’s cross-validation table, computed from those training flowers alone, to prune it. The one-standard-error rule takes the smallest tree whose cross-validated error is within one reported standard error of the minimum. It is a heuristic aimed at stability. It is not a confidence interval for the risk, and the test flowers are not used to choose the penalty.

set.seed(12006)
tree <- rpart::rpart(Species ~ ., data = train, method = "class",
                     control = rpart::rpart.control(cp = 0.001, xval = 10))
cp <- tree$cptable
j <- which.min(cp[, "xerror"])
cutoff <- cp[j, "xerror"] + cp[j, "xstd"]
chosen <- which(cp[, "xerror"] <= cutoff)[1]
small_tree <- rpart::prune(tree, cp = cp[chosen, "CP"])
print(cp)
#>       CP nsplit rel error  xerror    xstd
#> 1 0.5000      0   1.00000 1.17143 0.06054
#> 2 0.4714      1   0.50000 0.75714 0.07319
#> 3 0.0010      2   0.02857 0.07143 0.03117
pred <- predict(small_tree, test, type = "class")
c(number_of_splits = sum(small_tree$frame$var != "<leaf>"),
  test_error = mean(pred != test$Species))
#> number_of_splits   test_error
#>           2.0000       0.1111

The minimum cross-validated error is 0.07143, on the row with two splits. One reported standard error above that minimum is 0.07143+0.03117=0.1026. The cross-validated errors on the rows with zero and one split are 1.17143 and 0.75714, both above 0.1026, so the first row that meets the cutoff is the two-split tree, with complexity parameter 0.001. The complexity table is ordered from larger penalties to smaller ones, which is why the first row under the cutoff is the simpler tree among those that qualify. The pruned tree has two splits and test error 0.1111. The split thresholds are functions of this training sample. They move if the sample moves, and they are not botanical constants.

Key idea

A tree is a sequence of locally chosen partitions. Axis-aligned splits make the decisions readable, and they limit the shapes that can be represented efficiently. Pure leaves give zero training error only when purity is attainable and the growth rules allow it. Assess the pruned procedure on held-out observations, rather than treating training purity as the objective.

Used in

Taught in Classification trees
All 7 questions on Classification trees

Used in

See also