A tempting wrong turn

Topic: Classification trees · more questions on this topic
‹ Penalised impurity Decision trees and validation ›

A fully grown tree has apparent error rate 0. A colleague concludes that the classifier is perfect. Explain what is wrong and what you would report instead.

Worked answer (click to show)

An unrestricted tree can isolate distinct training inputs, so perfect training accuracy is possible. Conflicting labels at identical inputs prevent perfect purity. Practical limits on depth, on leaf size, and on splits can prevent it too. None of these training outcomes measures accuracy on new data. Select the tree’s complexity within the training and validation data, then report held-out or outer cross-validation performance. A binomial interval is appropriate for independent test outcomes conditional on a fixed fitted rule. It is not automatically valid for pooled cross-validation errors. Out-of-bag assessment requires a bootstrap construction such as a forest.