Laboratory: a forest and its out-of-bag observations

Section of Ensembles, random forests, and boosting, part of Statistical learning

Type	Section
Taught in	Ensembles, random forests, and boosting
Part	Statistical learning

The forest below is one recipe: 500 trees, two predictors offered at each split, grown on the training flowers. Each bootstrap tree omits some of those flowers. The out-of-bag prediction for a flower averages only the trees that did not train on it. The implementation draws the candidate predictors at each split. The test flowers are not used to choose T or k. With four iris measurements, k=2 is the classification default \sqrt{4}.

set.seed(13006)
forest <- randomForest::randomForest(
  Species ~ ., data = train, ntree = 500, mtry = 2, importance = TRUE)
pred <- predict(forest, test)
c(final_OOB_error = tail(forest$err.rate[, "OOB"], 1),
  test_error = mean(pred != test$Species))
#> final_OOB_error   test_error
#>        0.009524       0.088889
round(randomForest::importance(forest, type = 1), 3)
#>              MeanDecreaseAccuracy
#> Sepal.Length                9.695
#> Sepal.Width                 3.914
#> Petal.Length               34.745
#> Petal.Width                32.222

The final out-of-bag error on the training flowers is 0.009524, and the test error is 0.088889. Out-of-bag and test errors score different observations using different collections of trees, so they need not agree. The displayed importance(type = 1) values use the implementation’s default standardisation of permutation importance. They are not raw percentage-point increases in error. Petal length and petal width dominate this fit, at 34.745 and 32.222, against 9.695 for sepal length and 3.914 for sepal width. The perturbation measures predictive dependence within this fit, and correlated variables can mask or share importance. It does not identify causal effects.

Key idea

Averaging reduces variance only when the members disagree and are each reasonably accurate: the squared-error gap is their sample variance, and the variance of the average has floor \rho\sigma^2. A forest manufactures that disagreement with a bootstrap sample and a random set of k predictors at each split. AdaBoost manufactures it by reweighting, with \alpha_t=\frac12\log((1-\varepsilon_t)/\varepsilon_t), so that the learner just fitted has weighted error 1/2 on the next round’s problem.

Laboratory: a forest and its out-of-bag observations

Used in

See also