Variable importance
Topic of Ensembles, random forests, and boosting, part of Statistical learning
Variable importance
| Type | Topic |
| Taught in | Ensembles, random forests, and boosting |
| Part | Statistical learning |
A random-forest summary of how much each predictor contributes, such as the rise in out-of-bag error when that predictor’s values are randomly permuted. It describes the fitted forest’s use of a variable, not a causal effect.
In Ensembles, random forests, and boosting
An observation is absent from a size-n bootstrap sample with probability (1-1/n)^n\approx 0.368. Its out-of-bag prediction therefore uses about 36.8\% of the trees, not most of them. Aggregate only those trees, then compare the prediction with its label. This estimates performance of the specified fitting recipe, with finite-forest and resampling effects. If out-of-bag scores select among recipes, the selected minimum also contains selection optimism. Use a separate assessment of that selection procedure.
The average that lowers the variance also removes the single readable tree. Individual trees can still be inspected, and they were built to disagree, so reading one of them does not describe the forest. A partial substitute ranks the predictors. Count how often each variable is chosen for a split, or how much total impurity reduction it contributes, or how much the error rises when its values are permuted and the forest is applied again. The rank is a summary of predictive contribution inside this fit. It is not a ranking of causal effects, and correlated predictors can share or mask one another’s rank. The same two devices, random subsets of predictors and random subsets of cases, can be wrapped around base learners other than trees. Trees remain the usual choice because they are fast, unstable in the way the diversity term requires, and individually limited enough that no one tree dominates the average.
On a fixed iris training set, the out-of-bag error of a forest falls as trees are added and then levels off, separately for setosa, versicolor, virginica, and the overall rate. A permutation-importance bar on the same fit is largest for the two petal measurements. Those curves are one seeded fit, not a sampling distribution.
Used in
See also
Random forest · Out-of-bag estimation and variable importance · Squared error of an average · AdaBoost · The reweighted error is one half · Why averaging helps · Ensemble error decomposition · Where disagreement comes from