Random-forest tuning parameters
Section of Ensembles, random forests, and boosting, part of Statistical learning
Random-forest tuning parameters
| Type | Section |
| Taught in | Ensembles, random forests, and boosting |
| Part | Statistical learning |
Three quantities are still free: k, the number of predictors offered at each split; a control on tree size, such as a maximum depth or a minimum leaf size; and T, the number of trees. Rules of thumb often suffice. A search has to stay off the test set. Nested cross-validation puts the search inside the assessment. A simpler design uses one cross-validation to choose the triple and a separate test set to estimate the error of the chosen forest.
The variance formula says the second term falls as T grows and the correlation floor does not, so extra trees help until that floor dominates. They do not start to over-fit in the way an extra leaf of a single tree does. Computation grows linearly in T, both to train and to predict. The default in the randomForest implementation is 500 trees. Searching over T is rarely worth the fits. Computing time is the practical limit.
The variance formula wants \rho small, and a single tree wants enough variables to be accurate. So k has to be large enough for each tree to have a tolerable error and small enough for the trees to disagree. The usual suggestions are k=\sqrt{p} for classification and k=p/3 for regression, the defaults in randomForest.
A maximum depth is one control on tree size, and trees are often asymmetric, so a depth limit cuts some branches that still have cases to split and spares others that do not. The minimum number of observations in a leaf is the control in that implementation, with default 1 for classification and 5 for regression. The classification default places no limit on size: a leaf may be a single case. A maximum number of leaves is a related control and has no default.
The defaults tend to produce large trees, each fitted on its random variable sets, diverse because of the bootstrap and the restriction on k, and individually accurate enough that the first term of the diversity identity stays tolerable. Whether those defaults are enough depends on the problem: the complexity of the target, including the Bayes rule, the number of predictors, their associations with the response and with each other, and the sample size. A search over k and the leaf size is sometimes needed. The number of trees can stay at the default while that search is done.
Used in
See also
Squared error of an average · Random forest · AdaBoost · The reweighted error is one half · Why averaging helps · Ensemble error decomposition · Where disagreement comes from · Bagging and random forests