Why an ensemble might improve a tree

Topic: Ensembles, random forests, and boosting · more questions on this topic
‹ A tempting wrong turn

You compare one deep classification tree with a random forest built from many bootstrap samples.

  1. Explain how a random forest produces a prediction and why predictor subsampling can help.

  2. Explain what out-of-bag error estimates and one limitation if you use it repeatedly to select settings.

  3. Contrast the main training idea of boosting with bagging.

  1. Fit a tree to each bootstrap sample, considering a random subset of predictors at each split, and aggregate class votes or probabilities. Random predictor subsets reduce dependence between trees. Averaging or voting can reduce variance when the individual predictors are useful and imperfectly correlated. More trees do not remove systematic bias.

  2. For each observation, use predictions only from trees whose bootstrap sample omitted it, then aggregate the loss. This imitates held-out prediction for that forest construction. Repeatedly selecting settings by the same out-of-bag score can introduce selection optimism, so an independent assessment is still valuable.

  3. Bagging fits base learners to perturbed datasets without sequentially targeting the preceding fit’s errors. Boosting adds learners sequentially to reduce a specified loss, often by fitting reweighted errors or negative gradients. It is not another name for a vote over bootstrap trees.

Worked answers for neural units, backpropagation, networks in practice, support vector machines, high-dimensional analysis, and graphical models.