A tempting wrong turn

Topic: Ensembles, random forests, and boosting · more questions on this topic
‹ The exponential-loss minimiser Why an ensemble might improve a tree ›

A practitioner averages 500 copies of the same deterministic classifier trained on the same data, hoping for a variance reduction. What does the squared-error decomposition, or the variance formula for an average, predict? What are the ways a random forest and AdaBoost avoid this?

Worked answer (click to show)

Identical members have f_t=f for every t, so the diversity term is exactly 0 and the ensemble’s squared error equals the average member loss. There is no gain. In the variance formula, \rho=1 gives \operatorname{Var}(\bar f)=\sigma^2. A random forest breaks the correlation by training each tree on a different bootstrap sample and by restricting each split to a random subset of variables. AdaBoost makes each learner different by reweighting the observations, so that every new learner is trained on a problem on which the previous learner has weighted error exactly 1/2.