Ensemble error decomposition
Section of Ensembles, random forests, and boosting, part of Statistical learning
Ensemble error decomposition
| Type | Section |
| Taught in | Ensembles, random forests, and boosting |
| Part | Statistical learning |
Classification error is an indicator, and an average of indicators is not itself the indicator of a majority vote, so the clean algebra starts with a quantitative response. Suppose T models predict a number, with functions f_1,\ldots,f_T, and the ensemble predicts their average
f(x)=\frac1T\sum_{t=1}^T f_t(x).
Theorem 1 (Squared error of an average) If the response at this x is y, the squared error of the average splits into the average squared error of the members minus the sample variance of the members about their average:
\bigl(f(x)-y\bigr)^2 =\frac1T\sum_{t=1}^T\bigl(f_t(x)-y\bigr)^2 -\frac1T\sum_{t=1}^T\bigl(f_t(x)-f(x)\bigr)^2.
Write f_t-y=(f_t-f)+(f-y) and square:
(f_t-y)^2=(f_t-f)^2+2(f_t-f)(f-y)+(f-y)^2.
Average over t. The cross term vanishes because \frac1T\sum_t(f_t-f)=0 by definition of the average, and what remains is
\frac1T\sum_{t=1}^T(f_t-y)^2 =\frac1T\sum_{t=1}^T(f_t-f)^2+(f-y)^2,
which rearranges to the display. All three terms are non-negative. The squared error of the ensemble is therefore at most the average squared error of its members, and the gap is the diversity, the sample variance of the members about their average. The algebra used only the definition of the average and the expansion of a square. No model for y was required.
The design requirement is that the members be reasonably accurate, so that the first term is tolerable, and that they disagree, so that the diversity term is large. If every member were already the regression function, the diversity term would be zero and averaging would change nothing. The identity also applies componentwise to averaged class probabilities under squared error. It does not give the same guarantee for the zero–one loss after a majority vote, and it compares the ensemble with the average member, not with the best member. Diversity and individual accuracy cannot be varied independently without changing the first term. Adding noise merely to make predictors disagree need not help.
Used in
See also
Squared error of an average · Random forest · AdaBoost · The reweighted error is one half · Why averaging helps · Where disagreement comes from · Bagging and random forests · Why decorrelating the trees helps