Where disagreement comes from
Section of Ensembles, random forests, and boosting, part of Statistical learning
Where disagreement comes from
| Type | Section |
| Taught in | Ensembles, random forests, and boosting |
| Part | Statistical learning |
Separate variation caused by the dataset from variation caused by the fitting algorithm. A deterministic tree can be sensitive to a small data change. A neural network can also change because of its initial weights. Convexity addresses the optimisation problem, not sensitivity to data: a convex fit can still be unstable when the design is poorly conditioned. Evaluate stability in the setting at hand.
Two kinds of base learner do supply disagreement. One has many parameters relative to the sample and is hard to optimise, so different runs stop in different places. The other is constrained in size or structure and cannot fit the sample in only one way, so a small change in the data produces a different constrained fit. Trees are in the second group. Networks are in the first.
An average is already an ensemble in a setting that has nothing to do with classifiers. A signal from a distant spacecraft is received together with noise. Sending the same signal repeatedly and averaging the receptions lets independent noise cancel at the rate the central limit theorem describes, and leaves the signal. Each noisy reception is the analogue of one flawed classifier. The next two sections build the average in different ways: random forests, which average trees that were made to disagree, and boosting, which adds a learner aimed at the residual of the ones already fitted.
Used in
See also
Squared error of an average · Random forest · AdaBoost · The reweighted error is one half · Why averaging helps · Ensemble error decomposition · Bagging and random forests · Why decorrelating the trees helps