One round of AdaBoost

Topic: Ensembles, random forests, and boosting · more questions on this topic
‹ Out-of-bag counts The exponential-loss minimiser ›

Five observations x=1,\ldots,5 have labels y=(+1,+1,-1,-1,+1) and weights 0.2 each. The stump predicts +1 for x<2.5 and -1 otherwise. Compute \varepsilon_1, \alpha_1, the updated weights, and verify that the new weighted error is 0.5.

Worked answer (click to show)

The stump predicts (+,+,-,-,-) and the truth is (+,+,-,-,+), so only x=5 is wrong and \varepsilon_1=0.2. Then

\alpha_1=\tfrac12\log(0.8/0.2)=\tfrac12\log 4=0.693,

e^{-\alpha_1}=0.5, and e^{\alpha_1}=2. The un-normalised weights are 0.2\times 0.5=0.1 for each correct point and 0.2\times 2=0.4 for the wrong one. The normaliser is

Z_1=4(0.1)+0.4=0.8=2\sqrt{0.2\times 0.8}.

The normalised weights are 0.125 for x=1,2,3,4 and 0.5 for x=5. The weighted error of the same stump under the new weights is 0.5, all of it from x=5.