The error decomposition

Topic: Ensembles, random forests, and boosting · more questions on this topic
Variance of an average ›

Verify the squared-error decomposition numerically for member predictions f_t(x)=(1,2,6) and truth y=4. Then derive it in general.

Worked answer (click to show)

Here f=\frac13(1+2+6)=3. The left side is (f-y)^2=(3-4)^2=1. The average member loss is

\frac13\bigl[(1-4)^2+(2-4)^2+(6-4)^2\bigr] =\frac{9+4+4}{3} =\frac{17}{3} =5.667.

The diversity is

\frac13\bigl[(1-3)^2+(2-3)^2+(6-3)^2\bigr] =\frac{4+1+9}{3} =\frac{14}{3} =4.667.

Then 5.667-4.667=1, matching the left side. In general, f_t-y=(f_t-f)+(f-y), so

\frac1T\sum_t(f_t-y)^2 =\frac1T\sum_t(f_t-f)^2 +2(f-y)\frac1T\sum_t(f_t-f) +(f-y)^2.

The cross term vanishes because \frac1T\sum_t(f_t-f)=0, and

(f-y)^2 =\frac1T\sum_t(f_t-y)^2 -\frac1T\sum_t(f_t-f)^2.