Three impurities
A node contains 6, 2, and 2 observations of classes 1, 2, and 3. Compute its cross-entropy, Gini index, and misclassification rate. Show that \sum_g\hat p_g(1-\hat p_g)=1-\sum_g\hat p_g^2.
Worked answer (click to show)
The proportions are \hat p=(0.6,0.2,0.2). Cross-entropy, with the natural logarithm and the convention 0\log 0=0, is
-(0.6\log 0.6+2\cdot 0.2\log 0.2)=0.3065+0.6438=0.950.
The Gini index is
0.6(0.4)+2(0.2)(0.8)=0.24+0.32=0.56.
The misclassification rate is 1-0.6=0.4. For the identity,
\sum_g\hat p_g(1-\hat p_g) =\sum_g\hat p_g-\sum_g\hat p_g^2 =1-\sum_g\hat p_g^2,
since the proportions sum to 1. The check is 1-(0.36+0.04+0.04)=0.56.