Which split is better?
A node has 400 observations of each of two classes. Split A gives children with class counts (300,100) and (100,300). Split B gives (200,400) and (200,0). Show that the misclassification rate cannot distinguish them, and that the Gini index and the cross-entropy both prefer split B. Why is the pure child valuable?
The parent is (400,400), with misclassification 0.5, Gini index 0.5, and cross-entropy \log 2=0.693.
Misclassification. Split A: both children have error 100/400=0.25, so the weighted impurity is 0.25 and \Delta I=0.25. Split B: the child (200,400) has error 200/600=1/3 and weight 0.75; the child (200,0) has error 0; the weighted impurity is 0.25 and \Delta I=0.25. The two splits agree.
Gini. Split A: each child has 2(0.75)(0.25)=0.375, so \Delta I=0.5-0.375=0.125. Split B:
0.75\cdot 2\cdot\tfrac13\cdot\tfrac23=0.75\cdot\tfrac49=0.333, \qquad \Delta I=0.5-0.333=0.1667.
Cross-entropy. Split A: each child has -(0.75\log 0.75+0.25\log 0.25)=0.562, so \Delta I=0.693-0.562=0.131. Split B: the impure child has entropy 0.6365, the weighted impurity is 0.75\times 0.6365=0.477, and \Delta I=0.693-0.477=0.216. Both smooth criteria prefer B.
A pure child is a finished leaf that classifies perfectly, while the remaining child (200,400) can still be split further. Split A leaves two moderately impure nodes, and the misclassification rate does not register that B has made progress. The strict concavity of the Gini index and of entropy rewards that purity.