Node impurity
Definition taught in Classification trees, part of Statistical learning
Node impurity
| Type | Definition |
| Taught in | Classification trees |
| Part | Statistical learning |
| Questions using it | 2 |
For G classes, the impurity of node m is one of
\begin{align*} I_C(m)&=-\sum_{g=1}^G \hat p_{mg}\log\hat p_{mg}, &&\text{cross-entropy},\\ I_G(m)&=\sum_{g=1}^G \hat p_{mg}(1-\hat p_{mg}) =1-\sum_{g=1}^G \hat p_{mg}^2, &&\text{Gini index},\\ I_M(m)&=1-\hat p_{mg^*}, &&\text{misclassification rate}, \end{align*}
where g^*=\operatorname{arg\,max}_g\hat p_{mg}. In the cross-entropy, a term with \hat p_{mg}=0 is taken to be 0, the limiting value of p\log p as p\to 0. The logarithm may be in any fixed base. The worked arithmetic below uses the natural logarithm. A change of base multiplies every entropy, and every entropy reduction, by the same positive constant, so it does not change which split wins. A plot drawn in bits uses base 2.
Used in
- Taught in Classification trees, where the derivation and examples are.
- Also used in Ensembles, random forests, and boosting
- Questions that use it (2) — Which split is better? (answer) · Penalised impurity
- All 7 questions on Classification trees
See also
Cross-entropy · Cost-complexity pruning · Gini index · Building a classification tree · Choosing a split · Invariance to a change of units · Multi-way splits and missing values · Tree size and pruning