Invariance to a change of units
Section of Classification trees, part of Statistical learning
Invariance to a change of units
| Type | Section |
| Taught in | Classification trees |
| Part | Statistical learning |
Nearest neighbours changed their minds when a coordinate was rescaled, because Euclidean distance mixes the units. A tree never mixes units inside a split. Replacing predictor j by ax_j+b with a\ne 0 transforms every candidate cut on that variable and leaves \Delta I unchanged, so the sequence of variables and the order of splits stay the same. If a>0, the observations with x_j<\lambda are exactly the observations with ax_j+b<a\lambda+b. Every split partitions the data identically and has the same child compositions, hence the same \Delta I, at the transformed cut. If a<0, the left and right children swap. The reduction is symmetric in the two children, because replacing \hat p_L by 1-\hat p_L exchanges the two weighted terms, so \Delta I is unchanged. After the transformation the published cut-points transform with the data. The predictors do not need to be standardised before the fit.
A rotation is a different matter. The tree can cut only on coordinate axes, and a rotation moves those axes. A split on a single coordinate after rotation is a split on a linear combination before the rotation, which an axis-aligned tree cannot reproduce. The invariance is to strictly monotone affine changes of single coordinates, not to linear combinations. Training partitions are preserved by those coordinate transformations, up to ties and equivalent cuts. Predictions between observed values can differ when a nonlinear transformation changes the midpoint convention.
Used in
See also
Node impurity · Building a classification tree · Choosing a split · Multi-way splits and missing values · Tree size and pruning · What a fitted tree can be trusted to mean · Laboratory: prune using training cross-validation · Decision tree