Why the raw sum of squares cannot choose
Remark taught in K-means clustering, part of Statistical learning
Why the raw sum of squares cannot choose
| Type | Remark |
| Taught in | K-means clustering |
| Part | Statistical learning |
W, in either the pairwise or the unweighted version, is non-increasing in the number of clusters and reaches 0 at k=n, so minimising it over k always selects the largest k. The gap statistic corrects for this by comparing \log W_{\mathrm{data}}^{(k)} with what is expected for data without cluster structure, which also falls as k grows.
Used in
- Taught in K-means clustering, where the derivation and examples are.
- All 8 questions on K-means clustering
See also
Choosing K: the gap statistic · Lloyd’s K-means algorithm · Descent of Lloyd’s algorithm · K-means criterion · Within-cluster scatter · From pairs to cluster means · Why the partitions are not all searched · Initialising k-means