Choosing the bandwidth
Section of Hierarchical clustering and kernel density estimation, part of Statistical learning
Choosing the bandwidth
| Type | Section |
| Taught in | Hierarchical clustering and kernel density estimation |
| Part | Statistical learning |
| Questions using it | 3 |
The matrix H may be constrained to be diagonal, or left unconstrained. Either way the target is approximation of the density f that generated the sample. There is no unique choice, and the multivariate problem is not a coordinate-wise copy of the univariate one. Likelihood cross-validation, as on Model selection, fits the estimate on one part of the sample and scores the log-likelihood on the left-out part. The bandwidth with the highest held-out score is preferred. With a short sample the split is repeated over folds.
A standard distance between f and \widehat f, different from Kullback–Leibler divergence, is the mean integrated squared error,
\mathrm{MISE}(H) =\mathbb{E}\int_{\mathbb{R}^p}\bigl(\widehat f(\mathbf{x};H)-f(\mathbf{x})\bigr)^2\,d\mathbf{x},
regarded as a function of H. The optimal bandwidth minimises MISE over symmetric positive definite p\times p matrices. The expectation is over the sample that builds \widehat f. MISE can be estimated in several ways, including cross-validation. Most methods optimise H numerically. A full matrix has p(p+1)/2 free entries, which is already a difficult optimisation at moderate p. Large n adds a computational cost. Likelihood cross-validation, or a single validation set, is one practical selector. In high dimension one usually reduces the coordinates first, and that reduction discards information.
Reducing dimension before density estimation can make the smoothing problem manageable, and it changes the target to the projected distribution. Discarded directions cannot be reconstructed from the density of the retained scores alone. There is no universal dimension, such as six, at which estimation becomes impossible: difficulty depends on sample size, smoothness, and effective dimension. The software’s supported dimension is a separate constraint.
Without a smoothness restriction, many irregular densities are compatible with the same finite sample, and the sample does not choose among them. Smoothness is the assumption that lets the estimate take values between the observed points.
Used in
- Taught in Hierarchical clustering and kernel density estimation
- Also used in Descriptive statistics · Statistical classifiers and discriminant analysis · Neural units
- Questions that use it (3) — Kernel density by hand · A kernel is a density (answer) · Clustering and smoothing choices
- All 6 questions on Hierarchical clustering and kernel density estimation
See also
Epanechnikov kernel · Kernel density estimator · Single, complete, and average linkage · Hierarchical clustering · Agglomerative and divisive clustering · Properties of the linkages · Divisive clustering · Kernel density estimation