A tempting wrong turn

Topic: Hierarchical clustering and kernel density estimation · more questions on this topic
‹ A kernel is a density Clustering and smoothing choices ›

A student selects h by maximising \sum_i\log\widehat f(x_i;h) over h, where \widehat f is computed from all the data, including x_i. What happens, and how would you modify the criterion?

Worked answer (click to show)

At a training observation, the density includes a kernel centred exactly on that observation. For a univariate kernel scaled as K_h(u)=h^{-1}K(u/h), that self-contribution is (1/n)K(0)/h, which diverges as h\to 0. The training criterion increases without bound as h\to 0 and selects a spiky estimate that reproduces the sample, the same singularity as a mixture component collapsing on a data point.

The criterion should evaluate each point under a density that does not use it: the leave-one-out log-likelihood

\sum_i\log\widehat f_{-i}(x_i;h), \qquad \widehat f_{-i}(x_i;h)=\frac{1}{n-1}\sum_{l\ne i}K_h(x_i-x_l),

or the likelihood on a held-out validation set.