Gaussian kernel
Topic of Hierarchical clustering and kernel density estimation, part of Statistical learning · Also in Support vector machines
Gaussian kernel
| Type | Topic |
| Taught in | Hierarchical clustering and kernel density estimation · Support vector machines |
| Part | Statistical learning |
| Questions using it | 5 |
In density estimation, the standard normal density used as the kernel, with support the whole space. In a support vector machine, the radial basis kernel k(\mathbf{x},\mathbf{x}')=\exp\{-\|\mathbf{x}-\mathbf{x}'\|^2/(2\sigma^2)\}.
In Hierarchical clustering and kernel density estimation
The Epanechnikov kernel has bounded support, so a sample point contributes nothing beyond a finite radius. The normal kernel is the usual multivariate default, and its support is the whole space. A full matrix H can follow an elliptical cloud. A diagonal H, or a product of separate univariate smoothers, has axis-aligned contours. On a tilted cloud those contours place part of each kernel’s mass off the data. One H everywhere is a further restriction: a cloud whose shape changes from one region to another is smoothed with the same orientation in every region.
In Support vector machines
- Linear. K(x_i,x_j)=x_i^\top x_j, with \phi(x)=x itself.
- Polynomial of power d. K(x_i,x_j)=(1+x_i^\top x_j)^d, with \phi(x) of dimension \binom{p+d}{d}. For p=2 and d=2 that dimension is \binom{4}{2}=6, as in the expansion above. The higher-dimensional space still has intrinsic dimension p, because the map is not onto. Linear separators in it correspond to nonlinear separators in the original space.
- Gaussian, or radial basis function. K(x_i,x_j)=\exp\bigl(-\lVert x_i-x_j\rVert^2/(2\sigma^2)\bigr), with \phi(x) infinite-dimensional. Every point is mapped to a function, a Gaussian, and a combination of such functions for the support vectors is the separator. The Taylor expansion shows the infinite dimension:
\begin{aligned} e^{-\lVert x-x'\rVert^2/(2\sigma^2)} &=e^{-\lVert x\rVert^2/(2\sigma^2)}\,e^{-\lVert x'\rVert^2/(2\sigma^2)}\,e^{x^\top x'/\sigma^2}\\ &=e^{-\lVert x\rVert^2/(2\sigma^2)}\,e^{-\lVert x'\rVert^2/(2\sigma^2)} \sum_{n=0}^{\infty}\frac{(x^\top x')^n}{\sigma^{2n}\,n!}. \end{aligned}
This is a positive combination of the polynomial kernels (x^\top x')^n of every degree n, times a(x)a(x'), where a(x)=\exp\bigl[-\lVert x\rVert^2/(2\sigma^2)\bigr]. The paired product is a rank-one kernel. Either one-variable factor by itself is not a two-input kernel.
Used in
- Taught in Hierarchical clustering and kernel density estimation · Support vector machines
- Also used in Bayesian linear regression · Model selection for stochastic models · Support vector machines
- Questions that use it (5) — Kernel density by hand · A Gram matrix · Q9 (answer) · Q43 (answer) · Question 5 (answer)
- All 6 questions on Hierarchical clustering and kernel density estimation
See also
Kernel · Kernel density estimator · Kernel of a density · Radial basis function kernel · Single, complete, and average linkage · Hierarchical clustering · Agglomerative and divisive clustering · Properties of the linkages