Common kernels
Section of Support vector machines, part of Statistical learning
Common kernels
| Type | Section |
| Taught in | Support vector machines |
| Part | Statistical learning |
- Linear. K(x_i,x_j)=x_i^\top x_j, with \phi(x)=x itself.
- Polynomial of power d. K(x_i,x_j)=(1+x_i^\top x_j)^d, with \phi(x) of dimension \binom{p+d}{d}. For p=2 and d=2 that dimension is \binom{4}{2}=6, as in the expansion above. The higher-dimensional space still has intrinsic dimension p, because the map is not onto. Linear separators in it correspond to nonlinear separators in the original space.
- Gaussian, or radial basis function. K(x_i,x_j)=\exp\bigl(-\lVert x_i-x_j\rVert^2/(2\sigma^2)\bigr), with \phi(x) infinite-dimensional. Every point is mapped to a function, a Gaussian, and a combination of such functions for the support vectors is the separator. The Taylor expansion shows the infinite dimension:
\begin{aligned} e^{-\lVert x-x'\rVert^2/(2\sigma^2)} &=e^{-\lVert x\rVert^2/(2\sigma^2)}\,e^{-\lVert x'\rVert^2/(2\sigma^2)}\,e^{x^\top x'/\sigma^2}\\ &=e^{-\lVert x\rVert^2/(2\sigma^2)}\,e^{-\lVert x'\rVert^2/(2\sigma^2)} \sum_{n=0}^{\infty}\frac{(x^\top x')^n}{\sigma^{2n}\,n!}. \end{aligned}
This is a positive combination of the polynomial kernels (x^\top x')^n of every degree n, times a(x)a(x'), where a(x)=\exp\bigl[-\lVert x\rVert^2/(2\sigma^2)\bigr]. The paired product is a rank-one kernel. Either one-variable factor by itself is not a two-input kernel.
Used in
See also
Signed distance · Vapnik’s radius–margin bound · Hard-margin primal · Hard-margin dual · Soft-margin primal · Kernel · Linear separators and the margin · Distance to a hyperplane