Radial basis function kernel

Topic of Support vector machines, part of Statistical learning · Also in Support vector machines

Type	Topic
Taught in	Support vector machines · Support vector machines
Part	Statistical learning

k(\mathbf{x},\mathbf{x}')=\exp\{-\|\mathbf{x}-\mathbf{x}'\|^2/(2\sigma^2)\}, the default kernel of a support vector machine. Its feature space is infinite-dimensional. The width \sigma (or \gamma=1/(2\sigma^2)) sets how local the boundary is and is tuned inside the training data.

In Support vector machines

From Support vector machines

Linear. K(x_i,x_j)=x_i^\top x_j, with \phi(x)=x itself.
Polynomial of power d. K(x_i,x_j)=(1+x_i^\top x_j)^d, with \phi(x) of dimension \binom{p+d}{d}. For p=2 and d=2 that dimension is \binom{4}{2}=6, as in the expansion above. The higher-dimensional space still has intrinsic dimension p, because the map is not onto. Linear separators in it correspond to nonlinear separators in the original space.
Gaussian, or radial basis function. K(x_i,x_j)=\exp\bigl(-\lVert x_i-x_j\rVert^2/(2\sigma^2)\bigr), with \phi(x) infinite-dimensional. Every point is mapped to a function, a Gaussian, and a combination of such functions for the support vectors is the separator. The Taylor expansion shows the infinite dimension:

\begin{aligned} e^{-\lVert x-x'\rVert^2/(2\sigma^2)} &=e^{-\lVert x\rVert^2/(2\sigma^2)}\,e^{-\lVert x'\rVert^2/(2\sigma^2)}\,e^{x^\top x'/\sigma^2}\\ &=e^{-\lVert x\rVert^2/(2\sigma^2)}\,e^{-\lVert x'\rVert^2/(2\sigma^2)} \sum_{n=0}^{\infty}\frac{(x^\top x')^n}{\sigma^{2n}\,n!}. \end{aligned}

This is a positive combination of the polynomial kernels (x^\top x')^n of every degree n, times a(x)a(x'), where a(x)=\exp\bigl[-\lVert x\rVert^2/(2\sigma^2)\bigr]. The paired product is a rank-one kernel. Either one-variable factor by itself is not a two-input kernel.

In Support vector machines, continued

From Support vector machines

Cross a small grid of costs and kernel widths. Standardisation is recomputed in each fold and internal scaling is disabled so the transformation is explicit. The held-out fold is used only to score a candidate (C,\gamma). The labels that assess a setting are not the labels that train it. The helpers draw five stratified folds with seed 17006 and scale each test fold with the training-fold means and standard deviations. The objects train, test and iris_scaled are the same iris split as on the networks page.

fold <- stratified_folds(train$Species, 5, 17006)
grid <- expand.grid(cost = c(0.1, 1, 10), gamma = c(0.1, 0.5, 2))
grid$CV_error <- apply(grid, 1, function(pars) {
  pred <- character(nrow(train))
  for (f in 1:5) {
    a <- fold != f
    zz <- scale_train(train[a, 1:4], train[!a, 1:4])
    fit <- e1071::svm(zz$train, train$Species[a], kernel = "radial",
                      cost = pars[1], gamma = pars[2], scale = FALSE)
    pred[!a] <- as.character(predict(fit, zz$test))
  }
  mean(pred != train$Species)
})
print(grid)
#>    cost gamma CV_error
#> 1   0.1   0.1  0.15238
#> 2   1.0   0.1  0.03810
#> 3  10.0   0.1  0.04762
#> 4   0.1   0.5  0.07619
#> 5   1.0   0.5  0.02857
#> 6  10.0   0.5  0.02857
#> 7   0.1   2.0  0.07619
#> 8   1.0   2.0  0.04762
#> 9  10.0   2.0  0.06667
best <- grid[which.min(grid$CV_error), ]
svm_fit <- e1071::svm(iris_scaled$train, train$Species,
  kernel = "radial", cost = best$cost, gamma = best$gamma, scale = FALSE)
mean(predict(svm_fit, iris_scaled$test) != test$Species)
#> [1] 0.06667

The grid is intentionally modest. Two cells tie at cross-validated error 0.02857; which.min keeps the first, cost 1 and \gamma=0.5. The tuning score estimates a procedure trained on less data than the final fit, and minimising it adds selection optimism. The reported test error of the refit is 0.06667. A kernel decision score is not automatically a calibrated probability.

Key idea

The support vector machine chooses the separating hyperplane with the largest margin 2/\lVert w\rVert. The solution is a combination \sum_i\alpha_i y_i x_i of the training points whose multipliers are nonzero, found from a convex quadratic programme that involves the data only through inner products. Replacing those inner products by a kernel K(x,x') gives nonlinear classifiers in the original coordinates. The constant C prices margin violations against the squared norm.

Used in

Taught in Support vector machines · Support vector machines
All 10 questions on Support vector machines

In Support vector machines

In Support vector machines, continued

Used in

See also