Tune a radial kernel inside the training data

Section of Support vector machines, part of Statistical learning

Tune a radial kernel inside the training data

Type Section
Taught in Support vector machines
Part Statistical learning

Cross a small grid of costs and kernel widths. Standardisation is recomputed in each fold and internal scaling is disabled so the transformation is explicit. The held-out fold is used only to score a candidate (C,\gamma). The labels that assess a setting are not the labels that train it. The helpers draw five stratified folds with seed 17006 and scale each test fold with the training-fold means and standard deviations. The objects train, test and iris_scaled are the same iris split as on the networks page.

fold <- stratified_folds(train$Species, 5, 17006)
grid <- expand.grid(cost = c(0.1, 1, 10), gamma = c(0.1, 0.5, 2))
grid$CV_error <- apply(grid, 1, function(pars) {
  pred <- character(nrow(train))
  for (f in 1:5) {
    a <- fold != f
    zz <- scale_train(train[a, 1:4], train[!a, 1:4])
    fit <- e1071::svm(zz$train, train$Species[a], kernel = "radial",
                      cost = pars[1], gamma = pars[2], scale = FALSE)
    pred[!a] <- as.character(predict(fit, zz$test))
  }
  mean(pred != train$Species)
})
print(grid)
#>    cost gamma CV_error
#> 1   0.1   0.1  0.15238
#> 2   1.0   0.1  0.03810
#> 3  10.0   0.1  0.04762
#> 4   0.1   0.5  0.07619
#> 5   1.0   0.5  0.02857
#> 6  10.0   0.5  0.02857
#> 7   0.1   2.0  0.07619
#> 8   1.0   2.0  0.04762
#> 9  10.0   2.0  0.06667
best <- grid[which.min(grid$CV_error), ]
svm_fit <- e1071::svm(iris_scaled$train, train$Species,
  kernel = "radial", cost = best$cost, gamma = best$gamma, scale = FALSE)
mean(predict(svm_fit, iris_scaled$test) != test$Species)
#> [1] 0.06667

The grid is intentionally modest. Two cells tie at cross-validated error 0.02857; which.min keeps the first, cost 1 and \gamma=0.5. The tuning score estimates a procedure trained on less data than the final fit, and minimising it adds selection optimism. The reported test error of the refit is 0.06667. A kernel decision score is not automatically a calibrated probability.

Key idea

The support vector machine chooses the separating hyperplane with the largest margin 2/\lVert w\rVert. The solution is a combination \sum_i\alpha_i y_i x_i of the training points whose multipliers are nonzero, found from a convex quadratic programme that involves the data only through inner products. Replacing those inner products by a kernel K(x,x') gives nonlinear classifiers in the original coordinates. The constant C prices margin violations against the squared norm.

Used in

See also

Signed distance · Vapnik’s radius–margin bound · Hard-margin primal · Hard-margin dual · Soft-margin primal · Kernel · Linear separators and the margin · Distance to a hyperplane

Glossary and index