A tempting wrong turn

Topic: Support vector machines · more questions on this topic
‹ Interpreting multipliers Read the geometry ›

A practitioner fits a Gaussian-kernel machine with a very large \gamma and C and reports 100\% training accuracy, concluding the problem is solved. Explain what has probably happened and how to check it.

Worked answer (click to show)

With distinct inputs and \gamma=1/(2\sigma^2) very large, off-diagonal Gaussian-kernel entries approach zero. The mapped observations become nearly orthogonal, making interpolation possible at a sufficiently large cost parameter. Finite C penalises slack. It does not forbid it. Away from the training inputs, the narrow kernel contributions fade and the prediction approaches the sign of the intercept. This explains why perfect training accuracy can coexist with poor generalisation. It does not prove that it must do so on every distribution. Tune (C,\gamma) inside the training data and assess the selected procedure independently.