Hinge loss
Section of Support vector machines, part of Statistical learning
Hinge loss
| Type | Section |
| Taught in | Support vector machines |
| Part | Statistical learning |
At the optimum each \xi_i is as small as the constraints allow, so \xi_i=\max\{0,\,1-y_i f(x_i)\}. Substituting that value removes the slacks and leaves
\sum_{i=1}^{n}\max\{0,\,1-y_i f(x_i)\}+\frac{1}{2C}\lVert w\rVert^2,
a hinge loss plus a ridge penalty on w. Compare weight decay on networks in practice. The hinge is zero once the margin exceeds 1: a point that is correctly classified and far from the boundary does not pull the fit. Squared error would keep pulling. The binomial deviance penalises errors roughly linearly, as the hinge does, and it never becomes exactly zero. The factor 1/(2C) plays the role of the regularisation parameter \lambda in weight decay: small C is strong regularisation.
Used in
See also
Signed distance · Vapnik’s radius–margin bound · Hard-margin primal · Hard-margin dual · Soft-margin primal · Kernel · Linear separators and the margin · Distance to a hyperplane