Training error of a network
Section of Backpropagation, part of Statistical learning
Training error of a network
| Type | Section |
| Taught in | Backpropagation |
| Part | Statistical learning |
For classification the aim is a low error rate, or a related loss. For regression the aim is a small squared error. The performance measure used with the networks on this page is the sum of squared errors on the training data. Squared error is differentiable. It treats a numeric target the same way whether the target is a measurement or a class code. Any constant multiplier is absorbed into the learning rate, so the fitted direction does not depend on writing \tfrac12 in front of the sum. This page omits that factor. The omission is why a 2 appears in every gradient.
E=\sum_{i=1}^{n_{\mathrm{tr}}}\sum_{k=1}^{m} E_{ik} =\sum_{i=1}^{n_{\mathrm{tr}}}\sum_{k=1}^{m}\bigl(y_{ik}-f_{\mathrm{network},k}(x_i)\bigr)^2 =\sum_{i=1}^{n_{\mathrm{tr}}}\sum_{k=1}^{m}\bigl(y_{ik}-o_{ik}\bigr)^2,
where y_{ik} is the kth correct output for pattern i and o_{ik} is the kth network output. Dividing by m n_{\mathrm{tr}} gives the mean squared error, which does not affect the optimisation. The loss treats an error of a given size on each output as equally important. If that is the wrong weighting, the terms can be scaled by output.
For targets in \{0,1\}, squared error on a single output equals absolute error when the prediction is also in \{0,1\}, because the nonzero difference is 1. The display above is wider than that special case. With a sigmoid the output lies strictly between 0 and 1, and (y_{ik}-o_{ik})^2 is not a count of mistakes.
The free parameters are the weights and biases, and possibly the number of hidden units. The bias weights are connected to every hidden and output unit. Each is a weight from an extra input x_0 that is always 1.
Fix the architecture while differentiating. Its size is a separate choice. The weights form a vector in parameter space, and \nabla E collects one partial derivative per coordinate. In the Euclidean geometry of that space, -\nabla E is the direction of steepest local decrease for a fixed step length. If only one coordinate is changed, move opposite the sign of that coordinate’s derivative.
Used in
See also
Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer · Hidden–output weights