Early stopping
Topic of Networks in practice, part of Statistical learning · Also in Networks in practice
Early stopping
| Type | Topic |
| Taught in | Networks in practice · Networks in practice |
| Part | Statistical learning |
| Questions using it | 2 |
Stopping gradient-descent training when the error on a validation set stops falling, before the training error reaches its minimum. It acts as a regulariser, much like weight decay.
In Networks in practice
From Networks in practice
Initialisation and the scale of the inputs decide whether the sigmoid starts on its steep part, where the chain rule has a slope to multiply. Validation, early stopping, and weight decay decide how far that slope is followed. Architecture, decay, and the stopping rule are chosen on training and validation data. The assessment data are reserved for the procedure once those choices are frozen.
In Networks in practice, continued
From Networks in practice
Minimising the error on the training set is a strategy for obtaining good performance on new data. It does not always work, for two reasons that show up as different shapes of the training loss.
Overfitting. The model is a very good fit to the training data. It may have modelled some useful structure and has also modelled many irrelevant aspects. This is common with overly complex models.
Underfitting. The model is not a good fit to the training data. The size of E is a good indication. The model may be too simple, there may not be enough data, or the data may be so noisy that the output can never be predicted very accurately.
A common pattern is that training loss keeps falling after validation loss begins to rise. Early stopping selects a point along the optimisation path before the training objective is fully minimised. It restricts the fit indirectly. It is not generally equivalent to a specified L^2 penalty, and memorisation does not require every weight to be large. Its effect depends on the architecture, the initialisation, and the optimiser. More observations can help, although their usefulness depends on how they cover the prediction task.
The curve used for stopping is a validation curve. A test curve used to stop training has been spent as a tuning device, and it can no longer serve as an untouched assessment.
An explicit simulation with 20 observations and a 1–40–1 sigmoid network makes the pattern visible. Training runs for 30{,}000 epochs. Weight decay is 0 or 0.01. The training mean squared error keeps moving after the expected noisy-response error, averaged over a fixed uniform predictor grid, has stopped improving. The decay run stays closer to the known signal than the undecayed run.
Used in
- Taught in Networks in practice · Networks in practice
- Also used in Classification trees
- Questions that use it (2) — Reading the gap statistic (answer) · Sum of squares versus accuracy
- All 7 questions on Networks in practice
See also
Gradient descent · Validation set · Regularisation and weight decay · Overfitting · Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation