Generalisation of a network

Section of Networks in practice, part of Statistical learning

Type	Section
Taught in	Networks in practice
Part	Statistical learning

Minimising the error on the training set is a strategy for obtaining good performance on new data. It does not always work, for two reasons that show up as different shapes of the training loss.

Overfitting. The model is a very good fit to the training data. It may have modelled some useful structure and has also modelled many irrelevant aspects. This is common with overly complex models.

Underfitting. The model is not a good fit to the training data. The size of E is a good indication. The model may be too simple, there may not be enough data, or the data may be so noisy that the output can never be predicted very accurately.

A common pattern is that training loss keeps falling after validation loss begins to rise. Early stopping selects a point along the optimisation path before the training objective is fully minimised. It restricts the fit indirectly. It is not generally equivalent to a specified L^2 penalty, and memorisation does not require every weight to be large. Its effect depends on the architecture, the initialisation, and the optimiser. More observations can help, although their usefulness depends on how they cover the prediction task.

The curve used for stopping is a validation curve. A test curve used to stop training has been spent as a tuning device, and it can no longer serve as an untouched assessment.

An explicit simulation with 20 observations and a 1–40–1 sigmoid network makes the pattern visible. Training runs for 30{,}000 epochs. Weight decay is 0 or 0.01. The training mean squared error keeps moving after the expected noisy-response error, averaged over a fixed uniform predictor grid, has stopped improving. The decay run stays closer to the known signal than the undecayed run.

Used in

Taught in Networks in practice
All 7 questions on Networks in practice

Used in

See also