Constructing and training a network
Section of Networks in practice, part of Statistical learning
Constructing and training a network
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
To construct a neural network: choose a network topology; determine the number of hidden units, if any; create the network; learn the weight parameters; and evaluate the trained network through testing.
Generalisation
Main article: Generalisation of a network
Minimising the error on the training set is a strategy for obtaining good performance on new data. It does not always work, for two reasons that show up as different shapes of the training loss.
Overfitting. The model is a very good fit to the training data. It may have modelled some useful structure and has also modelled many irrelevant aspects. This is common with overly complex models.
Underfitting. The model is not a good fit to the training data. The size of E is a good indication. The model may be too simple, there may not be enough data, or the data may be so noisy that the output can never be predicted very accurately.
A common pattern is that training loss keeps falling after validation loss begins to rise. Early stopping selects a point along the optimisation path before the training objective is fully minimised. It restricts the fit indirectly. It is not generally equivalent to a specified L^2 penalty, and memorisation does not require every weight to be large. Its effect depends on the architecture, the initialisation, and the optimiser. More observations can help, although their usefulness depends on how they cover the prediction task.
The curve used for stopping is a validation curve. A test curve used to stop training has been spent as a tuning device, and it can no longer serve as an untouched assessment.
An explicit simulation with 20 observations and a 1–40–1 sigmoid network makes the pattern visible. Training runs for 30{,}000 epochs. Weight decay is 0 or 0.01. The training mean squared error keeps moving after the expected noisy-response error, averaged over a fixed uniform predictor grid, has stopped improving. The decay run stays closer to the known signal than the undecayed run.
Training, validation, and test sets
Main article: Training, validation, and test sets
To control generalisation the data are divided into a training set, a validation set, and a test set.
- The training set is used to optimise the weights.
- The validation set is monitored during or after training to choose when to stop, the number of nodes or weights, the learning rate, and the other tuning choices.
- The test set is used to assess how well the final model performs.
This is the same discipline as nested cross-validation. Anything chosen by looking at data is part of model building and must be assessed on data that played no part. Early stopping uses the validation error: stop at the epoch where it is smallest.
Regularisation and weight decay
Main article: Regularisation and weight decay
A weight penalty changes the objective itself. With L^2 regularisation one charges for large squared weights, just as ridge regression does for linear coefficients. This is a modelling preference for smaller parameter values in the chosen representation. Equivalent reparameterisations need not receive the same penalty. Under ordinary gradient descent, the resulting update contains a literal multiplicative decay. That equivalence should not be transferred automatically to an adaptive optimiser.
L^2 regularisation, also called weight decay, ridge regression (Hoerl and Kennard, 1970), or Tikhonov regularisation, adds a penalty proportional to the sum of squares of the weights. Optimise E+\lambda\lVert w\rVert^2 with \lambda>0 chosen by the user. After rescaling the tuning parameter, write the penalty as E+\tfrac{\lambda}{2}\lVert w\rVert^2. The gradient step becomes
w\leftarrow w-\varepsilon\Bigl(\frac{\partial E}{\partial w}+\lambda w\Bigr) =(1-\varepsilon\lambda)\,w-\varepsilon\frac{\partial E}{\partial w}.
When 0\le\varepsilon\lambda\le 1, the factor 1-\varepsilon\lambda shrinks a weight towards zero without reversing its sign. With no data-gradient term, t steps multiply it by (1-\varepsilon\lambda)^t. For \varepsilon=0.1, \lambda=0.5 and t=10, this factor is 0.95^{10}\approx 0.599. A step size is part of the statement: an excessive value can cause oscillation or divergence even in this simple recurrence. The penalty encourages small weights. It does not have the lasso’s thresholding mechanism. That mechanism is derived on high-dimensional analysis.
L^1 regularisation uses E+\lambda\sum|w|. The derivative at zero is replaced by a subgradient or a proximal update. Such algorithms can produce exact zeros. The non-differentiability changes the optimisation method. It does not make the penalty unusable.
NETtalk
Main article: NETtalk
NETtalk is an early example of learning a mapping from written context to a pronunciation code (Sejnowski and Rosenberg, 1986). A window of letters supplies the input and the central letter supplies the prediction target. The architecture with 203 inputs, 80 hidden units and 26 outputs has
(203+1)\,80+(80+1)\,26=18426
weights including biases. This count illustrates how quickly connections accumulate. Weight-decay experiments such as Krogh and Hertz (1992) motivate regularisation. A successful setting in one experiment is not a transferable value of \lambda.
Used in
See also
Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network · Output coding, softmax, and cross-entropy · Generalisation of a network