What has to be settled before a step
Section of Networks in practice, part of Statistical learning
What has to be settled before a step
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
Several quantities have to be settled before a gradient step is worth taking:
- weight initialisation, typically small random values with mean 0;
- batch or online updates;
- the learning rate \varepsilon>0, chosen for the scaling of the loss and the inputs;
- the stopping criterion;
- the number of layers;
- the number of nodes in hidden layers;
- the number of input–output examples.
A learning rate of 0.1 is an example of a step size. It is not a distinguished constant. The step size is a tuning choice. If the error oscillates, take a smaller step. If the weights barely move, a larger step may be tried.
Used in
See also
Softmax is multinomial logistic regression · Weight initialisation · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network · Output coding, softmax, and cross-entropy · Constructing and training a network · Generalisation of a network