Learning rate
Topic of Backpropagation, part of Statistical learning
Learning rate
| Type | Topic |
| Taught in | Backpropagation |
| Part | Statistical learning |
| Questions using it | 1 |
The step size \eta in a gradient-descent update \mathbf{w}\leftarrow\mathbf{w}-\eta\,\nabla E(\mathbf{w}). Too large and the error oscillates or diverges; too small and training crawls. A fixed rate leaves stochastic gradient descent fluctuating near a solution.
In Backpropagation
From Backpropagation
Definition 1 (Basic gradient-descent learning)
- Choose a random set of initial weights, for example from the uniform distribution on [-0.5,0.5].
- Repeat:
- Run every input training pattern through the network and record the output values of the hidden and output units.
- Calculate E for the current weights.
- Update every weight by w'=w-\varepsilon\frac{\partial E}{\partial w}=w+\Delta w, until a stopping condition holds: the performance is acceptable, or the weights are barely changing.
The learning rate \varepsilon>0 is the step size along minus the gradient. The value 0.1 is an example of a step size. It is not a distinguished constant.
In one dimension the error is a curve in the single weight. The update subtracts \varepsilon times the derivative, so the step length is \varepsilon times the slope. A steep region takes a longer step, and a large \varepsilon can jump over a valley. A flat region, where the derivative is near zero, barely moves. A start in one basin stays in that basin under a small enough step. Momentum adds a fraction of the previous step and can cross a flat ridge. A mini-batch replaces the full gradient by an average over a random subset and so adds noise to the gradient. A line search spends several evaluations of the loss on one step. A second-order step uses the Hessian and costs more per iteration.
A randomly sampled mini-batch can give an unbiased estimate of the gradient of the mean loss. That expectation, with the current weights held fixed, is the relevant property. The angle of one noisy draw is not a convergence argument. Its variance and the learning-rate schedule still affect the trajectory. One pass through all the training patterns is an epoch. NETtalk, on networks in practice, is the count used later.
For checking a gradient, perturb one weight at a time and compare the analytical derivative with [E(w+\delta)-E(w-\delta)]/(2\delta). A very large \delta causes approximation error. A very small one causes cancellation in floating-point arithmetic. Try a small range of steps and compare absolute as well as relative discrepancies, especially near zero derivatives. This check is expensive for training and useful for finding a missing sign or factor.
To first order, E(w+\Delta w)\approx E(w)+(\partial E/\partial w)\,\Delta w, which is decreased most for a given step length by taking \Delta w proportional to -\partial E/\partial w. What this establishes is a descent direction. The assumption that does the work is that the step is small enough for the remainder to stay smaller than the linear term. The XOR runs below show what the same update does when that assumption fails.
Used in
- Taught in Backpropagation
- Also used in Networks in practice
- Questions that use it (1) — One backpropagation update
- All 7 questions on Backpropagation
See also
Gradient descent · Stochastic gradient descent · Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · A single layer of sigmoid units