Stochastic gradient descent
Topic of Networks in practice, part of Statistical learning · Also in Networks in practice
Stochastic gradient descent
| Type | Topic |
| Taught in | Networks in practice · Networks in practice |
| Part | Statistical learning |
| Questions using it | 1 |
Gradient descent that updates the weights using the gradient of one observation’s loss, or of a small batch, chosen at random. Each step is cheap and noisy but unbiased for the full gradient; with a decreasing learning rate it converges under conditions.
In Networks in practice
From Networks in practice
A full gradient requires work over all training examples. With a large sample, more frequent and cheaper updates may be preferable. Online stochastic gradient descent samples an index i uniformly and uses its loss gradient. If E=\sum_i E_i, then
\mathbb{E}\bigl[n_{\mathrm{tr}}\nabla E_i\bigr]=\nabla E.
In Networks in practice, continued
From Networks in practice
The backpropagation update is in batch mode: the sums over patterns require all n_{\mathrm{tr}} training patterns to be fed in, and the hidden and output activations recorded, before any weight changes.
A full gradient requires work over all training examples. With a large sample, more frequent and cheaper updates may be preferable. Online stochastic gradient descent samples an index i uniformly and uses its loss gradient. If E=\sum_i E_i, then
\mathbb{E}\bigl[n_{\mathrm{tr}}\nabla E_i\bigr]=\nabla E.
This expectation is conditional on the current weights and on uniform sampling of the index. The gradient estimate is unbiased. Each proposed step need not reduce the full loss. For the average loss E/n_{\mathrm{tr}}, use \nabla E_i without the factor n_{\mathrm{tr}}. Confusing these two conventions changes the effective learning rate.
- Cost per update. One example requires less work than a full pass, so an online method can update weights sooner. A mini-batch shares this benefit while averaging some of the sampling noise.
- Noise and convergence. A noisy gradient can help leave some shallow regions, and it gives no general escape guarantee. With a fixed learning rate, fluctuations may persist near a solution. Convergence results require assumptions on the objective, the sampling, and the step sizes. Rare examples may be absent from short random runs. Shuffling each epoch ensures every training example is used during that epoch.
A compromise in common use is the mini-batch: the gradient is averaged over a small random subset of patterns. Batch mode uses the full sample gradient. Online mode uses one observation.
Used in
- Taught in Networks in practice · Networks in practice
- Also used in Neural units
- Questions that use it (1) — Online versus batch
- All 7 questions on Networks in practice
See also
Gradient descent · Unbiased estimator · Learning rate · Epoch · Mini-batch · Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation