Batch and online learning
Section of Networks in practice, part of Statistical learning
Batch and online learning
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
The backpropagation update is in batch mode: the sums over patterns require all n_{\mathrm{tr}} training patterns to be fed in, and the hidden and output activations recorded, before any weight changes.
A full gradient requires work over all training examples. With a large sample, more frequent and cheaper updates may be preferable. Online stochastic gradient descent samples an index i uniformly and uses its loss gradient. If E=\sum_i E_i, then
\mathbb{E}\bigl[n_{\mathrm{tr}}\nabla E_i\bigr]=\nabla E.
This expectation is conditional on the current weights and on uniform sampling of the index. The gradient estimate is unbiased. Each proposed step need not reduce the full loss. For the average loss E/n_{\mathrm{tr}}, use \nabla E_i without the factor n_{\mathrm{tr}}. Confusing these two conventions changes the effective learning rate.
- Cost per update. One example requires less work than a full pass, so an online method can update weights sooner. A mini-batch shares this benefit while averaging some of the sampling noise.
- Noise and convergence. A noisy gradient can help leave some shallow regions, and it gives no general escape guarantee. With a fixed learning rate, fluctuations may persist near a solution. Convergence results require assumptions on the objective, the sampling, and the step sizes. Rare examples may be absent from short random runs. Shuffling each epoch ensures every training example is used during that epoch.
A compromise in common use is the mini-batch: the gradient is averaged over a small random subset of patterns. Batch mode uses the full sample gradient. Online mode uses one observation.
Used in
See also
Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation · Preparing data for a network · Output coding, softmax, and cross-entropy · Constructing and training a network · Generalisation of a network