Mini-batch
Topic of Networks in practice, part of Statistical learning
Mini-batch
| Type | Topic |
| Taught in | Networks in practice |
| Part | Statistical learning |
A small random subset of the training data whose average gradient is used for one weight update. It sits between full-batch and one-observation online learning: cheaper than a full pass, less noisy than a single case.
In Networks in practice
From Networks in practice
- Cost per update. One example requires less work than a full pass, so an online method can update weights sooner. A mini-batch shares this benefit while averaging some of the sampling noise.
- Noise and convergence. A noisy gradient can help leave some shallow regions, and it gives no general escape guarantee. With a fixed learning rate, fluctuations may persist near a solution. Convergence results require assumptions on the objective, the sampling, and the step sizes. Rare examples may be absent from short random runs. Shuffling each epoch ensures every training example is used during that epoch.
Used in
- Taught in Networks in practice
- Also used in Backpropagation
- All 7 questions on Networks in practice
See also
Training set · Stochastic gradient descent · Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network