Online versus batch

Topic: Networks in practice · more questions on this topic
‹ Weight decay Sum of squares versus accuracy ›

For n_{\mathrm{tr}}=5000 patterns, each forward–backward pass costing one unit per pattern, compare the cost of 10{,}000 epochs of batch learning with 10{,}000 online iterations (one randomly chosen pattern each). What do the 10{,}000 online iterations amount to in epochs, and what does this suggest about when online learning is attractive?

Worked answer (click to show)

Each batch epoch passes all 5000 patterns, so 10{,}000 epochs cost 5\times 10^{7} pattern-passes. Online learning for 10{,}000 iterations costs 10^{4} pattern-passes, which is 10^{4}/5000=2 epochs’ worth of data, so the cost is 5000 times smaller. In terms of data seen, it is two epochs. Whether two epochs of single-pattern updates match thousands of batch epochs depends on the problem. Learning-rate schedules, noise, and redundancy can change the comparison. Equal iteration counts do not imply equal computational effort or equal optimisation accuracy.