Sum of squares versus accuracy

Topic: Networks in practice · more questions on this topic
‹ Online versus batch Counting for NETtalk ›

Explain how a network can have a smaller sum of squared errors than another but a lower classification accuracy. Think of outputs close to 0.5 and a threshold at 0.5. Why is classification accuracy on the validation set a more direct criterion for early stopping in a classification problem, and what is its disadvantage?

Worked answer (click to show)

The sum of squared errors measures how close the outputs are to the targets 0 and 1. Accuracy only counts which side of the threshold each output lies on. Take 100 patterns. Network A outputs 0.6 for every target-1 pattern and 0.4 for every target-0 pattern: perfect accuracy, but sum of squares 100\times 0.16=16. Network B outputs essentially exactly the target for 90 patterns and the opposite extreme for 10: accuracy 90\%, but sum of squares about 10\times 1=10<16. So B has the smaller sum of squares and the lower accuracy.

If the target is zero-one classification loss, validation accuracy addresses that target directly. Its disadvantages are that it is discrete and noisy, changing in steps of 1/n_{\mathrm{val}}, a poor guide when the validation set is small, and that it ignores how confident the predictions are. Cross-entropy or squared error on the validation set varies more smoothly.