Saturation
For a sigmoid output unit with squared-error loss, a pattern with target y=1 has net input z=-8. Compute the output and the gradient \partial E/\partial z, and compare with the gradient at z=-1. What does this say about recovering from a confidently wrong prediction, and how does cross-entropy loss change the picture?
At z=-8, o=g(-8)=3.35\times 10^{-4} and the error is 1-o\approx 0.9997. Then
\frac{\partial E}{\partial z} =-2(y-o)\,o(1-o) =-2(0.9997)(3.35\times 10^{-4})(0.9997) =-6.7\times 10^{-4}.
At z=-1, o=0.2689 and
\frac{\partial E}{\partial z} =-2(0.7311)(0.2689)(0.7311) =-0.2874.
The gradient at the confidently wrong point is about 430 times smaller, although the error is much larger. The unit is saturated and learns extremely slowly.
For cross-entropy loss -y\ln o-(1-y)\ln(1-o), one has \partial E/\partial z=o-y, which equals -0.9997 at z=-8 and -0.7311 at z=-1. The gradient grows with the error and is not damped by saturation.