Symmetric start
Show that if all hidden units of a network start with identical incoming weights and identical outgoing weights, then batch gradient descent keeps them identical for all time. What does this say about initialising all weights at 0?
Suppose hidden units q and q' have w_{jq}=w_{jq'} for all j and w_{qk}=w_{q'k} for all k. Then for every pattern u_{iq}=u_{iq'}, so h_{iq}=h_{iq'}, and the error signals
\delta_{iq}^{(h)}=h_{iq}(1-h_{iq})\sum_k w_{qk}\delta_{ik}^{(o)}
are identical for q and q'. The gradients \sum_i\delta_{iq}^{(h)} x_{ij} of their incoming weights are equal, and so are the gradients \sum_i\delta_{ik}^{(o)} h_{iq} of their outgoing weights. Both units receive the same update, so they remain identical. The network behaves as if it had one hidden unit, repeated.
If all weights start at 0, then h_{iq}=g(0)=0.5 for every hidden unit and every pattern. All hidden units are identical, and the output-layer gradients are also identical. The symmetry is never broken. A symmetry-breaking start is needed for these otherwise identical units to differentiate under this deterministic update. Random initialisation is one way to obtain it.