One backpropagation update

Topic: Backpropagation · more questions on this topic
‹ Saturation

A one-input network has h=g(ax+c), o=g(vh+b) and loss L=\tfrac12(o-y)^2, where g(z)=1/(1+e^{-z}).

1. Derive \partial L/\partial v and \partial L/\partial b.

2. Derive \partial L/\partial a, identifying the factors contributed by the chain rule.

3. Explain the sign and the role of the learning rate in one gradient-descent update.

Worked answer (click to show)

The loss in this exercise is half the squared error, so there is no extra factor of two.

1. With \delta_o=(o-y)o(1-o), the output-weight derivative is \delta_o h and the output-bias derivative is \delta_o. These contain the loss derivative, the sigmoid derivative, and the derivative of the net input.

\frac{\partial L}{\partial a}=(o-y)\,o(1-o)\,v\,h(1-h)\,x.

The factors traverse the path from the loss to the output, the output net input, the hidden activation, the hidden net input, and the weight a.

3. The update is a_{\mathrm{new}}=a-\eta\,\partial L/\partial a with \eta>0. The minus sign moves locally against increasing loss. A step that is too large may increase the loss or destabilise training. A step that is too small can be slow. The derivatives for a simultaneous update are evaluated at the same old parameters.