Delta form

Topic: Backpropagation · more questions on this topic
‹ Symmetric start Saturation ›

Verify that the input–hidden update equals -\varepsilon\sum_i \delta_{ib}^{(h)} x_{ia} with \delta^{(h)} as defined above.

Worked answer (click to show)

From the factored input–hidden update,

\Delta w_{ab} =2\varepsilon\sum_i e^{-u_{ib}} h_{ib}^2 x_{ia} \sum_k [y_{ik}-o_{ik}]\,e^{-z_{ik}} o_{ik}^2 w_{bk}.

Using e^{-u}h^2=h(1-h) and e^{-z}o^2=o(1-o),

\Delta w_{ab} =2\varepsilon\sum_i h_{ib}(1-h_{ib}) x_{ia} \sum_k w_{bk}[y_{ik}-o_{ik}]\,o_{ik}(1-o_{ik}).

With \delta_{ik}^{(o)}=-2(y_{ik}-o_{ik})o_{ik}(1-o_{ik}) one has 2[y_{ik}-o_{ik}]o_{ik}(1-o_{ik})=-\delta_{ik}^{(o)}, so

\Delta w_{ab} =-\varepsilon\sum_i x_{ia}\, h_{ib}(1-h_{ib})\sum_k w_{bk}\delta_{ik}^{(o)} =-\varepsilon\sum_i x_{ia}\,\delta_{ib}^{(h)}.