Input–hidden weights
Section of Backpropagation, part of Statistical learning
Input–hidden weights
| Type | Section |
| Taught in | Backpropagation |
| Part | Statistical learning |
Let w_{ab} connect input a\in\{0,\ldots,p\} to hidden unit b\in\{1,\ldots,l\}. This weight affects every output, because every output depends on h_{ib}. The chain rule therefore sums over outputs. Each contribution is a residual, times the output slope, times the outgoing weight w_{bk} that carried this hidden unit into that output, times the hidden slope, times the input x_{ia}.
\begin{aligned} \frac{\partial E_{ik}}{\partial w_{ab}} &=-2[y_{ik}-o_{ik}]\frac{\partial o_{ik}}{\partial w_{ab}},\\ \frac{\partial o_{ik}}{\partial w_{ab}} &=e^{-z_{ik}} o_{ik}^2\frac{\partial z_{ik}}{\partial w_{ab}} =e^{-z_{ik}} o_{ik}^2\, w_{bk}\,\frac{\partial h_{ib}}{\partial w_{ab}},\\ \frac{\partial h_{ib}}{\partial w_{ab}} &=g'(u_{ib})\frac{\partial u_{ib}}{\partial w_{ab}} =e^{-u_{ib}} h_{ib}^2\, x_{ia}. \end{aligned}
Summing over k and i,
\Delta w_{ab} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\sum_{k=1}^{m} \bigl(y_{ik}-o_{ik}\bigr)\,e^{-z_{ik}} o_{ik}^2\, w_{bk}\, e^{-u_{ib}} h_{ib}^2\, x_{ia}.
This is the update -\varepsilon\,\partial E/\partial w_{ab}, not the derivative itself. Its factors trace every path from the input weight to the loss. All are evaluated at the weights used in the forward pass. Mixing old activations with newly updated outgoing weights would not evaluate one gradient.
Pulling out the factors that do not depend on k,
\Delta w_{ab} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}} e^{-u_{ib}} h_{ib}^2\, x_{ia} \Biggl(\sum_{k=1}^{m}\bigl(y_{ik}-o_{ik}\bigr)\,e^{-z_{ik}} o_{ik}^2\, w_{bk}\Biggr).
The inner sum is the share of the output error that belongs to hidden unit b. Factoring it out does not change the value. It shows which pieces depend on the output index and which pieces are local to the hidden unit.
Used in
See also
Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer