Hidden–output weights
Section of Backpropagation, part of Statistical learning
Hidden–output weights
| Type | Section |
| Taught in | Backpropagation |
| Part | Statistical learning |
Let w_{bc} connect hidden unit b\in\{0,\ldots,l\} to output c\in\{1,\ldots,m\}, so b=0 is the output bias. Only the term E_{ic} depends on w_{bc}. The sigmoid slope contributes e^{-z_{ic}}o_{ic}^2, and the signal on this connection is the hidden activation h_{ib}.
\begin{aligned} \Delta w_{bc} &=-\varepsilon\frac{\partial E}{\partial w_{bc}} =-\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\frac{\partial E_{ic}}{\partial w_{bc}},\\ \frac{\partial E_{ic}}{\partial w_{bc}} &=-2[y_{ic}-o_{ic}]\frac{\partial o_{ic}}{\partial w_{bc}},\\ \frac{\partial o_{ic}}{\partial w_{bc}} &=g'(z_{ic})\frac{\partial z_{ic}}{\partial w_{bc}} =e^{-z_{ic}} o_{ic}^2\, h_{ib}. \end{aligned}
Hence
\Delta w_{bc} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\bigl(y_{ic}-o_{ic}\bigr)\,e^{-z_{ic}} o_{ic}^2\, h_{ib},
where h_{i0}=1 for the bias. The factor e^{-z}o^2 is again g'(z).
Used in
See also
Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer