Delta rule
Section of Backpropagation, part of Statistical learning
Delta rule
| Type | Section |
| Taught in | Backpropagation |
| Part | Statistical learning |
Both updates have the structure “input signal times local derivative times error signal”. Using g'(z)=g(z)\bigl(1-g(z)\bigr), which is the same identity as e^{-z}g(z)^2, define for pattern i the error signals
\delta_{ik}^{(o)}=-2\bigl(y_{ik}-o_{ik}\bigr)\,o_{ik}\bigl(1-o_{ik}\bigr), \qquad \delta_{iq}^{(h)}=h_{iq}\bigl(1-h_{iq}\bigr)\sum_{k=1}^{m} w_{qk}\,\delta_{ik}^{(o)}.
Then
\frac{\partial E}{\partial w_{qk}}=\sum_{i}\delta_{ik}^{(o)} h_{iq}, \qquad \frac{\partial E}{\partial w_{jq}}=\sum_{i}\delta_{iq}^{(h)} x_{ij},
and the update is \Delta w=-\varepsilon\,\partial E/\partial w. Because e^{-z}o^2=o(1-o),
\delta_{ik}^{(o)}=-2[y_{ik}-o_{ik}]\,e^{-z_{ik}} o_{ik}^2,
and \delta_{iq}^{(h)} equals the bracketed sum in the factored input–hidden update, times -2, times e^{-u}h^2. The output error is propagated backwards through the weights w_{qk}, and each hidden unit receives its share. That backward pass is where the name comes from.
Definition 1 (Backpropagation, batch form) For one pass through the data with current weights:
- Forward: for every pattern compute and store u_{iq}, h_{iq}, z_{ik}, o_{ik}.
- Output deltas: \delta_{ik}^{(o)} from the definition above.
- Backward: hidden deltas \delta_{iq}^{(h)} from the definition above.
- Gradients: G_{qk}=\sum_i \delta_{ik}^{(o)} h_{iq} and G_{jq}=\sum_i \delta_{iq}^{(h)} x_{ij}, with h_{i0}=x_{i0}=1.
- Update: w\leftarrow w-\varepsilon G for every weight.
The forward phase stores, for each training input, the hidden and output values and the weighted inputs that produced them. The backward phase multiplies those stored values by the deltas. Nothing in the algorithm minimises E in closed form. Each pass is one gradient step.
Example 1 (One backpropagation step, checked numerically) Question. A network has p=2 inputs, l=2 hidden sigmoid units, and m=1 sigmoid output. For the single training pattern x=(1,0), y=1 the weights are, with the first index 0 for the bias,
\begin{aligned} w_{01}&=0.1,& w_{11}&=0.4,& w_{21}&=-0.3,\\ w_{02}&=-0.2,& w_{12}&=0.3,& w_{22}&=0.2,\\ w_{01}^{\mathrm{out}}&=0.2,& w_{11}^{\mathrm{out}}&=0.5,& w_{21}^{\mathrm{out}}&=-0.4. \end{aligned}
Here w_{jq} is the weight from input j to hidden unit q, and w_{q1}^{\mathrm{out}} is the weight from hidden unit q (or the bias, q=0) to the output. Perform one gradient step with \varepsilon=0.5.
Solution. Forward. u_1=0.1+0.4(1)+(-0.3)(0)=0.5 and u_2=-0.2+0.3(1)+0.2(0)=0.1, so h_1=g(0.5)=0.6225 and h_2=g(0.1)=0.5250. Then
z=0.2+0.5(0.6225)-0.4(0.5250)=0.3012
and o=g(0.3012)=0.5748. The error is E=(1-0.5748)^2=0.1808.
Output delta. \delta^{(o)}=-2(1-0.5748)(0.5748)(0.4252)=-0.2079. Gradients of the output weights are \delta^{(o)}\times(1,h_1,h_2)=(-0.2079,-0.1294,-0.1091).
Hidden deltas.
\delta_1^{(h)}=h_1(1-h_1)\,w_{11}^{\mathrm{out}}\,\delta^{(o)} =0.2350\times 0.5\times(-0.2079)=-0.02443,
\delta_2^{(h)}=0.2494\times(-0.4)\times(-0.2079)=+0.02074.
The gradients of the input–hidden weights are \delta_q^{(h)}\times(1,x_1,x_2)=\delta_q^{(h)}\times(1,1,0): (-0.02443,-0.02443,0) for unit 1 and (0.02074,0.02074,0) for unit 2. The weights from the second input have zero gradient because x_2=0.
Check. Central differences with \delta=10^{-6} for all nine weights agree with these analytical gradients to 2\times 10^{-11}.
Update with \varepsilon=0.5:
\begin{aligned} w_{01}&=0.1+0.5(0.02443)=0.1122, & w_{11}&=0.4122,\\ w_{02}&=-0.2104, & w_{12}&=0.2896, \end{aligned}
and the output weights become 0.2+0.5(0.2079)=0.3039, 0.5+0.5(0.1294)=0.5647, -0.4+0.5(0.1091)=-0.3454. The new output is o=0.6175 and the error has fallen from 0.1808 to 0.1463.
Automatic differentiation applies the chain rule to the elementary operations of the computation. It evaluates the same derivative the algebra above organises by hand. The central-difference check is a separate numerical test of that derivative.
Used in
See also
Backpropagation, batch form · Basic gradient-descent learning · Squared error is not the logistic likelihood · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer