Backpropagation through one hidden layer

Section of Backpropagation, part of Statistical learning

Backpropagation through one hidden layer

Type Section
Taught in Backpropagation
Part Statistical learning

For a network with one hidden layer, learning has three phases: a feed-forward phase, an error calculation, and a weight modification. The algebra is the chain rule written along those phases. The object to keep is the update for one weight.

Feed forward. For each pattern i, with g(u)=1/(1+e^{-u}),

u_{iq}=\sum_{j=0}^{p} w_{jq} x_{ij},\qquad h_{iq}=g(u_{iq}),\qquad z_{ik}=\sum_{q=0}^{l} w_{qk} h_{iq},\qquad o_{ik}=g(z_{ik}),

with x_{i0}=h_{i0}=1.

Error. E_{ik}=[y_{ik}-o_{ik}]^2 and E=\sum_i\sum_k E_{ik}.

Weight modification. w=w_{\mathrm{old}}+\Delta w with \Delta w=-\varepsilon\,\partial E/\partial w.

Start at the loss and work backwards through each dependency. A hidden unit influences several outputs, so its derivative sums the contributions along all outgoing paths. This is sensitivity accounting. Once those sensitivities have been computed, gradient descent or another optimiser can use them.

Hidden–output weights

Main article: Hidden–output weights

Let w_{bc} connect hidden unit b\in\{0,\ldots,l\} to output c\in\{1,\ldots,m\}, so b=0 is the output bias. Only the term E_{ic} depends on w_{bc}. The sigmoid slope contributes e^{-z_{ic}}o_{ic}^2, and the signal on this connection is the hidden activation h_{ib}.

\begin{aligned} \Delta w_{bc} &=-\varepsilon\frac{\partial E}{\partial w_{bc}} =-\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\frac{\partial E_{ic}}{\partial w_{bc}},\\ \frac{\partial E_{ic}}{\partial w_{bc}} &=-2[y_{ic}-o_{ic}]\frac{\partial o_{ic}}{\partial w_{bc}},\\ \frac{\partial o_{ic}}{\partial w_{bc}} &=g'(z_{ic})\frac{\partial z_{ic}}{\partial w_{bc}} =e^{-z_{ic}} o_{ic}^2\, h_{ib}. \end{aligned}

Hence

\Delta w_{bc} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\bigl(y_{ic}-o_{ic}\bigr)\,e^{-z_{ic}} o_{ic}^2\, h_{ib},

where h_{i0}=1 for the bias. The factor e^{-z}o^2 is again g'(z).

Input–hidden weights

Main article: Input–hidden weights

Let w_{ab} connect input a\in\{0,\ldots,p\} to hidden unit b\in\{1,\ldots,l\}. This weight affects every output, because every output depends on h_{ib}. The chain rule therefore sums over outputs. Each contribution is a residual, times the output slope, times the outgoing weight w_{bk} that carried this hidden unit into that output, times the hidden slope, times the input x_{ia}.

\begin{aligned} \frac{\partial E_{ik}}{\partial w_{ab}} &=-2[y_{ik}-o_{ik}]\frac{\partial o_{ik}}{\partial w_{ab}},\\ \frac{\partial o_{ik}}{\partial w_{ab}} &=e^{-z_{ik}} o_{ik}^2\frac{\partial z_{ik}}{\partial w_{ab}} =e^{-z_{ik}} o_{ik}^2\, w_{bk}\,\frac{\partial h_{ib}}{\partial w_{ab}},\\ \frac{\partial h_{ib}}{\partial w_{ab}} &=g'(u_{ib})\frac{\partial u_{ib}}{\partial w_{ab}} =e^{-u_{ib}} h_{ib}^2\, x_{ia}. \end{aligned}

Summing over k and i,

\Delta w_{ab} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\sum_{k=1}^{m} \bigl(y_{ik}-o_{ik}\bigr)\,e^{-z_{ik}} o_{ik}^2\, w_{bk}\, e^{-u_{ib}} h_{ib}^2\, x_{ia}.

This is the update -\varepsilon\,\partial E/\partial w_{ab}, not the derivative itself. Its factors trace every path from the input weight to the loss. All are evaluated at the weights used in the forward pass. Mixing old activations with newly updated outgoing weights would not evaluate one gradient.

Pulling out the factors that do not depend on k,

\Delta w_{ab} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}} e^{-u_{ib}} h_{ib}^2\, x_{ia} \Biggl(\sum_{k=1}^{m}\bigl(y_{ik}-o_{ik}\bigr)\,e^{-z_{ik}} o_{ik}^2\, w_{bk}\Biggr).

The inner sum is the share of the output error that belongs to hidden unit b. Factoring it out does not change the value. It shows which pieces depend on the output index and which pieces are local to the hidden unit.

The delta form

Main article: Delta rule

Both updates have the structure “input signal times local derivative times error signal”. Using g'(z)=g(z)\bigl(1-g(z)\bigr), which is the same identity as e^{-z}g(z)^2, define for pattern i the error signals

\delta_{ik}^{(o)}=-2\bigl(y_{ik}-o_{ik}\bigr)\,o_{ik}\bigl(1-o_{ik}\bigr), \qquad \delta_{iq}^{(h)}=h_{iq}\bigl(1-h_{iq}\bigr)\sum_{k=1}^{m} w_{qk}\,\delta_{ik}^{(o)}.

Then

\frac{\partial E}{\partial w_{qk}}=\sum_{i}\delta_{ik}^{(o)} h_{iq}, \qquad \frac{\partial E}{\partial w_{jq}}=\sum_{i}\delta_{iq}^{(h)} x_{ij},

and the update is \Delta w=-\varepsilon\,\partial E/\partial w. Because e^{-z}o^2=o(1-o),

\delta_{ik}^{(o)}=-2[y_{ik}-o_{ik}]\,e^{-z_{ik}} o_{ik}^2,

and \delta_{iq}^{(h)} equals the bracketed sum in the factored input–hidden update, times -2, times e^{-u}h^2. The output error is propagated backwards through the weights w_{qk}, and each hidden unit receives its share. That backward pass is where the name comes from.

Definition 1 (Backpropagation, batch form) For one pass through the data with current weights:

  1. Forward: for every pattern compute and store u_{iq}, h_{iq}, z_{ik}, o_{ik}.
  2. Output deltas: \delta_{ik}^{(o)} from the definition above.
  3. Backward: hidden deltas \delta_{iq}^{(h)} from the definition above.
  4. Gradients: G_{qk}=\sum_i \delta_{ik}^{(o)} h_{iq} and G_{jq}=\sum_i \delta_{iq}^{(h)} x_{ij}, with h_{i0}=x_{i0}=1.
  5. Update: w\leftarrow w-\varepsilon G for every weight.

Article: Backpropagation, batch form →

The forward phase stores, for each training input, the hidden and output values and the weighted inputs that produced them. The backward phase multiplies those stored values by the deltas. Nothing in the algorithm minimises E in closed form. Each pass is one gradient step.

Example 1 (One backpropagation step, checked numerically) Question. A network has p=2 inputs, l=2 hidden sigmoid units, and m=1 sigmoid output. For the single training pattern x=(1,0), y=1 the weights are, with the first index 0 for the bias,

\begin{aligned} w_{01}&=0.1,& w_{11}&=0.4,& w_{21}&=-0.3,\\ w_{02}&=-0.2,& w_{12}&=0.3,& w_{22}&=0.2,\\ w_{01}^{\mathrm{out}}&=0.2,& w_{11}^{\mathrm{out}}&=0.5,& w_{21}^{\mathrm{out}}&=-0.4. \end{aligned}

Here w_{jq} is the weight from input j to hidden unit q, and w_{q1}^{\mathrm{out}} is the weight from hidden unit q (or the bias, q=0) to the output. Perform one gradient step with \varepsilon=0.5.

Solution. Forward. u_1=0.1+0.4(1)+(-0.3)(0)=0.5 and u_2=-0.2+0.3(1)+0.2(0)=0.1, so h_1=g(0.5)=0.6225 and h_2=g(0.1)=0.5250. Then

z=0.2+0.5(0.6225)-0.4(0.5250)=0.3012

and o=g(0.3012)=0.5748. The error is E=(1-0.5748)^2=0.1808.

Output delta. \delta^{(o)}=-2(1-0.5748)(0.5748)(0.4252)=-0.2079. Gradients of the output weights are \delta^{(o)}\times(1,h_1,h_2)=(-0.2079,-0.1294,-0.1091).

Hidden deltas.

\delta_1^{(h)}=h_1(1-h_1)\,w_{11}^{\mathrm{out}}\,\delta^{(o)} =0.2350\times 0.5\times(-0.2079)=-0.02443,

\delta_2^{(h)}=0.2494\times(-0.4)\times(-0.2079)=+0.02074.

The gradients of the input–hidden weights are \delta_q^{(h)}\times(1,x_1,x_2)=\delta_q^{(h)}\times(1,1,0): (-0.02443,-0.02443,0) for unit 1 and (0.02074,0.02074,0) for unit 2. The weights from the second input have zero gradient because x_2=0.

Check. Central differences with \delta=10^{-6} for all nine weights agree with these analytical gradients to 2\times 10^{-11}.

Update with \varepsilon=0.5:

\begin{aligned} w_{01}&=0.1+0.5(0.02443)=0.1122, & w_{11}&=0.4122,\\ w_{02}&=-0.2104, & w_{12}&=0.2896, \end{aligned}

and the output weights become 0.2+0.5(0.2079)=0.3039, 0.5+0.5(0.1294)=0.5647, -0.4+0.5(0.1091)=-0.3454. The new output is o=0.6175 and the error has fallen from 0.1808 to 0.1463.

Automatic differentiation applies the chain rule to the elementary operations of the computation. It evaluates the same derivative the algebra above organises by hand. The central-difference check is a separate numerical test of that derivative.

Used in

See also

Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Hidden–output weights

Glossary and index