A single layer of sigmoid units
Section of Backpropagation, part of Statistical learning
A single layer of sigmoid units
| Type | Section |
| Taught in | Backpropagation |
| Part | Statistical learning |
Consider a network with only one layer of weights and no hidden units, with sigmoid outputs o_{ik}=g(z_{ik}) and z_{ik}=\sum_{j=0}^{p} w_{jk} x_{ij}. Differentiate one weight w_{ab}, from input a to output b. Differentiate the sum of squares, keep the single output that depends on this weight, and multiply by the sigmoid slope and by \partial z_{ib}/\partial w_{ab}=x_{ia}.
\frac{\partial E}{\partial w_{ab}} =\sum_{i=1}^{n_{\mathrm{tr}}}\sum_{k=1}^{m} -2\bigl(y_{ik}-o_{ik}\bigr)\frac{\partial o_{ik}}{\partial w_{ab}}.
Only o_{ib} depends on w_{ab}, so the inner sum over outputs collapses to the term k=b. The sigmoid derivative g'(z)=e^{-z}g(z)^2 is the identity on the neural units page, and \partial z_{ib}/\partial w_{ab}=x_{ia}, so
\frac{\partial o_{ib}}{\partial w_{ab}}=g'(z_{ib})\,x_{ia}=e^{-z_{ib}} o_{ib}^2\, x_{ia}.
The gradient and the descent step are
\frac{\partial E}{\partial w_{ab}} =-2\sum_{i=1}^{n_{\mathrm{tr}}}\bigl(y_{ib}-o_{ib}\bigr)\,e^{-z_{ib}} o_{ib}^2\, x_{ia}, \qquad \Delta w_{ab} =2\varepsilon\sum_{i=1}^{n_{\mathrm{tr}}}\bigl(y_{ib}-o_{ib}\bigr)\,e^{-z_{ib}} o_{ib}^2\, x_{ia}.
Including the bias weights as w_{0k} means x_{i0}=1 for all patterns. The three displays are \partial E/\partial w_{ab} for squared error without a leading \tfrac12, under a logistic activation. The factor 2 is the derivative of a square. The factor e^{-z_{ib}} is the numerator of g'.
A common slip drops that numerator and writes \partial o/\partial w_{ab}=x_{ia}/(1+e^{-\sum_j w_{jk}x_{ij}})^2, concluding \Delta w_{ab}=2\varepsilon\sum_i [y_{ib}-o_{ib}]\,o_{ib}^2 x_{ia} with no factor e^{-z_{ib}}. The derivative of the logistic function is e^{-z}/(1+e^{-z})^2, so the numerator e^{-z} has been dropped. The correct result is the display above. The hidden–output update derived next contains e^{-z_{ic}}, and the two agree once the factor is restored. The error matters numerically: in the worked step below, the formula without the factor overstates a gradient component by e^{z}\approx 1.35 and fails the numerical gradient check.
Theorem 1 (Squared error is not the logistic likelihood) For one sigmoid output, squared-error fitting and Bernoulli likelihood fitting are different optimisation problems. The squared-error gradient includes the factor o_i(1-o_i), which becomes small at saturation. The gradient of the negative log-likelihood is \sum_i (o_i-y_i)x_{ia}, without that extra factor. The score of the positive log-likelihood has the opposite sign. Distinguish the gradient from the descent update. The likelihood itself is the logistic regression fit. The calculation here is the squared-error gradient, including through hidden layers.
Used in
See also
Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · Backpropagation through one hidden layer · Hidden–output weights