Squared error is not the logistic likelihood
Theorem taught in Backpropagation, part of Statistical learning
Squared error is not the logistic likelihood
| Type | Theorem |
| Taught in | Backpropagation |
| Part | Statistical learning |
For one sigmoid output, squared-error fitting and Bernoulli likelihood fitting are different optimisation problems. The squared-error gradient includes the factor o_i(1-o_i), which becomes small at saturation. The gradient of the negative log-likelihood is \sum_i (o_i-y_i)x_{ia}, without that extra factor. The score of the positive log-likelihood has the opposite sign. Distinguish the gradient from the descent update. The likelihood itself is the logistic regression fit. The calculation here is the squared-error gradient, including through hidden layers.
Used in
- Taught in Backpropagation, where the derivation and examples are.
- All 7 questions on Backpropagation
See also
Sigmoid function · Likelihood function · Hidden layer · Basic gradient-descent learning · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent