Local minima, plateaus, and symmetry
Theorem taught in Backpropagation, part of Statistical learning
Local minima, plateaus, and symmetry
| Type | Theorem |
| Taught in | Backpropagation |
| Part | Statistical learning |
The loss is generally nonconvex. A run can approach a local minimum, a saddle or a nearly flat region, or fail to settle. Permuting hidden units and their incident weights preserves the function. There are up to l! distinct permutations, fewer if some units coincide. Fully symmetric hidden-unit initialisation preserves that symmetry under deterministic gradient updates. Random initialisation breaks it. For sigmoids, replacing g(u) by g(-u)=1-g(u) also requires compensating outgoing weights and biases to preserve the function.
Used in
- Taught in Backpropagation, where the derivation and examples are.
- Also used in K-means clustering
- All 7 questions on Backpropagation
See also
Hidden layer · Sigmoid function · Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Training error of a network · Gradient descent · A single layer of sigmoid units