How many weights?
Section of Backpropagation, part of Statistical learning
How many weights?
| Type | Section |
| Taught in | Backpropagation |
| Part | Statistical learning |
A hidden layer that sees every input, and an output layer that sees every hidden unit, produces one weight per connection, plus a bias on every unit. The count is
(p+1)\,l+(l+1)\,m.
NETtalk had p=7\times 29=203 inputs, l=80 hidden units and m=26 output units: 204\times 80+81\times 26=16320+2106=18426 weights. The search for a global minimum of E is a search in that dimension.
A large parameter count raises estimation and optimisation questions. It does not prove that useful prediction is impossible. Symmetries can make individual weights non-identifiable even when the fitted function is well determined. A smaller network reduces one source of flexibility, and it can underfit or be harder to optimise. Compare functions and validation performance.
Theorem 1 (Local minima, plateaus, and symmetry) The loss is generally nonconvex. A run can approach a local minimum, a saddle or a nearly flat region, or fail to settle. Permuting hidden units and their incident weights preserves the function. There are up to l! distinct permutations, fewer if some units coincide. Fully symmetric hidden-unit initialisation preserves that symmetry under deterministic gradient updates. Random initialisation breaks it. For sigmoids, replacing g(u) by g(-u)=1-g(u) also requires compensating outgoing weights and biases to preserve the function.
The loss need not be convex, so a run can stop at a local minimum. The symmetry repeats each minimum, and a constant initialisation never breaks that symmetry. The start matters. The practical obstruction is often a flat region of the loss surface, where the gradient is near zero without the point being a minimiser. Saturation of a sigmoid is one way to land in such a region: g' is then close to zero, and the chain rule multiplies every upstream weight by that small factor.
Used in
See also
Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer