Weight initialisation
Section of Networks in practice, part of Statistical learning
Weight initialisation
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
| Questions using it | 1 |
A sigmoid has its largest slope near zero. Far into either tail, its derivative is small, so the chain rule transmits little change back through that unit. Initial net inputs should therefore be of moderate size. Different hidden units should start differently: units with identical incoming and outgoing weights remain interchangeable under deterministic symmetric updates, as shown for the batch rule on the backpropagation page. Independent random initialisation breaks that symmetry. Standardising predictors using training-set means and standard deviations makes a common initialisation scale more meaningful. It does not guarantee that every net input stays near zero. The variance calculation below is the scaling argument.
A variance argument
Main article: Variance argument for weight initialisation
Let each non-bias input have mean zero and variance one, and initialise the weights independently as w_{jq}\sim\operatorname{N}(0,\sigma^2), independently of the inputs. Write u_{iq}=\sum_{j=0}^{p} w_{jq} x_{ij}, where x_{i0}=1. Each summand has mean zero and second moment \sigma^2. Cross terms vanish because distinct weights are independent and centred:
\mathbb{E}\bigl(w_{jq} w_{kq} x_{ij} x_{ik}\bigr)=0\qquad(j\neq k).
The inputs themselves need not be uncorrelated. Consequently
\operatorname{Var}(u_{iq})=(p+1)\sigma^2.
Choosing \sigma=(p+1)^{-1/2} makes this marginal variance one. That is a scale calculation, not yet a tail-probability calculation. Conditional on a particular input vector, Gaussian weights give
u_{iq}\mid x_i\sim\operatorname{N}\bigl(0,\,\sigma^2[1+\lVert x_i\rVert^2]\bigr).
For an input with \lVert x_i\rVert^2=p, the proposed scale gives \mathbb{P}(|u_{iq}|\le 2\mid x_i)=2\Phi(2)-1\approx 0.9545. Across random inputs, the distribution is generally a mixture of normals with different variances. Unit marginal variance alone does not justify that same 95\% statement. Fan-in scaling is a useful starting point. Activation choice and depth still matter.
Used in
- Taught in Networks in practice
- Also used in Ensembles, random forests, and boosting · Backpropagation
- Questions that use it (1) — Initial weights
- All 7 questions on Networks in practice
See also
Basic gradient-descent learning · Softmax is multinomial logistic regression · What has to be settled before a step · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network · Output coding, softmax, and cross-entropy · Constructing and training a network