Variance argument for weight initialisation
Section of Networks in practice, part of Statistical learning
Variance argument for weight initialisation
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
Let each non-bias input have mean zero and variance one, and initialise the weights independently as w_{jq}\sim\operatorname{N}(0,\sigma^2), independently of the inputs. Write u_{iq}=\sum_{j=0}^{p} w_{jq} x_{ij}, where x_{i0}=1. Each summand has mean zero and second moment \sigma^2. Cross terms vanish because distinct weights are independent and centred:
\mathbb{E}\bigl(w_{jq} w_{kq} x_{ij} x_{ik}\bigr)=0\qquad(j\neq k).
The inputs themselves need not be uncorrelated. Consequently
\operatorname{Var}(u_{iq})=(p+1)\sigma^2.
Choosing \sigma=(p+1)^{-1/2} makes this marginal variance one. That is a scale calculation, not yet a tail-probability calculation. Conditional on a particular input vector, Gaussian weights give
u_{iq}\mid x_i\sim\operatorname{N}\bigl(0,\,\sigma^2[1+\lVert x_i\rVert^2]\bigr).
For an input with \lVert x_i\rVert^2=p, the proposed scale gives \mathbb{P}(|u_{iq}|\le 2\mid x_i)=2\Phi(2)-1\approx 0.9545. Across random inputs, the distribution is generally a mixture of normals with different variances. Unit marginal variance alone does not justify that same 95\% statement. Fan-in scaling is a useful starting point. Activation choice and depth still matter.
Used in
See also
Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Batch and online learning · Preparing data for a network · Output coding, softmax, and cross-entropy · Constructing and training a network · Generalisation of a network