Initial weights

Topic: Networks in practice · more questions on this topic
Softmax gradient ›

A hidden layer has fan-in p+1=26 (25 standardised inputs plus bias). What standard deviation should the initial weights have so that the net input has unit variance? What is the variance of the net input if the weights have standard deviation 0.5? For a fixed input with squared norm 25 and independent Gaussian initial weights, how likely is |u|>2?

Worked answer (click to show)

\operatorname{Var}(u)=(p+1)\sigma^2=1 gives \sigma=26^{-1/2}=0.196. With \sigma=0.5, \operatorname{Var}(u)=26\times 0.25=6.5, standard deviation 2.55. Conditional on the fixed input with squared norm 25 and Gaussian initial weights,

\mathbb{P}(|u|>2\mid x)=2\bigl[1-\Phi(2/2.55)\bigr]=2\bigl[1-\Phi(0.784)\bigr]=2(0.2165)=0.43.

Across repeated initialisations at that input, about 43\% of net inputs fall outside [-2,2], where the sigmoid derivative is below 0.105, so many units begin partly saturated.