Weight decay
A weight w=2 has \partial E/\partial w=0.4, with \varepsilon=0.1 and \lambda=0.5 in the decay update. Compute the updated weight with and without decay. What value of w makes the net update zero?
Worked answer (click to show)
Without decay, w'=2-0.1(0.4)=1.96. With decay,
w'=2-0.1(0.4+0.5\times 2)=2-0.14=1.86,
equivalently (1-0.05)\times 2-0.04=1.86. The net update is zero when \partial E/\partial w+\lambda w=0, that is
w=-\frac{1}{\lambda}\frac{\partial E}{\partial w}=-0.8.
The penalised stationary point balances the data gradient against the shrinkage force.