Regularisation and weight decay
Section of Networks in practice, part of Statistical learning
Regularisation and weight decay
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
A weight penalty changes the objective itself. With L^2 regularisation one charges for large squared weights, just as ridge regression does for linear coefficients. This is a modelling preference for smaller parameter values in the chosen representation. Equivalent reparameterisations need not receive the same penalty. Under ordinary gradient descent, the resulting update contains a literal multiplicative decay. That equivalence should not be transferred automatically to an adaptive optimiser.
L^2 regularisation, also called weight decay, ridge regression (Hoerl and Kennard, 1970), or Tikhonov regularisation, adds a penalty proportional to the sum of squares of the weights. Optimise E+\lambda\lVert w\rVert^2 with \lambda>0 chosen by the user. After rescaling the tuning parameter, write the penalty as E+\tfrac{\lambda}{2}\lVert w\rVert^2. The gradient step becomes
w\leftarrow w-\varepsilon\Bigl(\frac{\partial E}{\partial w}+\lambda w\Bigr) =(1-\varepsilon\lambda)\,w-\varepsilon\frac{\partial E}{\partial w}.
When 0\le\varepsilon\lambda\le 1, the factor 1-\varepsilon\lambda shrinks a weight towards zero without reversing its sign. With no data-gradient term, t steps multiply it by (1-\varepsilon\lambda)^t. For \varepsilon=0.1, \lambda=0.5 and t=10, this factor is 0.95^{10}\approx 0.599. A step size is part of the statement: an excessive value can cause oscillation or divergence even in this simple recurrence. The penalty encourages small weights. It does not have the lasso’s thresholding mechanism. That mechanism is derived on high-dimensional analysis.
L^1 regularisation uses E+\lambda\sum|w|. The derivative at zero is replaced by a subgradient or a proximal update. Such algorithms can produce exact zeros. The non-differentiability changes the optimisation method. It does not make the penalty unusable.
Used in
- Taught in Networks in practice
- Also used in Support vector machines · High-dimensional analysis
- All 7 questions on Networks in practice
See also
Early stopping · Regularisation · Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network