Regularisation

Topic of High-dimensional analysis, part of Statistical learning · Also in Networks in practice, Bayesian linear regression

Regularisation

Type Topic
Taught in High-dimensional analysis · Networks in practice · Bayesian linear regression
Part Statistical learning · Bayesian statistics
Questions using it 11

Adding a penalty on the size or roughness of the parameters to the fitting criterion, so that the fit trades goodness of fit against complexity: ridge (L^2), the lasso (L^1), weight decay in networks, the roughness penalty of a smoothing spline. A Bayesian prior has the same effect, with the MAP estimate as the penalised fit.

In High-dimensional analysis

From High-dimensional analysis

The rank obstruction comes first, then principal components, multiple testing, and penalised regression. Each changes the problem in a different way. Principal components choose a lower-dimensional representation. Multiple-testing procedures control specified error rates. Regularisation adds a preference among competing fits. The preference is part of the model. The ordinary least-squares fit, when the design has full rank, is linear regression. A single test of one coefficient or one mean is a hypothesis test. The multiple-testing correction, and the lasso objective, are derived here. The same L^1 penalty appears as the mode of a Laplace prior on Bayesian linear regression. That page identifies the penalty with the prior. This page derives the penalised least-squares criterion and the soft-thresholding map.

In Networks in practice

From Networks in practice

A weight penalty changes the objective itself. With L^2 regularisation one charges for large squared weights, just as ridge regression does for linear coefficients. This is a modelling preference for smaller parameter values in the chosen representation. Equivalent reparameterisations need not receive the same penalty. Under ordinary gradient descent, the resulting update contains a literal multiplicative decay. That equivalence should not be transferred automatically to an adaptive optimiser.

L^2 regularisation, also called weight decay, ridge regression (Hoerl and Kennard, 1970), or Tikhonov regularisation, adds a penalty proportional to the sum of squares of the weights. Optimise E+\lambda\lVert w\rVert^2 with \lambda>0 chosen by the user. After rescaling the tuning parameter, write the penalty as E+\tfrac{\lambda}{2}\lVert w\rVert^2. The gradient step becomes

w\leftarrow w-\varepsilon\Bigl(\frac{\partial E}{\partial w}+\lambda w\Bigr) =(1-\varepsilon\lambda)\,w-\varepsilon\frac{\partial E}{\partial w}.

When 0\le\varepsilon\lambda\le 1, the factor 1-\varepsilon\lambda shrinks a weight towards zero without reversing its sign. With no data-gradient term, t steps multiply it by (1-\varepsilon\lambda)^t. For \varepsilon=0.1, \lambda=0.5 and t=10, this factor is 0.95^{10}\approx 0.599. A step size is part of the statement: an excessive value can cause oscillation or divergence even in this simple recurrence. The penalty encourages small weights. It does not have the lasso’s thresholding mechanism. That mechanism is derived on high-dimensional analysis.

L^1 regularisation uses E+\lambda\sum|w|. The derivative at zero is replaced by a subgradient or a proximal update. Such algorithms can produce exact zeros. The non-differentiability changes the optimisation method. It does not make the penalty unusable.

In Bayesian linear regression

From Bayesian linear regression

The independent priors are

{\boldsymbol{\beta}}\sim\operatorname{N}_p({\boldsymbol{m}}_0,{\color{blue}\boldsymbol{V}}_0), \qquad \sigma^2\sim\operatorname{InvGamma}(a_0,b_0).

Under this specification the model is conditionally conjugate. Collecting quadratic terms in {\boldsymbol{\beta}} gives \begin{align*} {\color{blue}\boldsymbol{V}}_\beta^{-1} &=\frac{{\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}}}{\sigma^2}+{\color{blue}\boldsymbol{V}}_0^{-1},\\ {\boldsymbol{m}}_\beta &={\color{blue}\boldsymbol{V}}_\beta\left( \frac{{\color{blue}\boldsymbol{X}}^\top{\boldsymbol{y}}}{\sigma^2} +{\color{blue}\boldsymbol{V}}_0^{-1}{\boldsymbol{m}}_0\right), \end{align*} and hence \begin{aligned} {\boldsymbol{\beta}}\mid\sigma^2,{\boldsymbol{y}} &\sim\operatorname{N}_p({\boldsymbol{m}}_\beta,{\color{blue}\boldsymbol{V}}_\beta), \\ \sigma^2\mid{\boldsymbol{\beta}},{\boldsymbol{y}} &\sim\operatorname{InvGamma}\!\left( a_0+\frac n2, b_0+\frac12\operatorname{SSE}({\boldsymbol{\beta}})\right). \end{aligned} These full conditionals define a Gibbs sampler, but prior independence is not the usual jointly conjugate normal–inverse-Gamma family.

Definition 1 (Normal–inverse-Gamma prior) Full joint conjugacy is obtained from the scaled prior {\boldsymbol{\beta}}\mid\sigma^2 \sim\operatorname{N}_p({\boldsymbol{m}}_0,\sigma^2{\color{blue}\boldsymbol{V}}_0), \qquad \sigma^2\sim\operatorname{InvGamma}(a_0,b_0).

Article: Normal–inverse-Gamma prior →

Define \begin{aligned} {\color{blue}\boldsymbol{V}}_n^{-1}&={\color{blue}\boldsymbol{V}}_0^{-1}+{\color{blue}\boldsymbol{X}}^\top{\color{blue}\boldsymbol{X}}, \\ {\boldsymbol{m}}_n&={\color{blue}\boldsymbol{V}}_n ({\color{blue}\boldsymbol{V}}_0^{-1}{\boldsymbol{m}}_0+{\color{blue}\boldsymbol{X}}^\top{\boldsymbol{y}}). \end{aligned} The two quadratic forms combine as \begin{align*} &({\boldsymbol{y}}-{\color{blue}\boldsymbol{X}}{\boldsymbol{\beta}})^\top({\boldsymbol{y}}-{\color{blue}\boldsymbol{X}}{\boldsymbol{\beta}}) +({\boldsymbol{\beta}}-{\boldsymbol{m}}_0)^\top{\color{blue}\boldsymbol{V}}_0^{-1} ({\boldsymbol{\beta}}-{\boldsymbol{m}}_0)\\ &\quad={\boldsymbol{y}}^\top{\boldsymbol{y}}+{\boldsymbol{m}}_0^\top{\color{blue}\boldsymbol{V}}_0^{-1} {\boldsymbol{m}}_0 -2{\boldsymbol{\beta}}^\top({\color{blue}\boldsymbol{X}}^\top{\boldsymbol{y}}+{\color{blue}\boldsymbol{V}}_0^{-1}{\boldsymbol{m}}_0) +{\boldsymbol{\beta}}^\top{\color{blue}\boldsymbol{V}}_n^{-1}{\boldsymbol{\beta}}\\ &\quad=({\boldsymbol{\beta}}-{\boldsymbol{m}}_n)^\top{\color{blue}\boldsymbol{V}}_n^{-1} ({\boldsymbol{\beta}}-{\boldsymbol{m}}_n) +{\boldsymbol{y}}^\top{\boldsymbol{y}}+{\boldsymbol{m}}_0^\top{\color{blue}\boldsymbol{V}}_0^{-1}{\boldsymbol{m}}_0 -{\boldsymbol{m}}_n^\top{\color{blue}\boldsymbol{V}}_n^{-1}{\boldsymbol{m}}_n. \end{align*} The first line has the geometric reading

\lVert{\boldsymbol{y}}-{\color{blue}\boldsymbol{X}}{\boldsymbol{\beta}}\rVert_2^2 +\lVert{\boldsymbol{\beta}}-{\boldsymbol{m}}_0\rVert_{{\color{blue}\boldsymbol{V}}_0^{-1}}^2.

It adds squared data-space misfit to squared distance from the prior centre, measured in the prior-precision geometry. After completing the square, the coefficient-dependent term on the final line is \lVert{\boldsymbol{\beta}}-{\boldsymbol{m}}_n\rVert_{{\color{blue}\boldsymbol{V}}_n^{-1}}^2. Therefore, with \begin{align*} a_n&=a_0+\frac n2,\\ b_n&=b_0+\frac12\left( {\boldsymbol{y}}^\top{\boldsymbol{y}} +{\boldsymbol{m}}_0^\top{\color{blue}\boldsymbol{V}}_0^{-1}{\boldsymbol{m}}_0 -{\boldsymbol{m}}_n^\top{\color{blue}\boldsymbol{V}}_n^{-1}{\boldsymbol{m}}_n \right), \end{align*} the posterior remains normal–inverse-Gamma.

Theorem 1 (Normal–inverse-Gamma conjugacy) {\boldsymbol{\beta}}\mid\sigma^2,{\boldsymbol{y}} \sim\operatorname{N}_p({\boldsymbol{m}}_n,\sigma^2{\color{blue}\boldsymbol{V}}_n), \qquad \sigma^2\mid{\boldsymbol{y}}\sim\operatorname{InvGamma}(a_n,b_n).

Article: Normal–inverse-Gamma conjugacy →

A centred normal prior also explains ridge regularisation. If \sigma^2 is fixed and {\boldsymbol{\beta}}\sim\operatorname{N}_p({\boldsymbol{0}},\tau^2{\color{blue}\boldsymbol{I}}_p), then \begin{align*} -2\sigma^2\log p({\boldsymbol{\beta}}\mid{\boldsymbol{y}}) &=\operatorname{SSE}({\boldsymbol{\beta}}) +\frac{\sigma^2}{\tau^2}\lVert{\boldsymbol{\beta}}\rVert^2 +\text{a constant}\\ &=\lVert{\boldsymbol{y}}-{\color{blue}\boldsymbol{X}}{\boldsymbol{\beta}}\rVert_2^2 +\frac{\sigma^2}{\tau^2}\lVert{\boldsymbol{\beta}}\rVert_2^2 +\text{the same constant}. \end{align*} The MAP therefore minimises a ridge objective: Euclidean data-space error is balanced against squared Euclidean distance of the coefficient vector from the prior centre, here {\boldsymbol{0}}. A Laplace prior similarly produces an L^1 penalty related to the lasso. MAP keeps only the posterior mode; full Bayes retains coefficient uncertainty and averages predictions over the posterior.

In machine learning, priors are often introduced mainly as regularisers. That is a legitimate computational use, but it changes the emphasis: the MAP acts like a penalised estimate, while the rest of the posterior may be ignored. For statistical inference the posterior is reported, rather than only a penalised point estimate, because coefficient uncertainty and its dependence feed directly into predictions.

Used in

See also

Normal hierarchical shrinkage · Ridge regression · Maximum a posteriori estimate · Lasso · Regularisation and weight decay · Roughness penalty · Cubic smoothing spline · Effective degrees of freedom

Glossary and index