Newton–Raphson method
Topic of Logistic regression, part of Statistical learning · Also in Introduction to generalised linear models
Newton–Raphson method
| Type | Topic |
| Taught in | Logistic regression · Introduction to generalised linear models |
| Part | Statistical learning · Generalised linear models |
An iteration for solving U(\theta)=0: \theta^{(k+1)}=\theta^{(k)}-H(\theta^{(k)})^{-1}U(\theta^{(k)}), with H the Hessian of the log-likelihood. Fisher scoring replaces H by minus the expected information. For a canonical link the two coincide, and either one is iteratively reweighted least squares.
In Logistic regression
From Logistic regression
Newton–Raphson updates
\beta\leftarrow\beta+(X^\top W X)^{-1}X^\top(y-p).
In Introduction to generalised linear models
Fix a current value \boldsymbol{\beta}^{(0)}. Evaluate the mean, the variance and the slope of the mean at that value, and write \mu_i^{(0)}, V_i^{(0)} and \mu'_{(0),i} for those three numbers at observation i. The only unknown left in Equation (11) in Introduction to generalised linear models is the next \boldsymbol{\beta}. Replace the mean by its tangent at the current linear predictor, \mu({\boldsymbol{X}}_i^\top\boldsymbol{\beta}) \approx \mu_i^{(0)}+\mu'_{(0),i}\,{\boldsymbol{X}}_i^\top\bigl(\boldsymbol{\beta}-\boldsymbol{\beta}^{(0)}\bigr). Substitute that approximation into Equation (11) in Introduction to generalised linear models. The equation becomes linear in the unknown step \boldsymbol{\beta}-\boldsymbol{\beta}^{(0)}: \sum_{i=1}^n\frac{y_i-\mu_i^{(0)}}{V_i^{(0)}}\,\mu'_{(0),i}\,{\boldsymbol{X}}_i = \sum_{i=1}^n\frac{\bigl(\mu'_{(0),i}\bigr)^2}{V_i^{(0)}}\,{\boldsymbol{X}}_i{\boldsymbol{X}}_i^\top\bigl(\boldsymbol{\beta}-\boldsymbol{\beta}^{(0)}\bigr). Name the two sides. The working score and the working information at \boldsymbol{\beta}^{(0)} are \begin{align*} {\boldsymbol{U}}^{(0)} &=\sum_{i=1}^n\frac{y_i-\mu_i^{(0)}}{V_i^{(0)}}\,\mu'_{(0),i}\,{\boldsymbol{X}}_i,\\ {\color{blue}\boldsymbol{I}}^{(0)} &=\sum_{i=1}^n\frac{\bigl(\mu'_{(0),i}\bigr)^2}{V_i^{(0)}}\,{\boldsymbol{X}}_i{\boldsymbol{X}}_i^\top. \end{align*} The update is one step of weighted least squares, \boldsymbol{\beta}^{(1)}=\boldsymbol{\beta}^{(0)}+\bigl({\color{blue}\boldsymbol{I}}^{(0)}\bigr)^{-1}{\boldsymbol{U}}^{(0)}. Iterate until \boldsymbol{\beta} stops moving. The matrix {\color{blue}\boldsymbol{I}} is the expected negative Hessian of \ell, which is the Fisher information. The algorithm is Newton–Raphson with that expected second derivative in place of the observed one, and for that reason it is called Fisher scoring. It is the same algorithm for every exponential family we put in the random component. A model outside that class does not inherit it; you would write a separate optimiser.
Used in
See also
Likelihood function · Fisher scoring · Fisher information · Canonical link · Iteratively reweighted least squares · Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities