Iteratively reweighted least squares
Section of Logistic regression, part of Statistical learning
Iteratively reweighted least squares
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
Newton–Raphson updates
\beta\leftarrow\beta+(X^\top W X)^{-1}X^\top(y-p).
The update is the normal equation of a weighted least-squares problem, so each iteration can be computed by a regression routine. Define the working response
z=X\beta+W^{-1}(y-p).
Then
\begin{align*} X^\top W z &=X^\top W X\,\beta+X^\top(y-p),\\ (X^\top W X)^{-1}X^\top W z &=\beta+(X^\top W X)^{-1}X^\top(y-p), \end{align*}
which is exactly the Newton step. Each step is a weighted least-squares fit of z on X, with weights p_i(1-p_i). The weights are functions of the current \beta, through p_i, and they are recomputed every iteration. This is iteratively reweighted least squares. It typically converges in a handful of iterations.
For logistic regression, Fisher scoring and Newton–Raphson coincide, because the Hessian -X^\top W X does not depend on y except through the current probabilities, and those probabilities are functions of \beta and X. The expected Hessian and the observed Hessian are the same matrix once the variance of a Bernoulli trial is taken to be p_i(1-p_i). The working response is a device that makes one Newton step look like weighted least squares. It is recomputed from the current fit, and it is not itself the binary observation.
Example 1 (Newton’s method on four points) Question. Fit \log(p/(1-p))=\beta_0+\beta_1 x to x=(0,1,2,3) and y=(0,1,0,1).
Solution. The design matrix has rows (1,x_i). Start at \beta=0: every p_i=1/2, W=(1/4)I, and y-p=(-1/2,1/2,-1/2,1/2). The score is
X^\top(y-p) =\begin{pmatrix} -1/2+1/2-1/2+1/2\\ 0\cdot(-1/2)+1\cdot(1/2)+2\cdot(-1/2)+3\cdot(1/2) \end{pmatrix} =\begin{pmatrix}0\\1\end{pmatrix}.
Also X^\top X=\begin{pmatrix}4&6\\6&14\end{pmatrix}, so
X^\top W X=\frac14 X^\top X=\begin{pmatrix}1&1.5\\1.5&3.5\end{pmatrix}.
The determinant is 3.5-2.25=1.25, and
(X^\top W X)^{-1} =\frac{1}{1.25}\begin{pmatrix}3.5&-1.5\\-1.5&1\end{pmatrix}.
The Newton step is that inverse times (0,1)^\top,
\Delta\beta=\frac{1}{1.25}\begin{pmatrix}-1.5\\1\end{pmatrix} =\begin{pmatrix}-1.2\\0.8\end{pmatrix}.
After this first step \beta=(-1.2,0.8)^\top. The log-likelihood has risen from 4\log(1/2)=-2.7726 to -2.3526. Two more steps give (-1.356,0.904)^\top and (-1.3623,0.9082)^\top, with \ell=-2.3475, after which the score is zero to four decimal places. The asymptotic standard errors, the square roots of the diagonal of (X^\top W X)^{-1} at the solution, are 1.97 and 1.09. With four observations, nothing is distinguishable from zero at the usual scale.
Used in
See also
Newton–Raphson method · Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression · Interpretation and the decision boundary