Fisher scoring
Section of Introduction to generalised linear models, part of Generalised linear models
Fisher scoring
| Type | Section |
| Taught in | Introduction to generalised linear models |
| Part | Generalised linear models |
| Questions using it | 2 |
Fix a current value \boldsymbol{\beta}^{(0)}. Evaluate the mean, the variance and the slope of the mean at that value, and write \mu_i^{(0)}, V_i^{(0)} and \mu'_{(0),i} for those three numbers at observation i. The only unknown left in Equation (11) in Introduction to generalised linear models is the next \boldsymbol{\beta}. Replace the mean by its tangent at the current linear predictor, \mu({\boldsymbol{X}}_i^\top\boldsymbol{\beta}) \approx \mu_i^{(0)}+\mu'_{(0),i}\,{\boldsymbol{X}}_i^\top\bigl(\boldsymbol{\beta}-\boldsymbol{\beta}^{(0)}\bigr). Substitute that approximation into Equation (11) in Introduction to generalised linear models. The equation becomes linear in the unknown step \boldsymbol{\beta}-\boldsymbol{\beta}^{(0)}: \sum_{i=1}^n\frac{y_i-\mu_i^{(0)}}{V_i^{(0)}}\,\mu'_{(0),i}\,{\boldsymbol{X}}_i = \sum_{i=1}^n\frac{\bigl(\mu'_{(0),i}\bigr)^2}{V_i^{(0)}}\,{\boldsymbol{X}}_i{\boldsymbol{X}}_i^\top\bigl(\boldsymbol{\beta}-\boldsymbol{\beta}^{(0)}\bigr). Name the two sides. The working score and the working information at \boldsymbol{\beta}^{(0)} are \begin{align*} {\boldsymbol{U}}^{(0)} &=\sum_{i=1}^n\frac{y_i-\mu_i^{(0)}}{V_i^{(0)}}\,\mu'_{(0),i}\,{\boldsymbol{X}}_i,\\ {\color{blue}\boldsymbol{I}}^{(0)} &=\sum_{i=1}^n\frac{\bigl(\mu'_{(0),i}\bigr)^2}{V_i^{(0)}}\,{\boldsymbol{X}}_i{\boldsymbol{X}}_i^\top. \end{align*} The update is one step of weighted least squares, \boldsymbol{\beta}^{(1)}=\boldsymbol{\beta}^{(0)}+\bigl({\color{blue}\boldsymbol{I}}^{(0)}\bigr)^{-1}{\boldsymbol{U}}^{(0)}. Iterate until \boldsymbol{\beta} stops moving. The matrix {\color{blue}\boldsymbol{I}} is the expected negative Hessian of \ell, which is the Fisher information. The algorithm is Newton–Raphson with that expected second derivative in place of the observed one, and for that reason it is called Fisher scoring. It is the same algorithm for every exponential family we put in the random component. A model outside that class does not inherit it; you would write a separate optimiser.
For the binomial this rarely takes more than five or six iterations, and the start does not matter much. The cholesterol fit below takes four. A first reading does not require the iteration by hand; working a few steps by hand is still a reasonable exercise. Either way, the update is Equation (12) in Introduction to generalised linear models, and a GLM fit is that update and not a closed form.
Used in
- Taught in Introduction to generalised linear models
- Also used in Standard errors, and models for counts · Logistic regression
- Questions that use it (2) — Question 19 (answer) · C19
- All 9 questions on Introduction to generalised linear models
See also
Newton–Raphson method · Exponential family · Mean and variance of an exponential family · Generalised linear model · Canonical link · What a regression statement is hiding · Three examples the linear model does not fit · Runs in the past week