Score function

Topic of Introduction to generalised linear models, part of Generalised linear models · Also in Logistic regression

Score function

Type Topic
Taught in Introduction to generalised linear models · Logistic regression
Part Generalised linear models · Statistical learning
Questions using it 5

The gradient of the log-likelihood, U(\theta)=\partial\ell/\partial\theta. The maximum likelihood estimate solves U(\hat\theta)=0. At the true parameter the score has mean zero and variance equal to the Fisher information.

In Introduction to generalised linear models

From Introduction to generalised linear models

Differentiate with respect to \boldsymbol{\beta}. The chain rule runs through p_i and then through the linear predictor. Write \mu'(\eta) for the derivative of the mean function with respect to \eta={\boldsymbol{X}}^\top\boldsymbol{\beta}, so \partial p_i/\partial\boldsymbol{\beta}=\mu'({\boldsymbol{X}}_i^\top\boldsymbol{\beta})\,{\boldsymbol{X}}_i. Then \begin{align*} \frac{\partial\ell}{\partial\boldsymbol{\beta}} &=\sum_{i=1}^n\left\{\frac{y_i}{p_i}-\frac{1-y_i}{1-p_i}\right\}\mu'({\boldsymbol{X}}_i^\top\boldsymbol{\beta})\,{\boldsymbol{X}}_i\\ &=\sum_{i=1}^n\frac{y_i-p_i}{p_i(1-p_i)}\,\mu'({\boldsymbol{X}}_i^\top\boldsymbol{\beta})\,{\boldsymbol{X}}_i. \end{align*} Set the score equal to zero. The denominator p_i(1-p_i) is \mathrm{Var}(Y_i\mid {\boldsymbol{X}}_i). The same algebra, with the variance of the family in that slot, is the score equation for every GLM we have written down: \sum_{i=1}^n\frac{y_i-\mu_i}{\mathrm{Var}(Y_i\mid {\boldsymbol{X}}_i)}\,\mu'({\boldsymbol{X}}_i^\top\boldsymbol{\beta})\,{\boldsymbol{X}}_i={\boldsymbol{0}}. You should be able to recover Equation (11) in Introduction to generalised linear models from the Poisson log-likelihood and from the gamma log-likelihood. That is an exercise, not a new idea. For the normal identity-link model, \mu_i={\boldsymbol{X}}_i^\top\boldsymbol{\beta}, the derivative \mu' is 1, and the variance is the constant \sigma^2. Equation Equation (11) in Introduction to generalised linear models collapses to \frac{1}{\sigma^2}\sum_{i=1}^n (y_i-{\boldsymbol{X}}_i^\top\boldsymbol{\beta})\,{\boldsymbol{X}}_i={\boldsymbol{0}}, or {\color{blue}\boldsymbol{X}}^\top({\boldsymbol{y}}-{\color{blue}\boldsymbol{X}}\boldsymbol{\beta})={\boldsymbol{0}} once the rows {\boldsymbol{X}}_i^\top are stacked into the matrix {\color{blue}\boldsymbol{X}}. Those are the normal equations. The GLM score is the same residual, {\boldsymbol{y}}-\boldsymbol{\mu}, reweighted by the variance and by the slope of the mean function.

In Logistic regression

From Logistic regression

There is no closed form for the maximiser, so the next step is the derivative. Differentiate the reduced log-likelihood with \partial\eta_i/\partial\beta=x_i and

\frac{\partial}{\partial\eta_i}\log(1+e^{\eta_i})=\frac{e^{\eta_i}}{1+e^{\eta_i}}=p_i.

The score, a vector of length p+1, is

\frac{\partial\ell}{\partial\beta} =\sum_{i=1}^n(y_i-p_i)\,x_i =X^\top(y-p),

where X is the n\times(p+1) design matrix, y is the vector of labels, and p is the vector of fitted probabilities. Differentiating again, \partial p_i/\partial\beta=p_i(1-p_i)\,x_i by the logistic derivative \sigma'(\eta_i)=p_i(1-p_i). The Hessian, of size (p+1)\times(p+1), is therefore

\frac{\partial^2\ell}{\partial\beta\,\partial\beta^\top} =-\sum_{i=1}^n p_i(1-p_i)\,x_i x_i^\top =-X^\top W X,

where W=\mathrm{diag}\bigl(p_i(1-p_i)\bigr). The identity p_i(1-p_i)=\sigma'(\eta_i) is what puts the weights on the diagonal of W.

Theorem 1 (A finite stationary point is a global maximiser) At every finite coefficient vector the weights p_i(1-p_i) are positive. The Hessian -X^\top W X is negative semi-definite, so \ell is concave. If X has full column rank, W is positive definite on the column space of X and the Hessian is negative definite, so \ell is strictly concave. Any finite stationary point is then the unique global maximiser.

Article: A finite stationary point is a global maximiser →

Concavity does not prove that a finite stationary point exists, and it does not prove that an unguarded numerical iteration will find one.

Complete separation is a concrete failure. Suppose there is a direction v such that v^\top x_i>0 whenever y_i=1 and v^\top x_i<0 whenever y_i=0. Move the coefficient vector farther along that direction: the probabilities approach the observed zeroes and ones, and the log-likelihood approaches its upper bound 0 without attaining it at a finite vector. The bound holds because each Bernoulli factor is at most 1, so L(\beta)\le 1 and \ell(\beta)\le 0, with equality only in the limit where every fitted probability is exactly the observed label. The likelihood stays bounded. The coefficients diverge. Quasi-complete separation can also prevent a finite maximum likelihood estimate. Check convergence and separation before interpreting standard errors.

Used in

See also

Likelihood function · Maximum likelihood · Variance · Fisher information · Exponential family · Mean and variance of an exponential family · Generalised linear model · Canonical link

Glossary and index