Score and Hessian of logistic regression
Section of Logistic regression, part of Statistical learning
Score and Hessian of logistic regression
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
There is no closed form for the maximiser, so the next step is the derivative. Differentiate the reduced log-likelihood with \partial\eta_i/\partial\beta=x_i and
\frac{\partial}{\partial\eta_i}\log(1+e^{\eta_i})=\frac{e^{\eta_i}}{1+e^{\eta_i}}=p_i.
The score, a vector of length p+1, is
\frac{\partial\ell}{\partial\beta} =\sum_{i=1}^n(y_i-p_i)\,x_i =X^\top(y-p),
where X is the n\times(p+1) design matrix, y is the vector of labels, and p is the vector of fitted probabilities. Differentiating again, \partial p_i/\partial\beta=p_i(1-p_i)\,x_i by the logistic derivative \sigma'(\eta_i)=p_i(1-p_i). The Hessian, of size (p+1)\times(p+1), is therefore
\frac{\partial^2\ell}{\partial\beta\,\partial\beta^\top} =-\sum_{i=1}^n p_i(1-p_i)\,x_i x_i^\top =-X^\top W X,
where W=\mathrm{diag}\bigl(p_i(1-p_i)\bigr). The identity p_i(1-p_i)=\sigma'(\eta_i) is what puts the weights on the diagonal of W.
Theorem 1 (A finite stationary point is a global maximiser) At every finite coefficient vector the weights p_i(1-p_i) are positive. The Hessian -X^\top W X is negative semi-definite, so \ell is concave. If X has full column rank, W is positive definite on the column space of X and the Hessian is negative definite, so \ell is strictly concave. Any finite stationary point is then the unique global maximiser.
Concavity does not prove that a finite stationary point exists, and it does not prove that an unguarded numerical iteration will find one.
Complete separation is a concrete failure. Suppose there is a direction v such that v^\top x_i>0 whenever y_i=1 and v^\top x_i<0 whenever y_i=0. Move the coefficient vector farther along that direction: the probabilities approach the observed zeroes and ones, and the log-likelihood approaches its upper bound 0 without attaining it at a finite vector. The bound holds because each Bernoulli factor is at most 1, so L(\beta)\le 1 and \ell(\beta)\le 0, with equality only in the limit where every fitted probability is exactly the observed label. The likelihood stays bounded. The coefficients diverge. Quasi-complete separation can also prevent a finite maximum likelihood estimate. Check convergence and separation before interpreting standard errors.
Used in
See also
Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Iteratively reweighted least squares · Interpretation and the decision boundary · Linear discriminant analysis implies a logistic model