Inference, deviance, and model comparison

Section of Logistic regression, part of Statistical learning

Inference, deviance, and model comparison

Type Section
Taught in Logistic regression
Part Statistical learning

At a regular finite logistic maximum likelihood estimate, invert X^\top W X and take square roots of its diagonal entries to obtain asymptotic standard errors. The Wald statistic divides an estimated coefficient by its standard error. This is an approximation justified by sampling behaviour under the model, not a guarantee supplied by the numerical inverse. The Wald interval and the deviance comparison both assume that a finite maximum was attained. Under complete separation the coefficients diverge and those standard errors are not meaningful.

A comparison of two nested models uses the likelihood ratio rather than a single coefficient. Deviance, as a comparison of a fitted model with the saturated model, is on Paired comparisons, and how far a fit is from the data. For binary data the saturated model, with a free probability at each case, has log-likelihood 0: a fitted probability of 1 on a label 1, or 0 on a label 0, contributes \log 1=0. The deviance of a fitted model is therefore simply D=-2\ell. The null deviance D_0=-2\ell is the deviance of the model with only an intercept, and the residual deviance D=-2\ell is the deviance of the model with the full \beta. A higher likelihood is a smaller deviance. The drop D_0-D measures the improvement from the explanatory variables, and under the null that those variables have coefficients zero it is approximately \chi^2 with degrees of freedom equal to the number of coefficients added.

Which model to keep can also be decided by AIC or BIC, which penalise the likelihood by the number of parameters rather than by a significance cutoff. Those penalties are on Model selection for stochastic models. For this binomial fit, where the deviance is -2\ell,

\mathrm{AIC}=D+2k,

with k the number of fitted coefficients, intercept included. A categorical explanatory variable with m levels enters as m-1 binary indicators, the dummy-variable coding, with the all-zero pattern as the reference level. Continuous, binary, and categorical predictors are therefore all admissible in the same linear predictor.

Example 1 (Survival after heart failure) The following output is a supplied fit of death during follow-up on age and high-blood-pressure status, for 299 patients. Follow-up time is stored separately, so the outcome is not a common one-year event. The printed coefficients are the numbers to be interpreted. They are not re-estimated here.

Question. Read the odds ratios, the age interval, the deviance drop, and the AIC from the printed fit.

Solution. The supplied summary is

fit <- glm(death ~ age + hbp, data = heart, family = "binomial")
summary(fit)
#> Coefficients:
#>             Estimate Std. Error z value Pr(>|z|)
#> (Intercept) -3.70568    0.71072  -5.214 1.85e-07
#> age          0.04618    0.01110   4.159 3.20e-05
#> hbp          0.26920    0.26487   1.016    0.309
#> Null deviance: 375.35 on 298 degrees of freedom
#> Residual deviance: 354.97 on 296 degrees of freedom
#> AIC: 360.97
#> Number of Fisher Scoring iterations: 4
confint(fit)
#>                  2.5 %      97.5 %
#> (Intercept) -5.14257577 -2.34844333
#> age          0.02483923  0.06851426
#> hbp         -0.25344324  0.78698218

Increasing age by one year at fixed blood-pressure status multiplies the modelled odds by e^{0.04618}=1.0473. Ten years multiply them by e^{0.4618}=1.59. These are odds ratios. If the baseline probability is p and the odds multiplier is a, the new probability is ap/(1-p+ap). The printed age interval, exponentiated, gives an annual odds-ratio interval of approximately (e^{0.02484},e^{0.06851})=(1.025,1.071). For high blood pressure, the interval (e^{-0.2534},e^{0.7870})=(0.78,2.20) includes one. It does not establish that the association is exactly zero.

The deviance reduction is 375.35-354.97=20.38 on two degrees of freedom. The printed AIC is the residual deviance plus twice the number of coefficients,

354.97+2(3)=360.97.

These calculations check the interpretation of the supplied numbers. They are not causal effects, and they are not probabilities of death inside a fixed horizon.

Used in

See also

Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression · Iteratively reweighted least squares · Interpretation and the decision boundary

Glossary and index