Deviance

Section of Paired comparisons, and how far a fit is from the data, part of Generalised linear models

Deviance

Type	Section
Taught in	Paired comparisons, and how far a fit is from the data
Part	Generalised linear models
Questions using it	4

McCullagh and Nelder’s sentence is the right one: fitting a model replaces the data y_1,\ldots,y_n by fitted means \hat\mu_1,\ldots,\hat\mu_n that depend on fewer parameters than there are observations. A model with n parameters can repeat the sample exactly and then cannot predict anything else. The saturated fit is the baseline you compare with, not the fit you report.

The deviance is twice the drop in log-likelihood from the saturated fit to the model you actually fitted: D({\boldsymbol{y}};\hat{\boldsymbol{\mu}})=2\bigl\{\ell({\boldsymbol{y}};{\boldsymbol{y}})-\ell(\hat{\boldsymbol{\mu}};{\boldsymbol{y}})\bigr\}, up to the dispersion, which scales it. On the observation scale a residual of 5 is not the same thing for a Poisson mean of 2 and a Poisson mean of 50. The likelihood already knows that. For the families we have been using, the deviance contributions add up as follows. For the normal, \phi=\sigma^2, D=\sum_{i=1}^n(y_i-\hat\mu_i)^2, the residual sum of squares, because the factor of 2 cancels the \tfrac12 in the normal log-density. For the Poisson, D=2\sum_{i=1}^n\left\{y_i\log\frac{y_i}{\hat\mu_i}-(y_i-\hat\mu_i)\right\}. For a binomial count of y_i successes out of m_i, with fitted mean count m_i\hat p_i, D=2\sum_{i=1}^n\left\{y_i\log\frac{y_i}{m_i\hat p_i}+(m_i-y_i)\log\frac{m_i-y_i}{m_i(1-\hat p_i)}\right\}. The gamma deviance, in the mean-and-variance parameterisation \mathrm{Var}=\phi\mu^2, is D=2\sum_{i=1}^n\left\{-\log\frac{y_i}{\hat\mu_i}+\frac{y_i-\hat\mu_i}{\hat\mu_i}\right\}. These are not interchangable across families. A smaller deviance is a closer fit only inside one family. Two non-nested models, such as a logit and a probit, have no likelihood-ratio test. AIC is the comparison that remains: deviance plus twice the number of parameters, up to the usual constants.

Three links, one cholesterol slope

Main article: Three links, one cholesterol slope

Sex drops out, as we already found. Cholesterol alone, three links:

chol_file <- "data/cholest.csv"
if (!file.exists(chol_file)) {
  stop("Could not find data/cholest.csv. Set the working directory to the folder that contains it.")
}
chol <- read.csv(chol_file)
for (lnk in c("logit", "probit", "cloglog")) {
  fit <- glm(disease ~ cholesterol, data = chol, family = binomial(link = lnk))
  cat(lnk, "null", fit$null.deviance, "on", fit$df.null, "df,",
      "residual", fit$deviance, "on", fit$df.residual, "df,",
      "AIC", AIC(fit), "\n")
}

binomial logit:    null 137.63 on 99 df,  residual 114.04 on 98 df,  AIC 118.04
binomial probit:   null 137.63 on 99 df,  residual 114.12 on 98 df,  AIC 118.12
binomial cloglog:  null 137.63 on 99 df,  residual 111.90 on 98 df,  AIC 115.90

The null deviance does not depend on the link. It is the intercept-only model, one probability for everyone. Logit and probit are not separable on this sample.

Figure 1: The same cholesterol data under three links. The logit and the probit sit on each other. Complementary log-log stays a little lower through the middle and then climbs more steeply through the high-cholesterol cases.

In Figure the solid and dashed curves sit on each other. Complementary log-log has the smaller deviance and the smaller AIC. It rises more slowly and then more steeply, and it picks up the high-cholesterol ones a little better. We cannot test that improvement. The models are not nested. There is no constraint on the logit coefficients that turns the fitted curve into a probit.

Used in

• Taught in Paired comparisons, and how far a fit is from the data
• Also used in Chi-squared tests · Introduction to generalised linear models · Logistic regression · Ensembles, random forests, and boosting · Support vector machines · High-dimensional analysis · How to use the final preparation
• Questions that use it (4) — Reading the deviance (answer) · Q34 · Question 19 (answer) · C29
• All 4 questions on Paired comparisons, and how far a fit is from the data

See also

Residual deviance · Null deviance · Scaled deviance · Saturated model · Deviance residual · Akaike information criterion · Analysis of deviance · Bradley–Terry model · Three links, one cholesterol slope · Residuals in a GLM · Pearson residual

Glossary and index