Underdispersion
Topic of Standard errors, and models for counts, part of Generalised linear models · Also in Underdispersion, positive responses, and quasi-likelihood
Underdispersion
| Type | Topic |
| Taught in | Standard errors, and models for counts · Underdispersion, positive responses, and quasi-likelihood |
| Part | Generalised linear models |
| Questions using it | 1 |
Variance smaller than the model implies, for counts a variance below the mean. It is rarer than overdispersion. A quasi-Poisson model with \phi<1 or a Conway–Maxwell–Poisson model can describe it; a negative binomial cannot.
In Standard errors, and models for counts
Unless there is a strong reason to insist on a pure Poisson process, use
\mathrm{Var}(Y_i\mid {\boldsymbol{X}}_i)=\phi\mu_i=\phi\exp({\boldsymbol{X}}_i^\top\boldsymbol{\beta}).
Here \phi>1 is overdispersion and \phi<1 is underdispersion. The case \phi=1 is the Poisson. This is a quasi-Poisson model. The variance function is a legitimate description of spread. Inside the exponential family, a variance exactly proportional to the mean forces the Poisson, so there is no exponential-family distribution with a free \phi in Equation (4) in Standard errors, and models for counts. You cannot simulate a new count from a quasi-Poisson specification alone. We will use it as a description. A stricter view, which reserves the name model for something one can simulate from, would refuse the name. We will come back to that distinction when we treat quasi-likelihood properly. For fitting, glm only needs the mean and the variance, so family = quasipoisson runs.
In Underdispersion, positive responses, and quasi-likelihood
Cameron and Johansson (1997) recorded the number of takeover bids received by 126 US firms between 1978 and 1985. The covariates split into three groups. Defensive actions by the target: a lawsuit (leglrest), a proposed change in asset structure (rearest), a proposed change in ownership structure (finrest), and an invitation to a friendly third party, a white knight (whtknght). Characteristics of the firm: the bid premium (bidprem), the percentage of stock held by institutions (insthold), book value in billions (size), and its square. And one indicator for Department of Justice intervention (regulatn).
After those covariates, the counts are tightly bunched. A Poisson mean of 7 would happily produce anything from about 3 to 13. These firms, when the fit said 7, mostly produced 6, 7, or 8. That is underdispersion, which neither the Poisson nor the negative binomial can represent. The mean-parameterised Conway–Maxwell–Poisson can.
The R package that fitted it was withdrawn from CRAN in 2022, ahead of a replacement that did not arrive. The fit given here is a MATLAB fit of a log-linear mean. There is no formula interface. You build {\color{blue}\boldsymbol{X}} yourself: a column of ones and the nine covariates, 126 rows. The call returns the coefficient vector, the dispersion \nu, the implied rates \lambda(\mu,\nu), the maximised log-likelihood, and standard errors for \boldsymbol{\beta}. The estimates, standard errors, and ratios are
| term | estimate | std. error | z |
|---|---|---|---|
| intercept | 0.9897 | 0.4352 | 2.27 |
| legal defence | 0.2679 | 0.1228 | 2.18 |
| asset restructure | -0.1732 | 0.1547 | -1.12 |
| ownership change | 0.0677 | 0.1743 | 0.39 |
| white knight | 0.4813 | 0.1317 | 3.66 |
| bid premium | -0.6849 | 0.3075 | -2.23 |
| institutional holding | -0.3679 | 0.3466 | -1.06 |
| size | 0.1793 | 0.0476 | 3.77 |
| size squared | -0.0076 | 0.0025 | -3.05 |
| regulation | -0.0376 | 0.1302 | -0.29 |
The estimated dispersion is \hat\nu=1.754. Because \nu>1, the counts are tighter than a Poisson with the same mean, by the convention of this parameterisation.
The legal-defence coefficient is the easiest sentence. Two firms with the same size, the same premium, and the same other actions, one of which sues and one of which does not, differ by 0.2679 on the log-mean scale. The firm that sues is estimated to receive \exp(0.2679)=1.31 times as many bids. The ratio of estimate to standard error is 2.18, which clears 2 and so sits past the usual two-sided 5% cutoff on the z scale. The same reading applies to every other row. Size enters as a curve: the linear term is positive and the square is negative, so the mean rises with book value and then bends back. A one-line slope for size would hide that bend.
Why the underdispersion is plausible is the same selection story as the mines, in a different market. A private person cannot bid for a large firm. The pool of possible bidders shrinks as the target grows, and for the largest targets the count is often zero or one. Conditionally on the covariates, the count is more deterministic than a Poisson process. The Poisson model is the boundary case \nu=1. The fitted \hat\nu=1.754 lies above that boundary, which is the comparison a test of \nu=1 makes. The point to keep is that the mean is on the same scale as a Poisson log-linear model, so the coefficient 0.2679 means the same kind of thing.
Used in
- Taught in Standard errors, and models for counts · Underdispersion, positive responses, and quasi-likelihood
- Also used in Underdispersion, positive responses, and quasi-likelihood
- Questions that use it (1) — C28
- All 7 questions on Standard errors, and models for counts
See also
Overdispersion · Quasi-Poisson · Negative binomial distribution · Conway–Maxwell–Poisson distribution · Variance · Standard errors in a GLM · Binomial counts in a GLM · Count responses