Maximum likelihood under the wrong model
Section of Large-sample estimation, part of Statistical inference
Maximum likelihood under the wrong model
| Type | Section |
| Taught in | Large-sample estimation |
| Part | Statistical inference |
Suppose the true density is f_0, and the fitted family \{f(x;\theta):\theta\in\Omega\} does not contain f_0. The maximum likelihood estimator still maximises \ell(\theta)=\sum\log f(X_j;\theta). The average n^{-1}\ell(\theta) converges to \mathbb{E}_0\{\log f(X;\theta)\}, the expected log density under the true distribution. A value \theta^\star that uniquely maximises that expectation is the same value that minimises the Kullback–Leibler divergence
\int f_0(x)\log\frac{f_0(x)}{f(x;\theta)}\,dx.
So \widehat\theta converges to the parameter in the fitted family closest to the truth in that divergence, not to a parameter the true distribution does not have. The estimator remains meaningful. The standard error from 1/\mathcal{I}(\widehat\theta) does not. That standard error treats the fitted family as true, so it misses the extra variability of a misspecified model. A variance estimator that uses the empirical variance of the score, rather than the model information, is the repair used later for sandwich standard errors. The point here is the target: maximum likelihood fits the member of the family nearest the true distribution, and “nearest” means Kullback–Leibler.
A practical version of the same warning is the choice between a Poisson likelihood-ratio test and a t test for a mean count. If the counts really are Poisson with mean \lambda, the likelihood-ratio test of \lambda=\lambda_0 against \lambda>\lambda_0 rejects for a large sum, and it is the most powerful test of that pair. If the counts are not Poisson, that test need not have type I error near the nominal level. The one-sample t test of the same mean has the wrong likelihood, so it is less powerful when the Poisson model is right, and its type I error stays near the nominal level for a wide range of distributions with a finite variance, because it is a pivot built from the central limit theorem. Using the efficient test is right when the model is right. The asymptotic relative efficiency is the comparison under the model. It is not a guarantee after the model is dropped.
For large n, compare consistent asymptotically normal estimators by their limiting variances. In a regular model the maximum likelihood estimator achieves the Cramér–Rao bound in that limit. A smooth function of it is handled by multiplying the variance by the square of the derivative. If the family is wrong, the same estimator converges to the closest member of the family, and the model-based standard error is no longer the right uncertainty.
Used in
See also
Asymptotic relative efficiency · Asymptotic normality of maximum likelihood · Delta method