Asymptotic normality of maximum likelihood

Section of Large-sample estimation, part of Statistical inference

Type	Section
Taught in	Large-sample estimation
Part	Statistical inference

Let X_1,\ldots,X_n be independent with density or mass function f(x;\theta), and let i(\theta) be the information in one observation. Assume the regularity used for the information identities, strengthened by a third-derivative bound: on a neighbourhood of the true value \theta_0,

\Bigl|\frac{\partial^3}{\partial\theta^3}\log f(x;\theta)\Bigr|\le M(x)

with \mathbb{E}_{\theta_0} M(X) finite. Then a consistent root \widehat\theta_n of the likelihood equation S(\theta)=0 is asymptotically normal and efficient,

\sqrt n\,(\widehat\theta_n-\theta_0)\xrightarrow{d}\operatorname{N}\bigl(0,1/i(\theta_0)\bigr).

Equivalently, \widehat\theta_n is approximately \operatorname{N}(\theta_0,1/\mathcal{I}(\theta_0)) with \mathcal{I}(\theta_0)=n\,i(\theta_0). Efficient means that the limiting variance equals the Cramér–Rao bound. No other consistent, asymptotically normal estimator has a smaller limiting variance under these conditions, except for terms that \sqrt n washes out.

The expansion is Taylor’s formula for the score at \widehat\theta_n, about the true value. Because S(\widehat\theta_n)=0,

0=S(\theta_0)+(\widehat\theta_n-\theta_0)S'(\theta_0)+\tfrac12(\widehat\theta_n-\theta_0)^2 S'''(\theta_n^*),

with \theta_n^* between \widehat\theta_n and \theta_0. Rearrange:

\sqrt n\,(\widehat\theta_n-\theta_0)=\frac{n^{-1/2}S(\theta_0)}{-n^{-1}S'(\theta_0)-(2n)^{-1}(\widehat\theta_n-\theta_0)S'''(\theta_n^*)}.

The numerator is \sqrt n times the average of independent mean-zero scores. Each score has variance i(\theta_0), so the central limit theorem makes the numerator \operatorname{N}(0,i(\theta_0)) in the limit. The first term in the denominator converges in probability to i(\theta_0), because -n^{-1}\partial^2\ell/\partial\theta^2 is the average of the observed information contributions and that average converges to its expectation. The remainder has (\widehat\theta_n-\theta_0) in it, which tends to zero in probability by consistency, times an average that stays bounded in probability by the integrable majorant M. The denominator tends to i(\theta_0), and the ratio tends to \operatorname{N}(0,1/i(\theta_0)).

Two cautions belong with the theorem. Consistency of some root is a separate argument: the likelihood equation can have more than one root, and the data do not by themselves label which root is the consistent one. When the likelihood has a single maximum in an open parameter interval, that maximum is the root the theorem describes. A parameter on the boundary, such as a variance constrained to be positive and truly zero, is not this theorem. The uniform(0,\theta) maximum is not this theorem either, because the support depends on \theta and the score expansion does not apply. There \widehat\theta=M is still consistent, and it converges faster than 1/\sqrt n.

The normal approximation is how an interval is built when no exact pivot exists. For a discrete family the exact distribution of a statistic still depends on the unknown parameter after standardisation, so there is no pivot. Replacing \mathcal{I}(\theta_0) by \mathcal{I}(\widehat\theta) or by the observed information I(\widehat\theta) gives the standard error, and

\widehat\theta\pm z_{1-\alpha/2}\big/\sqrt{\mathcal{I}(\widehat\theta)}

is the Wald interval. The interval for a proportion is this formula for the Bernoulli information n/\{\widehat p(1-\widehat p)\}.

Solving S(\theta)=0 when there is no closed root is one Newton step from a current value \theta^{(k)}:

\theta^{(k+1)}=\theta^{(k)}+I(\theta^{(k)})^{-1}S(\theta^{(k)}),

because the observed information was defined as minus the second derivative, and the linear approximation to the score is S(\theta)\approx S(\theta^{(k)})-I(\theta^{(k)})(\theta-\theta^{(k)}). Fisher scoring uses \mathcal{I}(\theta^{(k)}) in place of I(\theta^{(k)}). For a binomial or Poisson canonical link those two matrices agree. One step from an estimator that is already within 1/\sqrt n of \theta_0 is enough to inherit the limiting normal law. Further steps matter for the numerical value, not for the asymptotic variance.

# Newton steps for a Poisson mean are unnecessary (the root is the sample
# mean), but the update is easy to see. Score n(xbar - lambda)/lambda,
# observed information (sum x)/lambda^2.
x <- c(0, 2, 1, 4, 1, 0, 3); n <- length(x); s <- sum(x)
lam <- 1
for (k in 1:5) {
  score <- s / lam - n
  info <- s / lam^2
  lam <- lam + score / info
}
c(lam, mean(x))

Used in

Taught in Large-sample estimation
All 2 questions on Large-sample estimation

Used in

See also