Cramér–Rao lower bound
Section of Information and bounds, part of Statistical inference
Cramér–Rao lower bound
| Type | Section |
| Taught in | Information and bounds |
| Part | Statistical inference |
| Questions using it | 2 |
Let T be any unbiased estimator of a differentiable function g(\theta), with finite variance. Assume the support of f does not depend on \theta, and that \mathbb{E}_\theta T=g(\theta) and \int L(\theta)\,dx=1 may both be differentiated under the integral. Then
\operatorname{Var}_\theta(T)\ge\frac{\{g'(\theta)\}^2}{\mathcal{I}(\theta)}.
The right-hand side is the Cramér–Rao lower bound, also called the minimum variance bound. For the parameter itself, g(\theta)=\theta, the bound is 1/\mathcal{I}(\theta).
The proof is Cauchy–Schwarz applied to the score. Unbiasedness says \mathbb{E}_\theta T=g(\theta). Differentiate under the integral, as the regularity conditions allow:
\mathbb{E}_\theta\bigl\{T\,S(\theta)\bigr\}=g'(\theta).
The score has mean zero, so the left side is the covariance of T and S(\theta). Cauchy–Schwarz gives
\{g'(\theta)\}^2=\bigl\{\operatorname{Cov}(T,S(\theta))\bigr\}^2\le\operatorname{Var}(T)\,\operatorname{Var}\{S(\theta)\}=\operatorname{Var}(T)\,\mathcal{I}(\theta),
which is the bound.
For a normal mean with known variance, \mathcal{I}(\mu)=n/\sigma^2 and the bound is \sigma^2/n. The sample mean has exactly that variance, so no unbiased estimator of \mu has a smaller variance at any \mu. For a Bernoulli mean the bound is \theta(1-\theta)/n, and the sample proportion has exactly that variance. For a Poisson mean the bound is \lambda/n, and \overline X meets it.
The bound is a statement about unbiased estimators. A biased estimator can have smaller mean squared error. The two variance estimators in Estimation are the example: the divisor n beats the unbiased divisor n-1 on mean squared error, and it does not attain a bound that was never claimed for it.
The bound is also only as good as the regularity conditions. For the uniform distribution on (0,\theta), M=\max X_i has variance of order 1/n^2, smaller than any bound of order 1/n. The derivative-under-the-integral step fails because the upper limit of the integral depends on \theta. A small variance in that problem is not a counter-example to the inequality. It is a case the inequality does not cover.
For a normal mean the score is proportional to \overline X-\mu, the bound is attained by \overline X, and the most powerful test of a larger mean rejects for large \overline X. The critical value is a normal quantile because that estimator is exactly normal. The test is derived on Most powerful tests, and the picture of the two densities is there.
Used in
- Taught in Information and bounds
- Also used in Rao–Blackwell and UMVU · Large-sample estimation
- Questions that use it (2) — Information for a Poisson mean · Where the bound does not apply
- All 2 questions on Information and bounds
See also
Fisher information · Score function · Rao–Blackwell theorem · Uniformly minimum-variance unbiased estimator