Unbiased estimator
Definition taught in Estimation, part of Statistical inference
Unbiased estimator
| Type | Definition |
| Taught in | Estimation |
| Part | Statistical inference |
| Properties | 2 |
| Worked examples | 3 |
An estimator T of \theta is unbiased when \mathbb{E}_\theta T=\theta for every \theta in the parameter space. The bias is \mathbb{E}_\theta T-\theta.
Properties
Moments of the sample mean and variance
Main article: Moments of the sample mean and variance
If X_1,\ldots,X_n are a random sample from a distribution with mean \mu and variance \sigma^2, then
\mathbb{E}\overline X=\mu,\qquad \operatorname{Var}(\overline X)=\frac{\sigma^2}{n},\qquad \mathbb{E} s^2=\sigma^2.
No normality is needed.
Least-squares estimators are unbiased
Main article: Least-squares estimators are unbiased
If \mathbb{E} Y_i=\beta_0+\beta_1x_i for every i, then \mathbb{E}\widehat\beta_1=\beta_1 and \mathbb{E}\widehat\beta_0=\beta_0. No assumption about the error distribution is needed.
Examples
Two variance estimators
Worked example from Estimation
Let X_1,\ldots,X_n be a normal random sample with n\ge 2.
Question. Compare the mean squared errors of s^2 and \widehat\sigma^2=((n-1)/n)s^2 as estimators of \sigma^2.
Solution. By the normal sample theorem, V=(n-1)s^2/\sigma^2 is \chi^2_{n-1}, so \operatorname{Var} V=2(n-1) and
\operatorname{Var}(s^2)=\frac{\sigma^4}{(n-1)^2}\,2(n-1)=\frac{2\sigma^4}{n-1}.
s^2 is unbiased, so this is its mean squared error. For \widehat\sigma^2, the variance is ((n-1)/n)^2 times as large and the bias is -\sigma^2/n:
\operatorname{MSE}(\widehat\sigma^2)=\frac{(n-1)^2}{n^2}\cdot\frac{2\sigma^4}{n-1}+\frac{\sigma^4}{n^2}=\frac{(2n-1)\sigma^4}{n^2}.
Since (2n-1)(n-1)=2n^2-3n+1<2n^2, the biased estimator has the smaller mean squared error for every n and every \sigma^2. With n=10 the two are 0.222\,\sigma^4 and 0.190\,\sigma^4. The unbiased estimator is preferred for other reasons: it is the one whose distribution gives the exact t and chi-squared pivots.
Bias of the two variance estimators
Question page: Bias of the two variance estimators
Let X_1,\ldots,X_n be a random sample with variance \sigma^2, and write s^2 for the bias-corrected sample variance. The maximum likelihood estimator of \sigma^2 in the normal model is \widehat\sigma^2=((n-1)/n)s^2. Find the bias of \widehat\sigma^2, taking as known that \mathbb{E} s^2=\sigma^2.
\mathbb{E}\widehat\sigma^2=\frac{n-1}{n}\mathbb{E} s^2=\frac{n-1}{n}\sigma^2.
The bias is \mathbb{E}\widehat\sigma^2-\sigma^2=-\sigma^2/n.
The expectation of the sample variance
Question page: The expectation of the sample variance
Let X_1,\ldots,X_n be a random sample with mean \mu and variance \sigma^2. Show that \sum_i (X_i-\overline X)^2=\sum_i (X_i-\mu)^2-n(\overline X-\mu)^2, and deduce that \mathbb{E} s^2=\sigma^2.
Write X_i-\overline X=(X_i-\mu)-(\overline X-\mu) and expand:
\sum_{i=1}^n(X_i-\overline X)^2=\sum_{i=1}^n(X_i-\mu)^2-2(\overline X-\mu)\sum_{i=1}^n(X_i-\mu)+n(\overline X-\mu)^2.
Since \sum(X_i-\mu)=n(\overline X-\mu), the middle term is -2n(\overline X-\mu)^2, which leaves \sum(X_i-\mu)^2-n(\overline X-\mu)^2.
Now \mathbb{E}(X_i-\mu)^2=\sigma^2 and \mathbb{E}(\overline X-\mu)^2=\operatorname{Var}\overline X=\sigma^2/n. Hence
\mathbb{E}\sum_{i=1}^n(X_i-\overline X)^2=n\sigma^2-n\cdot\frac{\sigma^2}{n}=(n-1)\sigma^2,
and \mathbb{E} s^2=\sigma^2.
Used in
- Taught in Estimation, where the derivation and examples are.
- All 5 questions on Estimation
See also
Estimator · Method of moments · Sample correlation coefficient · Likelihood function · Maximum likelihood · Invariance of maximum likelihood · Mean squared error · Consistent estimator