Estimation
Part of Statistical inference · 5 questions on this topic · What links here
Estimation
| Part | Statistical inference |
| Chapter | 2 of 7 in this part |
| Previous | Statistical inference |
| Next | Confidence intervals |
| Questions | 5 |
| Definitions | Estimator · Method of moments · Sample correlation coefficient · Likelihood function · Maximum likelihood · Unbiased estimator · Mean squared error · Consistent estimator |
| Results | Invariance of maximum likelihood |
An estimator is a statistic used as a guess at a parameter. The number it produces on the observed sample is the estimate. The two are written with the same symbol, and the difference is whether the sample is still random. \widehat\theta(X_1,\ldots,X_n) has a sampling distribution. \widehat\theta(x_1,\ldots,x_n) is a number.
The families below are the ones on the common distributions page, reached through a random sample. The two systematic ways of building an estimator are the method of moments and maximum likelihood. Once there are candidates, they are compared by bias, variance and mean squared error.
Estimates and estimators
For the milk cartons in Statistical inference, the model is a normal random sample with unknown mean \mu. The mean will never be known. It is plausible that \mu is close to the sample mean, 1.0512, but the sample median, 1.0460, is just as much a function of the data. Which is the better guess is a question about the two sampling distributions, not about these two numbers.
Definition 1 (Estimator) An estimator of \theta is a statistic T=T(X_1,\ldots,X_n) whose value is used as a guess at \theta. Its observed value t=T(x_1,\ldots,x_n) is the estimate.
The definition is deliberately broad. Any statistic is an estimator, and most are bad ones. The one requirement is that it can be computed from the data alone. For a Bernoulli sample with \mathbb{P}(X_i=1)=p, (X_1+X_4)/2 is an estimator of p, though a wasteful one. (X_1+\mathbb{E} X_4)/2 is not an estimator, because \mathbb{E} X_4=p is unknown.
Example 1 (Ten thermometers) Ten thermometers are drawn at random from a large stock and tested. The results are 0,1,1,0,1,0,0,1,0,0, where 1 means the thermometer meets the standard.
Question. Write down a model and an estimate of the proportion p of compliant thermometers in the stock.
Solution. Let X_i=1 if the ith thermometer works. Drawing at random from a large stock makes the X_i close to independent, with \mathbb{P}(X_i=1)=p. The natural estimator is the sample proportion
\widehat p=\frac1{10}\sum_{i=1}^{10}X_i,
and its observed value is 4/10=0.4. The estimator is the recipe that would be applied to tomorrow’s ten thermometers. The estimate is the number 0.4 from today’s.
Method of moments
Definition 2 (Method of moments) Suppose the first k population moments exist and determine \theta=(\theta_1,\ldots,\theta_k). The method of moments matches them to the corresponding sample moments,
\mathbb{E}_\theta(X^r)=\frac1n\sum_{i=1}^n x_i^r,\qquad r=1,\ldots,k,
and solves the k equations for \theta.
The population moments are functions of \theta, and the sample moments are numbers, so this is k equations in k unknowns. They are often nonlinear and solved numerically.
For a single mean, the first moment gives \widehat\mu=\overline X. For a mean and a variance together, \mathbb{E} X_i^2=\mu^2+\sigma^2, so the second sample moment produces
\widetilde\sigma^2=\frac1n\sum_{i=1}^n x_i^2-\overline x^2=\frac1n\sum_{i=1}^n (x_i-\overline x)^2=\frac{n-1}{n}s^2.
For independent Bernoulli trials with success probability p, the first moment is p itself, so the moment estimator is the sample proportion, as in the thermometer example.
Definition 3 (Sample correlation coefficient) For pairs (x_1,y_1),\ldots,(x_n,y_n), the sample correlation coefficient is
r=\frac{\sum_{i=1}^n(x_i-\overline x)(y_i-\overline y)}{\sqrt{\sum_{i=1}^n(x_i-\overline x)^2}\,\sqrt{\sum_{i=1}^n(y_i-\overline y)^2}}.
It lies in [-1,1], by the Cauchy–Schwarz inequality.
The sample correlation is a moment estimator. The population correlation is
\rho=\frac{\mathbb{E}(XY)-\mu_X\mu_Y}{\sigma_X\sigma_Y}.
Replace \mathbb{E}(XY) by \frac1n\sum x_iy_i, the means by sample means, and each variance by its divide-by-n moment estimator. The numerator becomes \frac1n\sum x_iy_i-\overline x\,\overline y=\frac1n\sum(x_i-\overline x)(y_i-\overline y), and the factors of 1/n cancel to give r. The same statistic returns in Linear regression, where r^2 is the proportion of variation explained by a fitted line.
Moment estimators are easy to write down and are consistent whenever the moments are continuous functions of \theta. They are not, in general, the estimators with the smallest variance, and they can fall outside the parameter space.
Maximum likelihood
Ten spark plugs are drawn from a large batch and two are defective. The obvious estimate of the defective proportion is 2/10. Maximum likelihood is a systematic route to that answer that also works when there is no obvious one.
Model the count as X\sim\operatorname{Bin}(10,p). The probability of what was observed, as a function of p, is
f(2;p)=\binom{10}{2}p^2(1-p)^8.
At p=0.4 this is 0.121. At p=0.3 it is 0.233, almost twice as large, and at p=0.2 it is 0.302. Some values of p make the observation more probable than others. Maximum likelihood picks the value that makes it most probable. Setting the derivative of \log f(2;p)=\text{const}+2\log p+8\log(1-p) to zero gives 2/p-8/(1-p)=0, that is \widehat p=0.2.
Definition 4 (Likelihood function) Let X_1,\ldots,X_n have joint mass or density function f(x_1,\ldots,x_n;\theta), and let x_1,\ldots,x_n be the observed values. The likelihood function is
L(\theta)=L(\theta;x_1,\ldots,x_n)=f(x_1,\ldots,x_n;\theta),\qquad \theta\in\Theta.
The formula is the same as the joint density. What changes is which argument varies. The density is a function of the data for a fixed parameter. The likelihood is a function of the parameter for the fixed, observed data. It is not a density in \theta and need not integrate to one.
For continuous data the probability of the exact observation is zero for every \theta, so the density takes its place. Suppose daily output of a chemical plant is \operatorname{N}(\mu,50) tonnes with \mu unknown, and only the mean over 50 days, \overline x=871, is recorded. Then \overline X\sim\operatorname{N}(\mu,1), and the likelihood is the density of \overline X at 871,
L(\mu)=\frac1{\sqrt{2\pi}}\exp\bigl\{-\tfrac12(871-\mu)^2\bigr\},
which is largest at \mu=871. Here the likelihood of a single summary is enough, because for a normal mean with known variance the sample mean carries all the information in the sample about \mu; that is the idea of a sufficient statistic.
Definition 5 (Maximum likelihood) For a random sample with density or mass function f(x;\theta), the likelihood is
L(\theta)=\prod_{i=1}^n f(x_i;\theta).
A maximum likelihood estimate is a value \widehat\theta that maximises L. The maximum likelihood estimator is the same formula with the random sample in place of the observed numbers.
The logarithm is strictly increasing, so \log L has the same maximiser, and a sum is easier to differentiate than a product:
\log L(\theta)=\sum_{i=1}^n\log f(x_i;\theta).
For a scalar parameter the likelihood equation is \partial\log L/\partial\theta=0, provided the maximum is interior and the derivative exists. For a vector parameter there is one equation per coordinate, \partial\log L/\partial\theta_j=0. A solution of the likelihood equations should still be checked to be a maximum rather than a minimum or a saddle.
Binomial
A single \operatorname{Bin}(n,p) count X=x has log-likelihood x\log p+(n-x)\log(1-p), up to a constant. The likelihood equation is
\frac xp-\frac{n-x}{1-p}=0,
so \widehat p=x/n, and the estimator is X/n. Maximum likelihood gives back the common-sense estimator.
Normal
For X_1,\ldots,X_n independent and \operatorname{N}(\mu,\sigma^2),
\log L(\mu,\sigma^2)=-\frac1{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2-\frac n2\log\sigma^2-\frac n2\log(2\pi).
The two likelihood equations are
\frac{\partial\log L}{\partial\mu}=\frac1{\sigma^2}\sum_{i=1}^n(x_i-\mu)=0, \qquad \frac{\partial\log L}{\partial\sigma^2}=\frac1{2\sigma^4}\sum_{i=1}^n(x_i-\mu)^2-\frac n{2\sigma^2}=0.
The first gives \widehat\mu=\overline x whatever the value of \sigma^2. Substituting into the second gives
\widehat\sigma^2=\frac1n\sum_{i=1}^n(x_i-\overline x)^2.
So the normal mean is estimated by the sample mean, and the maximum likelihood estimator of the variance divides by n, not by n-1. The two estimators coincide with the moment estimators. The derivative was taken with respect to \sigma^2, treated as the parameter; the next result shows that working with \sigma instead gives the same answer.
When the likelihood equation fails
Let X_1,\ldots,X_n be a random sample from the uniform distribution on (0,\theta). Each density is 1/\theta on (0,\theta), so
L(\theta)=\theta^{-n}\quad\text{if }\theta\ge\max_i x_i,\qquad L(\theta)=0\quad\text{otherwise}.
L is decreasing wherever it is positive, so its maximum is at the boundary, \widehat\theta=\max_i X_i. The derivative is never zero. The likelihood equation finds interior critical points only, and a maximum at the edge of the allowed region has to be found by looking at L directly. The moment estimator is different: \mathbb{E} X=\theta/2 gives \widetilde\theta=2\overline X, which can be smaller than the largest observation, an impossible value.
Proposition 1 (Invariance of maximum likelihood) If \widehat\theta is the maximum likelihood estimator of \theta and g is a function, then the maximum likelihood estimator of g(\theta) is g(\widehat\theta).
For a one-to-one g, write \nu=g(\theta) and reparameterise the likelihood as \widetilde L(\nu)=L(g^{-1}(\nu)). Because L is largest at \theta=\widehat\theta, \widetilde L is largest where g^{-1}(\nu)=\widehat\theta, that is at \nu=g(\widehat\theta). For a g that is not one-to-one the statement is taken as the definition of the estimator of g(\theta): maximise the largest likelihood over all \theta with g(\theta)=\nu, which again puts the maximum at g(\widehat\theta). Estimating \sigma by \sqrt{\widehat\sigma^2}, or a probability e^{-\lambda} by e^{-\widehat\lambda}, are the cases met most often.
Comparing estimators
Definition 6 (Unbiased estimator) An estimator T of \theta is unbiased when \mathbb{E}_\theta T=\theta for every \theta in the parameter space. The bias is \mathbb{E}_\theta T-\theta.
The sample mean is unbiased for \mu, by linearity, whenever the mean exists:
\mathbb{E}\overline X=\frac1n\bigl(\mathbb{E} X_1+\cdots+\mathbb{E} X_n\bigr)=\frac1n\,n\mu=\mu.
The bias-corrected sample variance s^2 is unbiased for \sigma^2, as shown in Statistical inference. The maximum likelihood estimator \widehat\sigma^2=((n-1)/n)s^2 therefore has expectation ((n-1)/n)\sigma^2 and bias -\sigma^2/n.
Unbiasedness says the estimator is right on average. It says nothing about how far a single estimate is likely to fall from \theta. A measure of that is the mean squared error.
Definition 7 (Mean squared error) The mean squared error of an estimator T of \theta is \operatorname{MSE}(T)=\mathbb{E}(T-\theta)^2. It splits as
\mathbb{E}(T-\theta)^2=\operatorname{Var}(T)+(\mathbb{E} T-\theta)^2.
To see the split, add and subtract \mathbb{E} T:
\mathbb{E}(T-\theta)^2=\mathbb{E}\{T-\mathbb{E} T\}^2+2\{\mathbb{E} T-\theta\}\,\mathbb{E}\{T-\mathbb{E} T\}+\{\mathbb{E} T-\theta\}^2.
The middle term is zero because \mathbb{E}\{T-\mathbb{E} T\}=0, which leaves the variance plus the squared bias.
The split is the variance–bias tradeoff. Removing a bias often raises the variance, and sometimes by more than the squared bias that was removed. An estimator with a small bias can then have a smaller mean squared error than an unbiased competitor. One estimator is more efficient than another when its mean squared error is no larger at every parameter value, and smaller at some.
Example 2 (Two variance estimators) Let X_1,\ldots,X_n be a normal random sample with n\ge 2.
Question. Compare the mean squared errors of s^2 and \widehat\sigma^2=((n-1)/n)s^2 as estimators of \sigma^2.
Solution. By the normal sample theorem, V=(n-1)s^2/\sigma^2 is \chi^2_{n-1}, so \operatorname{Var} V=2(n-1) and
\operatorname{Var}(s^2)=\frac{\sigma^4}{(n-1)^2}\,2(n-1)=\frac{2\sigma^4}{n-1}.
s^2 is unbiased, so this is its mean squared error. For \widehat\sigma^2, the variance is ((n-1)/n)^2 times as large and the bias is -\sigma^2/n:
\operatorname{MSE}(\widehat\sigma^2)=\frac{(n-1)^2}{n^2}\cdot\frac{2\sigma^4}{n-1}+\frac{\sigma^4}{n^2}=\frac{(2n-1)\sigma^4}{n^2}.
Since (2n-1)(n-1)=2n^2-3n+1<2n^2, the biased estimator has the smaller mean squared error for every n and every \sigma^2. With n=10 the two are 0.222\,\sigma^4 and 0.190\,\sigma^4. The unbiased estimator is preferred for other reasons: it is the one whose distribution gives the exact t and chi-squared pivots.
Example 3 (Two estimators of a uniform endpoint) Let X_1,\ldots,X_n be a random sample from the uniform distribution on (0,\theta), and let M=\max_i X_i.
Question. Compare the mean squared errors of the moment estimator 2\overline X and the maximum likelihood estimator M.
Solution. 2\overline X is unbiased, and \operatorname{Var} X_i=\theta^2/12, so
\operatorname{MSE}(2\overline X)=4\cdot\frac{\theta^2}{12n}=\frac{\theta^2}{3n}.
For the maximum, \mathbb{P}(M\le m)=(m/\theta)^n on (0,\theta), so M has density nm^{n-1}/\theta^n. Then \mathbb{E} M=n\theta/(n+1) and \mathbb{E} M^2=n\theta^2/(n+2), and
\operatorname{MSE}(M)=\mathbb{E} M^2-2\theta\mathbb{E} M+\theta^2=\frac{2\theta^2}{(n+1)(n+2)}.
The maximum is biased low, because it can never exceed \theta, but its mean squared error falls like 1/n^2 rather than 1/n. With n=10 the two are 0.033\,\theta^2 and 0.015\,\theta^2. The rescaled maximum (n+1)M/n is unbiased and has mean squared error \theta^2/\{n(n+2)\}, smaller still.
Definition 8 (Consistent estimator) A sequence of estimators T_n, one for each sample size n, is consistent for \theta when \mathbb{P}_\theta(|T_n-\theta|>\varepsilon)\to0 as n\to\infty, for every \varepsilon>0 and every \theta.
For a large sample the leading question is whether the estimator settles on \theta. If the mean squared error tends to zero, the estimator is consistent, by Chebyshev’s inequality: \mathbb{P}(|T_n-\theta|>\varepsilon)\le\operatorname{MSE}(T_n)/\varepsilon^2. The law of large numbers says that the sample mean is consistent for \mu, and the same limit passes to continuous functions of a consistent estimator, which is why moment estimators are usually consistent. Both estimators in each example above are consistent. Maximum likelihood estimators are consistent in the regular models used here.
Match moments, or maximise the likelihood. Then judge the estimator by its sampling distribution: bias, variance, and their combination, the mean squared error. Unbiased is not the same as best.
Questions on this topic
Open a question to attempt it. Its worked answer is folded away at the bottom of the page.
What links here
Pages (10)
- Statistical inference — via Estimation, mean squared error
- Hypothesis tests — via maximum likelihood estimate
- Chi-squared tests — via maximum likelihood estimator
- Linear regression — via maximum likelihood estimators, sample correlation coefficient, sample correlation
- Analysis of variance — via maximum likelihood estimators
- Descriptive statistics — via sample correlation coefficient
- Using R — via Estimation
- Estimation examples — via Estimation
- Bayesian hypotheses and model comparison — via MLE
- Standard errors, and models for counts — via maximum likelihood estimator
Questions and answers (16)
- Normal maximum likelihood estimates · Bias of the two variance estimators · An exponential rate · A Poisson mean · A shrunken sample mean · The correlation test is the slope test · Reading a correlation · Posterior mean under squared-error loss · Q2 · Q13 · Q27 (answer) · Q48 · Question 7 (answer) · Question 10 (answer) · Question 14 · Question 18 (answer)
See also
- Articles from this chapter: Estimator · Method of moments · Sample correlation coefficient · Likelihood function · Maximum likelihood · Invariance of maximum likelihood · Unbiased estimator · Mean squared error · Consistent estimator
- All statistical inference chapters
- Glossary and index
- Distribution sheet and formula lookup