Maximum likelihood

Definition taught in Estimation, part of Statistical inference

Maximum likelihood

Type Definition
Taught in Estimation
Part Statistical inference
Properties 1
Worked examples 4
Questions using it 12

For a random sample with density or mass function f(x;\theta), the likelihood is

L(\theta)=\prod_{i=1}^n f(x_i;\theta).

A maximum likelihood estimate is a value \widehat\theta that maximises L. The maximum likelihood estimator is the same formula with the random sample in place of the observed numbers.

Properties

Invariance of maximum likelihood

Main article: Invariance of maximum likelihood

If \widehat\theta is the maximum likelihood estimator of \theta and g is a function, then the maximum likelihood estimator of g(\theta) is g(\widehat\theta).

Examples

Two estimators of a uniform endpoint

Worked example from Estimation

Let X_1,\ldots,X_n be a random sample from the uniform distribution on (0,\theta), and let M=\max_i X_i.

Question. Compare the mean squared errors of the moment estimator 2\overline X and the maximum likelihood estimator M.

Solution. 2\overline X is unbiased, and \operatorname{Var} X_i=\theta^2/12, so

\operatorname{MSE}(2\overline X)=4\cdot\frac{\theta^2}{12n}=\frac{\theta^2}{3n}.

For the maximum, \mathbb{P}(M\le m)=(m/\theta)^n on (0,\theta), so M has density nm^{n-1}/\theta^n. Then \mathbb{E} M=n\theta/(n+1) and \mathbb{E} M^2=n\theta^2/(n+2), and

\operatorname{MSE}(M)=\mathbb{E} M^2-2\theta\mathbb{E} M+\theta^2=\frac{2\theta^2}{(n+1)(n+2)}.

The maximum is biased low, because it can never exceed \theta, but its mean squared error falls like 1/n^2 rather than 1/n. With n=10 the two are 0.033\,\theta^2 and 0.015\,\theta^2. The rescaled maximum (n+1)M/n is unbiased and has mean squared error \theta^2/\{n(n+2)\}, smaller still.

Normal maximum likelihood estimates

Question page: Normal maximum likelihood estimates

Let X_1,\ldots,X_n be independent and \operatorname{N}(\mu,\sigma^2), with both parameters unknown. Derive the maximum likelihood estimates of \mu and of \sigma^2.

Up to a constant,

\log L(\mu,\sigma^2)=-\frac1{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2-\frac n2\log\sigma^2.

Differentiating in \mu gives \sum_i(x_i-\mu)=0, so \widehat\mu=\overline x. Differentiating in \sigma^2 and substituting \overline x gives

\widehat\sigma^2=\frac1n\sum_{i=1}^n(x_i-\overline x)^2.

To check that this is a maximum, note that for fixed \sigma^2 the log-likelihood is a downward parabola in \mu, largest at \overline x whatever \sigma^2 is. With \mu=\overline x and v=\sigma^2, the profile -A/(2v)-\tfrac n2\log v with A=\sum(x_i-\overline x)^2 has derivative A/(2v^2)-n/(2v), positive for v<A/n and negative after. So (\overline x,\widehat\sigma^2) is the global maximum.

An exponential rate

Question page: An exponential rate

Let X_1,\ldots,X_n be a random sample from the exponential distribution with rate \lambda, density \lambda e^{-\lambda x} for x>0.

  1. Find the maximum likelihood estimator of \lambda, and check that it is a maximum.

  2. Find the method-of-moments estimator of \lambda.

  3. Use invariance to find the maximum likelihood estimator of the median lifetime (\log 2)/\lambda.

  1. The log-likelihood is

\log L(\lambda)=n\log\lambda-\lambda\sum_{i=1}^nx_i.

Its derivative n/\lambda-\sum x_i is zero at \lambda=n/\sum x_i=1/\overline x. The second derivative, -n/\lambda^2, is negative, so this is a maximum. The estimator is \widehat\lambda=1/\overline X.

  1. \mathbb{E} X=1/\lambda. Setting 1/\lambda=\overline x gives the same estimator, 1/\overline X.

  2. The median lifetime m solves 1-e^{-\lambda m}=1/2, so m=(\log2)/\lambda. By invariance its maximum likelihood estimator is (\log2)/\widehat\lambda=\overline X\log2.

A Poisson mean

Question page: A Poisson mean

Let X_1,\ldots,X_n be a random sample from \operatorname{Poisson}(\theta). Find the maximum likelihood estimator of \theta, show that it is unbiased, and find its mean squared error. Is it consistent?

The log-likelihood is \sum x_i\log\theta-n\theta-\sum\log x_i!, with derivative \sum x_i/\theta-n. Setting it to zero gives \widehat\theta=\overline X; the second derivative -\sum x_i/\theta^2 is negative, so it is a maximum. (If every x_i=0 the likelihood e^{-n\theta} is largest at \theta=0, which is again \overline x.)

\mathbb{E}\overline X=\theta, so the estimator is unbiased, and its mean squared error is its variance, \operatorname{Var}\overline X=\theta/n, because a Poisson variance equals its mean. That tends to zero, so \overline X is consistent, by Chebyshev’s inequality or by the law of large numbers.

Used in

See also

Random sample · Discrete distribution · Invariance of maximum likelihood · Pearson test with estimated parameters · Estimator · Method of moments · Sample correlation coefficient · Likelihood function

Glossary and index