Likelihood-ratio test

Topic of Estimation and inference for the multivariate normal, part of Statistical learning · Also in Paired comparisons, and how far a fit is from the data

Likelihood-ratio test

Type Topic
Taught in Estimation and inference for the multivariate normal · Paired comparisons, and how far a fit is from the data
Part Statistical learning · Generalised linear models
Questions using it 4

A test of a model against a nested submodel that compares their maximised likelihoods. Under regularity conditions 2\{\ell(\hat\theta)-\ell(\hat\theta_0)\} is approximately chi-squared, with degrees of freedom equal to the number of restrictions. A difference in deviances is this statistic.

In Estimation and inference for the multivariate normal

From Estimation and inference for the multivariate normal

Take n independent observations from \operatorname{N}_p(\boldsymbol{\mu},\Sigma), with both parameters unknown, and test

H_0:\boldsymbol{\mu}=\boldsymbol{\mu}_0 \qquad\text{against}\qquad H_1:\boldsymbol{\mu}\neq\boldsymbol{\mu}_0.

The unrestricted fit uses \bar{\mathbf{x}} and A/n. Under the null, the mean is fixed and only the covariance can adapt. The likelihood-ratio statistic compares those two best fits. The construction does not by itself establish uniform optimality against every multivariate alternative. Compare the maximised likelihood with \boldsymbol{\mu} fixed at \boldsymbol{\mu}_0 to the maximised likelihood with \boldsymbol{\mu} free:

\lambda=\frac{\max_\Sigma L(\boldsymbol{\mu}_0,\Sigma)}{\max_{\boldsymbol{\mu},\Sigma}L(\boldsymbol{\mu},\Sigma)}\in[0,1].

The numerator maximises over a subset of the denominator’s parameter space, so the ratio cannot exceed one. If it is small, imposing the proposed mean costs a great deal of fit.

Maximisation under H_0. Under H_0 only \Sigma is free. The derivative that produced \widehat\Sigma=A/n measures residuals from the centring point in use. With residuals measured from \boldsymbol{\mu}_0 rather than from a free mean, the same derivative gives

\widehat\Sigma_\omega=\frac1n\sum_{i=1}^n(\mathbf{x}_i-\boldsymbol{\mu}_0)(\mathbf{x}_i-\boldsymbol{\mu}_0)^\top.

Split those residuals at \bar{\mathbf{x}}:

\begin{aligned} \widehat\Sigma_\omega &=\frac1n\sum_{i=1}^n\bigl[(\mathbf{x}_i-\bar{\mathbf{x}})+(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)\bigr] \bigl[(\mathbf{x}_i-\bar{\mathbf{x}})+(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)\bigr]^\top\\ &=\frac1n\Bigl[A+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top\Bigr]. \end{aligned}

The cross terms vanish because \sum_i(\mathbf{x}_i-\bar{\mathbf{x}})=\mathbf{0}. The constrained covariance estimate is A/n plus the outer product of \bar{\mathbf{x}}-\boldsymbol{\mu}_0. The added term is positive semi-definite, and it is zero only when the sample mean equals the null mean.

Maximisation without restriction. With both parameters free, \widehat{\boldsymbol{\mu}}_\Omega=\bar{\mathbf{x}} and \widehat\Sigma_\Omega=A/n. The denominator of \lambda uses those two estimators.

The ratio. Substitute the two maximisers. The powers of 2\pi agree. The exponents also agree, for a reason that separates from the determinant algebra: at each maximiser the quadratic form in the exponent equals np. A scalar equals its own trace, and the trace is cyclic, so \mathbf{v}^\top M^{-1}\mathbf{v}=\operatorname{tr}(M^{-1}\mathbf{v}\mathbf{v}^\top). Apply that identity to the constrained exponent:

\begin{aligned} \sum_{i=1}^n(\mathbf{x}_i-\boldsymbol{\mu}_0)^\top\widehat\Sigma_\omega^{-1}(\mathbf{x}_i-\boldsymbol{\mu}_0) &=\operatorname{tr}\Bigl(\widehat\Sigma_\omega^{-1}\sum_{i=1}^n(\mathbf{x}_i-\boldsymbol{\mu}_0)(\mathbf{x}_i-\boldsymbol{\mu}_0)^\top\Bigr)\\ &=\operatorname{tr}\bigl(\widehat\Sigma_\omega^{-1}\cdot n\widehat\Sigma_\omega\bigr) =n\operatorname{tr}(I_p)=np. \end{aligned}

At the constrained maximiser, n\widehat\Sigma_\omega equals \sum_i(\mathbf{x}_i-\boldsymbol{\mu}_0)(\mathbf{x}_i-\boldsymbol{\mu}_0)^\top. The same steps with \widehat\Sigma_\Omega and residuals from \bar{\mathbf{x}} also give np, because n\widehat\Sigma_\Omega=A. Both exponential factors equal e^{-np/2} and cancel. Normality was used to write the likelihood. The cancellation itself is this trace identity at the two critical points. The likelihood ratio is a ratio of determinants,

\lambda=\frac{|\widehat\Sigma_\Omega|^{n/2}}{|\widehat\Sigma_\omega|^{n/2}} =\frac{|A|^{n/2}}{\bigl|A+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top\bigr|^{n/2}}.

A determinant identity. For conformable blocks with B invertible,

\begin{vmatrix}B&C\\ D&E\end{vmatrix} =\begin{vmatrix}B&C\\ 0&E-DB^{-1}C\end{vmatrix} =|B|\,|E-DB^{-1}C|.

The first equality subtracts DB^{-1} times the first block row from the second, which does not change the determinant. The same determinant, eliminated on the lower-right block instead, has the companion form |B|\,|E-DB^{-1}C|=|E|\,|B-CE^{-1}D|. Apply the identity with B=A, C=-\sqrt{n}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0), D=\sqrt{n}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top and E=1:

\begin{vmatrix} A & -\sqrt{n}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)\\ \sqrt{n}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top & 1 \end{vmatrix} =|A|\Bigl(1+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top A^{-1}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)\Bigr).

With E=1 the companion evaluation is |A+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top|, because -CE^{-1}D=+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top. The two evaluations therefore give

\bigl|A+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top\bigr| =|A|\Bigl(1+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top A^{-1}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)\Bigr).

This comparison is algebraic. Raise \lambda to the power 2/n:

\lambda^{2/n} =\frac{1}{1+n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top A^{-1}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)} =\frac{1}{1+t^2/(n-1)},

where A=(n-1)S has been used to name the quadratic form

t^2=n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top S^{-1}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0) =(n-1)\,n(\bar{\mathbf{x}}-\boldsymbol{\mu}_0)^\top A^{-1}(\bar{\mathbf{x}}-\boldsymbol{\mu}_0).

The two expressions agree because S^{-1}=(n-1)A^{-1}. Thus \lambda is a strictly decreasing function of t^2. Rejecting for small \lambda is the same test as rejecting for large t^2. The block determinant converts the ratio of covariance determinants into that quadratic form, and the conversion does not use normality. Normality enters when the threshold is given a probability. The statistic t^2 is n times the squared Mahalanobis distance from \bar{\mathbf{x}} to \boldsymbol{\mu}_0, computed with S. When p=1, S reduces to s^2 and t^2 is the square of the univariate t statistic.

In Paired comparisons, and how far a fit is from the data

From Paired comparisons, and how far a fit is from the data

The deviance is twice the drop in log-likelihood from the saturated fit to the model you actually fitted: D({\boldsymbol{y}};\hat{\boldsymbol{\mu}})=2\bigl\{\ell({\boldsymbol{y}};{\boldsymbol{y}})-\ell(\hat{\boldsymbol{\mu}};{\boldsymbol{y}})\bigr\}, up to the dispersion, which scales it. On the observation scale a residual of 5 is not the same thing for a Poisson mean of 2 and a Poisson mean of 50. The likelihood already knows that. For the families we have been using, the deviance contributions add up as follows. For the normal, \phi=\sigma^2, D=\sum_{i=1}^n(y_i-\hat\mu_i)^2, the residual sum of squares, because the factor of 2 cancels the \tfrac12 in the normal log-density. For the Poisson, D=2\sum_{i=1}^n\left\{y_i\log\frac{y_i}{\hat\mu_i}-(y_i-\hat\mu_i)\right\}. For a binomial count of y_i successes out of m_i, with fitted mean count m_i\hat p_i, D=2\sum_{i=1}^n\left\{y_i\log\frac{y_i}{m_i\hat p_i}+(m_i-y_i)\log\frac{m_i-y_i}{m_i(1-\hat p_i)}\right\}. The gamma deviance, in the mean-and-variance parameterisation \mathrm{Var}=\phi\mu^2, is D=2\sum_{i=1}^n\left\{-\log\frac{y_i}{\hat\mu_i}+\frac{y_i-\hat\mu_i}{\hat\mu_i}\right\}. These are not interchangable across families. A smaller deviance is a closer fit only inside one family. Two non-nested models, such as a logit and a probit, have no likelihood-ratio test. AIC is the comparison that remains: deviance plus twice the number of parameters, up to the usual constants.

Used in

See also

Likelihood function · Degrees of freedom · Statistic · Residual deviance · Invariance of the maximum likelihood estimator · Wishart distribution · Sampling law of the normal mean and scatter · Multivariate central limit theorem

Glossary and index