General linear hypothesis

Section of Linear hypotheses, part of Statistical inference

Type	Section
Taught in	Linear hypotheses
Part	Statistical inference

A linear hypothesis restricts \beta by

H_0: R\beta=0 \qquad\text{against}\qquad H_1: R\beta\ne 0,

where R is a known r\times p matrix of rank r. Each row of R is one linear constraint. Testing a single coefficient \beta_j=0 uses a row of R that picks out position j. Testing that J group means are equal uses J-1 contrasts and r=J-1.

Let \mathrm{RSS}_1 be the residual sum of squares of the unrestricted model, on \nu_1=n-p degrees of freedom. Let \mathrm{RSS}_0 be the residual sum of squares of the model refitted under R\beta=0, which has p-r free coefficients and \nu_0=n-(p-r) residual degrees of freedom. The difference \nu_0-\nu_1=r is the number of restrictions.

Under the normal model with known \sigma^2, the generalised likelihood ratio depends on the data only through \mathrm{RSS}_0-\mathrm{RSS}_1. The maximised log-likelihoods differ by

-\frac{1}{2\sigma^2}(\mathrm{RSS}_0-\mathrm{RSS}_1),

so -2\log\Lambda=(\mathrm{RSS}_0-\mathrm{RSS}_1)/\sigma^2. When H_0 is true, \mathrm{RSS}_1/\sigma^2\sim\chi^2_{n-p} and (\mathrm{RSS}_0-\mathrm{RSS}_1)/\sigma^2\sim\chi^2_r, and the two pieces are independent. Dividing each chi-squared variable by its degrees of freedom cancels \sigma^2:

F=\frac{(\mathrm{RSS}_0-\mathrm{RSS}_1)/r}{\mathrm{RSS}_1/(n-p)}\sim F_{r,\,n-p}.

The level-\alpha test rejects when F exceeds the upper \alpha point of that F law. Large F means the restrictions increased the residual sum of squares by more than sampling variation in \sigma^2 can explain.

This is the same F as the analysis-of-variance table. There the unrestricted model gives every group its own mean, \mathrm{RSS}_1=\mathrm{SS}_{\mathrm{error}}, and the restricted model forces the means to be equal, \mathrm{RSS}_0=\mathrm{SS}_{\mathrm{total}}. The numerator sum of squares is \mathrm{SS}_{\mathrm{treat}}. For a straight line, the unrestricted model is \beta_0+\beta_1 x and the restriction \beta_1=0 leaves an intercept only. Then \mathrm{RSS}_0-\mathrm{RSS}_1 is the regression sum of squares, r=1, and F is the square of the slope t statistic.

The table that goes with any such comparison is

Model	Degrees of freedom	Residual sum of squares	F
Alternative, R\beta\ne 0	n-p	\mathrm{RSS}_1
Null, R\beta=0	n-p+r	\mathrm{RSS}_0
Difference	r	\mathrm{RSS}_0-\mathrm{RSS}_1	\dfrac{(\mathrm{RSS}_0-\mathrm{RSS}_1)/r}{\mathrm{RSS}_1/(n-p)}

The null distribution is exact when the errors are independent and normal with common variance, and X is fixed. If the variance is known, (\mathrm{RSS}_0-\mathrm{RSS}_1)/\sigma^2 is compared with \chi^2_r directly and the denominator estimate of \sigma^2 is unnecessary. In either case the procedure is the generalised likelihood-ratio test for this normal likelihood.

Four points illustrate the arithmetic. Take x=(1,2,3,4) and y=(2,3,5,6). The least-squares line is \widehat y=0.5+1.4x. The residual sum of squares is

\mathrm{RSS}_1=(2-1.9)^2+(3-3.3)^2+(5-4.7)^2+(6-6.1)^2=0.20,

on 4-2=2 degrees of freedom. The intercept-only fit uses \overline y=4 and

\mathrm{RSS}_0=(2-4)^2+(3-4)^2+(5-4)^2+(6-4)^2=10.

Then

F=\frac{(10-0.20)/1}{0.20/2}=98

on (1,2) degrees of freedom. The upper 5\% point of F_{1,2} is 18.5, so the test rejects \beta_1=0. The slope is not a sampling fluctuation around a flat mean. With n=4 the F critical value is large, and a moderate slope would not have cleared it. The formula does not change when n is larger. Only the critical value and the residual degrees of freedom do.

x <- 1:4
y <- c(2, 3, 5, 6)
fit <- lm(y ~ x)
anova(fit)
# Df 1 and 2, Sum of Sq for x equal to 9.8, residuals 0.2, F = 98.

Used in

Taught in Linear hypotheses
All 2 questions on Linear hypotheses

Used in

See also