Fisher’s exact test

Section of Exact tests for tables, part of Statistical inference

Fisher’s exact test

Type Section
Taught in Exact tests for tables
Part Statistical inference

Consider a two-way table of counts. Fisher’s exact test conditions on the row totals and the column totals. Under the null hypothesis that the row classification is independent of the column classification, every table with those margins is equally likely in the following sense: the entries are a random allocation of the column labels to a fixed set of row labels, or the other way around.

The argument is easiest for two rows and two columns. Suppose n individuals have fixed row labels, n_1 in row 1 and n_2=n-n_1 in row 2, and a fixed number c of them carry a “success” in the column. If the column is independent of the row, those c successes are equally likely to be any subset of size c. The number K landing in row 1 is hypergeometric,

\mathbb{P}(K=k)=\frac{\binom{n_1}{k}\binom{n_2}{c-k}}{\binom{n}{c}},

for every k the margins allow, namely \max(0,c-n_2)\le k\le\min(n_1,c). The p-value is the probability, under this law, of a table at least as extreme as the one observed. A one-sided p-value sums the probabilities on the observed side of k. A two-sided p-value sums every table whose probability is no larger than the probability of the observed table.

The same reasoning applies to a larger table. Conditional on all margins, the null distribution is multiple hypergeometric: the count in the free cells is the number of ways to fill the table divided by the number of ways to arrange the margins. For a 2\times 2 table the sum can be done by hand. For a larger table the enumeration is the same sum carried out by a computer.

Here is a 2\times 2 table of operating-system preference by faculty, small enough to add up.

Science Arts Total
Windows 8 2 10
Mac 3 7 10
Total 11 9 20

Condition on the faculty margins 11 and 9 and the row margins 10 and 10. The Science–Windows count K has

\mathbb{P}(K=k)=\frac{\binom{10}{k}\binom{10}{11-k}}{\binom{20}{11}}, \qquad k=1,\ldots,10.

The observed value is k=8, and \mathbb{P}(K=8)=0.0322. The one-sided probability \mathbb{P}(K\ge 8) is 0.0349. The two-sided probability, the sum of every point mass no larger than 0.0322, is

\mathbb{P}(K\in\{1,2,3,8,9,10\})=0.0698.

At level 0.05 the one-sided test rejects independence in the direction of more Windows use in Science. The two-sided test does not. The chi-squared approximation on 1 degree of freedom is a different number and is less trustworthy with two expected counts equal to 9\times 10/20=4.5. The exact probability is the one to report.

Why the margins are conditioned on is part of the null, not a computational trick. If each person is assigned a faculty and then, independently, a computer, both margins are random and the table is multinomial. Conditioning on the margins is then optional, and the conditional test is still valid. If the faculty sizes were fixed by the design and the computers were handed out without regard to faculty, the margins really are fixed and the hypergeometric law is the exact null distribution, not an approximation to one. Either story gives the same conditional p-value. The stories differ in what else could have been tested.

tab <- matrix(c(8, 2, 3, 7), nrow = 2, byrow = TRUE,
              dimnames = list(os = c("Windows", "Mac"),
                              faculty = c("Science", "Arts")))
fisher.test(tab)                 # two-sided p = 0.0698
fisher.test(tab, alternative = "greater")   # one-sided p = 0.0349

A three-by-three table of the same kind, operating system by faculty with several dozen students, is the same test. The free cells are filled in every way consistent with the margins, each fill is weighted by its multiple-hypergeometric probability, and the p-value adds the fills that are at least as improbable as the observed one. That sum is what fisher.test computes. Doing it by listing tables is unnecessary once the function is available, and the logic is the 2\times 2 argument with more cells.

Used in

See also

Homogeneity and independence · Chi-squared test of independence

Glossary and index