Multiple testing
Topic of High-dimensional analysis, part of Statistical learning · Also in Analysis of variance, Analysis of variance examples
Multiple testing
| Type | Topic |
| Taught in | High-dimensional analysis · Analysis of variance · Analysis of variance examples |
| Part | Statistical learning · Statistical inference · Data analysis |
Running many tests at level \alpha makes false rejections likely: with m independent true null hypotheses, the chance of at least one is 1-(1-\alpha)^m. Corrections control either the family-wise error rate, as Bonferroni’s and Tukey’s methods do, or the false discovery rate, as Benjamini–Hochberg does.
In High-dimensional analysis
If m_0 null hypotheses are true and each has an exactly uniform null p-value, testing each at 0.05 gives \mathbb{E}(V)=0.05\,m_0. Independence is not needed for this expectation: use linearity on the rejection indicators. Valid conservative p-values instead satisfy \mathbb{P}(P_i\le t)\le t, giving an upper bound. Uniformity requires a correctly calibrated continuous null test. Continuity of a reported statistic alone is not enough. With 20{,}000 true nulls, the expected false-rejection count is therefore 1000.
The family-wise error rate is controlled at a level such as 0.05. The Bonferroni method exists, and it is often far too stringent when the number of tests is in the thousands. No particular value of a null p-value, including a very small one, is specially expected: under a continuous null the p-value is uniform.
Example 1 (The cost of Bonferroni) Question. If n=100 and p=20{,}000, what individual threshold does Bonferroni impose for a family-wise error rate of 0.05, and how large a two-sample difference does that demand?
Solution. The individual threshold is 0.05/20{,}000=2.5\times 10^{-6}. For a two-sample t-test with equal variances and 98 degrees of freedom this requires |t|\ge 5.00. With 50 observations in each group,
t=\frac{\bar x_1-\bar x_2}{s\sqrt{2/50}},
so |t|\ge 5.00 requires a difference between group means of 5.00\times s\times 0.2=1.00\,s: at least as large as the standard deviation within a group. This arithmetic concerns the observed standardised difference, not the power for a specified population effect. Moderate effects may fail to clear the threshold. An exploratory plot does not account for the 20{,}000 opportunities to select a striking difference.
It is often difficult to know how the set of variables found will be used. Sometimes the intended use is known. A laboratory may want to investigate up to 30 genes and would like a list of those that are most promising. Sorting the variables by their p-values and taking the smallest 30 does that. If the p-values are not sufficiently accurate, one can sort by the absolute value of the test statistic, for two-sided tests of one type. The problem is that one does not then know whether the list is wasting effort. Are any of the 30 relevant? Where should the line be drawn?
In Analysis of variance
From Analysis of variance
Comparing every pair of means with its own 95% interval inflates the chance that some interval misses. With five varieties there are ten pairs, and the chance that at least one of ten 95% intervals misses can be far above 5%. Tukey’s intervals control that simultaneous error.
Definition 1 (Tukey intervals) In a one-way design with k groups of equal size r, and n=kr, the probability is 1-\alpha that every one of the \binom k2 differences \mu_i-\mu_j simultaneously satisfies
\overline Y_{\cdot i}-\overline Y_{\cdot j}-D\sqrt{\mathrm{MS}_{\mathrm{error}}}<\mu_i-\mu_j<\overline Y_{\cdot i}-\overline Y_{\cdot j}+D\sqrt{\mathrm{MS}_{\mathrm{error}}}, \qquad D=\frac{Q_{k,\,n-k;\,1-\alpha}}{\sqrt r},
where Q_{k,\,n-k;\,1-\alpha} is the quantile of the studentised range for k groups and n-k error degrees of freedom.
The studentised range is the law of the largest minus the smallest of k independent standard normals, divided by an independent \sqrt{V/m} with V\sim\chi^2_m. Bounding the largest difference among the group means bounds all of them at once. A pair whose interval excludes zero is declared different at familywise level \alpha.
For the corn, k=5, r=3 and n-k=10, so Q_{5,10;0.95}=4.654 and the half-width of every interval is 4.654\sqrt{10.57/3}=8.74. The ten intervals for \mu_i-\mu_j are
| j=2 | j=3 | j=4 | j=5 | |
|---|---|---|---|---|
| i=1 | (-14.87,\ 2.60) | (-19.60,\ -2.13) | (-8.10,\ 9.37) | (-11.74,\ 5.74) |
| i=2 | (-13.47,\ 4.00) | (-1.97,\ 15.50) | (-5.60,\ 11.87) | |
| i=3 | (2.76,\ 20.24) | (-0.87,\ 16.60) | ||
| i=4 | (-12.37,\ 5.10) |
Two exclude zero: variety 3 yields more than variety 1 and more than variety 4. No other pair can be separated at familywise level 0.05.
In Analysis of variance examples
A significant F says the means are not all equal; it does not say which ones differ. Comparing every pair with its own test at level \alpha inflates the chance of a false difference, since with five groups there are ten pairs.
Definition 2 (Bonferroni correction) To make m comparisons with a familywise error rate of at most \alpha, run each at level \alpha/m. Equivalently, multiply each p-value by m, capping at 1.
The correction works because the probability that at least one of m events occurs is at most the sum of their probabilities. It is simple and always valid, but conservative when m is large. Tukey intervals, built for all pairwise differences of means, are sharper. For the cold sores, Tukey’s method at familywise level 0.05 separates only one pair: placebo against treatment 3, with difference 3.17 days and 95% simultaneous interval (0.57,\ 5.76). The Bonferroni version agrees: the t statistic for that pair is 3.59, beyond the corrected critical value t_{25;\,1-0.05/20}=3.08, and no other pair passes.
TukeyHSD(aov(times ~ treatment, data = coldsore))
pairwise.t.test(coldsore$times, coldsore$treatment, p.adjust.method = "bonferroni")The same question arises with unequal groups. Haemoglobin was measured for three forms of sickle-cell disease, in groups of 16 (HbSS), 15 (HbST) and 10 (HbSC). The analysis of variance gives a treatment sum of squares of 99.89 on 2 degrees of freedom and an error mean square of 1.00 on 38, so F=49.9 and the p-value is about 2.3\times10^{-11}. Bonferroni-corrected pairwise tests, with three comparisons each at level 0.05/3, then say which forms differ; the exercises work through them.
Used in
See also
Bonferroni correction · Family-wise error rate · False discovery rate · Benjamini–Hochberg · Tukey intervals · Null and alternative hypotheses · High-dimensional data · Rank bound · Family-wise control implies false-discovery control