Why a fixed threshold fails
Section of High-dimensional analysis, part of Statistical learning
Why a fixed threshold fails
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
If m_0 null hypotheses are true and each has an exactly uniform null p-value, testing each at 0.05 gives \mathbb{E}(V)=0.05\,m_0. Independence is not needed for this expectation: use linearity on the rejection indicators. Valid conservative p-values instead satisfy \mathbb{P}(P_i\le t)\le t, giving an upper bound. Uniformity requires a correctly calibrated continuous null test. Continuity of a reported statistic alone is not enough. With 20{,}000 true nulls, the expected false-rejection count is therefore 1000.
The family-wise error rate is controlled at a level such as 0.05. The Bonferroni method exists, and it is often far too stringent when the number of tests is in the thousands. No particular value of a null p-value, including a very small one, is specially expected: under a continuous null the p-value is uniform.
Example 1 (The cost of Bonferroni) Question. If n=100 and p=20{,}000, what individual threshold does Bonferroni impose for a family-wise error rate of 0.05, and how large a two-sample difference does that demand?
Solution. The individual threshold is 0.05/20{,}000=2.5\times 10^{-6}. For a two-sample t-test with equal variances and 98 degrees of freedom this requires |t|\ge 5.00. With 50 observations in each group,
t=\frac{\bar x_1-\bar x_2}{s\sqrt{2/50}},
so |t|\ge 5.00 requires a difference between group means of 5.00\times s\times 0.2=1.00\,s: at least as large as the standard deviation within a group. This arithmetic concerns the observed standardised difference, not the power for a specified population effect. Moderate effects may fail to clear the threshold. An exploratory plot does not account for the 20{,}000 opportunities to select a striking difference.
It is often difficult to know how the set of variables found will be used. Sometimes the intended use is known. A laboratory may want to investigate up to 30 genes and would like a list of those that are most promising. Sorting the variables by their p-values and taking the smallest 30 does that. If the p-values are not sufficiently accurate, one can sort by the absolute value of the test statistic, for two-sided tests of one type. The problem is that one does not then know whether the list is wasting effort. Are any of the 30 relevant? Where should the line be drawn?
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from