Single-variable tests
Section of High-dimensional analysis, part of Statistical learning
Single-variable tests
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
In many studies the scientific question is which coordinates are associated with the response, not the size of a prediction error. Association is not causation. With two groups, one with a disease and one healthy, the question may be which explanatory variables have a different distribution in each group. With a continuous response, it may be which variables are most strongly related to it. The variables that survive are candidates for further study.
The simplest approach is to work through the explanatory variables one by one. When the response is nominal and the explanatory variable is continuous, analysis of variance is the usual test if its conditions are met: independence of observations, approximate normality within each group, and similar variances. A transformation sometimes achieves that. When it cannot, a rank test such as Kruskal–Wallis asks for less. With two groups, Welch’s t-test allows unequal group variances. When both variables are nominal, a chi-squared or Fisher exact test is the usual choice. For a continuous response and a categorical explanatory variable, analysis of variance again. When both are continuous, linear regression, perhaps after transformation. The result is a p-value for every explanatory variable.
A single one of those tests, with its null distribution and its p-value, is the procedure on hypothesis tests. The difficulty here is the threshold, because the same rule is applied to thousands of coordinates at once.
Why a fixed threshold fails
Main article: Why a fixed threshold fails
If m_0 null hypotheses are true and each has an exactly uniform null p-value, testing each at 0.05 gives \mathbb{E}(V)=0.05\,m_0. Independence is not needed for this expectation: use linearity on the rejection indicators. Valid conservative p-values instead satisfy \mathbb{P}(P_i\le t)\le t, giving an upper bound. Uniformity requires a correctly calibrated continuous null test. Continuity of a reported statistic alone is not enough. With 20{,}000 true nulls, the expected false-rejection count is therefore 1000.
The family-wise error rate is controlled at a level such as 0.05. The Bonferroni method exists, and it is often far too stringent when the number of tests is in the thousands. No particular value of a null p-value, including a very small one, is specially expected: under a continuous null the p-value is uniform.
Example 1 (The cost of Bonferroni) Question. If n=100 and p=20{,}000, what individual threshold does Bonferroni impose for a family-wise error rate of 0.05, and how large a two-sample difference does that demand?
Solution. The individual threshold is 0.05/20{,}000=2.5\times 10^{-6}. For a two-sample t-test with equal variances and 98 degrees of freedom this requires |t|\ge 5.00. With 50 observations in each group,
t=\frac{\bar x_1-\bar x_2}{s\sqrt{2/50}},
so |t|\ge 5.00 requires a difference between group means of 5.00\times s\times 0.2=1.00\,s: at least as large as the standard deviation within a group. This arithmetic concerns the observed standardised difference, not the power for a specified population effect. Moderate effects may fail to clear the threshold. An exploratory plot does not account for the 20{,}000 opportunities to select a striking difference.
It is often difficult to know how the set of variables found will be used. Sometimes the intended use is known. A laboratory may want to investigate up to 30 genes and would like a list of those that are most promising. Sorting the variables by their p-values and taking the smallest 30 does that. If the p-values are not sufficiently accurate, one can sort by the absolute value of the test statistic, for two-sided tests of one type. The problem is that one does not then know whether the list is wasting effort. Are any of the 30 relevant? Where should the line be drawn?
The false discovery rate
Main article: False discovery rate
False discovery rate control offers a different error criterion (Benjamini and Hochberg, 1995). It concerns the expected fraction of false rejections among all rejections. Write m for the number of tests, keeping p available for a p-value only when there is no ambiguity.
| Declared non-significant | Declared significant | Total | |
|---|---|---|---|
| H_0 true | U | V | M_0 |
| H_0 false | T | S | M_1 |
| Total | m-R | R | m |
The table separates the unknown truth from the random decisions. In this frequentist formulation the set of true nulls, and hence M_0 and M_1, is fixed but unknown. The counts U,V,T,S,R vary across repeated data sets. For M_0>0, the average null rejection probability is \bar\alpha=\mathbb{E}(V)/M_0. This average need not equal a common nominal level for every test. When M_1>0, the corresponding average type II error probability and power are
\bar\beta=\frac{\mathbb{E}(T)}{M_1}, \qquad \text{average power}=1-\bar\beta=\frac{\mathbb{E}(S)}{M_1}.
The family-wise error rate is \mathrm{FWER}=\mathbb{P}(V\ge 1). The false discovery proportion is Q=V/R when R>0 and Q=0 when V=R=0: the proportion of the variables declared significant that are in fact null. A declared-significant hypothesis is a false discovery when its null hypothesis is true. The false discovery rate is its expected value,
\mathrm{FDR}=\mathbb{E}(Q)=\mathbb{E}\Bigl(\frac{V}{R}\Bigm| R>0\Bigr)\mathbb{P}(R>0).
Set the false discovery proportion to zero when there are no rejections. This convention makes the rate well-defined even under the complete null. In contrast, \mathbb{E}(V/R\mid R>0) conditions on making at least one discovery. The two quantities differ by the factor \mathbb{P}(R>0), which can be small. Neither quantity bounds the false fraction in every individual realised list.
Theorem 1 (Family-wise control implies false-discovery control) When all null hypotheses are true (m_0=m), \mathrm{FDR}=\mathrm{FWER}. In general \mathrm{FDR}\le\mathrm{FWER}.
Article: Family-wise control implies false-discovery control →
If all nulls are true, S=0 and so V=R. If V=0 then Q=0. If V\ge 1 then Q=1. So \mathbb{P}(V\ge 1)=\mathbb{E}(Q). In general, I(V\ge 1)\ge Q: if V=0 both sides are 0; if V\ge 1 the left side is 1 and Q=V/R\le 1. Taking expectations, \mathbb{P}(V\ge 1)\ge\mathbb{E}(Q).
Any procedure that controls the family-wise error rate therefore controls the false discovery rate. If one controls only the false discovery rate, the family-wise error rate can be considerably higher. That may be of no concern, and the benefit can be a large increase in the power of the overall procedure.
The Benjamini–Hochberg procedure
Main article: Benjamini–Hochberg procedure
Test m null hypotheses H_1,\ldots,H_m whose p-values are P_1,\ldots,P_m. Before the data are collected these are random variables. Order them P_{(1)}\le P_{(2)}\le\cdots\le P_{(m)} and let H_{(i)} be the hypothesis whose p-value is P_{(i)}.
Definition 1 (Benjamini–Hochberg) Let k be the largest i for which P_{(i)}\le (i/m)\,q^*. Then reject all H_{(i)} for i=1,\ldots,k. Reject none if no such i exists.
Theorem 2 (Benjamini–Hochberg controls the false discovery rate) Suppose the p-values are mutually independent and each true-null p-value is valid: \mathbb{P}(P_i\le t)\le t for 0\le t\le 1. The procedure at q^*\in(0,1) satisfies
\mathrm{FDR}\le\frac{m_0}{m}q^*\le q^*.
Equality holds in the first inequality for exactly uniform null p-values (Benjamini and Hochberg, 1995; Benjamini and Yekutieli, 2001).
Article: Benjamini–Hochberg controls the false discovery rate →
The argument below is for independent p-values with exactly uniform null p-values. The denominator R depends on all the tests, including the one whose false-rejection probability is wanted. Remove that dependence by setting the selected null p-value to zero. This gives a rejection count determined only by the other p-values. The cancellation is then between its appearance in the threshold and in the denominator.
Write q=q^* and let N be the true-null set. For i\in N, let R_i be the rejection count after replacing P_i by zero. Then R_i\ge 1. If H_i is rejected, lowering its already-rejected p-value cannot create a new crossing at an index greater than R, so R_i=R. Conversely, if P_i\le q R_i/m, restoring P_i retains at least R_i values below that threshold. It cannot increase the count above the count obtained with zero, so R=R_i and H_i is rejected. Hence
\frac{V}{R\vee 1}=\sum_{i\in N}\frac{I(P_i\le q R_i/m)}{R_i}.
Condition on all p-values except P_i. Independence and exact null uniformity give \mathbb{P}(P_i\le q R_i/m\mid R_i)=q R_i/m, so each summand has expectation q/m. Summing gives \mathrm{FDR}=m_0 q/m. For super-uniform null p-values, replace equality by an upper bound in the conditional probability.
Independence is sufficient, not necessary. The procedure also controls the false-discovery rate under positive regression dependence on the true nulls. Arbitrary correlation is not that condition. For arbitrary dependence, and for valid marginal p-values, replace q^* by q^*/c(m), where c(m)=\sum_{j=1}^{m} 1/j. Reject through the largest i satisfying P_{(i)}\le i q^*/[m\,c(m)]. The price of the broader guarantee is a smaller threshold.
Example 2 (Benjamini–Hochberg by hand) Question. Ten sorted p-values are 0.0002, 0.0009, 0.0041, 0.0121, 0.0230, 0.0302, 0.0420, 0.2400, 0.5200, 0.8900. Apply the procedure at m=10 and q^*=0.05, and compare with Bonferroni and with the unadjusted rule.
Solution. The thresholds i q^*/m are 0.005, 0.010, 0.015, and so on up to 0.050. Comparing: p_{(5)}=0.0230\le 0.025 holds, p_{(6)}=0.0302\le 0.030 fails by a hair, and p_{(7)}=0.0420>0.035. The largest i meeting the condition is k=5: reject the five smallest. Bonferroni at 0.05/10=0.005 would reject only three (0.0002, 0.0009, 0.0041), and the unadjusted rule at 0.05 would reject seven. The Benjamini–Yekutieli thresholds are those divided by c(10)=2.929 and give k=3. The rule takes the largest i for which p_{(i)} lies on or below the line i q^*/m and rejects everything up to it. In this example every earlier p-value is also below its own threshold. That need not happen.
One simulated family of 1{,}000 tests shows the same comparison as a picture: ordered p-values against the line i q/m, and rejection counts for the unadjusted rule, Bonferroni, and Benjamini–Hochberg. The step-up rule rejects more than Bonferroni and fewer than the unadjusted threshold.
Marginal testing can identify several correlated coordinates because each has an association with the response. A lasso fit instead considers them jointly and may retain only some interchangeable predictors. Neither procedure is guaranteed to select an entire correlated group, or exactly one representative. Their objectives differ before any particular selection occurs.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from