False discovery rate
Section of High-dimensional analysis, part of Statistical learning
False discovery rate
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
| Questions using it | 1 |
False discovery rate control offers a different error criterion (Benjamini and Hochberg, 1995). It concerns the expected fraction of false rejections among all rejections. Write m for the number of tests, keeping p available for a p-value only when there is no ambiguity.
| Declared non-significant | Declared significant | Total | |
|---|---|---|---|
| H_0 true | U | V | M_0 |
| H_0 false | T | S | M_1 |
| Total | m-R | R | m |
The table separates the unknown truth from the random decisions. In this frequentist formulation the set of true nulls, and hence M_0 and M_1, is fixed but unknown. The counts U,V,T,S,R vary across repeated data sets. For M_0>0, the average null rejection probability is \bar\alpha=\mathbb{E}(V)/M_0. This average need not equal a common nominal level for every test. When M_1>0, the corresponding average type II error probability and power are
\bar\beta=\frac{\mathbb{E}(T)}{M_1}, \qquad \text{average power}=1-\bar\beta=\frac{\mathbb{E}(S)}{M_1}.
The family-wise error rate is \mathrm{FWER}=\mathbb{P}(V\ge 1). The false discovery proportion is Q=V/R when R>0 and Q=0 when V=R=0: the proportion of the variables declared significant that are in fact null. A declared-significant hypothesis is a false discovery when its null hypothesis is true. The false discovery rate is its expected value,
\mathrm{FDR}=\mathbb{E}(Q)=\mathbb{E}\Bigl(\frac{V}{R}\Bigm| R>0\Bigr)\mathbb{P}(R>0).
Set the false discovery proportion to zero when there are no rejections. This convention makes the rate well-defined even under the complete null. In contrast, \mathbb{E}(V/R\mid R>0) conditions on making at least one discovery. The two quantities differ by the factor \mathbb{P}(R>0), which can be small. Neither quantity bounds the false fraction in every individual realised list.
Theorem 1 (Family-wise control implies false-discovery control) When all null hypotheses are true (m_0=m), \mathrm{FDR}=\mathrm{FWER}. In general \mathrm{FDR}\le\mathrm{FWER}.
Article: Family-wise control implies false-discovery control →
If all nulls are true, S=0 and so V=R. If V=0 then Q=0. If V\ge 1 then Q=1. So \mathbb{P}(V\ge 1)=\mathbb{E}(Q). In general, I(V\ge 1)\ge Q: if V=0 both sides are 0; if V\ge 1 the left side is 1 and Q=V/R\le 1. Taking expectations, \mathbb{P}(V\ge 1)\ge\mathbb{E}(Q).
Any procedure that controls the family-wise error rate therefore controls the false discovery rate. If one controls only the false discovery rate, the family-wise error rate can be considerably higher. That may be of no concern, and the benefit can be a large increase in the power of the overall procedure.
Used in
- Taught in High-dimensional analysis
- Questions that use it (1) — Selection in many dimensions
- All 9 questions on High-dimensional analysis
See also
Bonferroni correction · Family-wise error rate · Multiple testing · Benjamini–Hochberg · Tukey intervals · Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n