Top Highlights

• The method presents a major advance in addressing the oft-asked question, “how many permutations are required?” Even if a small number of permutations have been conducted, the investigator can be confident that this source of variance is reflected in the CI estimation, thereby adequately quantifying uncertainty in the FDR.
• Also, the approach can be applied directly to statistics with uncharacterized distributions, bypassing the need for p-values entirely. Thus, there is no assumption of uniform or unbiased p-values. The main assumption is that permuted results accurately reflect the null.
• The appropriateness of parametric distributions becomes a much more challenging issue in large-scale inference settings because the investigator is forced to work in the extreme tails to adjust for multiplicity. This problem is sometimes addressed by severe transformations such as quantile normalization (Becker et al., 2012), which can cause a loss...

Domain