a) Learned weights of IHW atα = 0.1 applied to the RNA-seq dataset by Bottomly et al. [13] analyzed with DESeq2. The mean of the normalized counts for each gene is used as the covariate. Genes with low counts get assigned low weight. Note that for each value of the covariate, five potentially different weights were learned because of the splitting scheme (E2). The trend is consistent.b) Weights learned by IHW atα = 0.1 for the quantitative proteomics dataset [15]. As covariate we use the total number of peptides quantified for each protein.

a-b) Results from applying thet-test to Normal samples (n = 2 × 5, σ =1) with either the same mean (true nulls, π₀ = 0.95) or means differing by the effect size indicated on the x-axis (π₁ = 0.05). The pooled variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line). These plots are analogous toFigure 2 g-h, but with further methods.

a) The y-axis shows the actual FDR. All methods lose type I error control for low values of the effect size, though they regain it at higher effect sizes.

b) Power analysis for the t-test scenario.

Panelsc-f highlight size-investing. Data were simulated as in [7]: p-values were computed from the one-sided z-test, with Z_i∼N(0,1) under the null and Z_i∼N(ξ_i,1) under the alternative, with the covariate ξ_i∼ U [1, ξ_max]. The fraction of alternatives was π₁ = 0.1. The x-axis represents the simulation parameter ξ_max.

c,e) The y-axis shows the actual FDR. In this case, naive IHW, Clfdr and FDRreg are slightly anticonservative.

d,f) The logarithm of the power increase compared to BH is shown as a function of ξ_max. Large values of ξ_max necessitate a size-investing strategy. Methods which cannot apply such a strategy, including SBH, TST-GBH, LSL-GBH, FDRreg and Greedy Independent Filtering, have power even lower than BH or at best equal to it.

a) Cumulative distribution function of the p-value for tests of two hypotheses H₁ and H₂. The CDF in the case of H₁ is stochastically smaller (more “powerful”) than that in the case of H₂. This difference is due to different π₀s and alternative distributions for the two hypothesis tests.

b) Situation similar to panela, but here the difference in CDFs is due to different π₀ solely, while the alternative distributions are the same.

c) Graphs of p-value as a function of tdr (this plot is analogous toSupplementary Fig. 4d-f with flipped axes) for the 2 hypotheses in panela. Note that for a high fixed tdr value, the corresponding p-value for hypothesis H₁ is greater than for H₂. As the tdr decreases, eventually these two functions cross and the inequality changes direction. This explains the size-investing strategy.

d) Analogous to panelc, for the hypotheses in panelb. Because here only π₀ is different among H₁ and H₂, no size-investing can occur.

-c) These three density curves correspond to panelsb-d inFigure 3 based on simulated scenarios.d-f) Note that the procedure of calculating tdr(t) based on Equation (1) can be applied for each t. Thus the tdr is a deterministic function of the p-value and these panels show the graph of this function for scenariosa-c.g-i) Histograms of simulated p-values based on the distribution ina-c. In practice we observe only one of these p-values for each hypothesis. Therefore, this distribution can only be estimated under simplifying assumptions and by sharing information across the hypotheses.j-l) Histogram of the tdr values, i.e., of the transformed p-values (by Equation (1) in the main text). The tdr-values are also random variables. Notice that for high signal situations the tdr histogram gives a much better split than the corresponding p-value histogram. But again, since we do not know the distribution, in practice we cannot observe these random variables (i. e., they are not statistics, in contrast to the p-values), and we can only approximate them by using assumptions as above.

a) This panel benchmarks type I error control if all null hypotheses are true. Shown is the true FWER against the nominal significance level α; the dashed line indicates the identity function. Both Bonferroni and IHW- Bonferroni control the FWER. Panelsb-c look at implications of different effect sizes. The two-samplet-test was applied to Normal samples (n = 2 × 5, σ = 1) with either the same mean (true nulls) or means differing by the effect size indicated on the x-axis (true alternatives). The fraction of true alternatives was 0.05. The pooled empirical variance was used as the covariate. Rejections were made at the nominal level α = 0.1 (dashed line).b) The y-axis shows the actual FWER, which is controlled for both methods.c) A power analysis. IHW-Bonferroni shows dramatic power improvements compared to Bonferroni. Panelsd-e highlight the size-investing simulation: p-values were computed from the one-sided z-test, with Z_i∼N(0,1) under the null and Z_i∼N(ξ_i,1) under the alternative, with the covariate ξ_i∼ U [1, ξ_max]. The fraction of alternatives was 0.1. The x- axis represents the simulation parameter ξ_max.d) The y-axis shows the logarithm of the actual FWER, as well as the nominal level log₁₀(α) = −1. FWER is again controlled by both methods.e) The power analysis again shows a power increase for IHW-Bonferroni.

Supplementary Figures 1–5, Supplementary Table 1 and Supplementary Notes 1–7 (PDF 675 kb)

IHW and IHWpaper R packages for implementing the methodology described in this paper and for reproducing the results. (ZIP 15107 kb)