arXiv:2106.15743v3 Announce Type: replace-cross Abstract: We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric. BONuS is an adaptive and interactive knockoff-type method that helps improve the testing power while controlling the false discovery rate (FDR). The BONuS procedure is closely connected to many existing FDR-controlling methods, such as the counting knockoffs procedure and Benjamini-Hochberg procedure with conformal p-values. Contrary to procedures that start with a $p$-value for each hypothesis, BONuS analyzes the entire data set to adaptively estimate an optimal $p$-value transform based on an empirical Bayes model. Moreover, it can iteratively refine the model fit by peeling off synthetic null statistics to reduce noise in a highly adaptively way. Our method controls FDR in finite samples even if the empirical Bayes model is incorrect or the estimation is poor. An extension, the Double BONuS procedure, validates the empirical Bayes model to guard against power loss due to model misspecification. We demonstrate the power gain of BONuS procedure on multi-trait GWAS, asset pricing, and LLM evaluation.