BONuS: Multiple multivariate testing with a data-adaptive test statistic
summary
The gist
We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric.
In short
BONuS is a new adaptive multiple testing procedure for multivariate or nonparametric tests. It improves power and controls the false discovery rate by analyzing the entire dataset to estimate an optimal p-value transform using an empirical Bayes model, rather than relying on pre-defined univariate p-values.
Key concepts
- Bag-Of-Null-Statistics (BONuS)
- BONuS is an adaptive knockoff method for multiple testing. It works by hiding synthetic controls drawn from the null distribution within the real data and iteratively updating an estimator to find a rejection region, halting when a certain significance level is met.
- Empirical Bayes Model
- This model is used to adaptively estimate an optimal p-value transform. Instead of assuming a fixed transformation, this model learns the best way to summarize multivariate data based on the observed data itself, improving testing accuracy.
- FDR Control Guarantees
- The method guarantees control of the false discovery rate (FDR) at level alpha in both fixed-effects and two-groups models. This robust control holds even if the analyst uses an incorrect prior model for the empirical Bayes estimation, relying on martingale arguments.
Terminology used across episodes
This episode discusses
- BONuS: Multiple multivariate testing with a data-adaptive test statistic · Paper Radio
- A Generalized Benjamini-Hochberg Procedure for Multivariate Hypothesis Testing
- Hypothesis Testing for High-Dimensional Multinomials: A Selective Review
- Hypothesis Testing For Densities and High-Dimensional Multinomials: Sharp Local Minimax Rates
- Family learning: nonparametric statistical inference with parametric efficiency
- A New Approach for Large Scale Multiple Testing with Application to FDR Control for Graphically Structured Hypotheses
- Multiple testing with the structure adaptive Benjamini-Hochberg algorithm
- DAGGER: A sequential algorithm for FDR control on DAGs
- A Power and Prediction Analysis for Knockoffs with Lasso Statistics
The paper
BONuS: Multiple multivariate testing with a data-adaptive test statistic · Read on arXiv
Department of Statistics, UC Berkeley · Department of Statistics, Stanford University · Department of Statistics and Data Sciences, UT Austin
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "BONuS: Multiple multivariate testing with a data-adaptive test statistic".
Jane: Here's the gist of the paper. We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric.
Tom: BONuS is an adaptive and interactive knockoff-type method that helps improve the testing power while controlling the false discovery rate (FDR), and is closely connected to the “counting knockoffs” procedure analyzed in Weinstein et al.
Jane: (2017).
Tom: Contrary to procedures that start with a p-value for each hypothesis, our method analyzes the entire data set to adaptively estimate an optimal p-value transform based on an empirical Bayes model.
Title and authors: Jane: To summarize, the BONuS procedure begins by generating a set of synthetic controls from the null distribution and hiding them among your real statistics. Then, it proceeds iteratively, revealing more information to an analyst who updates an estimator based on new data as it comes in.
Lu: The core mechanic is that as soon as an estimator of the false discovery proportion falls below a specified significance level alpha, the analyst stops and rejects all null hypotheses where the real statistics are in that rejection region.
Tom: It’s like a game where you start by guessing, then you keep getting more clues from new data points until you can confidently decide which null hypotheses to reject based on that learning.
Meng: So if I were deploying this, the first thing I'd worry is how many synthetic controls we need to generate up front, since that involves a permutation of the data set X(one) through X(n) and ne synthetic controls <ref:2106.15743#pg1>.
Lalam: They do suggest using a Double BONuS procedure to validate this empirical Bayes model by introducing a second group of synthetic nulls, which is meant to guard against power loss due to model misspecification.
Jane: That validation step is key because it tests whether the empirical Bayes model they are using actually works well for picking the right test statistic.
Tom: It sounds like they’re not just relying on one method; they’re building in a check to make sure their learning mechanism isn't leading them down a dead end with bad assumptions.
Lu: They are essentially showing that this interactive approach, which pools information across hypotheses, can nearly recover the performance of an oracle test, even when using the EM principal component analysis on the full data set.
The paper's summary: Jane: The paper highlights that BONuS improves upon standard procedures by starting with a full dataset analysis rather than just one p-value for each hypothesis upfront.
Tom: That’s the difference, right? Most methods give you a p-value for each test and then we have to figure out how to combine those p-values later. BONuS does that learning process inside the procedure itself.
Lu: The paper shows that in multivariate Gaussian simulation, when using seven components for their model selection, they attained twenty one four discoveries in S at a level of zero point zero five false discovery rate control.
Meng: Twenty one four discoveries? That number sounds promising, but what if we tried to run this on real traffic where the underlying structure isn't perfectly Gaussian?
Lalam: The paper also shows that BONuS is especially suitable for high-dimensional settings and multinomial testing where an appropriate test statistic is usually unavailable, achieving power gains over agnostic and oracle procedures.
Jane: That’s what I mean by the structural information part; it lets the method learn the underlying structure from the data even when we don't have a pre-defined test statistic.
Tom: So they are claiming power gains over agnostic tests in situations where picking a good test statistic is incredibly hard, which is exactly where we need help.
Lu: They also address some limitations, like assuming independence across hypotheses and requiring null hypotheses to be simple rather than composite, which points toward future work.
The paper's improvements: Jane: So to wrap up this paper on BONuS: it’s a new adaptive empirical Bayes framework that lets you test many multivariate or nonparametric hypotheses by learning the optimal p-value transform from the data itself.
Tom: The main implication is that we can get better power and keep FDR control even in complicated settings where picking a test statistic is tough, provided we have enough data to let the learning happen.
Lu: The Double BONuS procedure is important because it validates whether that empirical Bayes model you’re using actually works before you rely on the power gains.
Meng: If we look at deployment, the paper suggests choosing a number of rejections based on n over alpha, which keeps the minimum number of rejections roughly the same across different methods.
Lalam: The final conclusion is that while it shows power gains depend heavily on our model choice, for any two pairs of test statistics, the ratio involving those statistics converges to zero, implying the power gain isn't guaranteed in every example.
Jane: So BONuS gives us a powerful tool for when we need to test many hypotheses at once and have a complex data structure without having to pre-specify everything.
Tom: It’s solid work that shows how learning from the data can actually improve testing power in these tough multivariate scenarios, even if it't conditional on our model assumptions holding true.
Lu: And they point toward the future by noting that assuming independence across hypotheses and requiring simple null hypotheses are things they need to resolve for more real-world use.
Meng: That makes sense, because those assumptions don't hold in genetics or medical imaging where we usually test many things simultaneously and with correlated data.
Lalam: Exactly, so BONuS is a strong step forward in using empirical Bayes to adapt testing procedures across different types of multivariate problems.
Conclusion: Jane: So we’ve been talking about BONuS, which is this new framework for multiple multivariate testing that learns the best way to test things based on all your data instead of starting with a fixed p-value for every hypothesis.
Tom: Yeah, it’s all about adapting the test statistic itself as you get more information from the data set and controlling that false discovery rate, which is a big deal when your experiments aren't simple univariate tests.
Lu: The core idea is this interactive knockoff-type method where you start generating synthetic controls drawn from the null distribution and then gradually reveal them while updating an estimator based on what you’ve seen so far.
Meng: From a deployment standpoint, it sounds like a lot of upfront computation to generate those initial synthetic controls, which is something I gotta think about for real-world shipping.
Lalam: But the paper shows that they guarantee robust false discovery rate control at level alpha whether or not you use the right model for the prior.
Jane: That robustness is important, because it means even if your initial assumptions about how things are distributed aren't perfect, the control should still hold up.
Tom: The power gain they show in multivariate Gaussian simulation, hitting twenty one four discoveries with a zero point zero five false discovery rate control using a seven component model, that’s the kind of number we need to see when we’re dealing with high-dimensional data.
Lu: That specific result relies on choosing the right model structure, and they showed how it helps select that optimal structure by running a Double BONuS procedure.
Meng: It sounds like they are suggesting that using PCA across different dimensions can help you cherry-pick the best model structure for capturing low rank information in those Gaussian distributions.
Jane: So, for someone just listening, this paper is showing how we can test many complex hypotheses at once without having to guess the exact test statistic beforehand and still keep a tight grip on false positives.
Tom: It’s about moving away from the naive approach of getting one p-value per hypothesis and instead letting the entire data set drive the learning process for better testing.
Lu: And they did point out some limitations, though, like BONuS assumes independence across hypotheses and requires null hypotheses to be simple rather than composite.
Meng: I’d ask what happens when you run this on real traffic data where everything is definitely correlated and the nulls are complex scenarios?
Jane: That’s the next big question for researchers, figuring out how to make BONuS work when those independence assumptions break down in fields like genetics or medical imaging.
Tom: Anyway, that’s what we saw with BONuS: it’s a method that adapts the test statistic based on empirical Bayes modeling to improve power while controlling the false discovery rate.
Lu: It's a strong tool for high dimensional settings where picking an appropriate test statistic is usually not available and the cost of an agnostic test is huge.
Lalam: The results show that this method can achieve power gains over both agnostic and oracle procedures in multinomial testing and even in metabolic syndrome GWAS applications.
Jane: So it’s a concrete way to see how learning from the data can lead to better results when you're dealing with complex, multi-faceted problems.
Tom: Yeah, so keep an eye on this BONuS work: it’s showing us that adaptive methods built around empirical Bayes are getting pretty useful for multivariate testing.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates