BONuS: Multiple multivariate testing with a data-adaptive test statistic

arXiv:2106.15743 · stat.ME, math.ST, stat.TH · Submitted 2021-06-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "BONuS: Multiple multivariate testing with a data-adaptive test statistic".

Jane: Here's the gist of the paper. We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric.

Tom: BONuS is an adaptive and interactive knockoff-type method that helps improve the testing power while controlling the false discovery rate (FDR), and is closely connected to the “counting knockoffs” procedure analyzed in Weinstein et al.

Jane: (2017).

Tom: Contrary to procedures that start with a p-value for each hypothesis, our method analyzes the entire data set to adaptively estimate an optimal p-value transform based on an empirical Bayes model.

Title and authors: Jane: To summarize, the BONuS procedure begins by generating a set of synthetic controls from the null distribution and hiding them among your real statistics. Then, it proceeds iteratively, revealing more information to an analyst who updates an estimator based on new data as it comes in.

Lu: The core mechanic is that as soon as an estimator of the false discovery proportion falls below a specified significance level alpha, the analyst stops and rejects all null hypotheses where the real statistics are in that rejection region.

Tom: It’s like a game where you start by guessing, then you keep getting more clues from new data points until you can confidently decide which null hypotheses to reject based on that learning.

Meng: So if I were deploying this, the first thing I'd worry is how many synthetic controls we need to generate up front, since that involves a permutation of the data set X(one) through X(n) and ne synthetic controls <ref:2106.15743#pg1>.

Lalam: They do suggest using a Double BONuS procedure to validate this empirical Bayes model by introducing a second group of synthetic nulls, which is meant to guard against power loss due to model misspecification.

Jane: That validation step is key because it tests whether the empirical Bayes model they are using actually works well for picking the right test statistic.

Tom: It sounds like they’re not just relying on one method; they’re building in a check to make sure their learning mechanism isn't leading them down a dead end with bad assumptions.

Lu: They are essentially showing that this interactive approach, which pools information across hypotheses, can nearly recover the performance of an oracle test, even when using the EM principal component analysis on the full data set.

The paper's summary: Jane: The paper highlights that BONuS improves upon standard procedures by starting with a full dataset analysis rather than just one p-value for each hypothesis upfront.

Tom: That’s the difference, right? Most methods give you a p-value for each test and then we have to figure out how to combine those p-values later. BONuS does that learning process inside the procedure itself.

Lu: The paper shows that in multivariate Gaussian simulation, when using seven components for their model selection, they attained twenty one four discoveries in S at a level of zero point zero five false discovery rate control.

Meng: Twenty one four discoveries? That number sounds promising, but what if we tried to run this on real traffic where the underlying structure isn't perfectly Gaussian?

Lalam: The paper also shows that BONuS is especially suitable for high-dimensional settings and multinomial testing where an appropriate test statistic is usually unavailable, achieving power gains over agnostic and oracle procedures.

Jane: That’s what I mean by the structural information part; it lets the method learn the underlying structure from the data even when we don't have a pre-defined test statistic.

Tom: So they are claiming power gains over agnostic tests in situations where picking a good test statistic is incredibly hard, which is exactly where we need help.

Lu: They also address some limitations, like assuming independence across hypotheses and requiring null hypotheses to be simple rather than composite, which points toward future work.

The paper's improvements: Jane: So to wrap up this paper on BONuS: it’s a new adaptive empirical Bayes framework that lets you test many multivariate or nonparametric hypotheses by learning the optimal p-value transform from the data itself.

Tom: The main implication is that we can get better power and keep FDR control even in complicated settings where picking a test statistic is tough, provided we have enough data to let the learning happen.

Lu: The Double BONuS procedure is important because it validates whether that empirical Bayes model you’re using actually works before you rely on the power gains.

Meng: If we look at deployment, the paper suggests choosing a number of rejections based on n over alpha, which keeps the minimum number of rejections roughly the same across different methods.

Lalam: The final conclusion is that while it shows power gains depend heavily on our model choice, for any two pairs of test statistics, the ratio involving those statistics converges to zero, implying the power gain isn't guaranteed in every example.

Jane: So BONuS gives us a powerful tool for when we need to test many hypotheses at once and have a complex data structure without having to pre-specify everything.

Tom: It’s solid work that shows how learning from the data can actually improve testing power in these tough multivariate scenarios, even if it't conditional on our model assumptions holding true.

Lu: And they point toward the future by noting that assuming independence across hypotheses and requiring simple null hypotheses are things they need to resolve for more real-world use.

Meng: That makes sense, because those assumptions don't hold in genetics or medical imaging where we usually test many things simultaneously and with correlated data.

Lalam: Exactly, so BONuS is a strong step forward in using empirical Bayes to adapt testing procedures across different types of multivariate problems.

Conclusion: Jane: So we’ve been talking about BONuS, which is this new framework for multiple multivariate testing that learns the best way to test things based on all your data instead of starting with a fixed p-value for every hypothesis.

Tom: Yeah, it’s all about adapting the test statistic itself as you get more information from the data set and controlling that false discovery rate, which is a big deal when your experiments aren't simple univariate tests.

Lu: The core idea is this interactive knockoff-type method where you start generating synthetic controls drawn from the null distribution and then gradually reveal them while updating an estimator based on what you’ve seen so far.

Meng: From a deployment standpoint, it sounds like a lot of upfront computation to generate those initial synthetic controls, which is something I gotta think about for real-world shipping.

Lalam: But the paper shows that they guarantee robust false discovery rate control at level alpha whether or not you use the right model for the prior.

Jane: That robustness is important, because it means even if your initial assumptions about how things are distributed aren't perfect, the control should still hold up.

Tom: The power gain they show in multivariate Gaussian simulation, hitting twenty one four discoveries with a zero point zero five false discovery rate control using a seven component model, that’s the kind of number we need to see when we’re dealing with high-dimensional data.

Lu: That specific result relies on choosing the right model structure, and they showed how it helps select that optimal structure by running a Double BONuS procedure.

Meng: It sounds like they are suggesting that using PCA across different dimensions can help you cherry-pick the best model structure for capturing low rank information in those Gaussian distributions.

Jane: So, for someone just listening, this paper is showing how we can test many complex hypotheses at once without having to guess the exact test statistic beforehand and still keep a tight grip on false positives.

Tom: It’s about moving away from the naive approach of getting one p-value per hypothesis and instead letting the entire data set drive the learning process for better testing.

Lu: And they did point out some limitations, though, like BONuS assumes independence across hypotheses and requires null hypotheses to be simple rather than composite.

Meng: I’d ask what happens when you run this on real traffic data where everything is definitely correlated and the nulls are complex scenarios?

Jane: That’s the next big question for researchers, figuring out how to make BONuS work when those independence assumptions break down in fields like genetics or medical imaging.

Tom: Anyway, that’s what we saw with BONuS: it’s a method that adapts the test statistic based on empirical Bayes modeling to improve power while controlling the false discovery rate.

Lu: It's a strong tool for high dimensional settings where picking an appropriate test statistic is usually not available and the cost of an agnostic test is huge.

Lalam: The results show that this method can achieve power gains over both agnostic and oracle procedures in multinomial testing and even in metabolic syndrome GWAS applications.

Jane: So it’s a concrete way to see how learning from the data can lead to better results when you're dealing with complex, multi-faceted problems.

Tom: Yeah, so keep an eye on this BONuS work: it’s showing us that adaptive methods built around empirical Bayes are getting pretty useful for multivariate testing.

Department of Statistics, UC Berkeley · Department of Statistics, Stanford University · Department of Statistics and Data Sciences, UT Austin

stat.ME, math.ST, stat.TH

Submitted: 2021-06-29

Updated: 2026-11-10

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric.

Key concepts

Bag-Of-Null-Statistics (BONuS)
BONuS is an adaptive knockoff method for multiple testing. It works by hiding synthetic controls drawn from the null distribution within the real data and iteratively updating an estimator to find a rejection region, halting when a certain significance level is met.
Empirical Bayes Model
This model is used to adaptively estimate an optimal p-value transform. Instead of assuming a fixed transformation, this model learns the best way to summarize multivariate data based on the observed data itself, improving testing accuracy.
FDR Control Guarantees
The method guarantees control of the false discovery rate (FDR) at level alpha in both fixed-effects and two-groups models. This robust control holds even if the analyst uses an incorrect prior model for the empirical Bayes estimation, relying on martingale arguments.

Terminology

Summary

We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric. p. 0 BONuS is an adaptive and interactive knockoff-type method that helps improve the testing power while controlling the false discovery rate (FDR), and is closely connected to the “counting knockoffs” procedure analyzed in Weinstein et al. p. 0 (2017). Contrary to procedures that start with a p-value for each hypothesis, our method analyzes the entire data set to adaptively estimate an optimal p-value transform based on an empirical Bayes model. p. 0

Multiple Testing Challenges

In the literature of multiple testing, it is customary to begin with one p-value for each of n null hypotheses as the primitive inputs and then focus on designing or analyzing methods for processing them. In many scientific problems, however, each of the n experiments yields multivariate data, and it is unclear a priori how best to summarize each one with a univariate p-value. p. 0 As the dimension of each problem grows, an agnostic p-value transformation may yield little power unless an exceptionally strong signal is present. p. 0 In a single multivariate testing problem, we cannot avoid paying the price of agnosticism without prior knowledge of which alternatives are more likely to occur. p. 1

The BONuS Procedure Mechanics

The BONuS procedure begins by either generating a set of ne synthetic controls drawn from the null distribution, Xf(1),..., Xf(ne) i.i.d. ∼ f0(x), and then hiding them among the real statistics X(1),. p. 5.., X(n). Formally, the analyst observes a permuted version of the data set, Z = Π X(1),. p. 5.., X(n), Xf(1),..., Xf(ne), where Π is a uniformly random permutation on n+ = n + ne elements. p. 5 The procedure proceeds iteratively, gradually revealing more information to the analyst, who continually updates an estimator Tb: X → R of either LRΛ(x) or fmix(x)/f0(x) as new information arrives. p. 5 As soon as an estimator of FDP falls below a pre-specified significance level α, the analyst halts the procedure and rejects all null hypotheses H(i) for which X(i) is in the current rejection region. p. 5

FDR Control Guarantees

Theorem 2 controls FDR in the fixed-effects model (1), and also in the two-groups model (2) for any π0 and Λ, whether or not the analyst uses a correctly specified model for the prior. The proof relies on optional stopping arguments involving martingales derived from hypergeometric random variables. p. 8 Specifically, FDR is controlled both conditionally and marginally regardless of whether the analyst specifies a correct model for Λ. p. 3

Power and Model Selection

The gain in power depends on the model we choose. The Double BONuS procedure is proposed as a validation scheme for assessing the power gain of different variants of the BONuS procedure by introducing a second group of synthetic nulls. p. 0 In multivariate Gaussian simulation, the optimal choice is the model with 7 components, where we attained 214 ‘discoveries’ in S for a level 0. p. 105 FDR control. p. 1

Future Directions

For future work, there are two remaining issues with the work that we feel it is especially pressing to resolve. First, like many other FDR-controlling multiple testing methods, BONuS assumes independence across the hypotheses, an unrealistic assumption in many of the most popular applications of multiple testing in genetics and medical imaging. p. 1 Second, BONuS requires the null hypotheses to be simple rather than composite, and does not allow for incorporating covariates. p. 2

Convergence Rates of MLE

Theorem 5 shows that the convergence rates of Gbn to G∗ are exponential in log n. Specifically, Pλbn − λ∗kΨb nkkΨ∗

k > C1 log n/n 1/2 ≤ exp(-c1 log n). This result is crucial for evaluating the consistency of the estimator. p. 6

Double BONuS Implementation Details

The Double BONuS procedure involves three steps, including running each variant of BONuS on Z and Ze. The method recommends choosing ˜n in the order of n/α, unless constraint in computational power or storage forbids us from doing so. p. 0 This ensures that the minimum number of rejections for the Storey-BONuS method is roughly the same. p. 1 The optimal choice in the simulation was a model with 7 components, where we attained 214 ‘discoveries’ in S. p. 1

Multivariate Multinomial Testing

BONuS is especially suitable for a high dimensional setting where an appropriate test statistic is typically not available and the cost of an agnostic test is huge. In the simulation of multinomial tests, the empirical FDR comparison shows that BONuS can achieve power gains over agnostic and oracle procedures. p. 1 The optimal choice is the model with 7 components, where we attained 214 ‘discoveries’ in S. p. 0 p. 0

Metabolic Syndrome GWAS Application

In the study of metabolic syndrome, BONuS is applied to identify SNPs related to the condition across multiple phenotypes. The empirical FDR comparison shows that BONuS can achieve power gains over agnostic and oracle procedures. p. 1 The optimal choice is the model with 7 components, where we attained 214 ‘discoveries’ in S.

Convergence Proof Summary

The proof of Proposition 8 shows that Case 3 cannot happen because it leads to a contradiction regarding the first-order identifiability of the Gaussian distribution with only covariance parameter. This implies that not all coefficients of ∂αf1/∂Ψα (xΨ∗n) go to 0 as 1 ≤ α ≤ 2. p. 3 The final conclusion is that the maximum of these coefficients leads to a contradiction, showing that Case 3 cannot happen.

Final Conclusion on Power

The results indicate that the power gain depends on the model we choose. The final result shows that for any two pairs (u1, v1) and (u2, v2), the ratio involving the test statistic converges to zero. p. 1 p. 1 This implies that the power gain is not guaranteed to be significant in every example. p. 3 p. 3

Convergence Rate of Test Statistics

The convergence rate of the test statistics is shown to be exponential in log n. The final result shows that for any two pairs (u1, v1) and (u2, v2), the ratio involving the test statistic converges to zero. This implies that the power gain is not guaranteed to be significant in every example.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive Multiple Testing Framework: The system can implement a Bag-Of-Null-Statistics (BONuS) procedure to perform multiple testing where each hypothesis is multivariate or nonparametric, adaptively estimating an optimal p-value transform based on an empirical Bayes model to improve testing power while controlling the false discovery rate.

  2. Robust FDR Control in Finite Samples: The system guarantees robust finite sample control of the false discovery rate at level alpha whether or not the empirical Bayes working model for the prior is correctly specified, as shown by Theorem 2 and its proof relying on optional stopping arguments related to hypergeometric random variables.

  3. Cross-Validation for Model Selection: The system can utilize a Double BONuS procedure, which involves running multiple variants of BONuS on mixed data sets to select the most rejections, helping the analyst select the most reasonable modeling assumptions among several competitors while maintaining FDR control.

  4. Power Optimization via Structural Information: By employing Double BONuS with principal component analysis (PCA) across different dimensions, the system can cherry-pick an optimal model structure that captures low rank information in multivariate Gaussian distributions, leading to substantial power gains over agnostic tests.

  5. High Dimensional and Nonparametric Testing: The system is specifically designed for high dimensional settings and multinomial testing where appropriate test statistics are unavailable, allowing it to achieve significant power improvements even in relatively low dimensions by learning the underlying structure from the data.

Sources

Related papers