Boosting e-BH via conditional calibration
summary
The gist
Boosting e-BH via Conditional Calibration (e-BH-CC) This paper introduces a novel general framework, termed e-BH-CC…
In short
The e-BH-CC framework enhances empirical Bayes (e-BH) to increase statistical power without sacrificing its False Discovery Rate control. It uses Conditional Calibration (CC) to transform original e-values into 'boosted' values, which are guaranteed to be at least as powerful. This method works across various dependence structures and testing scenarios, offering superior performance compared to standard methods.
Key concepts
- Conditional Calibration (CC)
- A technique that boosts e-values by identifying a sufficient statistic for each null hypothesis. This allows the procedure to evaluate the conditional joint distribution of e-values under a specific null hypothesis, enabling the creation of boosted e-values that are more powerful than the originals.
- General Framework for Arbitrary Dependence
- The method is designed to work across many different multiple testing problems, regardless of how dependent the test results are. It achieves this by leveraging partial information about the dependence structure, allowing it to be applied in settings like parametric testing or model-free selection.
- Guaranteed FDR Control
- A crucial guarantee ensuring that even after boosting e-values, applying the standard e-BH procedure still keeps the false discovery rate below a set level ($\alpha$). This is proven by showing that the sets of data points remain jointly equal in distribution when conditioned on specific variables.
- Boosted E-Values (e_b)
- The new e-values derived from the calibration process. These values are constructed to possess superior power compared to the initial e-values ($\mathbf{e}$). They serve as the input for a standard e-BH procedure that maintains strict FDR control.
Terminology used across episodes
This episode discusses
- Boosting e-BH via conditional calibration · Paper Radio
- Multiple testing under negative dependence
- Adaptive Storey's null proportion estimator
- Simultaneous Hypothesis Testing Using Internal Negative Controls with An Application to Proteomics
- Model-free selective inference under covariate shift via weighted conformal p-values
- The numeraire e-variable and reverse information projection
- A General Framework for Multiple Testing via E-value Aggregation and Data-Dependent Weighting
- Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers
- Improving knockoffs with conditional calibration
- Randomized and Exchangeable Improvements of Markov's, Chebyshev's and Chernoff's Inequalities
- On Controlling the False Discovery Rate in Multiple Testing of the Means of Correlated Normals Against Two-Sided Alternatives
- E-backtesting · Paper Radio
- Time-uniform central limit theory and asymptotic confidence sequences
- A Power and Prediction Analysis for Knockoffs with Lasso Statistics
- More powerful multiple testing under dependence via randomization
- Online multiple testing with e-values
- Post-selection inference for e-value based confidence intervals
- On the existence of powerful p-values and e-values for composite hypotheses
The paper
Boosting e-BH via conditional calibration · Read on arXiv
Junu Lee, Zhimei Ren
University of Pennsylvania
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Boosting e-BH via conditional calibration".
Jane: Here's the gist of the paper. Detailed Research Summary: Boosting e-BH via Conditional Calibration (e-BH-CC) This paper introduces a novel general framework, termed e-BH-CC, designed to significantly enhance the statistical power of the empirical Bayes (e-BH) procedure while rigorously maintaining its established False Discovery Rate (FDR) control guarantee, even under arbitrary dependence structures between e-values.
Tom: The core mechanism enabling this power boost is Conditional Calibration (CC), which transforms the original e-values into "boosted e-values" that are guaranteed to be at least as powerful as the originals.
Jane: Key Theoretical Contributions and Methodology: 1.
Tom: General Framework for Arbitrary Dependence: The framework is designed to operate across a wide class of multiple testing problems by leveraging partial information about the dependence structure.
Title and authors: Tom: So we're looking at this paper from Junu Lee and Zhimei Ren called "Boosting e-BH via conditional calibration." Basically, it tackles a big problem with e-values, which are used for multiple testing to control the false discovery rate.
Jane: It sounds like they found a way to make these e-values much more powerful without messing up that FDR control guarantee they already have. What's the core idea here?
Tom: The core idea is this technique called conditional calibration, where you take those original e-values and transform them into something new, the boosted e-values.
Lu: It’s about finding a sufficient statistic for each null hypothesis, which lets them look at the conditional joint distribution of those e-values given that statistic under the null hypothesis.
Meng: So they are essentially using some extra information about how dependent things are to adjust the original values up to a point where they have better power. What kind of dependence structures were they testing this on?
Tom: They've shown it works across three different types of multiple testing problems: parametric models, conditional independence testing in the model-X setting, and model-free conformalized selection.
Jane: That’s a lot of ground to cover for a paper with just this abstract. How does this calibration actually work in practice?
Tom: They define a critical value, b cj, based on an expectation involving that conditional distribution, and then they construct the boosted e-values, e b, using that calibration.
Lu: The crucial part is that by doing this construction, the authors guarantee that these new boosted e-values are at least as powerful as the original ones, which is what we call "no less—and are often more."
Meng: But power gains often come with a cost in terms of false discoveries, so how do they ensure the FDR control doesn't get blown out when you use these new values?
Tom: They formally establish this guarantee through Proposition one which shows that applying the standard e-BH procedure to these newly constructed boosted values still results in a rejection set with an FDR control level of alpha <ref:2404.17562#pg1>.
Jane: So, even though they made the underlying evidence stronger for each test, the overall process still respects that target false discovery rate. That seems like a big deal.
Tom: It really is because they show that e-BH-CC doesn't just offer more power; it’s much more efficient when you look at how much of your FDR budget you actually spend across all the tests.
Title and authors: Lu: They compare marginal boosting against e-BH-CC and find that conditional calibration is far more efficient in utilizing the FDR budget, leading to higher realized FDR control than its simpler counterpart.
Meng: For someone deploying this at scale, that efficiency matters because it means you get better discovery for the same statistical budget you're willing to spend on errors.
Tom: And they connect this work to other established procedures, showing that the dBH procedure is equivalent to applying e-BH to a specific set of derived e-values.
Jane: They also show a deterministic relationship between their weighted conformal e-value and the one from the Weighted Conformal Selection, proving it's always greater than or equal to it.
Lu: That connection through Lemma three is interesting because it links these new calibrated values directly back to established methods like WCS <ref:2404.17562#pg1>.
Meng: When I think about deployment, the paper mentions applications in real-world observational data for identifying individuals with specific counterfactual outcomes, which sounds useful in a practical sense.
Tom: And they also show superior performance when dealing with online testing and outlier detection compared to methods like WCS or knockoff filters.
Jane: It seems they’ve built a framework that's quite general, not just tied to one specific type of data model, which is something we need for real research.
Lu: The versatility across parametric models, conditional independence testing, and conformalized selection shows this isn't just a fix for one niche; it’s a general method.
Tom: And they address the challenge of sparsity and threshold phenomena by incorporating Monte Carlo error control using anytime-valid confidence sequences or filtering based on preliminary test statistics.
Meng: That addresses my concern about power loss when things get sparse, which is a real issue when you're trying to ship models that have to handle high-dimensional data.
Jane: And they also show how you can generate derandomized e-values from knockoffs by averaging multiple runs of the knockoff filter, which handles the variability in those randomized procedures.
Lu: So it seems they’ve addressed the practical hurdles around reproducibility and robustness when dealing with methods like model-X knockoffs.
Tom: They keep emphasizing that this framework provides a robust way to overcome the practical power limitations of standard e-BH by recalibrating evidence intelligently using conditional information.
Jane: It really sounds like they've taken a theoretically sound tool and given it the practical muscle it needed for real-world application.
Title and authors: Tom: They’ve shown that you can get better statistical performance without having to completely abandon the rigorous FDR control you need for reliable inference.
Lu: The key here is that this isn't just about tweaking a formula; it's about fundamentally rethinking how we use conditional information to boost evidence in multiple testing settings.
Meng: For deployment, what I’m hearing is that if you can get access to partial information about the dependence structure, this method becomes a strong candidate for more reliable testing pipelines.
Tom: Exactly. If we have that kind of partial structure info, this e-BH-CC approach should give us a nice bump in power while keeping the FDR tight.
Jane: So, to wrap up on this paper titled "Boosting e-BH via conditional calibration," it’s essentially a general framework that boosts the power of e-BH by conditionally calibrating those original e-values based on partial dependence information.
Tom: It works across three problem classes and guarantees FDR control, even under arbitrary dependence structures.
Jane: It seems like this method moves us closer to having more reliable multiple testing tools when we can't assume a perfect structure for our data.
Lu: The connection they make to other methods like WCS and dBH suggests this is part of a larger picture connecting different ways to derive robust e-values.
Meng: From an engineering standpoint, the mention of running on real traffic and the handling of sparsity issues makes this relevant for production systems, even if the theoretical bounds are what matter most right now.
Tom: The results show it significantly improves power over base methods and is more efficient in using up that false discovery rate budget compared to marginal boosting.
Jane: It’s a solid paper because they don't just claim a better number; they prove how the mechanism works and why the FDR control holds under these diverse settings.
Lu: The future work, I guess, will be exploring even more complex dependence structures or maybe extending this calibration concept to even broader classes of testing problems.
Meng: If we can get a more concrete benchmark run on actual production traffic, that would solidify its value beyond the simulations they ran on one thousand replications <ref:2404.17562#pg3>.
Tom: Well, it’s clear this paper provides a robust methodology for boosting e-BH while keeping the FDR control you need.
Jane: It's a solid piece of work for anyone interested in making multiple testing more practical and powerful under messy data conditions.
The paper's summary: Tom: So, to wrap up on this paper, they’ve introduced a method called e-BH-CC that takes your original e-values and calibrates them using conditional information to make them more powerful.
Jane: Right, so instead of just using the e-values as they are, you use some extra data—the dependence structure—to adjust those values up, making the tests stronger without messing up the false discovery rate control.
Tom: Exactly. The core mechanism is this conditional calibration technique, where they find a sufficient statistic for each null hypothesis to see how that affects the joint distribution of those e-values under that null hypothesis.
Lu: It’s about finding a critical value, b cj, based on an expectation involving that conditional distribution, and then building new boosted e-values, e b, from that calibration.
Jane: The big win here is they guarantee these new boosted values are at least as powerful as the original ones; they're "no less—and often more."
Meng: But how do you know this doesn't blow up your false discovery rate when you use these stronger numbers? That’s the big question, right?
Tom: The paper shows that applying the standard e-BH procedure to these new boosted values still keeps the false discovery rate control at alpha. They prove this using a proposition that shows the sets of data points are jointly equal in distribution under specific conditions.
Lu: It’s a general framework designed to work across three different types of problems: testing under parametric models, conditional independence testing in the model-X setting, and even model-free conformalized selection.
Jane: That generality is key because it means this isn't just a fix for one specific type of data; it’s meant to handle a lot of different multiple testing scenarios.
Tom: And they show that conditional calibration is way more efficient than just marginal boosting when you look at how much of your total false discovery rate budget you actually spend across all the tests.
Meng: Efficiency matters because I’m shipping models to production, and if this method uses less of my budget while getting higher power, it’s worth deploying.
Lu: They also connect this work to other established procedures, showing that the dBH procedure is equivalent to applying e-BH to a specific set of derived e-values.
Jane: And they prove that their weighted conformal e-value is deterministically greater than or equal to the one from Weighted Conformal Selection, which links it back to WCS.
Tom: So, this paper provides a robust methodology for boosting e-BH while keeping the false discovery rate control you need under arbitrary dependence structures.
Jane: It seems like this method moves us closer to having more reliable multiple testing tools when we can't assume a perfect structure for our data.
The paper's improvements: Tom: So, we’re looking at how they suggest improving the base e-BH procedure using this conditional calibration technique across different problem types.
Jane: Right, so beyond just boosting power generally, what are these specific suggestions for making the method better for different kinds of testing?
Lu: They focus on applying it to three distinct problem classes: first, testing under parametric models; second, conditional independence testing within the model-X setting; and third, model-free conformalized selection.
Tom: That’s a lot of ground to cover. The paper explicitly shows how they identify sufficient statistics for each of those three areas to calibrate the e-values accordingly.
Meng: But what are the practical implications here? Does this mean we have to know *exactly* which problem class we're in just to use this boosting thing?
Lu: Not exactly. The framework is designed to be general, so if you have partial information about the dependence structure, you can apply it across these areas.
Jane: So, if I’m running a test that doesn't fit perfectly into those three boxes—say, something new—can I still use this approach?
Lu: You can still use the general idea of conditional calibration to adjust your e-values based on whatever partial information you have about how those tests depend on each other.
Tom: The authors also suggest a way to handle power loss when things get sparse, using Monte Carlo error control with anytime-valid confidence sequences or filtering based on preliminary test statistics.
Meng: That’s good for production. I hate it when the model just tanks under high-dimensional data because of sparsity. A filter approach sounds like something we could actually put into a pipeline to monitor performance in real time.
Lu: And they also suggest generating derandomized e-values from knockoffs by averaging multiple runs, which fixes the high selection variability you see with randomized procedures like model-X knockoffs.
Jane: So, this isn't just about making the numbers look better; it’s about building a more robust pipeline that handles messy data and sparse situations better.
Tom: It shifts the focus from just finding one good baseline to creating a flexible calibration system that adapts to the specific tests you're running.
Meng: If this framework can handle online testing and outlier detection better than methods like knockoff filters, then it could actually improve how we detect anomalies in live data streams.
Lu: The paper also shows connections to existing methods, like proving that their weighted conformal e-value is deterministically greater than or equal to the one from Weighted Conformal Selection.
Jane: So, the implication is that this isn't just a new trick; it’s part of a bigger picture connecting different ways we derive these robust e-values.
Tom: It shows that if you can get access to partial information about dependence, you should be able to get better power while keeping the false discovery rate tight.
Conclusion: Tom: So, to wrap up this discussion on "Boosting e-BH via conditional calibration," we’ve seen how they take original e-values and adjust them using conditional information to get stronger test results while keeping that false discovery rate control intact.
Jane: Exactly, it’s a general method that works across three different testing problem types, parametric models, model-X settings, and conformalized selection.
Lu: The main implication is that if you have partial information about the dependence structure between your tests, you can use this framework to boost your power while maintaining rigor.
Meng: For deployment at scale, the efficiency gain means we’re using our budget better, which is exactly what I need when I’m trying to manage a high-throughput system.
Tom: And they showed it outperforms simpler methods like marginal boosting in how much of the false discovery rate budget you actually spend across all your tests.
Jane: It really shows that this isn't just tweaking a formula; it’s about fundamentally rethinking how we use conditional data to boost evidence in multiple testing.
Lalam: I'm confident this direction is solid, as the math they presented for the FDR guarantee holds up under those specific conditions.
Lu: It does, and they even proved deterministic relationships with established procedures like Weighted Conformal Selection, which is a nice link for representation learning.
Meng: The robustness against sparsity and threshold phenomena that they included in their experiments is what I’m most interested in; we deal with that all the time when shipping models.
Tom: So, to summarize, this paper introduces e-BH-CC as a robust way to improve power by intelligently calibrating evidence based on partial dependence info.
Jane: It moves us closer to having more reliable multiple testing tools when our data doesn't perfectly fit standard assumptions.
Tom: It’s a solid piece of work, and it opens the door for using this calibration idea in other complex machine learning scenarios.
Lalam: I think the connection they make to dBH equivalence is quite important for connecting these different ways to derive robust e-values.
Lu: Looking ahead, exploring even more complex dependence structures or extending this calibration concept to even broader classes of testing problems seems like a natural next step for this line of research.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates