Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning

arXiv:2310.07973 · stat.ME, math.OC, stat.AP, stat.ML · Submitted 2023-10-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning".

Paper summary: Tom: So, we're looking at this paper today called "Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning." Essentially, it tackles how researchers use machine learning to find people who will get the most benefit from a treatment, or who might actually be harmed by it.

Jane: That sounds really practical for clinical trials or even policy decisions where you need to know exactly who gets what. But the paper points out that the usual way of doing this has some serious problems because those conditional average treatment effect estimates are often noisy and biased.

Lu: Exactly, and they point out that when you use the same data to both select a group based on an ML score and then estimate the effect for just that selected group, you run into a multiple testing problem. That's not statistically sound if you're trying to find those exceptional responders (this paper, Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Meng: From a deployment side, I worry about how reliable those ML scores are when we move them from a research setting to actual production where data might be messy. If the score is unreliable, the whole subgroup identification process falls apart.

Lalam: The core idea they are proposing is developing uniform confidence bands for estimating what they call the GATES, which is just the group average treatment effect among individuals grouped by that prioritization score. They claim this gives a statistical guarantee regardless of how you choose that generic machine learning algorithm (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Tom: So what's the big claim here? They aren't just suggesting a better way to do things; they are providing a statistical performance guarantee for subgroup identification. That means they can identify that specific subgroup with a certain probability, no matter how messy the underlying ML algorithm is.

Jane: That sounds like it solves the problem of relying too much on those noisy CATE estimates when trying to select patients based on them. They are essentially building a way to be statistically sure about your subgroup identification even when you're using generic machine learning models (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Lu: Theoretically, they tackle this by showing that finding those uniform confidence bands is actually equivalent to deriving the distribution of a particular functional of induced order statistics. They then use that idea to show that GATES are non-negatively correlated with each other no matter how you build your treatment prioritization score.

Meng: But what does "non-negatively correlated" mean for us in practice? Does it just mean that if one group is likely to respond well, the others are too? That sounds like a strong assumption for real data.

Paper summary: Lalam: The paper suggests they can bound this correlation using something called a Wiener process through something called Donsker's theorem. This mathematical framework then leads them to construct a uniform confidence band in Theorem one that minimizes the area of the resulting confidence region by approximating it with a specific functional form (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning) <ref:2310.07973#pg2>.

Tom: So, they are minimizing the area of that confidence region using this approximation. That sounds like they're making the most efficient use of their statistical tools when trying to set those thresholds for selecting responders.

Jane: It’s interesting because they claim this approach works even if you don't have a performance guarantee on your initial treatment prioritization score, which is something that usually trips people up (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Lu: They extend their Theorem one to mean-adjusted individual level statistics like the Population Average Prescriptive Effect curve by replacing the Wiener process with a Brownian bridge in the confidence band construction. This shows they can apply this concept to other metrics too, like cost adjusted effects (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Meng: Applying it to things like cost adjusted effects is where I see a real win for production systems because we often have to optimize based on more than just raw efficacy. But what about the uncertainty of the scoring rule itself? How do you know if your empirical ML score is actually good enough for this guarantee?

Lalam: They handle that by introducing alternative assumptions, like assumption three A and four A, which allow them to evaluate performance using an estimated prioritization score by assuming a uniform margin condition on the true treatment prioritization score and a rate condition on the convergence of that estimated score (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Tom: That's smart because it lets us look at real-world deployment scenarios where our ML model might be slightly off, not just in a perfect theoretical world. So, what did they find when they actually tested this?

Jane: They ran a simulation study using data from the two thousand sixteen Atlantic Causal Inference Conference Competition and found that these uniform confidence bands reach nominal coverage for every dataset size down to n equals one hundred while pointwise confidence bands suffered significant undercoverage (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Lu: That's a strong empirical result because it shows the method holds up across different data sizes, which is usually where these things break down. Pointwise bands are way worse.

Meng: But I still wonder about the real-world complexity of setting those thresholds based on this GATES estimation. If you’re deploying this, how hard is it to calculate that average effect for every potential subgroup?

Paper summary: Lalam: They characterize the selected subgroup by estimating the average value of covariates among them, like E(Xj F(S) greater than or equal to one minus p star), which gives you an interpretable description of those exceptional responders through covariate profiles (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Jane: So, if we find this subgroup, we can describe *why* they are responding well by looking at their average characteristics. The clinical trial example showed that the average tumor size in the selected group was four point two centimeters larger than the population average, which was statistically significant at a ten percent alpha level (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Tom: That's concrete. So, if you are trying to target a treatment effectively, this paper suggests you don't have to rely on just one noisy ML score; you use these uniform confidence bands to get a statistical guarantee on the group average treatment effect (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Jane: And when we look at the authors and the title of this paper, "Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning," it really sounds like they are giving us a robust toolkit instead of just another black box model (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Lu: It’s about moving away from that two-step process where you estimate the conditional average treatment effect first and *then* select individuals, because that compresses everything into one score too early (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Meng: For me, the practical implication is reducing the risk when we use these subgroup analyses in a live setting. If this method gives us a statistical guarantee about how much effect we're actually seeing, that makes deployment much safer (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Lalam: It’s useful when your evaluation sample size is limited, or when the ML models give you biased or noisy estimates of treatment efficacy, and when the proportion of exceptional responders is relatively small (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Tom: So to wrap up this discussion on "Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning," it’s about giving us a rigorous statistical performance guarantee when we select an optimal subset of a population who are predicted to benefit most from a treatment (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Jane: It’s about exploiting modern machine learning algorithms while providing that rigorous performance guarantee, which is super useful when you need to be sure about your subgroup identification results (Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning).

Conclusion: Tom: So we’ve been looking at how researchers try to find those exceptional responders using machine learning, and this paper is called "Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning."

Jane: It basically gives a statistical guarantee that you can actually find that subgroup reliably, no matter which generic machine learning algorithm you use.

Tom: So what does that mean in plain terms? It means they’re not just guessing which patients are good; they’re giving you a mathematical bound on how sure you can be about the average treatment effect for the group you pick.

Lu: They tackle the problem of picking individuals based on an ML score, which is usually a risky two-step process, and they show how to make that selection statistically sound.

Meng: I’m still thinking about deployment here. If this method gives us a guarantee, does it mean we can actually trust the results when we run it on real messy data?

Lalam: The authors address that uncertainty by introducing some specific assumptions about the scoring rule, which lets you evaluate performance even if your ML score isn't perfect.

Jane: And they did show empirical validation using simulation data, showing that their method achieves nominal coverage for every dataset size down to a thousand samples. That’s pretty strong for a methodology like this.

Tom: So the numbers are there, and they show it works across different sample sizes, not just in textbook scenarios.

Lu: What’s also interesting is how they handle the uncertainty of the scoring rule itself by assuming certain margin conditions on that score. It shows a deep dive into bounding that estimation error.

Meng: I wonder what the practical cost of running this kind of uniform confidence band calculation is when you have millions of potential subgroups to check.

Lalam: They do characterize those selected groups by estimating the average value of specific covariates among them, which lets you describe *why* they responded well through their profiles.

Jane: That’s a useful piece of information for clinicians or doctors; it gives you a reason for the response beyond just the treatment itself.

Tom: So we’ve seen how they build this guarantee, and what the simulation results show about its reliability across different data sizes. Next up, we look at how this translates to actual medical applications in prostate cancer trials.

Michael Lingzhi Li, Kosuke Imai

Harvard Business School · Harvard University

stat.ME, math.OC, stat.AP, stat.ML

Submitted: 2023-10-12

Updated: 2026-11-17

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: Across a wide array of disciplines, many researchers use machine learning (ML) algorithms to identify a subgroup of individuals who are likely to benefit from a treatment the most (“exceptional…

Key concepts

Grouped Average Treatment Effect (GATES)
This is the average treatment effect calculated among a specific subgroup of individuals defined by their prioritization score. It measures how much better or worse the treatment performs for this selected group compared to others.
Uniform Confidence Bands
These are statistical boundaries used to estimate the true GATES with a guaranteed level of accuracy. The method constructs these bands uniformly across different possible effect sizes, providing a reliable statistical guarantee for subgroup identification.
Treatment Prioritization Score
This is the score generated by a generic machine learning algorithm that ranks individuals based on their predicted response to treatment. The method works even if this score does not have a guaranteed performance level.
Neyman’s Repeated Sampling Framework
This theoretical framework is used to establish the statistical foundation for the method, assuming only randomization of treatment and random sampling of units. It allows the researchers to find a quantile cutoff with a specified probability.

Terminology

Summary

Across a wide array of disciplines, many researchers use machine learning (ML) algorithms to identify a subgroup of individuals who are likely to benefit from a treatment the most (“exceptional responders”) or those who are harmed by it. p. 0

Where the paper says it

Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning∗ Michael Lingzhi Li† Kosuke Imai‡ September 3, 2025 Abstract Across a wide array of disciplines, many researchers use machine learning (ML) algorithms to identify a subgroup of individuals who are likely to benefit from a treatment the most (“exceptional responders”) or those who are harmed by it. A common approach to this subgroup identification problem consists of two steps.

The proposed methodology develops uniform confidence bands for estimating the group average treatment effect sorted by generic ML algorithm (GATES) to provide a statistical guarantee for subgroup identification regardless of how the effect size is chosen. p. 0
Where the paper says it

To address these challenges, we develop uniform confidence bands for estimation of the group average treatment effect sorted by generic ML algorithm (GATES). Using these uniform confidence bands, researchers can identify, with a statistical guarantee, a subgroup whose GATES exceeds a certain effect size, regardless of how this effect size is chosen.

Problem Context

A common approach to subgroup identification involves two steps: first, estimating the conditional average treatment effect (CATE) using an ML algorithm, and second, selecting individuals based on that estimate. This two-step procedure compresses rich heterogeneity into a one-dimensional score and may be suboptimal for identifying exceptional responders. p. 1

Where the paper says it

For example, one might define the exceptional responders as those in the top quantile of the score and report the average treatment effect within this group. Such a two-step procedure inevitably compresses rich heterogeneity into a one-dimensional score and may therefore be suboptimal in identifying exceptional responders.

The Grouped Average Treatment Effect (GATES) is the average treatment effect among individuals grouped by the prioritization score. p. 1
Where the paper says it

, 2022; Imai et al. In the literature, the average treatment effect among individuals grouped by the prioritization score is referred to as the Grouped Average Treatment Effect or GATES (Chernozhukov et al.

This approach faces challenges such as statistical uncertainty in estimating GATES and a multiple testing problem when utilizing the same data to select a group and estimate its effect. p. 2
Where the paper says it

Third, accounting for this statistical uncertainty is difficult because utilizing the same data to both evaluate the GATES and select the group of potential exceptional responders suffers from a multiple testing problem. Although this problem can be alleviated by setting aside a large holdout set for evaluation, such an approach is often not a feasible solution given the limited sample size of randomized control trials.

Methodology Development

The core of the method involves developing uniform confidence bands for estimating the GATES among a subgroup defined by a quantile of the treatment prioritization score. This methodology is based on Neyman’s repeated sampling framework, assuming only randomization of treatment and random sampling of individual units without requiring any modeling assumption. p. 2

Where the paper says it

For example, our methodology enables researchers to choose the quantile cutoff such that the GATES among the selected exceptional responders exceeds a certain threshold with a pre-specified probability. The proposed methodology is based on Neyman’s repeated sampling framework (Neyman, 1923), assuming only randomization of treatment and random sampling of individual units without requiring any modeling assumption or repeated sampling methods.

The goal is to find a quantile cutoff such that the GATES among the selected exceptional responders exceeds a certain threshold with a pre-specified probability. p. 2
Where the paper says it

The proposed uniform confidence bands are valid regardless of the way in which the quantile value is chosen, thereby providing a statistical guarantee for a generic objective function to be optimized when selecting the group of individuals who are either most positively or most negatively affected by the treatment. For example, our methodology enables researchers to choose the quantile cutoff such that the GATES among the selected exceptional responders exceeds a certain threshold with a pre-specified probability.

Theoretical Foundation

The development relies on showing that the problem of finding uniform confidence bands is equivalent to deriving the distribution of a particular functional of induced order statistics. The authors show that GATES are non-negatively correlated with one another regardless of how the treatment prioritization score is constructed, allowing for bounding by a Wiener process through Donsker’s theorem. p. 3

Where the paper says it

While the resulting asymptotic distributions are in general difficult to characterize, we show that the GATES are non-negatively correlated with one another regardless of how the treatment prioritization score is constructed. This allows us to uniformly bound the GATES by a Wiener process through an application of Donsker’s theorem.

This leads to the construction of a uniform confidence band in Theorem 1, which minimizes the area of the resulting confidence region by approximating it via a specific functional form. p. 3
Where the paper says it

This allows us to uniformly bound the GATES by a Wiener process through an application of Donsker’s theorem. Although the resulting uniform confidence bands are not unique, we propose a specific choice that yields a narrow band for a reasonable range of quantile values.

Empirical Validation

A simulation study using data from the 2016 Atlantic Causal Inference Conference (ACIC) Competition demonstrates that the proposed uniform confidence bands reach nominal coverage for every dataset size down to n = 100 while pointwise confidence bands suffer from significant undercoverage. In a clinical trial of late-stage prostate cancer, the methodology was applied to identify a subgroup whose 5. p. 0

Where the paper says it

We analyze a clinical trial of late-stage prostate cancer and find a relatively large proportion of exceptional responders. Key Words: causal inference, exceptional responders, heterogeneous treatment effects, treatment prioritization, uniform confidence bands ∗The proposed methodology is implemented through an open-source R package, evalITR, which is freely available for download at the Comprehensive R Archive Network (CRAN; https://CRAN.

0mg estrogen treatment would increase total survival by more than 15 months on average with statistical significance. p. 1
Where the paper says it

In the literature, the average treatment effect among individuals grouped by the prioritization score is referred to as the Grouped Average Treatment Effect or GATES (Chernozhukov et al. As a motivating study, we consider a well-known randomized clinical trial described by Byar and Green (1980) concerning the effect of administering a type of synthetic estrogen, called diethylstilbestrol, to prolong total survival among late-stage prostate cancer patients.

Generalizability and Extension

The proposed methodology is applicable to any generic treatment prioritization score, including those without a performance guarantee. Furthermore, Theorem 1 is extended to mean-adjusted individual-level statistics like the Population Average Prescriptive Effect (PAPE) curve by replacing the Wiener process with a Brownian bridge in the confidence band construction. p. 4

Where the paper says it

Building on the up-lift modeling literature (e. , Radcliffe, 2007), the authors propose the Population Average Prescriptive Effect (PAPE) curve as a performance measure of an ITR relative to the non-individualized treatment rule that treats the same proportion of randomly selected individuals.

The resulting uniform confidence bands can be applied to other evaluation metrics, such as cost-adjusted effects. p. 0
Where the paper says it

We analyze a clinical trial of late-stage prostate cancer and find a relatively large proportion of exceptional responders. Key Words: causal inference, exceptional responders, heterogeneous treatment effects, treatment prioritization, uniform confidence bands ∗The proposed methodology is implemented through an open-source R package, evalITR, which is freely available for download at the Comprehensive R Archive Network (CRAN; https://CRAN.

Characterization of Subgroup

The method allows for characterizing the selected subgroup by estimating the average value of covariates among them, such as E(Xj F(S) ≥ 1 − p∗). This enables an interpretable description of exceptional responders through covariate profiles. p. 2

Where the paper says it

The proposed uniform confidence bands are valid regardless of the way in which the quantile value is chosen, thereby providing a statistical guarantee for a generic objective function to be optimized when selecting the group of individuals who are either most positively or most negatively affected by the treatment. For example, our methodology enables researchers to choose the quantile cutoff such that the GATES among the selected exceptional responders exceeds a certain threshold with a pre-specified probability.

The results in the clinical trial showed that the average tumor size of the selected group was 4. p. 3
Where the paper says it

We further extend our theoretical results beyond the GATES to other causal quantities of interest, widening the applicability of the proposed methodology. Finally, we apply the proposed methodology to the aforementioned randomized clinical trial (Section 4).

2 cm2 larger than the population average, which was statistically significant at the α = 10% level. p. 4
Where the paper says it

Building on the up-lift modeling literature (e. , Radcliffe, 2007), the authors propose the Population Average Prescriptive Effect (PAPE) curve as a performance measure of an ITR relative to the non-individualized treatment rule that treats the same proportion of randomly selected individuals.

Estimation Uncertainty Handling

The authors address estimation uncertainty of the scoring rule by introducing alternative assumptions (Assumption 3A and 4A) that allow for evaluation of empirical performance using an estimated prioritization score. This involves assuming a uniform margin condition on the true treatment prioritization score and a rate condition on the convergence of the estimated prioritization score. p. 6

Where the paper says it

7, we discuss the condition, under which our inference remains valid even when one is interested in evaluating the empirical performance of the true population scoring rule. For now, the only assumption we impose is that the resulting treatment prioritization score is continuous.

This ensures that the uniform confidence bands are valid even when evaluating an empirical scoring rule deployed in real-world applications. p. 5
Where the paper says it

One might, therefore, argue that inference about selected subgroup should integrate the sampling variability of the estimated scoring rule. Our goal, however, is to evaluate the empirical performance of a fixed scoring rule that can actually be deployed in the real world rather than the “true” scoring rule that can never be attained.

Conclusion

The proposed methodology provides a statistical performance guarantee when selecting an optimal subset of a target population who are predicted to benefit most from a treatment of interest. It is particularly useful when the evaluation sample is of limited size, machine learning models provide biased or noisy estimates of treatment efficacy, and/or the proportion of exceptional responders is relatively small. p. 0

Where the paper says it

A simulation study shows that the proposed uniform confidence bands are reasonably informative and have an appropriate empirical coverage even when the sample size is as small as 100. We analyze a clinical trial of late-stage prostate cancer and find a relatively large proportion of exceptional responders.

The empirical application demonstrates that this methodology can exploit modern ML algorithms while providing a rigorous performance guarantee. p. 3
Where the paper says it

, Bertsimas et al. This empirical application demonstrates that our methodology can exploit the power of modern ML algorithms while providing a statistical performance guarantee.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Uniform statistical performance guarantee for subgroup identification: The system can identify exceptional responders by selecting a quantile of treatment prioritization score such that the GATES among those who are identified as benefiting most from the treatment is at least as great as the threshold c with probability at least one minus alpha.

  2. Robustness to scoring rule estimation error: The methodology allows for evaluation even when using an estimated scoring rule by replacing Assumption 3 and 4 with Assumption 3A and 4A, which includes a margin condition on the true treatment prioritization score and a rate condition on the convergence of the estimated prioritization score.

  3. Enhanced performance in small sample sizes: The system achieves nominal coverage for uniform confidence bands down to sample size n = 100, demonstrating that our uniform confidence bands reach the nominal coverage for every dataset size down to n = 100 while the pointwise confidence bands suffer from a significant undercoverage.

  4. Interpretation of subgroup characteristics: The system can characterize selected subgroups by estimating the average value of covariate Xj among the selected group, E(Xj F(S) ≥ 1 − p∗), allowing for interpretable descriptions such as the average tumor size of the selected group is 4.2 cm2 larger than the population average.

  5. Application to mean-adjusted statistics: The methodology can be extended to evaluate mean adjusted individual-level statistics like the Population Average Prescriptive Effect (PAPE) by constructing an estimator for PsiM(p) that can be constructed from Psibn(p) in the same fashion as how a Brownian bridge on [0, 1] is constructed from a Wiener process.

Abstract

Across a wide array of disciplines, many researchers use machine learning (ML) algorithms to identify a subgroup of individuals who are likely to benefit from a treatment the most (``exceptional responders'') or those who are harmed by it. A common approach to this subgroup identification problem consists of two steps. First, researchers estimate the conditional average treatment effect (CATE) using an ML algorithm. Next, they use the estimated CATE to select those individuals who are predicted to be most affected by the treatment, either positively or negatively. Unfortunately, CATE estimates are often biased and noisy. In addition, utilizing the same data to both identify a subgroup and estimate its group average treatment effect results in a multiple testing problem. To address these challenges, we develop uniform confidence bands for estimation of the group average treatment effect sorted by generic ML algorithm (GATES). Using these uniform confidence bands, researchers can identify, with a statistical guarantee, a subgroup whose GATES exceeds a certain effect size, regardless of how this effect size is chosen. The proposed methodology uses random treatment assignment, random sampling of units, and continuity and moment conditions, without requiring consistency or monotonicity of the ML score. We derive a Wiener representation of the estimation error, with corrections for the empirical subgroup threshold and the fixed treatment sample size. Estimating this representation gives asymptotically exact uniform coverage using Gaussian draws, without refitting the ML model. We also develop uniform bands for mean-adjusted treatment effects and other evaluation metrics. A simulation design and an application to a clinical trial of late-stage prostate cancer illustrate the subgroup identification problem.

Sources

Related papers