Inference for Batched Adaptive Experiments
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Inference for Batched Adaptive Experiments".
Warren: Here's the gist of the paper. Adaptive experiments pose challenges for causal inference This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals Inference Methodology The paper introduces a BOLS test statistic designed for inference in adaptive experiments, which provides a precisionequalizing aggregation of per-period treatment-control differences under heteroskedasticity This combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals The asymptotic normality of the heteroskedastic BOLS statistic, defined by Zhet = ∆b − c SE(∆) b = 1/√T X T t=1 zt,het, is established as N(0, 1) Feasible implementation involves using consistent estimators for arm- and period-specific variances to construct the confidence interval CI1−α(∆) = ∆b F ± Zˆ het,1−α/2 SE(∆b F) Statistical Foundations The variance of the period difference is given by Var(∆ˆ t) ≡ σ21t/N1t + σ20t/N0t = 1/(nt
σ21πt + σ20(1 − πt): The authors construct a weighted average effect estimate ∆ = ˆ X T t=1 wt S ∆ˆ t, where wt = 1/√vt and S = X T s=1 ws The conditional variance of the weighted estimator is derived as Var(∆) = ˆ X T t=1 wt S2 vt = 1/S2 X T t=1 w2t vt = T/S2.
Title and authors: Ruth: So, we're looking at this paper now, "Inference for Batched Adaptive Experiments," by Kempera and Afscharb from the University of Mannheim and ZEW, IZA in Germany. The main thing they're proposing here is a specific test statistic called the BOLS test statistic to handle causal inference in adaptive experiments when the outcomes have different variances across treatment arms.
Warren: Right, so it’s about fixing that challenge where you have sequential decision making—where you adapt your treatment assignment based on what happened in previous batches—and trying to get valid results even when the variance of those outcomes isn't constant. It sounds like they are tackling the heteroskedasticity issue head-on with this BOLS statistic.
Iris: From my side, I see they are aiming for a precision equalizing aggregation of per-period treatment control differences under that heteroskedasticity condition. That means instead of just averaging things up naively, they're normalizing the per-period z-statistics to get one combined test statistic that can give us asymptotically valid confidence intervals.
Ruth: Exactly, and they define this combined test statistic as a normalized average of those heteroskedastic per-period z-statistics. This is a key step because it allows them to construct those confidence intervals correctly even when the underlying variances are different between the treatment arms.
Warren: And they establish the asymptotic normality of this BOLS statistic, defining it as Zhet equals delta b minus c divided by the square root of T times X T t equals one zt het. They say that this Zhet converges to a normal distribution with a mean of zero and a standard deviation of one.
Iris: That’s important because it means for large batch sizes, we can use this distribution to approximate the actual finite-sample distribution when we want to build those confidence intervals. They connect it back to work by Hirano and Porter from two thousand twenty-five who showed that any limit distribution from joint choices of adaptive assignment rules and statistics can fit into a unified Gaussian experiment <ref:2512.10156#pg2,joint choices of adaptive assignment rules and statistics can>.
Ruth: So, the practical implication here is that researchers can now use this statistic to build those confidence intervals for treatment effects in adaptive experiments under heteroskedasticity. It moves inference beyond what was possible when we only had homoskedastic outcomes in these sequential designs.
Warren: How does that change things for someone who is actually running an experiment? They have to construct the confidence interval CI sub alpha delta b F plus Z hat het over two, divided by the standard error of delta b F. That gives them a concrete way to calculate the range for their treatment effect estimate based on this BOLS statistic.
Title and authors: Iris: And they also show a way to handle that variance structure through a weighted average effect estimate called delta equals hat X T t equals one wt S delta hat t, where the weight w is one over the square root of vt. This weighting scheme helps derive the conditional variance of that weighted estimator, which is important for understanding the precision.
Ruth: It sounds like they are giving practitioners a tool to deal with outcomes that are inherently variable by treatment arm without having to assume those variances are all identical across every single period or batch. That’s a big practical win for designing experiments where variance can naturally differ between groups.
Warren: But I wonder about the cost of using this. They mention that feasible implementation involves using consistent estimators for the arm- and period-specific variances to construct those confidence intervals, which means you still need good estimates of those variances in the first place to make this work reliably.
Iris: That’s a fair point. The paper does acknowledge that the practical application depends on having consistent estimators for those specific variances, so it’s not just a plug-and-play tool; it requires careful estimation beforehand. They also discuss how they can dynamically adjust weights based on batch sizes and treatment shares, which lets the AI focus its learning on periods where the evidence is strongest or when sample sizes are most balanced.
Ruth: That dynamic weighting mechanism seems interesting because it allows the system to be smarter about how much influence each period has on the final effect estimate depending on how big that batch was or what share of treatments was in it. It moves us toward more targeted learning in these complex settings.
Warren: So, if we look at what this means for policy evaluation, the paper suggests that AI systems evaluating policies in adaptive settings can now use this heteroskedastic BOLS statistic to avoid overrejecting null hypotheses when outcome variances aren't constant across treatment arms. Simulation results even matched nominal five percent rejection rates, which is a pretty strong indicator of its robustness.
Iris: That robustness is key for policy work, because it means we don't waste resources making decisions based on false signals just because the data structure was messy. But they also flag a limitation: the paper shows that when you have fixed T and large batch sizes, this asymptotic normality holds well, but their claim relies on those large batch sizes being achieved.
Ruth: So, if we're talking about what this means for decision making systems, it means AI agents can now make more reliable decisions in adaptive settings where the treatment assignment depends on past outcomes and those outcome variances are known to differ between treatments. It makes the entire decision process more trustworthy under those conditions.
Title and authors: Warren: It’s about building a test statistic that is not just looking at one period or one arm, but aggregating information across all periods while correctly weighting them by their specific variance structure under heteroskedasticity. That’s the core mechanism they are presenting for this paper, "Inference for Batched Adaptive Experiments."
Iris: And to wrap up on the methodology, they show how to construct the confidence interval CI sub alpha delta b F plus Z hat het over two using that heteroskedastic BOLS statistic and those consistent estimators. It’s a concrete procedure for when you need that precision.
Ruth: So, this paper gives us a formalized way to perform inference in adaptive experiments that respects the complexities of differing outcome variances across treatments by using this BOLS test statistic and its associated confidence intervals. It addresses how sequential learning works in practice while keeping causal inference sound.
Warren: And for the future, they point toward feasible implementation strategies, which include those dynamic weight adjustments we talked about earlier to optimize how much each period counts in the final result. That lets the system be more efficient when learning from a sequence of batches.
Iris: I think one thing they emphasize is that this approach provides a way to approximate the finite-sample distribution using the asymptotic results, which is what we need when we're actually running these experiments today with limited data. They’re giving us a bridge between theory and practice for these adaptive designs.
Ruth: So, to summarize, "Inference for Batched Adaptive Experiments" provides a BOLS test statistic that aggregates per-period z-statistics under heteroskedasticity to construct asymptotically valid confidence intervals for treatment effects in adaptive settings. It’s a solid piece of work for anyone dealing with sequential decision making where variance isn't constant.
Warren: It’s a tool for getting more reliable estimates when the experiment is adaptive and the outcomes have varying spreads across treatments. We need to keep an eye on how this gets implemented in real-world AI applications because that’s where the payoff will be felt.
Iris: I just think their work on establishing asymptotic normality for this statistic is the foundation here, showing that even with all those adaptive rules and variances, we can still rely on a Gaussian limit experiment for inference.
Ruth: Exactly. We're going to take a quick break and then talk about how this BOLS statistic fits into the broader picture of policy learning with confidence, or perhaps we can pivot to something else entirely depending on where the audience wants to go next.
The paper's summary: Ruth: So it’s about giving researchers a way to calculate those confidence intervals even when things get messy and the spread of outcomes changes depending on what happened before.
Warren: Exactly. It handles that heteroskedasticity issue by creating this combined test statistic, which is basically a normalized average of per-period tests.
Iris: And they establish that this statistic is asymptotically normal, meaning for big enough experiments, we can trust the math behind it to build valid confidence intervals.
Ruth: That’s the theoretical side. Now, what does this actually mean for someone trying to set up an experiment with AI? It means you don't have to assume every single batch or treatment arm has the same level of noise in your data just to get a reliable answer.
Warren: The cost is that you still have to estimate those specific variances for each arm and period, which adds another layer of estimation work on top of the decision-making process.
Iris: And they point out that the method works best when you can dynamically adjust weights based on how many observations are in a batch or what share of treatments is in that batch. That lets the AI focus its learning where it's most informative right now.
Ruth: I think that dynamic weighting is really interesting for sequential decision-making systems, because it lets the AI be smarter about which past information to prioritize when making a new treatment assignment.
Warren: It’s about efficiency in learning, though I wonder how robust this whole framework is if the underlying assumptions about those variances are slightly off in the real world.
Iris: The authors say they show simulation results matching nominal five percent rejection rates, which suggests that for practical policy evaluation, you can rely on this tool to avoid making false decisions just because your outcome spreads are uneven.
Ruth: So it moves inference past what was possible before, letting AI agents make more trustworthy decisions when the treatment assignments themselves depend on previous outcomes and those outcomes have different variances.
Warren: It’s a concrete way to build that precision, but we still need to watch how this gets implemented when everyone starts using these types of adaptive strategies in high-stakes environments.
The paper's improvements: Tom: So we're looking at how they suggest improving this BOLS test statistic for adaptive experiments, moving beyond just getting an initial valid interval to making the whole process more dynamic and useful.
Ruth: They introduce a mechanism for dynamically adjusting weights based on batch sizes and treatment shares, which lets the AI focus its learning where it’s most informative or when sample sizes are most balanced.
Warren: That sounds like a way to optimize the learning schedule so you aren't spending too much computational effort on periods that don't actually matter as much for the final effect estimate.
Iris: It addresses the problem of having to guess how much influence each period has when you’re dealing with sequential data, which is a big headache in these adaptive settings.
Ruth: By giving the AI this control over its own weighting scheme, we can make those confidence intervals even more tailored to the specific structure of our experiment.
Warren: It’s about making the learning process itself more efficient so that when an AI agent makes a decision, it’s based on the most relevant recent evidence, not just blindly averaging everything together.
Iris: This dynamic adjustment ties back into how we get those conditional variances, because if you know the weight structure is changing, your estimate of how precise your result is also changes over time.
Ruth: It’s about building a more flexible inference framework that works not just for a single static experiment, but for the entire sequence of adaptive decisions.
Warren: So it means less computational waste and potentially more reliable results when you're running long-running, complex adaptive systems.
Iris: But they do have a caveat—the implementation still requires consistent estimators for those arm and period specific variances to get the best result from this dynamic weighting structure.
Ruth: That’s the trade-off, right? You gain a smarter learning schedule but you still need good initial estimates for those variances to make it all work smoothly.
Warren: It’s a practical constraint, I guess. But overall, they're giving researchers more granular control over how they process the data as it comes in.
Conclusion: Ruth: To wrap things up, the paper "Inference for Batched Adaptive Experiments" gives us a robust way to perform causal inference in adaptive settings by using that BOLS test statistic under heteroskedasticity conditions.
Warren: It shows that we can construct those confidence intervals asymptotically valid, even when the treatment assignment depends on past outcomes and those outcomes have different variances across arms.
Iris: From my econometrician view, the core contribution is establishing that asymptotic normality for this combined test statistic, which is what makes us confident in using it for policy evaluation.
Ruth: This means AI systems evaluating policies can now use this statistic to avoid overrejecting null hypotheses when outcome variances aren't constant across treatment arms, which is a huge step toward trustworthy recommendations.
Warren: The numbers show that simulation results matched nominal five percent rejection rates, which gives us some confidence in its robustness for policy work.
Iris: I just want to stress that this relies on having consistent estimators for those specific variances beforehand, so it’s not just a magic tool you can plug into anything without doing the necessary preliminary estimation work.
Ruth: So the implication is that sequential decision-making systems can now make more reliable decisions under complex uncertainty structures by using this method.
Warren: It's a tool for getting more reliable estimates when the experiment is adaptive and the outcomes have varying spreads across treatments, but we still need to watch how this gets implemented in real-world AI applications.
Iris: I think their work on establishing that Gaussian limit distribution is the foundation here, showing that even with all those adaptive rules and variances, we can still rely on a Gaussian experiment for inference.
Ruth: So ultimately, "Inference for Batched Adaptive Experiments" provides a formalized way to perform inference in adaptive experiments that respects the complexities of differing outcome variances by using this BOLS test statistic.
Warren: It’s a solid piece of work for anyone dealing with sequential learning where variance isn't constant, and it sets up some interesting challenges for future work on even more complex decision processes.
Jan Kempera, Davud Rostam-Afschar
University of Mannheim · ZEW
econ.EM, cs.LG, stat.ME
Submitted: 2025-12-10
Updated: 2026-11-01
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 76/100
The gist: Adaptive experiments pose challenges for causal inference This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments The…
Key concepts
- BOLS Test Statistic
- This is a specific statistical test designed for adaptive experiments. It combines per-period z-statistics from different time points into a single, normalized average. This aggregation helps create a robust estimate of the overall treatment effect even when the variance of outcomes changes over time.
- Heteroskedasticity
- This refers to situations where the variability (variance) of the data is not constant across all periods or treatment groups. In adaptive experiments, this means that some time points might have much more or less noisy outcomes than others, which standard tests cannot handle well.
- Asymptotic Validity
- This means that as the experiment gets larger (more data), the confidence intervals constructed using this statistic will become increasingly accurate and reliable. The paper proves that the test statistic is asymptotically normal, ensuring that the calculated confidence intervals are trustworthy in large-scale applications.
Terminology
Summary
Adaptive experiments pose challenges for causal inference This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals
Inference Methodology
The paper introduces a BOLS test statistic designed for inference in adaptive experiments, which provides a precisionequalizing aggregation of per-period treatment-control differences under heteroskedasticity This combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals The asymptotic normality of the heteroskedastic BOLS statistic, defined by Zhet = ∆b − c SE(∆) b = 1/√T X T t=1 zt,het, is established as N(0, 1) Feasible implementation involves using consistent estimators for arm- and period-specific variances to construct the confidence interval CI1−α(∆) = ∆b F ± Zˆ het,1−α/2 SE(∆b F)
Statistical Foundations
The variance of the period difference is given by Var(∆ˆ t) ≡ σ21t/N1t + σ20t/N0t = 1/(nt [σ21πt + σ20(1 − πt)] The authors construct a weighted average effect estimate ∆ = ˆ X T t=1 wt S ∆ˆ t, where wt = 1/√vt and S = X T s=1 ws The conditional variance of the weighted estimator is derived as Var(∆) = ˆ X T t=1 wt S2 vt = 1/S2 X T t=1 w2t vt = T/S2.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved Causal Inference for Sequential Decision Making Systems: The system can now construct asymptotically valid confidence intervals for treatment effects in adaptive experiments under heteroskedasticity by using the BOLS selection weighted statistic. This allows AI agents to make more reliable decisions when their treatment assignments depend on past outcomes and outcome variances differ between treatments.
-
Enhanced Robustness in Policy Evaluation: AI systems evaluating policies in adaptive settings can now utilize the heteroskedasticity-robust BOLS statistic to avoid overrejecting null hypotheses, as demonstrated by the simulation results matching nominal 5% rejection rates. This leads to more trustworthy policy recommendations when outcome variances are not constant across treatment arms.
-
Dynamic Weight Adjustment for Learning: The system gains a mechanism for dynamically adjusting weights based on batch sizes and treatment shares, as detailed in Table A.1, allowing it to assign appropriate influence to different periods within the experiment. This enables the AI to focus its learning on periods where the evidence is most informative or when sample sizes are most balanced.
Abstract
The advantages of adaptive experiments have led to their rapid adoption in economics, other fields, and among practitioners. However, adaptive experiments pose challenges for causal inference. This note suggests a BOLS (batched ordinary least squares) test statistic for inference on treatment effects in adaptive experiments. The statistic provides a precision-equalizing aggregation of per-period treatment-control differences under heteroskedasticity. The combined test statistic is a linear combination of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals. We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.
Sources
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs