Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice

summary

The gist

We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true…

In short

The paper develops a theory for sample splitting in A/B testing, where test data is divided into two parts to measure performance when true test impacts are unknown. It finds that these estimators are biased for full-sample performance but accurately estimate the sample-split version of the impact. The work provides methods for inference and characterizes the trade-offs involved.

Key concepts

Sample Splitting
This is a technique where data from an A/B test is divided into two non-overlapping groups or 'splits.' One split is used to implement a proposed methodology, while the other split helps create an unbiased estimate of the true impact of the test. This allows performance evaluation without knowing what the true outcome would have been.
Sample-Split Estimators
These are statistical methods that use sample splitting data to estimate performance. While they might not give you the best possible result if you had all the data, they consistently estimate how well a sample-split version of the impact would perform. The paper analyzes their biases and how they relate to full-sample performance.
Plug-in Methodologies
This is a shortcut for certain statistical methods that only rely on summary statistics rather than individual unit data. Instead of splitting the actual units, this approach samples directly from the joint distribution of estimates derived from the full dataset, simplifying how sample-split data can be generated.
Bias-Variance Trade-off
This refers to the balance between a statistical method's tendency to produce biased results and its sensitivity to random fluctuations in the data. The paper examines how choosing a sample-split design affects this trade-off, showing how different splitting choices influence the accuracy and reliability of performance measurements.

Terminology used across episodes

This episode discusses

The paper

Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice · Read on arXiv

Ryan Kessler, James McQueen, Miikka Rokkanen

Amazon

We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved. We show that sample-split estimators are generally biased for full-sample performance but consistently estimate sample-split analogues of it. We derive their asymptotic distributions, construct valid confidence intervals, and characterize the bias-variance trade-offs underlying sample-split design choices. We validate our theoretical results through simulations and provide implementation guidance for A/B testing products seeking to evaluate new estimators and decision rules.

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Evaluating A/B Testing Methodologies via Sample Splitting".

Warren: Here's the gist of the paper. We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved <ref:2512.03366#pg2> The single most informative sentence about the paper's subject and finding is that sample-split estimators are generally biased for full-sample performance but consistently estimate sample-split analogues of it <ref:2512.03366#pg3>.

Ruth: Theoretical Foundation The paper introduces a theoretical framework to characterize what sample splitting methods can measure and how to conduct valid inference in A/B testing environments <ref:2512.03366#pg5>.

Warren: The core idea is that each A/B test is divided into two non-overlapping splits: one for implementing the proposed methodology and another for constructing an independent, unbiased proxy for the true impact <ref:2512.03366#pg9>.

Ruth: This approach allows performance measurement when true impacts are unobserved without making strong distributional assumptions <ref:2512.03366#pg10>.

Paper summary: Ruth: So to recap, this paper establishes a theoretical framework for sample splitting in A/B testing environments where data for each test gets partitioned into two splits so we can measure methodological performance when we don't know the true impacts of those tests >

Warren: The central thesis is that while sample-split estimators are generally biased when compared to full-sample performance, they reliably estimate their own sample-split analogues >

Iris: They then move on to derive the asymptotic distributions for these estimators and show how to build valid confidence intervals for them based on this framework >

Ruth: Beyond just the numbers, the authors characterize the trade-offs between bias and variance that you face when you decide on a specific sample-split design >

Warren: And they validated all of this by running simulations, which is important because it shows how these concepts actually behave in real scenarios and gives some implementation advice for products >

Iris: It also points out that common sample splitting practices, like splitting data equally or using hundreds of random partitions, are often not the best choices for measurement >

Ruth: So what this means is that if you're developing a new estimator or a decision rule, you can use this framework to test its performance under these specific split conditions without needing the true impacts upfront >

Conclusion: Warren: So, thinking about the title, "Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice," it really frames this as a practical guide for anyone working on A/B testing products >

Ruth: It's about taking something that sounds theoretical, like sample splitting, and giving you the tools to actually use it in a way that makes sense for decision-making >

Iris: The implication is that instead of just running one big test and hoping for the best, you can use this method to systematically compare different statistical approaches >

Warren: It gives product builders a principled way to conduct hypothesis testing or power calculations when you're comparing two different ways of measuring impact >

Ruth: So it’s about making sure that your evaluation process itself is sound, even when the underlying reality of the test results is hidden from you >

More episodes

← Home