Rewarding Noise: Incentives in Decentralized A/B Testing

arXiv:2610.19512 · econ.TH · Submitted 2026-10-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Rewarding Noise: Incentives in Decentralized A/B Testing".

Warren: Here's the gist of the paper. Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact The paper argues that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience They formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas The Core Mechanism The central argument revolves around the interaction between two actors: the product team, which chooses what and how to test, and the experience owner, who sets evaluation rules and launch criteria This interaction is modeled as a principal-agent problem where the product team seeks to maximize its private reward based on measured test outcomes The experience owner seeks to maximize the true impact of launched features while bearing complexity costs The scoring rule, which maps estimated impacts to a score, is the key mechanism that shapes the product team’s testing behavior Incentives and Scoring Rules The paper contrasts two scoring rules: raw scoring, which takes estimated impacts at face value, and posterior-mean scoring, which pulls estimates toward a prior mean in proportion to their sampling noise The model predicts that raw scoring creates a reward to noise, allowing product teams to increase their expected reward by running many low-precision tests Conversely, correctly calibrated posterior-mean scoring removes this reward to noise and aligns the two actors’ objectives when complexity costs are zero Empirical Findings The authors test these model predictions using data from Amazon A/B tests before and after a shift to posterior-mean-based evaluation in 2024 Consistent with the model, they document patterns where test volume declined while precision and the dispersion of true impacts increased, resulting in a more than two-fold increase in signal-to-noise ratios These shifts imply an approximately 60 percent increase in the value of testing on the experience, despite more than 50 percent fewer launches Methodological Implications The results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place The paper contributes to the literature on principal-agent problems in experimentation by grounding misalignment in the scoring rule rather than just product team preferences Furthermore, it suggests that as automation and AI reduce testing costs, these incentive effects may become more important Theoretical Results The theoretical results demonstrate that the experience owner’s choice of scoring rule shapes the product team’s testing behavior through four propositions Proposition 3 shows that posterior-mean scoring removes the reward to noise, meaning the gap between objectives is strictly smaller than under raw scoring Proposition 4 establishes a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas when switching from raw to posterior-mean scoring Proposition 2 shows that under vanishing testing costs, raw scoring drives the team toward minimum ambition and maximum feasible test volume with diverging sampling variance Empirical Application The empirical application confirms Proposition 4 by showing evidence consistent with the model's predictions on an Amazon experience that shifted from approximately raw to posterior-mean scoring in 2024 They estimate that test volume fell by approximately 15 percent and the sampling standard error fell by over 30 percent, leading to a more than two-fold increase in the signal-to-noise ratio for both metrics These findings imply that choosing κ in each period to approximate observed launch rates and setting cX = 0, the value of testing increased by approximately 60 percent for both metrics The results are robust across various falsification tests, including one on an experience that continued to use raw scoring throughout the analysis period Limitations The paper acknowledges several limitations, including assuming true impacts are normally distributed when they may have fatter-than-normal tails It also notes that the change in scoring rule coincided with a change in launch criteria, making it difficult to isolate the effect of the scoring rule change per se The authors also admit that their estimates of some model parameters are imprecise, particularly for sigma 2 in the pre-change period The optimal launch threshold could partially compensate for a suboptimal scoring rule if the experience owner jointly optimizes both instruments References Azevedo, Eduardo M., Alex Deng, José L.

Ruth: Montiel Olea, Justin Rao, and E.

Warren: Glen Weyl, “A/B Testing with Fat Tails,” Journal of Political Economy, 2020 Bates, Stephen, Michael I.

Ruth: Jordan, Michael Sklar, and Jake A.

Paper summary: Ruth: To recap, "Rewarding Noise: Incentives in Decentralized A/B Testing" argues that low statistical power in firms running A/B tests isn't just a technical problem; it can be a symptom of misaligned incentives between the product teams and the experience owners.

Warren: Specifically, product teams want to maximize their private reward by choosing what and how to test on customer experiences, while the experience owner sets the rules for evaluation and launch criteria to optimize that experience.

Iris: The central thesis is that raw impact estimates create a reward for sampling noise, which allows product teams to run many low-precision tests.

Ruth: Conversely, using correctly calibrated posterior-mean evaluation removes this reward for noise and creates a local incentive for the product team to focus on fewer, more precise tests of more ambitious ideas.

Warren: The paper formalizes this interaction with a principal-agent model and shows that raw scoring rewards noise while posterior-mean scoring aligns the objectives when complexity costs are zero.

Iris: This matters because it suggests that A/B testing methodologies directly shape not only measurement and launch decisions but which tests are pursued in the first place.

Ruth: It’s important to remember that they test this using data from Amazon A/B tests run before and after a two thousand twenty-four shift to posterior-mean evaluation, finding patterns consistent with their model predictions <ref:2610.19512#pg1>.

Warren: They saw test volume decline while precision and the dispersion of true impacts increased, resulting in a more than two-fold increase in signal-to-noise ratios.

Iris: These shifts imply an approximately sixty percent increase in the value of testing on the experience, despite there being more than fifty percent fewer launches.

Conclusion: Ruth: So, looking at "Rewarding Noise: Incentives in Decentralized A/B Testing" by Kessler et al., the main point is that we need to think about the incentives driving testing decisions in decentralized settings.

Warren: Simply put, if you don't adjust how you score your test results based on sampling noise, the system might encourage teams to run lots of noisy tests instead of focusing on meaningful ones.

Iris: The implication for me is that we have to be careful about what variation is doing the work here; we need to ensure that our evaluation rules aren't accidentally rewarding low-precision estimates.

Ruth: That’s right, and it suggests that shifting from raw scoring to posterior-mean scoring creates a real local incentive for product teams to stop running dozens of noisy tests and start focusing on fewer, more precise experiments.

Warren: It’s about how the structure of the evaluation system can actively steer testing behavior in a way we didn't expect.

Iris: And while they showed evidence for this shift on Amazon data, one thing they flagged is that their results depend heavily on whether the launch criteria also changed at the same time as they adjusted their scoring rule.

Ruth: That’s right, so it’s a nuanced finding—the scoring rule change isn't the only variable at play when we look at real-world testing shifts.

Warren: Ultimately, this paper points toward designing A/B testing systems where the rules are set up to reward precision over sheer volume of low-quality tests.

econ.TH

Submitted: 2026-10-14

Updated: 2026-10-27

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact The paper argues that in decentralized

Key concepts

Principal-Agent Model: This model describes the relationship between two parties: the product team (the agent) who chooses what and how to test, and the experience owner (the principal) who sets evaluation rules. It formalizes how their competing interests influence testing behavior.
Raw Scoring: This scoring rule treats estimated impacts as true, regardless of sampling noise. This creates a reward for product teams to run many tests with low precision because they can maximize their private reward by increasing the volume of experiments.
Posterior-Mean Scoring: This rule adjusts estimates by pulling them toward a prior mean, scaled by the sampling noise. This mechanism removes the incentive to exploit noise, aligning product teams' objectives with the experience owner's goal of finding true impact.

Terminology

Actor — 6× in this paper · explained in 3 episodes
signal-to-noise ratios — 5× in this paper · explained in 2 episodes
Bayesian Inference — 2× in this paper · explained in 7 episodes
Variance Reduction — 2× in this paper · explained in 2 episodes
Causal Discovery — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Grounding — 1× in this paper · explained in 3 episodes
Robustness — 1× in this paper · explained in 6 episodes

Summary

Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact The paper argues that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience They formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas

The Core Mechanism

The central argument revolves around the interaction between two actors: the product team, which chooses what and how to test, and the experience owner, who sets evaluation rules and launch criteria This interaction is modeled as a principal-agent problem where the product team seeks to maximize its private reward based on measured test outcomes The experience owner seeks to maximize the true impact of launched features while bearing complexity costs The scoring rule, which maps estimated impacts to a score, is the key mechanism that shapes the product team’s testing behavior

Incentives and Scoring Rules

The paper contrasts two scoring rules: raw scoring, which takes estimated impacts at face value, and posterior-mean scoring, which pulls estimates toward a prior mean in proportion to their sampling noise The model predicts that raw scoring creates a reward to noise, allowing product teams to increase their expected reward by running many low-precision tests Conversely, correctly calibrated posterior-mean scoring removes this reward to noise and aligns the two actors’ objectives when complexity costs are zero

Empirical Findings

The authors test these model predictions using data from Amazon A/B tests before and after a shift to posterior-mean-based evaluation in 2024 Consistent with the model, they document patterns where test volume declined while precision and the dispersion of true impacts increased, resulting in a more than two-fold increase in signal-to-noise ratios These shifts imply an approximately 60 percent increase in the value of testing on the experience, despite more than 50 percent fewer launches

Methodological Implications

The results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place The paper contributes to the literature on principal-agent problems in experimentation by grounding misalignment in the scoring rule rather than just product team preferences Furthermore, it suggests that as automation and AI reduce testing costs, these incentive effects may become more important

Theoretical Results

The theoretical results demonstrate that the experience owner’s choice of scoring rule shapes the product team’s testing behavior through four propositions Proposition 3 shows that posterior-mean scoring removes the reward to noise, meaning the gap between objectives is strictly smaller than under raw scoring Proposition 4 establishes a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas when switching from raw to posterior-mean scoring Proposition 2 shows that under vanishing testing costs, raw scoring drives the team toward minimum ambition and maximum feasible test volume with diverging sampling variance

Empirical Application

The empirical application confirms Proposition 4 by showing evidence consistent with the model's predictions on an Amazon experience that shifted from approximately raw to posterior-mean scoring in 2024 They estimate that test volume fell by approximately 15 percent and the sampling standard error fell by over 30 percent, leading to a more than two-fold increase in the signal-to-noise ratio for both metrics These findings imply that choosing κ in each period to approximate observed launch rates and setting cX = 0, the value of testing increased by approximately 60 percent for both metrics The results are robust across various falsification tests, including one on an experience that continued to use raw scoring throughout the analysis period

Limitations

The paper acknowledges several limitations, including assuming true impacts are normally distributed when they may have fatter-than-normal tails It also notes that the change in scoring rule coincided with a change in launch criteria, making it difficult to isolate the effect of the scoring rule change per se The authors also admit that their estimates of some model parameters are imprecise, particularly for sigma 2 in the pre-change period The optimal launch threshold could partially compensate for a suboptimal scoring rule if the experience owner jointly optimizes both instruments

References

Azevedo, Eduardo M., Alex Deng, José L. Montiel Olea, Justin Rao, and E. Glen Weyl, “A/B Testing with Fat Tails,” Journal of Political Economy, 2020 Bates, Stephen, Michael I. Jordan, Michael Sklar, and Jake A. Soloff, “Principal-Agent Hypothesis Testing,” arXiv preprint arXiv:2205.06812 Cator, Eric A. and Erik W. van Zwet, “The Significance Filter, the Winner’s Curse and the Need to Shrink,” Statistica Neerlandica, 2021 Chassang, Sylvain, Gerard Padro I Miquel, and Erik Snowberg, “Selective Trials: A PrincipalAgent Approach to Randomized Controlled Experiments,” American Economic Review, 2012 Deng, Alex, “Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments,” in “Proceedings of the 24th International Conference on World Wide Web” WWW ’15 Companion Association for Computing Machinery New York, NY, USA 2015 Efron, Bradley and Carl Morris, “Limiting the Risk of Bayes and Empirical Bayes Estimators – Part I: The Bayes Case,” Journal of the American Statistical Association, 1971 Gelman, Andrew and John Carlin, “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors,” Perspectives on Psychological Science, 2014 Ghosh, Sourobh and Dmitry Taubinsky, “Increasing Profits by Inducing Greater Risk-Taking in Project Selection,” Amazon Working Paper, 2025 Gordon, Robert D., “Values of Mills’ Ratio of Area to Bounding Ordinate and of the Normal Probability Integral for Large Values of the Argument,” The Annals of Mathematical Statistics, 1941 Gupta, Somit, Ronny Kohavi, Diane Tang, Ya Xu, Reid Andersen, Eytan Bakshy, Niall Cardin, Sumita Chandran, Nanyu Chen, Dominic Coey, Mike Curtis Heidemanns et al., “Improving Bayesian Inference in Weblab,” Amazon Working Paper, 2024 Larsen et al., “Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology,” The American Statistician, 2024 Masoero et al.

Improvements for AI systems

Improve test allocation by using posterior-mean scoring instead of raw scoring to create a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas. This shifts testing behavior from running many low-precision tests to focusing on higher quality, more significant experiments.
Optimize the selection of which features to test by incorporating the experience owner's objective into the product team's decision-making process via posterior-mean scoring. The model shows that posterior-mean scoring removes the reward to noise, aligning objectives when complexity costs associated with launches are zero.
Develop a dynamic testing protocol where sampling variance is explicitly modeled as a choice variable, allowing for the adjustment of test volume based on incentives. This enables the system to avoid the trap of high-volume, low-precision experimentation that occurs under raw scoring with vanishing testing costs.

Abstract

Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact. We argue that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience. We formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise. Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas. Using data on A/B tests run at Amazon, we document patterns consistent with these model predictions. After an Amazon experience shifted to posterior-mean-based evaluation in 2024, test volume declined while both precision and the dispersion of true impacts increased, more than doubling signal-to-noise ratios. Our results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place.