Rewarding Noise: Incentives in Decentralized A/B Testing

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Rewarding Noise: Incentives in Decentralized A/B Testing".

Warren: Here's the gist of the paper. Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact The paper argues that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience They formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas The Core Mechanism The central argument revolves around the interaction between two actors: the product team, which chooses what and how to test, and the experience owner, who sets evaluation rules and launch criteria This interaction is modeled as a principal-agent problem where the product team seeks to maximize its private reward based on measured test outcomes The experience owner seeks to maximize the true impact of launched features while bearing complexity costs The scoring rule, which maps estimated impacts to a score, is the key mechanism that shapes the product team’s testing behavior Incentives and Scoring Rules The paper contrasts two scoring rules: raw scoring, which takes estimated impacts at face value, and posterior-mean scoring, which pulls estimates toward a prior mean in proportion to their sampling noise The model predicts that raw scoring creates a reward to noise, allowing product teams to increase their expected reward by running many low-precision tests Conversely, correctly calibrated posterior-mean scoring removes this reward to noise and aligns the two actors’ objectives when complexity costs are zero Empirical Findings The authors test these model predictions using data from Amazon A/B tests before and after a shift to posterior-mean-based evaluation in 2024 Consistent with the model, they document patterns where test volume declined while precision and the dispersion of true impacts increased, resulting in a more than two-fold increase in signal-to-noise ratios These shifts imply an approximately 60 percent increase in the value of testing on the experience, despite more than 50 percent fewer launches Methodological Implications The results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place The paper contributes to the literature on principal-agent problems in experimentation by grounding misalignment in the scoring rule rather than just product team preferences Furthermore, it suggests that as automation and AI reduce testing costs, these incentive effects may become more important Theoretical Results The theoretical results demonstrate that the experience owner’s choice of scoring rule shapes the product team’s testing behavior through four propositions Proposition 3 shows that posterior-mean scoring removes the reward to noise, meaning the gap between objectives is strictly smaller than under raw scoring Proposition 4 establishes a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas when switching from raw to posterior-mean scoring Proposition 2 shows that under vanishing testing costs, raw scoring drives the team toward minimum ambition and maximum feasible test volume with diverging sampling variance Empirical Application The empirical application confirms Proposition 4 by showing evidence consistent with the model's predictions on an Amazon experience that shifted from approximately raw to posterior-mean scoring in 2024 They estimate that test volume fell by approximately 15 percent and the sampling standard error fell by over 30 percent, leading to a more than two-fold increase in the signal-to-noise ratio for both metrics These findings imply that choosing κ in each period to approximate observed launch rates and setting cX = 0, the value of testing increased by approximately 60 percent for both metrics The results are robust across various falsification tests, including one on an experience that continued to use raw scoring throughout the analysis period Limitations The paper acknowledges several limitations, including assuming true impacts are normally distributed when they may have fatter-than-normal tails It also notes that the change in scoring rule coincided with a change in launch criteria, making it difficult to isolate the effect of the scoring rule change per se The authors also admit that their estimates of some model parameters are imprecise, particularly for sigma 2 in the pre-change period The optimal launch threshold could partially compensate for a suboptimal scoring rule if the experience owner jointly optimizes both instruments References Azevedo, Eduardo M., Alex Deng, José L.

Ruth: Montiel Olea, Justin Rao, and E.

Warren: Glen Weyl, “A/B Testing with Fat Tails,” Journal of Political Economy, 2020 Bates, Stephen, Michael I.

Ruth: Jordan, Michael Sklar, and Jake A.

Paper summary: Ruth: To recap, "Rewarding Noise: Incentives in Decentralized A/B Testing" argues that low statistical power in firms running A/B tests isn't just a technical problem; it can be a symptom of misaligned incentives between the product teams and the experience owners.

Warren: Specifically, product teams want to maximize their private reward by choosing what and how to test on customer experiences, while the experience owner sets the rules for evaluation and launch criteria to optimize that experience.

Iris: The central thesis is that raw impact estimates create a reward for sampling noise, which allows product teams to run many low-precision tests.

Ruth: Conversely, using correctly calibrated posterior-mean evaluation removes this reward for noise and creates a local incentive for the product team to focus on fewer, more precise tests of more ambitious ideas.

Warren: The paper formalizes this interaction with a principal-agent model and shows that raw scoring rewards noise while posterior-mean scoring aligns the objectives when complexity costs are zero.

Iris: This matters because it suggests that A/B testing methodologies directly shape not only measurement and launch decisions but which tests are pursued in the first place.

Ruth: It’s important to remember that they test this using data from Amazon A/B tests run before and after a two thousand twenty-four shift to posterior-mean evaluation, finding patterns consistent with their model predictions <ref:2610.19512#pg1>.

Warren: They saw test volume decline while precision and the dispersion of true impacts increased, resulting in a more than two-fold increase in signal-to-noise ratios.

Iris: These shifts imply an approximately sixty percent increase in the value of testing on the experience, despite there being more than fifty percent fewer launches.

Conclusion: Ruth: So, looking at "Rewarding Noise: Incentives in Decentralized A/B Testing" by Kessler et al., the main point is that we need to think about the incentives driving testing decisions in decentralized settings.

Warren: Simply put, if you don't adjust how you score your test results based on sampling noise, the system might encourage teams to run lots of noisy tests instead of focusing on meaningful ones.

Iris: The implication for me is that we have to be careful about what variation is doing the work here; we need to ensure that our evaluation rules aren't accidentally rewarding low-precision estimates.

Ruth: That’s right, and it suggests that shifting from raw scoring to posterior-mean scoring creates a real local incentive for product teams to stop running dozens of noisy tests and start focusing on fewer, more precise experiments.

Warren: It’s about how the structure of the evaluation system can actively steer testing behavior in a way we didn't expect.

Iris: And while they showed evidence for this shift on Amazon data, one thing they flagged is that their results depend heavily on whether the launch criteria also changed at the same time as they adjusted their scoring rule.

Ruth: That’s right, so it’s a nuanced finding—the scoring rule change isn't the only variable at play when we look at real-world testing shifts.

Warren: Ultimately, this paper points toward designing A/B testing systems where the rules are set up to reward precision over sheer volume of low-quality tests.