Rewarding Noise: Incentives in Decentralized A/B Testing
The paper investigates how testing incentives in decentralized organizations lead to low statistical power and noisy decisions. It shows that when product teams maximize private rewards, they favor many low-precision tests. Switching to a posterior-mean scoring rule removes this reward for noise, leading product teams to focus on fewer, more precise tests of ambitious ideas.