Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
cs.CL, cs.AI, stat.AP
Submitted: 2026-08-03
Updated: 2026-09-11
Comments: Accepted as a workshop paper at https://www.aiagentbehavior.com/
License: http://creativecommons.org/licenses/by/4.0/
The gist: A/B testing remains the standard for rolling out new features in the technology industry.
Terminology
Abstract
A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a Simulated Randomized Controlled Trial (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by about 77 times; a within-subject design---where each agent is exposed to both arms---reduces standard errors by about 2.4 times. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.
Sources
- LLM Social Simulations Are a Promising Research Method
- Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation
- SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce
- HumanStudy-Bench: Towards AI Agent Design for Participant Simulation
- AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents
- LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings
- ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders
- Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts
- LLM Agents as Social Scientists: A Human-AI Collaborative Platform for Social Science Automation
- Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering