Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel

arXiv:2608.00979 · cs.AI, cs.CL, cs.GT · Submitted 2026-08-02 · Read on arXiv

Yohei Nakajima

cs.AI, cs.CL, cs.GT

Submitted: 2026-08-02

Comments: 19 pages, 5 figures. Project site: https://yoheinakajima.github.io/synthetic-players/ ; code, data, registrations, review record, and zero-call replay capsule: https://github.com/yoheinakajima/synthetic-players

Code: https://github.com/yoheinakajima/synthetic-players

Project page: https://yoheinakajima.github.io/synthetic-players

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Composition — 23× in this paper · explained in 2 episodes
• Cooperation — 21× in this paper · explained in 2 episodes
• Provenance — 6× in this paper · explained in 5 episodes
• Generator — 5× in this paper · explained in 2 episodes
• Robustness — 3× in this paper · explained in 6 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Calibration — 2× in this paper · explained in 9 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 14 episodes
• Benchmark — 1× in this paper · explained in 2 episodes
• Causal Inference — 1× in this paper · explained in 5 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Homogeneity — 1× in this paper · explained in 2 episodes
• Homogenization — 1× in this paper · explained in 2 episodes
• Non-stationarity — 1× in this paper · explained in 2 episodes
• Scaffold — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• Evaluating Counterfactual Strategic Reasoning in Large Language Models
• This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
• Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments
• The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study
• Same Game, Different Story: A Minimal Conservative Strategic Robustness Benchmark for Large Language Model Agents
• Strategies of cooperation and defection in five large language models
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
• Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference
• The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems