Artificial Societies Benchmark: A Validation Framework for Synthetic Research
cs.CL
Submitted: 2026-09-24
Updated: 2026-10-01
Comments: 36 pages, 9 figures, 9 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions.
Terminology
Abstract
A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions. We introduce the Artificial Societies Benchmark to help researchers assess whether synthetic populations support their intended analyses. The framework combines eleven tests across internal, construct, and external validity, drawing on twenty human sources and comparing nine language models. It connects each research use to the evidence it requires and tests how results change with the information we supply about respondents. Importantly, strong performance in one domain does not establish fidelity in the others. Models often answer too consistently, compress response scales, and alter relationships between traits whilst richer profiles improve prediction for some models and worsen it for others. The resulting scorecard helps researchers identify which aspects of a synthetic population can support their analysis and where researchers need further human evidence.
Sources
- Consequential Behaviour and Representational Fairness in the Validation of Synthetic Research
- LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
- Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
- Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering