Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Total Simulated Survey Error".
Jane: Here's the gist of the paper. The Total Simulated Survey Error (TS2E) Framework is introduced as a conceptual framework extending the Total Survey Error (TSE) framework from quantitative measurement theory to systematically account for measurement and representation errors in surveys generated by Large Language Models (LLMs).
Tom: This framework aims to move research beyond simply asking whether LLMs can simulate humans toward precisely determining when, for whom, and with what errors they perform.
Jane: The TS2E framework enumerates errors across four distinct phases of the LLM-generated survey lifecycle: Before Fielding (Questionnaire Design), Fielding (LLM Selection and Inference), After Fielding (Response Processing), and Evaluation.
Tom: Errors are categorized into measurement errors—issues in how a construct is simulated—and representation errors—issues in how a target population is accessed.
Paper summary: Tom: So, wrapping up the "Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models" paper, it’s essentially a comprehensive system for diagnosing errors in surveys made by large language models. It moves beyond simple questions about capability and gives us a structured way to map those errors onto the specific steps of designing and running an LLM-based survey.
Jane: The authors are Indira Sen, Georg Ahnert, Leah von der Heyde, Jana Lasser, Bernd Weiß, and Markus Strohmaier from various universities and research hubs. They introduce this Total Simulated Survey Error framework which systematically accounts for both measurement errors in how a construct is simulated and representation errors in accessing the target population.
Lu: What they argue is that this framework provides a natural foundation for reasoning about the quality of LLM-generated survey data by distinguishing between errors caused by researcher design choices and those caused by inherent limitations of the AI technology itself.
Meng: For someone focused on shipping models, it means knowing exactly where to focus your effort when you see low accuracy in simulated outputs—is it a bad persona prompt, or is the underlying LLM just struggling with that specific type of measurement?
Tom: It gives designers a way to systematically reason about the quality of their data by linking survey operations directly to statistical error components, which lets them do trade-off analysis among competing error sources.
Jane: In simple terms, this paper tells us how to move from vaguely knowing that LLMs might be biased in surveys to having a precise tool that shows us exactly where the bias is coming from and what we can actually control.
Conclusion: Tom: So we’ve been looking at this framework for Total Simulated Survey Error from the paper "Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models."
Jane: It basically takes all those messy errors in LLM surveys and puts them into four steps—before fielding, during fielding, after fielding, and evaluation.
Tom: Right. So it’s not just about getting a good answer; it’s about tracking exactly *where* the mistake happened in the whole process.
Lu: The authors are trying to move past just saying LLMs can simulate humans and actually figuring out when they fail for specific groups or under certain conditions.
Meng: From my side, it’s interesting because it suggests we can start debugging our own model outputs by mapping them onto these design choices—like how the question was written or who the AI is pretending to be.
Jane: So, instead of just accepting a low accuracy number, you can look at the framework and see if that error came from a bad persona setup, or maybe a poor choice in how you phrased the task itself.
Tom: Exactly. It helps designers isolate which specific design choices cause the most damage to the final estimate and helps you understand those inherent LLM limitations.
Lu: The study used a multiverse analysis on vote choice data from an election to show that incorporating certain attitudes into the persona construction actually made a real difference in prediction accuracy.
Jane: That’s concrete. It shows that tweaking how you define the target group can significantly improve the simulation results, which is really practical for anyone using these tools.
Tom: It gives designers a diagnostic tool to see which error source—measurement error or representation error—is causing the bigger problem in their survey setup right now.
Meng: For us shipping models, it means we have to be more careful about those persona constructions because that’s where a lot of the simulation errors seem to cluster.
Jane: It also points out some common mistakes people make when they evaluate these surveys, like assuming the ground truth is perfect or picking just one single best metric without checking other things.
Tom: So we can start demanding more documentation from the teams building these simulations about their choices and how those choices map to statistical error components.
Lu: It’s a way to perform retrospective evaluation and plan future designs by explicitly linking survey operations to those statistical error components, which is pretty powerful for iterative improvement.
Jane: This whole idea is important because it forces us to think about the entire survey lifecycle as a series of potential points of failure, not just one big final answer.
Tom: And there are ethical bits too—it touches on privacy and consent when you’re simulating individuals versus broad groups.
Jane: It’s a lot to digest, but really it gives us a clearer language to talk about the quality and limitations of what these large language models can produce in survey contexts.
University of Mannheim · GESIS - Leibniz Institute for the Social Sciences · University of Graz · University of Duisburg-Essen · Complexity Science Hub
cs.CY
Submitted: 2026-09-09
Updated: 2026-11-16
Comments: Preprint, 19 pages in main text
Code: https://github.com/dess-mannheim/ts2e_
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 88/100
The gist: The Total Simulated Survey Error (TS2E) Framework is introduced as a conceptual framework extending the Total Survey Error (TSE) framework from quantitative measurement theory to systematically…
Key concepts
- Measurement Errors
- These are issues related to how a specific concept is simulated or measured in the survey. They include errors in question design, such as poor specification, and errors in persona construction, where the simulated respondent does not accurately reflect the target population being studied.
- Representation Errors
- These occur when there is a mistake in how the target population is accessed or represented by the LLM. For example, a persona simulation error happens when the LLM misrepresents who it is simulating, leading to inaccurate data about that group.
- Ground Truth Fallacy
- This fallacy involves assuming that a single 'correct' answer exists for every survey question. The framework warns against this by suggesting designers must account for multiple possible correct answers or acknowledge the inherent uncertainty in simulated responses.
- Context Drift Fallacy
- This error arises when the context of the simulation changes unexpectedly during the process. Designers are advised to meticulously document all decision choices and LLM characteristics used, as unrecorded shifts can introduce significant, unaccounted-for errors into the final estimates.
Terminology
Summary
The Total Simulated Survey Error (TS2E) Framework is introduced as a conceptual framework extending the Total Survey Error (TSE) framework from quantitative measurement theory to systematically account for measurement and representation errors in surveys generated by Large Language Models (LLMs). p. 1 Building on quantitative measurement theory, and specifically the Total Survey Error (TSE) framework
[Groves and Lyberg, 2010], we adapt the well-established distinction between measurement errors (errors in how a
2
For brevity, we interchangeably use ‘LLM-generated surveys’, ‘LLM-generated survey responses’, and ‘LLM-simulated
responses. ’ While LLMs can also be used for other parts of the survey lifecycle (cf. More broadly, we hope the framework offers a common
vocabulary facilitating interdisciplinary research in LLM-generated surveys. In that role, the framework could support
the development of reporting and evaluation standards and benchmarks for LLM-generated survey data, and help move
the field from asking whether LLMs can simulate humans toward asking, more precisely, when, for whom, and with
what errors.Where the paper says it
Where the paper says it
The TS2E framework enumerates errors across four distinct phases of the LLM-generated survey lifecycle: Before Fielding (Questionnaire Design), Fielding (LLM Selection and Inference), After Fielding (Response Processing), and Evaluation. p. 5 WORKING PAPER - SEPTEMBER 10, 2026
LLM
Inference
Construct Target
Population
Processed LLM
Response
Final
Measurement
Persona
Simulation
Error
Response
Generation
Error
Response
Processing
Error
Questionnaire Design Specification
Error
Response Processing
Reweighted LLM
Respondents
Adjustment
Error
Reweighting
Persona
Construction
Error
Persona Construction
Measurement Representation
Question
Articulation
Error
Prompt:
Task
Prompt:
Personas
LLM
Response
LLM
Respondents
Task Articulation
Fielding or
Simulation
After
Fielding
Before
Fielding
Evaluation
Single Best
Metric
Fallacy
Context
Drift
Fallacy
Ground
Truth
Fallacy
Figure 3: The Total Simulated Survey Error (TS2E) Framework: Representation and Measurement Errors, and
Evaluation Fallacies. We show an idealized lifecycle of simulated survey with different design choices and artifacts. WORKING PAPER - SEPTEMBER 10, 2026
construct is defined, operationalized and answered by respondents) and representation errors (errors in how a welldefined target population is accessed and approximated) to the specific affordances of current LLM technology. The
TSE framework has previously been extended to novel quantitative social science data and methods Weiß et al.Where the paper says it
Where the paper says it
Specific measurement errors identified include:
-
Questionnaire Design Error: This encompasses
Specification error,
defined as the gap between the theoretical construct and the question used to measure it; andArticulation error,
defined as delta(q, t), which is the gap between the question and the task formulated to prompt the LLM. -
Persona Construction Error: Defined as delta(U, fpersona(fmap(u)u in U)), this is the gap between the target population (U) and the constructed sample of personas (fpersona).
Specific representation errors identified include:
-
Persona Simulation Error: A representation error where the LLM misrepresents the respondent being simulated.
-
Response Generation Error: A measurement error arising from incorrect LLM responses to a given task.
The paper utilizes an empirical case study simulating vote choice during the 2024 Presidential Election in the U. p. 2 The frameworks also introduces
evaluation fallacies that might plague the validation of LLM-generated survey responses, especially when they are
benchmarked against human responses. In addition to illustrating these errors with theoretical examples, we provide an
empirical case study that applies the framework using a multiverse analysis over simulating vote choice during the 2024
Presidential Election in the U. WORKING PAPER - SEPTEMBER 10, 2026
LLM
Inference
Construct Target
Population
Processed LLM
Response
Final
Measurement
Persona
Simulation
Error
Response
Generation
Error
Response
Processing
Error
Questionnaire Design Specification
Error
Response Processing
Reweighted LLM
Respondents
Adjustment
Error
Reweighting
Persona
Construction
Error
Persona Construction
Measurement Representation
Question
Articulation
Error
Prompt:
Task
Prompt:
Personas
LLM
Response
LLM
Respondents
Task Articulation
Fielding or
Simulation
After
Fielding
Before
Fielding
Evaluation
Single Best
Metric
Fallacy
Context
Drift
Fallacy
Ground
Truth
Fallacy
Figure 3: The Total Simulated Survey Error (TS2E) Framework: Representation and Measurement Errors, and
Evaluation Fallacies. We show an idealized lifecycle of simulated survey with different design choices and artifacts.Where the paper says it
Where the paper says it
The TS2E framework serves as a practical diagnostic tool for simulation designers to isolate which design choices most degrade estimates and to document errors stemming from inherent LLM limitations. p. 2 Beyond organizing the fast-moving literature on potentials and pitfalls in LLM-based survey simulations, the TS2E
framework is intended as a practical diagnostic tool. By decomposing error by source and by lifecycle stage, it
enables simulation designers to isolate which design choices most degrade their estimates, to document errors that
lie beyond their control because they are due to inherent LLM limitations, and to reason about trade-offs among
competing error sources—much as the TSE framework reframed survey design as an optimization problem over
multiple error components rather than a single quality metric. Beyond organizing the fast-moving literature on potentials and pitfalls in LLM-based survey simulations, the TS2E
framework is intended as a practical diagnostic tool. By decomposing error by source and by lifecycle stage, it
enables simulation designers to isolate which design choices most degrade their estimates, to document errors that
lie beyond their control because they are due to inherent LLM limitations, and to reason about trade-offs among
competing error sources—much as the TSE framework reframed survey design as an optimization problem over
multiple error components rather than a single quality metric.Where the paper says it
Where the paper says it
The paper also outlines several evaluation fallacies, such as the Ground Truth Fallacy, Single Best Metric Fallacy, and Context Drift Fallacy. p. 5 WORKING PAPER - SEPTEMBER 10, 2026
LLM
Inference
Construct Target
Population
Processed LLM
Response
Final
Measurement
Persona
Simulation
Error
Response
Generation
Error
Response
Processing
Error
Questionnaire Design Specification
Error
Response Processing
Reweighted LLM
Respondents
Adjustment
Error
Reweighting
Persona
Construction
Error
Persona Construction
Measurement Representation
Question
Articulation
Error
Prompt:
Task
Prompt:
Personas
LLM
Response
LLM
Respondents
Task Articulation
Fielding or
Simulation
After
Fielding
Before
Fielding
Evaluation
Single Best
Metric
Fallacy
Context
Drift
Fallacy
Ground
Truth
Fallacy
Figure 3: The Total Simulated Survey Error (TS2E) Framework: Representation and Measurement Errors, and
Evaluation Fallacies. We show an idealized lifecycle of simulated survey with different design choices and artifacts. Beyond organizing the fast-moving literature on potentials and pitfalls in LLM-based survey simulations, the TS2E
framework is intended as a practical diagnostic tool. By decomposing error by source and by lifecycle stage, it
enables simulation designers to isolate which design choices most degrade their estimates, to document errors that
lie beyond their control because they are due to inherent LLM limitations, and to reason about trade-offs among
competing error sources—much as the TSE framework reframed survey design as an optimization problem over
multiple error components rather than a single quality metric. Total Simulated Survey Error: Designing and Diagnosing Survey
Responses from Large Language Models
Indira Sen1∗
, Georg Ahnert1
, Leah von der Heyde2
, Jana Lasser3
, Bernd Weiß4
, and Markus Strohmaier1,2,5
1University of Mannheim, 2 GESIS - Leibniz Institute for the Social Sciences,
3University of Graz, 4 University of Duisburg-Essen, 5Complexity Science Hub
September 10, 2026
ABSTRACT
Large Language models (LLMs), having been trained on vast amounts of human-generated data, may
encode the attitudes and behaviors of these humans. As such, LLMs show promise in mimicking
human-like patterns that facilitate their use in simulating people in a wide variety of contexts. Sensitivity, also used interchangeably with robustness [Ye et al. , 2026], refers to changes in LLM output in the
face of minute and inconsequential changes in input [Sclar et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The TS2E framework is applicable across various LLM uses, including content analysis, behavioral experiments, and generative agent-based models (GABMs), noting that certain errors are transferable across contexts. p. 0 Trained on vast corpora of human-generated text,
LLMs encode rich, if imperfect, representations of human attitudes, knowledge, and behavioral tendencies, and have
demonstrated an ability to produce human-like outputs across a wide range of tasks. This capacity has prompted
the development of approaches that use LLMs as proxies for humans in social science contexts, dubbed as “in-silico
samples” [Argyle et al. The LLM generates output based on this
prompt, which can range from simple single-token responses for closed survey questions to complex free-text responses
that describe the behavior of the entity they are simulating [Ahnert et al. LLMs are imperfect simulators — their simulations of certain groups and certain constructs are worse, but not
necessarily systematically [Boelaert et al. Beyond organizing the fast-moving literature on potentials and pitfalls in LLM-based survey simulations, the TS2E
framework is intended as a practical diagnostic tool. By decomposing error by source and by lifecycle stage, it
enables simulation designers to isolate which design choices most degrade their estimates, to document errors that
lie beyond their control because they are due to inherent LLM limitations, and to reason about trade-offs among
competing error sources—much as the TSE framework reframed survey design as an optimization problem over
multiple error components rather than a single quality metric.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Trace error origins across the full lifecycle of an LLM-generated survey by using a framework that
distinguishes errors due to design choices of a researcher from those due to inherent LLM limitations.
This allows designers tosystematically identify, trace, and document limitations in LLM-generated survey responses.
-
Implement systematic error decomposition by separating errors into measurement and representation errors. This distinction is crucial because they
call for different remedies,
allowing designers to address how the construct is measured separately from how the target population is approximated. -
Create a diagnostic tool that allows simulation designers to
isolate which design choices most degrade their estimates, to document errors that lie beyond their control because they are due to inherent LLM limitations, and to reason about trade-offs among competing error sources.
This moves the field fromasking whether LLMs can simulate humans toward asking, more precisely, when, for whom, and with what errors.
-
Develop a checklist for documenting LLM-generated survey data that extends existing guidelines to include issues specific to survey simulation. This checklist should guide designers in
documenting the simulation context, their decision choices, as well as characteristics of the LLMs they use.
-
Integrate reliability and sensitivity testing into evaluation protocols. Designers should assess
reliability which stresses changes in LLM output to the exact same input and design configurations
andsensitivity to inconsequential changes, e.g., the question phrasing,
rather than relying on single performance scores.
Abstract
Large Language models (LLMs), having been trained on vast amounts of human-generated data, may encode the opinions, attitudes, and behaviors of these humans. As such, LLMs show promise in mimicking human-like patterns that facilitate their use in simulating people in a wide variety of contexts. One such context is using LLMs as 'silicon samples', i.e., proxies of people in answering survey questions to establish public opinion, design policies, or use as (social) scientific data. However, several critical questions of social biases, generalization, and technical limitations remain, further complicated by a vast design space open to simulation designers. In addition, LLM-simulated surveys often lean on existing survey infrastructure, e.g., survey questions and samples. These artifacts might not be suitable for LLMs or themselves might have errors that are then inherited by the LLM-powered survey simulations. As a result, we lack a systematic understanding of the design space of LLM-generated surveys as well as how these decisions interplay with inherent LLM limitations. In this article, we aim to systematically identify, trace, and document limitations in LLM-generated survey responses. Building on traditions in the quantitative social sciences, specifically survey methodology and measurement theory, we investigate threats to the validity of LLM-generated survey responses. To do so, we design a framework that enumerates conceptual errors and systematic biases that can occur at different stages of the survey simulation lifecycle. Our framework - called the Total Simulated Survey Error (TS2E) Framework - provides a unified and end-to-end perspective on errors in LLM-generated survey data. We illustrate the framework through a theoretical and empirical case study, which enables survey simulation designers to systematically understand, identify and reflect on errors in LLM-generated surveys.
Sources
- Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
- LLM Social Simulations Are a Promising Research Method
- Emergent LLM behaviors are observationally equivalent to data leakage
- Centaur: a foundation model of human cognition
- Post-training makes large language models less human-like
- Digital trace data collection through data donation
- The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions
- Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
- Validating LLMs in social science: Epistemic threats and emerging norms
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- Learning from Convenience Samples: Fine-Tuning LLMs to Predict Voting Choices from Biased Survey Samples
- Imputation of Public Opinion Data with Large Language Models
- SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
- This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
- AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks
- Virtual Personas for Language Models via an Anthology of Backstories
- Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity
- AI Automatons: AI Systems Intended to Imitate Humans
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance