HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning
cs.AI, cs.CL, cs.CY
Submitted: 2025-10-16
Updated: 2026-08-31
Comments: Accepted to EMNLP 2026 Main Conference
License: http://creativecommons.org/licenses/by/4.0/
The gist: Simulating human reasoning in open-ended tasks has long been a central aspiration in AI and cognitive science.
Terminology
Abstract
Simulating human reasoning in open-ended tasks has long been a central aspiration in AI and cognitive science. While large language models now approximate human responses at scale, they remain tuned to population-level consensus, often erasing the individuality of reasoning styles and belief trajectories. To advance the vision of more human-like reasoning in machines, we introduce HugAgent (HUman-Grounded AGENT Benchmark), which rethinks human reasoning simulation along three dimensions: (i) from averaged to individualized reasoning, (ii) from behavioral mimicry to cognitive alignment, and (iii) from vignette-based to open-ended data. The benchmark evaluates whether a model can predict a specific person's behavioral responses and the underlying reasoning dynamics in out-of-distribution scenarios, given partial evidence of their prior views. HugAgent combines structured questionnaires with semi-structured think-aloud interviews to collect ecologically valid belief states, belief updates, and reasoning traces from human participants. Our experiments reveal a clear asymmetry: models recover a person's belief state from their own context reasonably well, but struggle to predict belief updates under intervention. Cross-person and cross-domain controls trace this gap to associative matching within a topic rather than identity-consistent reasoning, suggesting that progress requires better-calibrated change detection, not simply more context. We scope the benchmark to self-reported belief reasoning in three policy domains: healthcare, surveillance, and zoning. The benchmark, along with its complete data collection pipeline and companion chatbot, is open-sourced as HugAgent (https://github.com/jajamoa/HugAgent) and TraceYourThinking (https://github.com/jajamoa/trace-your-thinking).
Sources
- Relational inductive biases, deep learning, and graph networks
- ToMBench: Benchmarking Theory of Mind in Large Language Models
- Towards Measuring the Representation of Subjective Global Opinions in Language Models
- DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
- Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
- Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making
- HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
- WikiWhy: Answering and Explaining Cause-and-Effect Questions
- Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning
- CommunityLM: Probing Partisan Worldviews from Language Models
- MMToM-QA: Multimodal Theory of Mind Question Answering
- Lyfe Agents: Generative agents for low-cost real-time social interactions
- FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
- Simulating Society Requires Simulating Thought
- LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
- Machine Theory of Mind
- ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
- A Roadmap to Pluralistic Alignment
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems