Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies
cs.CL
Submitted: 2025-10-13
Updated: 2025-10-13
Comments: 34 pages, 32figures
Journal ref: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 50207-50243 (2026)
DOI: 10.18653/2026.emnlp-main.2208
License: http://creativecommons.org/licenses/by/4.0/
The gist: Social deduction games like Werewolf combine language, reasoning, and strategy, providing a testbed for studying natural language and social intelligence.
Terminology
Abstract
Social deduction games like Werewolf combine language, reasoning, and strategy, providing a testbed for studying natural language and social intelligence. However, most studies reduce the game to LLM-based self-play, yielding templated utterances and anecdotal cases that overlook the richness of social gameplay. Evaluation further relies on coarse metrics such as survival time or subjective scoring due to the lack of quality reference data. To address these gaps, we curate a high-quality, human-verified multimodal Werewolf dataset containing over 100 hours of video, 32.4M utterance tokens, and 15 rule variants. Based on this dataset, we propose a novel strategy-alignment evaluation that leverages the winning faction's strategies as ground truth in two stages: 1) Speech evaluation, formulated as multiple-choice-style tasks that assess whether the model can adopt appropriate stances across five dimensions of social ability; and 2) Decision evaluation, which assesses the model's voting choices and opponent-role inferences. This framework enables a fine-grained evaluation of models' linguistic and reasoning capabilities, while capturing their ability to generate strategically coherent gameplay. Our experiments show that state-of-the-art LLMs show diverse performance, with roughly half remain below 0.50, revealing clear gaps in deception and counterfactual reasoning. We hope our dataset further inspires research on language, reasoning, and strategy in multi-agent interaction.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction
- AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeek-V3 Technical Report
- Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Game
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- GPT-4o System Card
- Gemma 3 Technical Report
- A Novel Weighted Ensemble Learning Based Agent for the Werewolf Game
- Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria
- AvalonBench: Evaluating LLMs Playing the Game of Avalon
- The Stepwise Deception: Simulating the Evolution from True News to Fake News with LLM Agents
- Playing the Werewolf game with artificial intelligence for language understanding
- Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
- Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey
- Enhance Reasoning for Large Language Models in the Game Werewolf
- Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf
- Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data