Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

Abstract

Theory of Mind (ToM) is essential for agent interactions, yet existing evaluations either rely on static scenarios that oversimplify mental-state reasoning or interactive settings that provide limited diagnostic insight. We present Avalon-ToM-Bench, a fine-grained benchmark that operationalizes ToM through the asymmetric-information mechanics of The Resistance: Avalon. Rather than evaluating end-to-end gameplay, Avalon-ToM-Bench decomposes ToM into a 2 × 2 taxonomy spanning epistemic versus motivational reasoning and inference versus action, using human-crafted, perspective-constrained queries. Benchmarking 28 LLMs reveals three insights: First, models show strong game-rule comprehension but weaker ToM abilities. Second, mechanistic analyses via linear probing and activation steering demonstrate that models frequently represent correct mental-state inferences in their hidden states but fail to express them during generation (linear probes recover 77–82% accuracy versus 62–70% from the models’ own chain-of-thought), indicating that ToM limitations are primarily a problem of expression rather than representation. Third, dedicated reasoning training yields substantial improvements whereas test-time chain-of-thought provides only marginal gains (+11.0 versus +1.1 points on average), suggesting that robust ToM depends on a learned reasoning policy rather than increased inference-time deliberation.

Introduction

The evaluation of Theory of Mind (ToM) in Large Language Models (LLMs) demands a shift from treating the capability as a single, indivisible trait to recognizing it as a spectrum of distinct cognitive abilities. However, prevailing methodologies often miss this nuance. Traditional benchmarks frequently probe superficial patterns through static, overly simplistic scenarios—such as explicitly querying a character’s emotional state or visual perspective—stripping away the asymmetric information, deception, and multi-party dynamics inherent to real-world social reasoning. On the other hand, while hidden-role games offer rich, interactive environments, evaluating models strictly on end-to-end performance (e.g., win rates or action prediction) sacrifices interpretability. Such outcome-driven metrics blur the lines between genuine mind-reading and orthogonal skills like strategic planning, persuasion, or logical memory, making it impossible to isolate true ToM deficits from broader statistical biases.

To address these limitations, we introduce Avalon-ToM-Bench, utilizing the mechanics of The Resistance: Avalon as a controlled diagnostic environment rather than a gameplay arena. We map ToM across two fundamental axes: the nature of the targeted mental state (Epistemic knowledge versus Motivational intent) and its cognitive execution (passive Inference versus active Action). This yields four distinct quadrants of asymmetric-information reasoning. To guarantee objective assessment, Avalon-ToM-Bench queries models using highly constrained, human-verified propositions that restrict available evidence exclusively to the designated player’s point of view and the public interaction history, penalizing models that rely on omniscient context. We further verify that these axes are empirically dissociable: under permutation testing, item difficulty loads on the epistemic dimension whereas the benefit of chain-of-thought loads on the orthogonal inference–action dimension, confirming that the taxonomy carves ToM at meaningful joints rather than imposing an arbitrary partition.

Related Work

Theory of Mind and its Evaluation in LLMs. Early LLM studies treated ToM as a monolithic capability probed by short, bias-controlled question-answering. Later benchmarks broadened coverage—conversational asymmetry, long-form narratives, fine-grained behaviors, and higher-order reasoning—but still frame ToM as passive reading and entangle epistemic knowledge with motivational intent. Adversarial stress tests expose its brittleness, yet why models fail, and how to mitigate it, remains open.

Social Deduction Games as Reasoning Environments. Hidden-role games are long-standing testbeds for reasoning under information asymmetry, from pre-LLM game-theoretic agents to a fast-growing line of LLM players. These validate belief inference only through end-to-end game success, conflating it with planning, persuasion, and memory. Game-based evaluations instead target strategic competence or score behavioral alignment to human play; either way their gold labels are behavioral rather than the objective correctness of a mental-state deduction.

By cleanly decoupling ToM from general game-playing heuristics, Avalon-ToM-Bench is, to the best of our knowledge, the first framework to operationalize mental-state reasoning as a multidimensional, testable skill. We benchmark 28 distinct LLMs—encompassing various sizes, reasoning paradigms, and open/closed weights—uncovering three primary insights:

• Rule Mastery Does Not Equal Social Intelligence: Models struggle with perspective-constrained ToM (performance 50 − 60%) despite recalling Avalon’s mechanics near-perfectly (average rule comprehension accuracy ≈85%).

• The Latent Competence Gap: Linear probing and activation steering reveal that models internally compute the correct deduction but fail to surface it—probes recover 77–82% accuracy versus 62–70% from the model’s own chain-of-thought, suggesting that ToM failure may be one of expression, not representation.

• Robust ToM Requires Learned Policies, Not Just Deliberation: Test-time chain-of-thought yields volatile, marginal gains (+1.1 pts), whereas reasoning-trained models improve consistently (+11.0 pts)—robust ToM depends on a learned reasoning policy, not more inference-time tokens.

Probing and Steering Internal Representations. A complementary interpretability line reads and writes model computation directly: linear probes recover latent task information, models encode knowledge their generations omit, and activation steering shows such directions are causal. We bring these tools to ToM to show that mental-state inferences are internally represented yet under-expressed—a mechanistic axis unseen in prior ToM benchmarks.

Avalon-ToM-Bench

Avalon-ToM-Bench treats The Resistance: Avalon—a hidden-role game whose secret roles give each seat asymmetric private information over a shared public history of proposals, votes, and mission outcomes—not as gameplay to be won but as a measurement instrument. We decompose ToM into four abilities on a 2 × 2 taxonomy (§3.1) and probe each with human-authored, perspective-constrained binary statements whose labels are objectively verifiable against the rules, the designated seat’s information, and the public history.

A 2 × 2 Taxonomy of Theory of Mind

We organize ToM along two axes. The first is the nature of the mental state being reasoned about: Epistemic—what a seat (a player’s role-defined vantage point) can perceive or know (beliefs, information access)—versus Motivational—what a seat wants or intends (goals, motives). The second is how that mental state is engaged: Inference, where the target mental state exists from the role and board structure and must be read off with no communicative act involved, versus Action, where a strategic act—a public claim, a sabotage choice—is itself the object of reasoning and must be produced or interpreted as a deliberate move. Crossing the two axes yields four dissociable abilities, each of which is naturally instantiated by a distinct Avalon scenario:

1. Perspective-Taking (Epistemic × Inference): The Merlin–Percival–Morgana structure makes adversarial roles deliberately confusable; the task is to read off what a given seat can legitimately know under these constraints—a purely epistemic inference with no strategic act.

2. Strategic Signaling (Epistemic × Action): The Lady of the Lake holder privately learns a player’s faction and then makes a public, unverifiable claim; the evidence is epistemic but the object of reasoning is a communicative act—revealing, concealing, faking, or decoding that claim.

3. Intention Attribution (Motivational × Inference): Suspicion redirection scenarios reveal no identity directly; the model must abduce a hidden motive from an observed action.

4. Tacit Coordination (Motivational × Action): The EVIL sabotage protocol (priority Assassin > Morgana > Mordred) requires EVIL teammates to converge, without communication, on who plays the FAIL card—a coordinated act that must mesh with teammates’ shared intent.

These placements are design-driven—for each cell we selected the Avalon mechanic that most naturally instantiates it—and, crucially, not arbitrary: our decomposition analysis (§4.2) finds that item difficulty loads on the Epistemic axis while the benefit of chain-of-thought loads on the orthogonal Inference/Action axis, confirming that the taxonomy carves ToM at empirically dissociable joints.

Perspective-Constrained Task Formulation

Let R denote the Avalon ruleset, G the complete game state, H the public interaction history, and s a candidate inference. Most items designate a player i, either through an explicit perspective instruction Pi or directly in the target query. The role-visibility rules induce an information projection Ii = Πi(G) containing only what player i can legitimately know, and the model must evaluate s using evidence restricted to this projection: ŷ = M(R, Pi, G, H, s), ŷ ∈ TRUE, FALSE.

Although the model is shown the complete state G, valid evidence is confined to Ii and H; it must reconstruct the player’s epistemic state rather than treat every stated identity as accessible knowledge. A role-conditioned statement is labeled TRUE when it is valid under R, Ii, H, and any explicit scenario assumptions, and FALSE if it violates the rules, relies on inaccessible information, or makes an unsupported belief or strategic inference. Statements from the same scenario are evaluated independently rather than as competing multiple-choice options. A small number of EVIL-Protocol items are observer-level: they test whether a protocol consequence follows from the complete state without assigning a viewpoint, and are assessed directly against R, G, and H.

Evaluation. Each instance presents the game setup, relevant history, and one target statement; the gold label and reference rationale are withheld from the model, which outputs a single TRUE/FALSE judgment. We report accuracy and, because some diagnostic subsets are class-skewed, macro-averaged F1, which exposes any residual TRUE/FALSE prior. All items are run zero-shot with a fixed system/user prompt template.

Data Generation and Validation

The benchmark is built from real games rather than synthetic vignettes: the authors play and record full games and then, reviewing each recorded snapshot seat-by-seat and turn-by-turn, draft and independently cross-validate perspective-constrained statements against the rules, the seat’s role-filtered information, and the public history. In total it comprises 408 independently evaluated binary statements across 97 scenarios and the four dimensions, near-balanced at roughly 50% TRUE, plus a 34-item rule-recall control set—not counted among the 408—that isolates basic rule recall from perspective-sensitive reasoning.

Experiments

We evaluate 28 models spanning proprietary and open-weight families, reasoning-trained and standard instruction-tuned variants, and sizes from 0.6B up. Each is prompted zero-shot with a fixed template—game setup, public history, and one perspective-constrained statement—and returns a TRUE/FALSE judgment. We report accuracy and macro-F1 per ability on the full balanced set (408 items, 50% TRUE per ability), so a residual TRUE/FALSE prior surfaces as an Acc–F1 gap. A companion rules exam tests recall of Avalon’s mechanics, separating game knowledge from reasoning about minds.

Models Read the Rules but Not the Mind

The leaderboard reveals three findings:

• A wide, structured spread. Proprietary reasoning models lead (Gemini-2.5-Pro 89.7, GPT-5 88.5, Gemini-2.5-Flash 86.8), the weakest open models fall to the mid-50s, and class means track capability: 84.1 (proprietary), 71.9 (open reasoning), 66.5 (open non-reasoning). Reasoning training and scale both help, yet no open model reaches the proprietary frontier and even the best leave headroom on the harder abilities.

• Rule mastery is not social intelligence. Almost every model answers the rules exam well (mean ≈83.3%; most ≥88%) yet scores far lower on ToM items that apply those same rules from a constrained point of view. The failures are thus deficits of mental-state reasoning, not missing domain knowledge—and a benchmark that did not separate the two would conflate them.

• The abilities are not one skill. Averaged over models, accuracy orders as Intention Attribution (78.2) > Tacit Coordination (75.8) > Strategic Signaling (70.4) > Perspective-Taking (63.5), with Perspective-Taking consistently hardest and Intention Attribution easiest across the roster. ToM here behaves as dissociable abilities rather than a single trait.

The Taxonomy Carves ToM at Dissociable Joints

The per-ability ordering could reflect an arbitrary difficulty ranking. To test whether the two axes of our taxonomy truly capture distinct structure, we decompose two per-cell quantities over the 2 × 2 grid—item difficulty (mean accuracy) and the benefit of chain-of-thought—into main effects for the Epistemic/Motivational axis and the Inference/Action axis, and assess each effect with a permutation test that shuffles cell labels.

The two quantities load on orthogonal axes. Difficulty is driven by the Epistemic/Motivational axis (−8.5 pp, p < 0.001): epistemic abilities (Perspective-Taking, Strategic Signaling), which demand reconstructing what another seat can perceive, are reliably harder, while the Inference/Action axis has no significant effect on difficulty (−3.2 pp, p = 0.14). The benefit of chain-of-thought shows the mirror-image pattern: it is governed by the Inference/Action axis (−7.9 pp, p = 0.008)—deliberation helps the Action abilities, where a strategic move must be produced or decoded, more than the Inference abilities—while the Epistemic axis has no significant effect (−1.2 pp, p = 0.71). This double dissociation confirms that the taxonomy partitions ToM along meaningful, empirically separable dimensions rather than imposing an arbitrary grouping: what makes an item hard and what deliberation can fix are controlled by different axes of the design. A significant difficulty interaction (−9.3 pp, p = 0.027) further indicates that Perspective-Taking—epistemic and inference—is harder than the sum of the two main effects would predict, consistent with it being the single hardest cell.

Ablation Studies

Beyond ranking models, Avalon-ToM-Bench is designed to localize where and why Theory of Mind breaks. We report two analyses, conducted on the open-weight models whose internal states we can access. The first shows that the correct answer is present in a model’s internal representation yet under-expressed at its output: ToM failure is largely a problem of expression, not representation. The second shows that test-time chain-of-thought does little to close this gap, whereas dedicated reasoning training does—implicating the learned reasoning policy rather than the amount of inference-time deliberation.

Internal Representations as Stronger Predictors of ToM

If low accuracy reflected an inability to represent the answer, the answer should be as absent from a model’s internal states as from its output. We find the opposite through three distinct perspectives:

• A linear probe beats the model’s own reasoning. We train logistic probes on the last-token hidden states of the Qwen3 models (every layer, five-fold cross-validation) to predict the correct TRUE/FALSE label, and compare each probe against the same model’s chain-of-thought output on the identical items. Interestingly, the probe is substantially more accurate (≈77–…

Based on this paper, here are the specific improvements I can make to AI systems:

• Improvement: Train models to reason exclusively from a designated agent's information projection (what that agent can legitimately know), rather than from omniscient context.

• Capability: The improved system can answer questions about what another agent believes, knows, or intends—while explicitly ignoring privileged information it has access to. This is critical for multi-agent systems where each agent must act on partial information.

• Improvement: Add a training objective or decoding strategy that forces the model to verbalize what its internal representations already encode. The paper shows probes recover 77–82% accuracy vs. 62–70% from chain-of-thought, meaning the answer exists internally but isn't surfaced.

• Capability: The system can reliably output its true internal reasoning rather than producing under-expressed or incorrect final answers. This reduces the latent competence gap and improves reliability in high-stakes reasoning tasks.

• Improvement: Prioritize training-time reasoning instruction (e.g., explicit perspective-taking fine-tuning) over relying on chain-of-thought prompting at inference. The paper shows +11.0 pp from reasoning training vs. +1.1 pp from CoT, with CoT causing 27% answer flips.

• Capability: The system achieves robust, consistent ToM reasoning without needing long inference-time deliberation. This is more compute-efficient and less volatile in production settings.

• Improvement: Integrate a 2×2 diagnostic (Epistemic/Motivational × Inference/Action) into the system's self-assessment or evaluation pipeline, so failures are localized to specific cognitive sub-skills rather than treated as a monolithic ToM score.

• Capability: The system can identify whether it struggles with perspective-taking (epistemic inference), strategic signaling (epistemic action), intention attribution (motivational inference), or tacit coordination (motivational action)—enabling targeted retraining or runtime fallback strategies.

• Improvement: Implement a lightweight, inference-time intervention that adds a learned TRUE–FALSE direction to the residual stream at the best probe layer, lifting accuracy from 54–74% to 77–83% across model sizes.

• Capability: The system can be steered to surface latent ToM knowledge on demand, without retraining. This is useful for specialized applications (e.g., negotiation agents, tutoring systems) where ToM accuracy is critical but full retraining is impractical.

• Improvement: When a system is uncertain, inject the ground-truth answer into context and observe whether it improves. The paper shows +12.6 pp average improvement with no backfires across 28 models.

• Capability: The system can be used in human-in-the-loop settings where a supervisor provides hints. It also serves as a diagnostic: if hint injection doesn't help, the failure is representational (not just expressive), indicating a deeper training deficiency.

• Improvement: Separate domain-knowledge recall from social reasoning in evaluation. The paper shows models score 85% on rule recall but only 50–60% on perspective-constrained ToM.

• Capability: The system can be diagnosed to distinguish missing knowledge from missing reasoning. This prevents wasted training on knowledge the model already has, focusing instead on the actual deficit—mental-state reasoning under constraint.

• Improvement: Train on scenarios with deliberate confusability (e.g., Merlin–Percival–Morgana structures) where adversarial roles are intentionally ambiguous, forcing the model to reason about what others can plausibly know rather than what is true.

• Capability: The system can operate effectively in adversarial or deceptive environments (e.g., fraud detection, cybersecurity, competitive negotiation) where agents must reason about others' beliefs without being misled by surface-level information.

Summary of what the improved AI system can do: It can reason about other agents' mental states from their perspective, surface its latent knowledge reliably, improve ToM through training rather than inference-time tokens, self-diagnose which specific ToM sub-skill it lacks, be steered to enhance ToM on demand, and operate robustly in deceptive multi-agent environments—all while distinguishing knowledge gaps from reasoning gaps.

Theory of Mind (ToM) is essential for agent interactions, yet existing evaluations either rely on static scenarios that oversimplify mental-state reasoning or interactive settings that provide limited diagnostic insight. We present Avalon-ToM-Bench, a fine-grained benchmark that operationalizes ToM through the asymmetric-information mechanics of The Resistance: Avalon. Rather than evaluating end-to-end gameplay, it decomposes ToM into a 2 times 2 taxonomy -- epistemic versus motivational reasoning crossed with inference versus action -- using human-crafted, perspective-constrained queries. Benchmarking 28 LLMs reveals three insights: 1) Reasoning, not knowledge. Models show strong game-rule comprehension but markedly weaker ToM abilities, isolating failures to social reasoning rather than missing domain knowledge. 2) Expression, not representation. Mechanistic analyses via linear probing and activation steering show that models frequently represent correct mental-state inferences in their hidden states but fail to express them during generation -- linear probes recover 77-82% accuracy versus 62-70% from the models' own chain-of-thought. 3) Policy, not deliberation. Dedicated reasoning training yields substantial improvements whereas test-time chain-of-thought provides only marginal gains (+11.0 versus +1.1 points on average), suggesting that robust ToM depends on a learned reasoning policy rather than increased inference-time deliberation.