MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games

arXiv:2607.10645 · cs.CL, cs.MA, cs.SI · Submitted 2026-07-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games".

Jane: Here's the gist of the paper. An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes <ref:2607.10645#pg2> Pruning redundant reasoning tokens MafiaScope is presented as a non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind within social deduction games like Mafia <ref:2607.10645#pg4>.

Tom: It allows researchers to distinguish between an agent losing by misreading the game or by wasting a correct read, a distinction that is invisible through outcomes and transcripts <ref:2607.10645#pg3>.

Jane: The system works after every public utterance by having every agent privately answer structured probe questions whose answers never re-enter the game and are scored against the ground truth the engine already knows <ref:2607.10645#pg4>.

Tom: How the training signal is built The core of MafiaScope consists of three decoupled layers: a game engine, a probe engine, and a viewer connected by JSONL logs <ref:2607.10645#pg4>.

Title and authors: Tom: Moving on to the title and authors, we have "MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games," which sounds pretty descriptive of what they’re trying to do.

Jane: It’s a mouthful, but the core idea is that this isn't just about playing Mafia; it's about using that game as a way to measure how agents are reasoning socially > <ref:2607.10645#pg1>.

Lu: The authors are working on this by building a whole infrastructure—a game engine, a probe engine, and a viewer connected by JSONL logs > <ref:2607.10645#pg4>. They’re essentially creating an open testbed for studying these complex social deduction games with LLM agents.

Meng: So they've built the mechanics to turn the game into data, which is what I usually focus on when we think about deployment > <ref:2607.10645#pg4>. The infrastructure itself, the way they wrote those logs in append-only JSONL, is important for how you ingest and analyze this information later.

Tom: Right. It’s about building a structured way to capture every single step of the game so we can probe it later without disrupting the flow of the actual play > <ref:2607.10645#pg4>. It’s about turning an event into a data point that can be analyzed in depth > <ref:2607.10645#pg1>.

Jane: And what this means for us is that we have a standardized way to run these kinds of experiments, which is super valuable when we are trying to compare different agent architectures or model families > <ref:2607.10645#pg4>. It gives us a common language for studying theory of mind in AI.

Lu: I think the related work section points out that existing methods like DeepRole already maintained posterior beliefs over role assignments inside a purpose-built planner, but MafiaScope probes and scores those beliefs differently > <ref:2607.10645#pg2>. They’re not just tracking what the agent believes; they are actively testing that belief against something else.

Meng: So they're adding a layer of active questioning to what was already there, which is useful because it moves it from passive observation to active interrogation > <ref:2607.10645#pg2>. It’s about forcing the model to articulate its reasoning rather than just letting it run its course.

Tom: And that leads right into the practical improvements they propose, which are where the real utility for us comes in—how we actually use this tool beyond just reading the paper > <ref:2607.10645#pg4>.

The paper's summary: Jane: So, to recap what we've covered, MafiaScope is essentially a non-invasive belief probing workflow designed to measure machine theory of mind within social deduction games > <ref:2607.10645#pg1>.

Lu: That means they’ve built a probe engine that runs after every public event and privately asks agents structured questions—things like role beliefs and planned actions > <ref:2607.10645#pg4>.

Meng: And these answers are scored against the ground truth that the engine already knows, so it lets you distinguish between losing by misreading or wasting a correct read > <ref:2607.10645#pg1>.

Tom: That distinction is invisible through just looking at outcomes or transcripts, which is a huge gap they’re trying to fill > <ref:2607.10645#pg1>. They can tell if the agent made a mistake in reading the situation or if it just got stuck in an unproductive loop > <ref:2607.10645#pg1>.

Jane: And then they have this replayable inspection environment where you can look at beliefs per utterance, per game, and even across games with views for calibration and deception > <ref:2607.10645#pg4>. It lets you see the agent's internal state in real time as it plays > <ref:2607.10645#pg4>.

Lu: And a reusable benchmark is also a key contribution here: a cross-model corpus of probed games plus case studies showing what outcome-only evaluation misses, like confidence detached from accuracy > <ref:2607.10645#pg4>.

Meng: It sounds like they’ve formalized the gap between what we measure and what we actually care about, which is important because outcome-only evaluation is often misleading > <ref:2607.10645#pg4>. It’s about measuring the belief dynamics themselves > <ref:2607.10645#pg4>.

Tom: And that means if you’re trying to build a model that can reason socially, you need tools like this to check if it's reasoning correctly or just mimicking a correct final output > <ref:2607.10645#pg1>.

Jane: So, in short, they are giving us the infrastructure to inspect the internal reasoning of these agents during social deduction games without having to change how they fundamentally operate > <ref:2607.10645#pg4>.

The paper's improvements: Lu: The authors suggest a few key improvements focused on adding better analysis layers to what’s already there, like better ways to study belief dynamics > <ref:2607.10645#pg4>. They want to go beyond just showing the game and the basic probes > <ref:2607.10645#pg4>.

Meng: I think one of those is developing a dynamic belief graph visualization that lets researchers inspect the ground-truth graph, one agent’s subjective graph, and the filtered game log over a shared timeline > #six <ref:2607.10645#pg2>. That would be really powerful for tracking how Villager-side agents start with no opinion, which was seventy-four point nine percent "Unknown" in round zero > <ref:2607.10645#pg4>.

Tom: That visualization sounds like a way to visualize that continuous-time graph of timestamped, confidence-weighted belief edges > <ref:2607.10645#pg4>. It lets you scrub timelines and inspect agent divergence in a very intuitive way > #six <ref:2607.10645#pg2>.

Jane: And they also talk about probe chaining re-injecting previous assessments can strengthen commitment regardless of new evidence > <ref:2607.10645#pg4>. That’s a mechanism we can use to test if initial beliefs have lasting influence on subsequent decisions.

Lu: That suggests that even if the agent gets new data, its initial internal state still plays a role in how it processes that data > <ref:2607.10645#pg4>. It's about testing the persistence of belief > <ref:2607.10645#pg4>.

Meng: And they also mention a flip-rate floor of thirty-four point eight percent which comes from probing five times each point, reweighted to the corpus composition, which shapes the measurement > <ref:2607.10645#pg4>. That’s a concrete metric that shows how robust this measurement is across different setups > <ref:2607.10645#pg4>.

Tom: So they are focusing on making sure these metrics aren't just arbitrary numbers but actually reflect real dynamics in the system, which is what separates a good experiment from noise > #six <ref:2607.10645#pg2>. It’s about measuring the actual system behavior, not just some abstract concept of reasoning quality > #six <ref:2607.10645#pg2>.

Jane: And I also want to talk about how they grade beliefs, not wins, which is a big shift from just looking at the final outcome as they are making those points > #four. That means we can finally look at the nuance of their internal reasoning > #four.

Lu: So when you combine all these ideas—the dynamic graph visualization and the belief edge analysis and that’s what makes this paper interesting for researchers in representation learning > #six <ref:2607.10645#pg2>. It’s about seeing how agents build their understanding over time > #four.

Meng: And I just want to make sure we're talking about things that actually matter for deployment, like what breaks at scale or how much computation this probing adds to the system > #four.

Conclusion: Tom: Alright, wrapping up this look at the MafiaScope paper. The main implication is that we have tools to inspect agents' beliefs in social deduction games without changing their core operation > <ref:2607.10645#pg1>.

Jane: We’re gaining a new way to grade reasoning by looking at these belief trajectories, which are continuous graphs of timestamped, confidence-weighted belief edges > #four.

Lu: So we can track how those Villager-side agents start with no opinion and converge on the Mafia as more evidence accumulates > #four. It gives researchers a dynamic view of machine reasoning in these scenarios > #six <ref:2607.10645#pg2>.

Meng: For production systems, I just want to make sure we're not adding too much overhead when trying to run this level of probing against live traffic or complex workloads > #four.

Lalam: I’m pretty sure that with the MafiaScope method, we can finally see how agents build their understanding over time on the game board > #four. It's a huge step forward in understanding machine theory of mind > #four.

Tom: We have this non-invasive belief probing workflow that lets us score beliefs against ground truth, which tells us whether an agent lost by misreading or wasting a correct read > #one <ref:2607.10645#pg1,whether an agent lost by misreading>. That’s the core mechanism we need to be excited about for AI research > #one <ref:2607.10645#pg1>.

Jane: It shifts the focus from just checking the final result to understanding the actual process behind that result > #four. That’s a big shift in how we evaluate agents on social tasks, and it opens up a lot of avenues for future work > #four.

Ilia Karpov

HSE University

cs.CL, cs.MA, cs.SI

Submitted: 2026-07-12

Updated: 2026-10-30

Code: https://github.com/karpovilia/mafiascope

Project page: https://karpovilia.github.io/mafiascope

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing…

Key concepts

MafiaScope
A non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind in social deduction games. It works by having agents privately answer structured questions after every public statement, allowing researchers to score beliefs against known ground truth.
Probe Engine
The component that pauses the game after each public event to privately interrogate alive agents using structured probes. These probes ask specific questions about an agent's role beliefs, suspicion rankings, and planned actions without those answers re-entering the game.
Belief Trajectories
The measurable path of an agent's opinions over time. For Villager-side agents, this trajectory shows they start with no opinion and gradually converge on identifying the Mafia role as more evidence accumulates in the game.

Terminology

Summary

An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes <ref:2607. p. 0

Where the paper says it

MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games Ilia Karpov HSE University karpovilia@gmail. com Abstract An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes.

10645#pg2>

Pruning redundant reasoning tokens

MafiaScope is presented as a non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind within social deduction games like Mafia <ref:2607.10645#pg4>. It allows researchers to distinguish between an agent losing by misreading the game or by wasting a correct read, a distinction that is invisible through outcomes and transcripts <ref:2607. p. 0

Where the paper says it

com Abstract An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind: it can tell whether an agent lost by misreading the game or by wasting a correct read, a distinction invisible to outcomes and transcripts.

10645#pg3>. The system works after every public utterance by having every agent privately answer structured probe questions whose answers never re-enter the game and are scored against the ground truth the engine already knows <ref:2607. p. 0
Where the paper says it

, 2025); questioning the agent in-game changes its subsequent behaviour. We present MafiaScope, a reusable workflow for belief probing and counterfactual replay in interactive LLM agents, demonstrated on the social deduction game Mafia:1 after every public utterance, every agent privately answers structured questions whose answers never re-enter the game and are scored against the ground truth the engine already knows; second-order beliefs (“Casey suspects me”) are scored against the target’s own same-step reports.

10645#pg4>.

How the training signal is built

The core of MafiaScope consists of three decoupled layers: a game engine, a probe engine, and a viewer connected by JSONL logs <ref:2607.10645#pg4>. The game engine simulates a classic Mafia loop for n players where agents are parameterized by backend and language <ref:2607. p. 2

Where the paper says it

A classic Mafia loop (night kill / doctor save → day discussion → public vote) for n players with roles Mafia, Doctor, Villager. Players are LLM agents parameterized by backend and language (EN/RU prompt packs).

10645#pg4>. The probe engine pauses the game after every public event to privately interrogate each alive agent using structured probes defined in a YAML DSL <ref:2607. p. 2
Where the paper says it

4 The Probe Engine Asking an agent what it believes normally changes what it does next. The probe engine removes this cost: after each public utterance and private night action, and once per round, it pauses the game and privately interrogates each alive agent.

10645#pg4>. These probes include role beliefs, role assessment, suspicion ranking, planned action, and social map <ref:2607. p. 3
Where the paper says it

Answers are parsed as JSON, tolerant of markdown fences and truncation; both raw and parsed forms are logged. The default probe set includes role_beliefs and role_assessment (first-order role guesses with confidence), suspicion_ranking, planned_action (post-turn intent), and social_map (second-order: each player’s attitude toward me); full templates: Appendix A; the case-study corpus runs five probes.

10645#pg4>.

Where the gains come from

The system's utility is demonstrated through several key features that go beyond simple outcome evaluation <ref:2607.10645#pg4>. These contributions include a non-invasive belief probing workflow, a replayable inspection environment, and a reusable benchmark for studying belief dynamics <ref:2607. p. 1

Where the paper says it

Contributions. (1) A non-invasive beliefprobing workflow: a probe engine, configured in a small YAML DSL, that turns any run of the game into a fully indexed log of every agent’s evolving beliefs; (2) a replayable inspection environment: an interactive viewer that makes beliefs inspectable per utterance, per game and across games, with calibration, deception and replay views; (3) a reusable benchmark for studying belief dynamics: a released cross-model corpus of probed games and counterfactual forks, plus a case study showing what outcome-only evaluation misses (confidence detached from accuracy, over-predicted suspicion, votes locked in by a wrong picture of the world).

10645#pg4>. The viewer allows agents to inspect beliefs per utterance, per game and across games, with calibration, deception and replay views <ref:2607. p. 1
Where the paper says it

Contributions. (1) A non-invasive beliefprobing workflow: a probe engine, configured in a small YAML DSL, that turns any run of the game into a fully indexed log of every agent’s evolving beliefs; (2) a replayable inspection environment: an interactive viewer that makes beliefs inspectable per utterance, per game and across games, with calibration, deception and replay views; (3) a reusable benchmark for studying belief dynamics: a released cross-model corpus of probed games and counterfactual forks, plus a case study showing what outcome-only evaluation misses (confidence detached from accuracy, over-predicted suspicion, votes locked in by a wrong picture of the world).

10645#pg4>. The counterfactual replay experiment finds that single votes rarely flip the outcome but concentrates where the agent had read the game right <ref:2607. p. 0
Where the paper says it

5 times, and an injected-utterance replay experiment finds that single votes rarely flip the outcome: flips concentrate where the agent had read the game right, while votes made under a wrong picture of the world barely vary when resampled and change nothing. Engine, viewer, the recorded games and counterfactual forks are released under an open licence; code: https:// github.

10645#pg4>.

The case study across two model families with tens of thousands of parsed probe answers shows that stated confidence is poorly calibrated under this elicitation, with agents over-predicting being suspected 1.5 times <ref:2607. p. 0

Where the paper says it

Mafia concentrates the challenge: a hidden informed minority, the Mafia, covertly kills one 1 https://en. org/wiki/Mafia_(party_ game) 1 arXiv:2607.

10645#pg4>. Furthermore, an injected-utterance replay experiment reveals that single votes rarely flip the outcome, concentrating where the agent had read the game right <ref:2607. p. 0
Where the paper says it

In a case study across two model families with tens of thousands of parsed probe answers, stated confidence is poorly calibrated under this elicitation, agents over-predict being suspected 1. 5 times, and an injected-utterance replay experiment finds that single votes rarely flip the outcome: flips concentrate where the agent had read the game right, while votes made under a wrong picture of the world barely vary when resampled and change nothing.

10645#pg4>. The findings grade beliefs, not wins <ref:2607. p. 5
Where the paper says it

A second deepseek-chat batch of 30 games with a different probe wording and token budget and a 100-game gpt-4o-mini corpus serve as replication corpora. Mafia won 31 of 32 here versus 17 of 30 in the 30-game batch; the findings grade beliefs, not wins.

10645#pg4>. The belief trajectories are measurable, showing Villager-side agents starting with no opinion and converging on the Mafia as evidence accumulates <ref:2607. p. 5
Where the paper says it

F1: Belief trajectories are measurable. Villager-side agents start with no opinion (74.

10645#pg4>.

The calibration view shows that accuracy stays flat around 45% below confidence 80 (F2) across the corpus, indicating that stated confidence is not a probability until recalibrated <ref:2607.10645#pg4>. The over-prediction ratio for innocent agents against Mafia is stable across thresholds, showing an over-prediction of 1. p. 5

Where the paper says it

F3: Agents over-predict being suspected. Each social-map prediction (“B suspects me”) is graded against B’s own same-step assessment of the predictor: the gap between the panels of Figure 3.

53 times <ref:2607. p. 0
Where the paper says it

Mafia concentrates the challenge: a hidden informed minority, the Mafia, covertly kills one 1 https://en. org/wiki/Mafia_(party_ game) 1 arXiv:2607.

10645#pg4>. The vote-coupling details show that the top-1 vote lands inside the voter’s committed Mafia set in 71. p. 5
Where the paper says it

F4: Probed beliefs track votes. An innocent agent’s day vote lands on the top suspect of its latest pre-vote suspicion ranking in 64.

2% for innocent voters <ref:2607. p. 0
Where the paper says it

Mafia concentrates the challenge: a hidden informed minority, the Mafia, covertly kills one 1 https://en. org/wiki/Mafia_(party_ game) 1 arXiv:2607.

10645#pg4>.

The engine, probes, viewer and dataset are MITlicensed, and the release holds 557 recorded games and 512 counterfactual forks <ref:2607.10645#pg4>. The performance shows that the probe’s gain over such rules separates from zero only on the two larger corpora <ref:2607. p. 5

Where the paper says it

Simple transcript rules do well: a vote for the most-accused player reaches 58. The probe’s gain over such rules separates from zero only on the two larger corpora (Appendix C).

10645#pg4>. The system is intended both as an experimental platform and as a practical inspection environment <ref:2607. p. 2
Where the paper says it

Showcase traces of all three (transcript-only) ship in the dataset and demo. MafiaScope is intended both as an experimental platform and as a practical inspection environment: a probe battery attaches as YAML configuration without touching agent prompts.

10645#pg4>. The engine, viewer, the recorded games and counterfactual forks are released under an open licence <ref:2607. p. 0
Where the paper says it

5 times, and an injected-utterance replay experiment finds that single votes rarely flip the outcome: flips concentrate where the agent had read the game right, while votes made under a wrong picture of the world barely vary when resampled and change nothing. Engine, viewer, the recorded games and counterfactual forks are released under an open licence; code: https:// github.

10645#pg4>.

The testbed studies deception in a fictional, consented game frame among artificial agents with no human subjects involved <ref:2607.10645#pg4>. Research into how LLMs deceive and detect deception is dual-use, releasing measurement tooling rather than optimized deception policies <ref:2607. p. 6

Where the paper says it

Research into how LLMs deceive and detect deception is dual-use: the same insights that protect systems against manipulation could also be used to manipulate. We release measurement tooling (probes, scoring, visualization) rather than optimized deception policies, and the dataset contains only synthetic game dialogue.

10645#pg4>. The testbed measures beliefs inside single games with out-ofband probing and replay, unlike concurrent MindGames live arena <ref:2607. p. 1
Where the paper says it

The concurrent MindGames live arena (Wang et al. , 2026) ranks models on social and strategic reasoning over largescale multi-agent trajectories; MafiaScope instead measures beliefs inside single games with out-ofband probing and replay.

10645#pg4>. The engine server is shipped as a container image for the fork API server <ref:2607. p. 4
Where the paper says it

Forks start from the viewer’s Branch button or the command line; branches arrive as ordinary games, render as a branch tree, and can be forked again. Forking needs the engine server (shipped as a container image); the hosted static demo is read-only.

10645#pg4>. The data pipeline and visualizer join game and probe logs into per-step viewer states for the single-page viewer <ref:2607. p. 2
Where the paper says it

prepare_viewer. py joins game and probe logs into per-step viewer states for the single-page viewer (§5).

10645#pg4>. The architecture shows that all recorded data lives in these files <ref:2607. p. 2
Where the paper says it

py Figure 2: Architecture: engine and probe engine write append-only JSONL; the viewer is built from the logs and sends fork requests back. All recorded data lives in these files.

10645#pg4>.

The system's performance and scalability are demonstrated by the fact that the viewer is static, states, metrics and calibration are precomputed at build time <ref:2607.10645#pg4>. Rendering many agents is cheap while reading them is not, as the belief graph gets cluttered up to a dozen agents <ref:2607. p. 4

Where the paper says it

Rendering many agents is cheap; reading them is not. Past about a dozen agents the belief graph gets cluttered.

10645#pg4>. The probe missingness is transport-driven, with 98. p. 9
Where the paper says it

8% “Unknown”), but recall plateaus at 21-29% across rounds, never above the alive-Mafia chance share, and committed accuracy declines in late rounds: the no-opinion start is general, the convergence is model-specific. Probe missingness is transport-driven: on the pinned corpus 13,815 of 20,199 probe calls returned parsed answers; 98.

3% of missing answers being API transport errors that never delivered a model answer <ref:2607. p. 9
Where the paper says it

Probe missingness is transport-driven: on the pinned corpus 13,815 of 20,199 probe calls returned parsed answers; 98. 3% of the missing answers are API transport errors that never delivered a model answer, 99.

10645#pg4>. The architecture shows that the viewer is built from the logs and sends fork requests back <ref:2607. p. 2
Where the paper says it

py JSONL logs: game / introspection / state viewer (D3 SPA) + fork API server pause / ask fork / replay prepare_viewer. py Figure 2: Architecture: engine and probe engine write append-only JSONL; the viewer is built from the logs and sends fork requests back.

10645#pg4>. The system's capabilities are contrasted with other systems, showing it scores privately probed beliefs against engine ground truth <ref:2607.10645#pg4>. The architecture shows that the probe engine writes append-only JSONL <ref:2607. p. 2
Where the paper says it

py JSONL logs: game / introspection / state viewer (D3 SPA) + fork API server pause / ask fork / replay prepare_viewer. py Figure 2: Architecture: engine and probe engine write append-only JSONL; the viewer is built from the logs and sends fork requests back.

10645#pg4>.

The case study shows that the 30-game batch replicates the shape of belief trajectories (81.8% “Unknown” in round 0) <ref:2607. p. 5

Where the paper says it

Villager-side agents start with no opinion (74. 9% “Unknown” in round 0) and converge on the Mafia as evidence accumulates: recall of the true Mafia rises from 5.

10645#pg4>. The probe chaining re-injecting previous assessments can strengthen commitment regardless of evidence <ref:2607. p. 9
Where the paper says it

B First-Order Trajectories, Probe Loss and Dynamics Details Two artefacts inflate the trajectory in Table 2 independently of any real belief improvement. First, probe chaining re-injects the agent’s previous assessment into the next question; this anchoring can strengthen commitment regardless of evidence.

10645#pg4>. The flip-rate floor of 34. p. 5
Where the paper says it

The probe logs also read as a continuous-time graph of timestamped, confidence-weighted belief edges: about 1,500 edge events per game, exported in temporal-graph-network format (Rossi et al. The top-suspect flip rate is 48.

8% comes from 40 frozen-context points probed 5 times each, reweighted to the corpus’s pair composition <ref:2607. p. 4
Where the paper says it

Bifurcation experiment. For each selected vote we resample the agent’s utterance 500 times from the frozen pre-action context at temperature 1.

10645#pg4>. The probe budgets shape the measurement, with truncation occurring under lower token caps <ref:2607. p. 5
Where the paper says it

F5: Probe budgets shape the measurement. Under the 30-game batch’s 400-token cap, 73% of role-assessment answers truncated mid-JSON (repair recovered 4,118; Appendix B).

10645#pg4>. The belief dynamics are read as a continuous-time graph of timestamped, confidence-weighted belief edges <ref:2607. p. 5
Where the paper says it

Belief dynamics as a temporal graph. The probe logs also read as a continuous-time graph of timestamped, confidence-weighted belief edges: about 1,500 edge events per game, exported in temporal-graph-network format (Rossi et al.

10645#pg4>. The system's evaluation measures are computed on the fly in the viewer, allowing researchers to scrub timelines and inspect agent divergence <ref:2607.10645#pg4>.

The counterfactual replay tests whether a belief mattered, with 320 forks applied to 16 innocent day votes <ref:2607.10645#pg4>. Three findings emerge: one utterance rarely changes the outcome, and flips concentrate where the assessment was right <ref:2607. p. 4

Where the paper says it

First, one utterance rarely changes the outcome: about 6% of forks flip the winner in either quadrant (all 16 parents were Mafia wins). Second, the flips differ in kind: at policy points they concentrate on the vote itself, P(Villagers win) 15.

10645#pg4>. The bifurcation experiment shows that at policy points, flips concentrate on the vote itself, while at perception points such a vote does not help <ref:2607. p. 4
Where the paper says it

First, one utterance rarely changes the outcome: about 6% of forks flip the winner in either quadrant (all 16 parents were Mafia wins). Second, the flips differ in kind: at policy points they concentrate on the vote itself, P(Villagers win) 15.

10645#pg4>. The loss was decided before the utterance at perception points, showing they are behaviourally locked in <ref:2607. p. 4
Where the paper says it

0025, pooled over forks); at perception points such a vote does not help (5. Third, perception points are behaviourally locked in: at all 8 the factual vote matches the modal resample target, which captures on average 87% of the 500 draws.

10645#pg4>. The system's results grade beliefs, not wins <ref:2607.10645#pg4>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve agent evaluation by distinguishing between misreading a game and wasting a correct read by using MafiaScope to score beliefs against ground truth, which can tell whether an agent lost by misreading the game or by wasting a correct read, a distinction invisible to outcomes and transcripts.

  2. Enhance model calibration for stated confidence scores by using the calibration view in the viewer, which bins stated confidence against actual accuracy per agent or corpus-wide and identifies that agents over-predict being suspected 1.5 times.

  3. Develop counterfactual replay capabilities where an agent's subsequent behavior is tested by injecting alternative utterances, allowing researchers to see how votes change: flips concentrate where the agent had read the game right, while votes made under a wrong picture of the world barely vary when resampled and change nothing.

  4. Create a dynamic belief graph visualization that allows researchers to inspect ground-truth graph, one agent’s subjective graph, and the filtered game log over a shared timeline to track how Villager-side agents start with no opinion (74.9% “Unknown” in round 0) and converge on the Mafia as evidence accumulates.

  5. Implement an introspection mode that allows an agent to view its own beliefs against others' reports, enabling detection of deception: "Impersonate mode re-renders the entire interface from inside one agent, filtering the log to what the agent can observe and the graph to its own beliefs, and contrasting two ego-panels: what it believes others think of it (“believed”) against their actual same-step reports (“actual”)."

  6. Introduce a workflow for measuring belief dynamics by analyzing belief edges as a continuous-time graph of timestamped, confidence-weighted belief edges, allowing researchers to study suspicion volatility (mean L1 shift between consecutive suspicion vectors on common support, death-forced flips excluded from the flip rate).

Abstract

An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and its public claims may differ from privately elicited assessments. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for multi-agent reasoning. After each public utterance, living agents privately answer configurable structured questions on copies of their available conversations. Probing is non-invasive with respect to the acting context: neither the questions nor the answers enter the histories used to continue the game. Role assessments are scored against roles known to the engine; second-order predictions are compared with other agents' reports. An interactive visualizer renders the assessment trajectories: impersonate mode focuses on one agent's information and reports, panels chart timeline-aligned accuracy and calibration, and counterfactual replay branches from saved states. An offline heuristic links shifts in collective suspicion to episodes for inspection. Experiments span multiple models, languages, and probing configurations; the demonstration presents episode inspection, calibration, probe-repeatability and probe/no-probe checks, and utterance-replacement replay. Code: https://github.com/karpovilia/mafiascope Demo: https://karpovilia.github.io/mafiascope Video: https://vimeo.com/1208920221

Sources

Related papers