MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games".
Jane: Here's the gist of the paper. An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes <ref:2607.10645#pg2> Pruning redundant reasoning tokens MafiaScope is presented as a non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind within social deduction games like Mafia <ref:2607.10645#pg4>.
Tom: It allows researchers to distinguish between an agent losing by misreading the game or by wasting a correct read, a distinction that is invisible through outcomes and transcripts <ref:2607.10645#pg3>.
Jane: The system works after every public utterance by having every agent privately answer structured probe questions whose answers never re-enter the game and are scored against the ground truth the engine already knows <ref:2607.10645#pg4>.
Tom: How the training signal is built The core of MafiaScope consists of three decoupled layers: a game engine, a probe engine, and a viewer connected by JSONL logs <ref:2607.10645#pg4>.
Title and authors: Tom: Moving on to the title and authors, we have "MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games," which sounds pretty descriptive of what they’re trying to do.
Jane: It’s a mouthful, but the core idea is that this isn't just about playing Mafia; it's about using that game as a way to measure how agents are reasoning socially > <ref:2607.10645#pg1>.
Lu: The authors are working on this by building a whole infrastructure—a game engine, a probe engine, and a viewer connected by JSONL logs > <ref:2607.10645#pg4>. They’re essentially creating an open testbed for studying these complex social deduction games with LLM agents.
Meng: So they've built the mechanics to turn the game into data, which is what I usually focus on when we think about deployment > <ref:2607.10645#pg4>. The infrastructure itself, the way they wrote those logs in append-only JSONL, is important for how you ingest and analyze this information later.
Tom: Right. It’s about building a structured way to capture every single step of the game so we can probe it later without disrupting the flow of the actual play > <ref:2607.10645#pg4>. It’s about turning an event into a data point that can be analyzed in depth > <ref:2607.10645#pg1>.
Jane: And what this means for us is that we have a standardized way to run these kinds of experiments, which is super valuable when we are trying to compare different agent architectures or model families > <ref:2607.10645#pg4>. It gives us a common language for studying theory of mind in AI.
Lu: I think the related work section points out that existing methods like DeepRole already maintained posterior beliefs over role assignments inside a purpose-built planner, but MafiaScope probes and scores those beliefs differently > <ref:2607.10645#pg2>. They’re not just tracking what the agent believes; they are actively testing that belief against something else.
Meng: So they're adding a layer of active questioning to what was already there, which is useful because it moves it from passive observation to active interrogation > <ref:2607.10645#pg2>. It’s about forcing the model to articulate its reasoning rather than just letting it run its course.
Tom: And that leads right into the practical improvements they propose, which are where the real utility for us comes in—how we actually use this tool beyond just reading the paper > <ref:2607.10645#pg4>.
The paper's summary: Jane: So, to recap what we've covered, MafiaScope is essentially a non-invasive belief probing workflow designed to measure machine theory of mind within social deduction games > <ref:2607.10645#pg1>.
Lu: That means they’ve built a probe engine that runs after every public event and privately asks agents structured questions—things like role beliefs and planned actions > <ref:2607.10645#pg4>.
Meng: And these answers are scored against the ground truth that the engine already knows, so it lets you distinguish between losing by misreading or wasting a correct read > <ref:2607.10645#pg1>.
Tom: That distinction is invisible through just looking at outcomes or transcripts, which is a huge gap they’re trying to fill > <ref:2607.10645#pg1>. They can tell if the agent made a mistake in reading the situation or if it just got stuck in an unproductive loop > <ref:2607.10645#pg1>.
Jane: And then they have this replayable inspection environment where you can look at beliefs per utterance, per game, and even across games with views for calibration and deception > <ref:2607.10645#pg4>. It lets you see the agent's internal state in real time as it plays > <ref:2607.10645#pg4>.
Lu: And a reusable benchmark is also a key contribution here: a cross-model corpus of probed games plus case studies showing what outcome-only evaluation misses, like confidence detached from accuracy > <ref:2607.10645#pg4>.
Meng: It sounds like they’ve formalized the gap between what we measure and what we actually care about, which is important because outcome-only evaluation is often misleading > <ref:2607.10645#pg4>. It’s about measuring the belief dynamics themselves > <ref:2607.10645#pg4>.
Tom: And that means if you’re trying to build a model that can reason socially, you need tools like this to check if it's reasoning correctly or just mimicking a correct final output > <ref:2607.10645#pg1>.
Jane: So, in short, they are giving us the infrastructure to inspect the internal reasoning of these agents during social deduction games without having to change how they fundamentally operate > <ref:2607.10645#pg4>.
The paper's improvements: Lu: The authors suggest a few key improvements focused on adding better analysis layers to what’s already there, like better ways to study belief dynamics > <ref:2607.10645#pg4>. They want to go beyond just showing the game and the basic probes > <ref:2607.10645#pg4>.
Meng: I think one of those is developing a dynamic belief graph visualization that lets researchers inspect the ground-truth graph, one agent’s subjective graph, and the filtered game log over a shared timeline > #six <ref:2607.10645#pg2>. That would be really powerful for tracking how Villager-side agents start with no opinion, which was seventy-four point nine percent "Unknown" in round zero > <ref:2607.10645#pg4>.
Tom: That visualization sounds like a way to visualize that continuous-time graph of timestamped, confidence-weighted belief edges > <ref:2607.10645#pg4>. It lets you scrub timelines and inspect agent divergence in a very intuitive way > #six <ref:2607.10645#pg2>.
Jane: And they also talk about probe chaining re-injecting previous assessments can strengthen commitment regardless of new evidence > <ref:2607.10645#pg4>. That’s a mechanism we can use to test if initial beliefs have lasting influence on subsequent decisions.
Lu: That suggests that even if the agent gets new data, its initial internal state still plays a role in how it processes that data > <ref:2607.10645#pg4>. It's about testing the persistence of belief > <ref:2607.10645#pg4>.
Meng: And they also mention a flip-rate floor of thirty-four point eight percent which comes from probing five times each point, reweighted to the corpus composition, which shapes the measurement > <ref:2607.10645#pg4>. That’s a concrete metric that shows how robust this measurement is across different setups > <ref:2607.10645#pg4>.
Tom: So they are focusing on making sure these metrics aren't just arbitrary numbers but actually reflect real dynamics in the system, which is what separates a good experiment from noise > #six <ref:2607.10645#pg2>. It’s about measuring the actual system behavior, not just some abstract concept of reasoning quality > #six <ref:2607.10645#pg2>.
Jane: And I also want to talk about how they grade beliefs, not wins, which is a big shift from just looking at the final outcome as they are making those points > #four. That means we can finally look at the nuance of their internal reasoning > #four.
Lu: So when you combine all these ideas—the dynamic graph visualization and the belief edge analysis and that’s what makes this paper interesting for researchers in representation learning > #six <ref:2607.10645#pg2>. It’s about seeing how agents build their understanding over time > #four.
Meng: And I just want to make sure we're talking about things that actually matter for deployment, like what breaks at scale or how much computation this probing adds to the system > #four.
Conclusion: Tom: Alright, wrapping up this look at the MafiaScope paper. The main implication is that we have tools to inspect agents' beliefs in social deduction games without changing their core operation > <ref:2607.10645#pg1>.
Jane: We’re gaining a new way to grade reasoning by looking at these belief trajectories, which are continuous graphs of timestamped, confidence-weighted belief edges > #four.
Lu: So we can track how those Villager-side agents start with no opinion and converge on the Mafia as more evidence accumulates > #four. It gives researchers a dynamic view of machine reasoning in these scenarios > #six <ref:2607.10645#pg2>.
Meng: For production systems, I just want to make sure we're not adding too much overhead when trying to run this level of probing against live traffic or complex workloads > #four.
Lalam: I’m pretty sure that with the MafiaScope method, we can finally see how agents build their understanding over time on the game board > #four. It's a huge step forward in understanding machine theory of mind > #four.
Tom: We have this non-invasive belief probing workflow that lets us score beliefs against ground truth, which tells us whether an agent lost by misreading or wasting a correct read > #one <ref:2607.10645#pg1,whether an agent lost by misreading>. That’s the core mechanism we need to be excited about for AI research > #one <ref:2607.10645#pg1>.
Jane: It shifts the focus from just checking the final result to understanding the actual process behind that result > #four. That’s a big shift in how we evaluate agents on social tasks, and it opens up a lot of avenues for future work > #four.
Ilia Karpov
HSE University
cs.CL, cs.MA, cs.SI
Submitted: 2026-07-12
Updated: 2026-10-30
Code: https://github.com/karpovilia/mafiascope
Project page: https://karpovilia.github.io/mafiascope
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing…
Key concepts
- MafiaScope
- A non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind in social deduction games. It works by having agents privately answer structured questions after every public statement, allowing researchers to score beliefs against known ground truth.
- Probe Engine
- The component that pauses the game after each public event to privately interrogate alive agents using structured probes. These probes ask specific questions about an agent's role beliefs, suspicion rankings, and planned actions without those answers re-entering the game.
- Belief Trajectories
- The measurable path of an agent's opinions over time. For Villager-side agents, this trajectory shows they start with no opinion and gradually converge on identifying the Mafia role as more evidence accumulates in the game.
Terminology
Summary
An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes <ref:2607. p. 0 MafiaScope: Non-Invasive, Time-Resolved Belief Probing
for LLM Agents in Social Deduction Games
Ilia Karpov
HSE University
karpovilia@gmail. com
Abstract
An LLM agent’s public behaviour reveals little about its social reasoning: an agent that
votes correctly may be guessing, and an agent
that lies well leaves no trace of what it actually believes.Where the paper says it
Pruning redundant reasoning tokens
MafiaScope is presented as a non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind within social deduction games like Mafia <ref:2607.10645#pg4>. It allows researchers to distinguish between an agent losing by misreading the game or by wasting a correct read, a distinction that is invisible through outcomes and transcripts <ref:2607. p. 0 com
Abstract
An LLM agent’s public behaviour reveals little about its social reasoning: an agent that
votes correctly may be guessing, and an agent
that lies well leaves no trace of what it actually believes. We present MafiaScope, an open
testbed that turns the social deduction game
Mafia into a measurement instrument for machine Theory of Mind: it can tell whether an
agent lost by misreading the game or by wasting a correct read, a distinction invisible to outcomes and transcripts. , 2025); questioning the agent
in-game changes its subsequent behaviour. We present MafiaScope, a reusable workflow for
belief probing and counterfactual replay in interactive LLM agents, demonstrated on the social deduction game Mafia:1
after every public utterance,
every agent privately answers structured questions
whose answers never re-enter the game and are
scored against the ground truth the engine already
knows; second-order beliefs (“Casey suspects me”)
are scored against the target’s own same-step reports.Where the paper says it
Where the paper says it
How the training signal is built
The core of MafiaScope consists of three decoupled layers: a game engine, a probe engine, and a viewer connected by JSONL logs <ref:2607.10645#pg4>. The game engine simulates a classic Mafia loop for n players where agents are parameterized by backend and language <ref:2607. p. 2 A classic Mafia loop
(night kill / doctor save → day discussion → public vote) for n players with roles Mafia, Doctor,
Villager. Players are LLM agents parameterized
by backend and language (EN/RU prompt packs). 4 The Probe Engine
Asking an agent what it believes normally changes
what it does next. The probe engine removes this
cost: after each public utterance and private night
action, and once per round, it pauses the game and
privately interrogates each alive agent. Answers are parsed as JSON, tolerant of markdown
fences and truncation; both raw and parsed forms
are logged. The default probe set includes role_beliefs
and role_assessment (first-order role guesses
with confidence), suspicion_ranking,
planned_action (post-turn intent), and
social_map (second-order: each player’s attitude toward me); full templates: Appendix A; the
case-study corpus runs five probes.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The system's utility is demonstrated through several key features that go beyond simple outcome evaluation <ref:2607.10645#pg4>. These contributions include a non-invasive belief probing workflow, a replayable inspection environment, and a reusable benchmark for studying belief dynamics <ref:2607. p. 1 Contributions. (1) A non-invasive beliefprobing workflow: a probe engine, configured in a
small YAML DSL, that turns any run of the game
into a fully indexed log of every agent’s evolving beliefs; (2) a replayable inspection environment: an interactive viewer that makes beliefs inspectable per utterance, per game and across games,
with calibration, deception and replay views; (3) a
reusable benchmark for studying belief dynamics:
a released cross-model corpus of probed games
and counterfactual forks, plus a case study showing
what outcome-only evaluation misses (confidence
detached from accuracy, over-predicted suspicion,
votes locked in by a wrong picture of the world). Contributions. (1) A non-invasive beliefprobing workflow: a probe engine, configured in a
small YAML DSL, that turns any run of the game
into a fully indexed log of every agent’s evolving beliefs; (2) a replayable inspection environment: an interactive viewer that makes beliefs inspectable per utterance, per game and across games,
with calibration, deception and replay views; (3) a
reusable benchmark for studying belief dynamics:
a released cross-model corpus of probed games
and counterfactual forks, plus a case study showing
what outcome-only evaluation misses (confidence
detached from accuracy, over-predicted suspicion,
votes locked in by a wrong picture of the world). 5 times,
and an injected-utterance replay experiment
finds that single votes rarely flip the outcome:
flips concentrate where the agent had read the
game right, while votes made under a wrong
picture of the world barely vary when resampled and change nothing. Engine, viewer, the
recorded games and counterfactual forks are released under an open licence; code: https://
github.Where the paper says it
Where the paper says it
Where the paper says it
The case study across two model families with tens of thousands of parsed probe answers shows that stated confidence is poorly calibrated under this elicitation, with agents over-predicting being suspected 1.5 times <ref:2607. p. 0 Mafia concentrates the challenge: a hidden
informed minority, the Mafia, covertly kills one
1
https://en. org/wiki/Mafia_(party_
game)
1
arXiv:2607. In a case
study across two model families with tens of
thousands of parsed probe answers, stated confidence is poorly calibrated under this elicitation,
agents over-predict being suspected 1. 5 times,
and an injected-utterance replay experiment
finds that single votes rarely flip the outcome:
flips concentrate where the agent had read the
game right, while votes made under a wrong
picture of the world barely vary when resampled and change nothing. A second deepseek-chat batch of 30 games with
a different probe wording and token budget and a
100-game gpt-4o-mini corpus serve as replication
corpora. Mafia won 31 of 32 here versus 17 of 30
in the 30-game batch; the findings grade beliefs,
not wins. F1: Belief trajectories are measurable. Villager-side agents start with no opinion (74.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The calibration view shows that accuracy stays flat around 45% below confidence 80 (F2) across the corpus, indicating that stated confidence is not a probability until recalibrated <ref:2607.10645#pg4>. The over-prediction ratio for innocent agents against Mafia is stable across thresholds, showing an over-prediction of 1. p. 5 F3: Agents over-predict being suspected. Each social-map prediction (“B suspects me”) is
graded against B’s own same-step assessment of
the predictor: the gap between the panels of Figure 3. Mafia concentrates the challenge: a hidden
informed minority, the Mafia, covertly kills one
1
https://en. org/wiki/Mafia_(party_
game)
1
arXiv:2607. F4: Probed beliefs track votes. An innocent
agent’s day vote lands on the top suspect of its latest pre-vote suspicion ranking in 64. Mafia concentrates the challenge: a hidden
informed minority, the Mafia, covertly kills one
1
https://en. org/wiki/Mafia_(party_
game)
1
arXiv:2607.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The engine, probes, viewer and dataset are MITlicensed, and the release holds 557 recorded games and 512 counterfactual forks <ref:2607.10645#pg4>. The performance shows that the probe’s gain over such rules separates from zero only on the two larger corpora <ref:2607. p. 5 Simple transcript rules do well:
a vote for the most-accused player reaches 58. The probe’s gain over such rules separates from
zero only on the two larger corpora (Appendix C). Showcase traces of all
three (transcript-only) ship in the dataset and demo. MafiaScope is intended both as an experimental platform and as a practical inspection environment: a probe battery attaches as YAML configuration without touching agent prompts. 5 times,
and an injected-utterance replay experiment
finds that single votes rarely flip the outcome:
flips concentrate where the agent had read the
game right, while votes made under a wrong
picture of the world barely vary when resampled and change nothing. Engine, viewer, the
recorded games and counterfactual forks are released under an open licence; code: https://
github.Where the paper says it
Where the paper says it
Where the paper says it
The testbed studies deception in a fictional, consented game frame among artificial agents with no human subjects involved <ref:2607.10645#pg4>. Research into how LLMs deceive and detect deception is dual-use, releasing measurement tooling rather than optimized deception policies <ref:2607. p. 6 Research into how LLMs deceive and
detect deception is dual-use: the same insights that
protect systems against manipulation could also
be used to manipulate. We release measurement
tooling (probes, scoring, visualization) rather than
optimized deception policies, and the dataset contains only synthetic game dialogue. The concurrent
MindGames live arena (Wang et al. , 2026) ranks
models on social and strategic reasoning over largescale multi-agent trajectories; MafiaScope instead
measures beliefs inside single games with out-ofband probing and replay. Forks start from
the viewer’s Branch button or the command line;
branches arrive as ordinary games, render as a
branch tree, and can be forked again. Forking needs
the engine server (shipped as a container image);
the hosted static demo is read-only. prepare_viewer. py joins game and probe
logs into per-step viewer states for the single-page
viewer (§5). py
Figure 2: Architecture: engine and probe engine write
append-only JSONL; the viewer is built from the logs
and sends fork requests back. All recorded data lives in
these files.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The system's performance and scalability are demonstrated by the fact that the viewer is static, states, metrics and calibration are precomputed at build time <ref:2607.10645#pg4>. Rendering many agents is cheap while reading them is not, as the belief graph gets cluttered up to a dozen agents <ref:2607. p. 4 Rendering many agents is cheap; reading
them is not. Past about a dozen agents the belief
graph gets cluttered. 8% “Unknown”),
but recall plateaus at 21-29% across rounds, never
above the alive-Mafia chance share, and committed
accuracy declines in late rounds: the no-opinion
start is general, the convergence is model-specific. Probe missingness is transport-driven: on the
pinned corpus 13,815 of 20,199 probe calls returned parsed answers; 98. Probe missingness is transport-driven: on the
pinned corpus 13,815 of 20,199 probe calls returned parsed answers; 98. 3% of the missing answers are API transport errors that never delivered
a model answer, 99. py
JSONL logs: game /
introspection / state
viewer (D3 SPA)
+ fork API server
pause / ask
fork / replay prepare_viewer. py
Figure 2: Architecture: engine and probe engine write
append-only JSONL; the viewer is built from the logs
and sends fork requests back. py
JSONL logs: game /
introspection / state
viewer (D3 SPA)
+ fork API server
pause / ask
fork / replay prepare_viewer. py
Figure 2: Architecture: engine and probe engine write
append-only JSONL; the viewer is built from the logs
and sends fork requests back.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The case study shows that the 30-game batch replicates the shape of belief trajectories (81.8% “Unknown” in round 0) <ref:2607. p. 5 Villager-side agents start with no opinion (74. 9%
“Unknown” in round 0) and converge on the Mafia
as evidence accumulates: recall of the true Mafia
rises from 5. B First-Order Trajectories, Probe Loss
and Dynamics Details
Two artefacts inflate the trajectory in Table 2 independently of any real belief improvement. First,
probe chaining re-injects the agent’s previous assessment into the next question; this anchoring
can strengthen commitment regardless of evidence. The
probe logs also read as a continuous-time graph
of timestamped, confidence-weighted belief edges:
about 1,500 edge events per game, exported in
temporal-graph-network format (Rossi et al. The top-suspect flip rate is 48. Bifurcation experiment. For each selected vote
we resample the agent’s utterance 500 times from
the frozen pre-action context at temperature 1. F5: Probe budgets shape the measurement. Under the 30-game batch’s 400-token cap, 73%
of role-assessment answers truncated mid-JSON
(repair recovered 4,118; Appendix B). Belief dynamics as a temporal graph. The
probe logs also read as a continuous-time graph
of timestamped, confidence-weighted belief edges:
about 1,500 edge events per game, exported in
temporal-graph-network format (Rossi et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The counterfactual replay tests whether a belief mattered, with 320 forks applied to 16 innocent day votes <ref:2607.10645#pg4>. Three findings emerge: one utterance rarely changes the outcome, and flips concentrate where the assessment was right <ref:2607. p. 4 First, one utterance rarely
changes the outcome: about 6% of forks flip the
winner in either quadrant (all 16 parents were
Mafia wins). Second, the flips differ in kind: at
policy points they concentrate on the vote itself,
P(Villagers win) 15. First, one utterance rarely
changes the outcome: about 6% of forks flip the
winner in either quadrant (all 16 parents were
Mafia wins). Second, the flips differ in kind: at
policy points they concentrate on the vote itself,
P(Villagers win) 15. 0025, pooled over
forks); at perception points such a vote does not
help (5. Third, perception points
are behaviourally locked in: at all 8 the factual
vote matches the modal resample target, which captures on average 87% of the 500 draws.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve agent evaluation by distinguishing between misreading a game and wasting a correct read by using MafiaScope to score beliefs against ground truth, which
can tell whether an agent lost by misreading the game or by wasting a correct read, a distinction invisible to outcomes and transcripts.
-
Enhance model calibration for stated confidence scores by using the calibration view in the viewer, which
bins stated confidence against actual accuracy per agent or corpus-wide
and identifies that agentsover-predict being suspected 1.5 times.
-
Develop counterfactual replay capabilities where an agent's subsequent behavior is tested by injecting alternative utterances, allowing researchers to see how votes change:
flips concentrate where the agent had read the game right, while votes made under a wrong picture of the world barely vary when resampled and change nothing.
-
Create a dynamic belief graph visualization that allows researchers to inspect
ground-truth graph, one agent’s subjective graph, and the filtered game log over a shared timeline
to track howVillager-side agents start with no opinion (74.9% “Unknown” in round 0) and converge on the Mafia as evidence accumulates.
-
Implement an introspection mode that allows an agent to view its own beliefs against others' reports, enabling detection of deception: "Impersonate mode re-renders the entire interface from inside one agent, filtering the log to what the agent can observe and the graph to its own beliefs, and contrasting two ego-panels: what it believes others think of it (“believed”) against their actual same-step reports (“actual”)."
-
Introduce a workflow for measuring belief dynamics by analyzing belief edges as a
continuous-time graph of timestamped, confidence-weighted belief edges,
allowing researchers to studysuspicion volatility (mean L1 shift between consecutive suspicion vectors on common support, death-forced flips excluded from the flip rate).
Abstract
An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and its public claims may differ from privately elicited assessments. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for multi-agent reasoning. After each public utterance, living agents privately answer configurable structured questions on copies of their available conversations. Probing is non-invasive with respect to the acting context: neither the questions nor the answers enter the histories used to continue the game. Role assessments are scored against roles known to the engine; second-order predictions are compared with other agents' reports. An interactive visualizer renders the assessment trajectories: impersonate mode focuses on one agent's information and reports, panels chart timeline-aligned accuracy and calibration, and counterfactual replay branches from saved states. An offline heuristic links shifts in collective suspicion to episodes for inspection. Experiments span multiple models, languages, and probing configurations; the demonstration presents episode inspection, calibration, probe-repeatability and probe/no-probe checks, and utterance-replacement replay. Code: https://github.com/karpovilia/mafiascope Demo: https://karpovilia.github.io/mafiascope Video: https://vimeo.com/1208920221
Sources
- WOLF: Werewolf-based Observations for LLM Deception and Falsehoods
- Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction
- Looking Inward: Language Models Can Learn About Themselves by Introspection
- AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game
- Deceive, Detect, and Disclose: Large Language Models Play Mini-Mafia
- Among Us: A Sandbox for Measuring and Detecting Agentic Deception
- Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4
- Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
- Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf
- Language Models (Mostly) Know What They Know
- AvalonBench: Evaluating LLMs Playing the Game of Avalon
- Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models
- Bayesian Social Deduction with Graph-Informed Language Models
- Position: Theory of Mind Benchmarks are Broken for Large Language Models
- Temporal Graph Networks for Deep Learning on Dynamic Graphs
- Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind
- Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
- MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
- Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data