MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games

summary

The gist

An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing…

In short

MafiaScope is a system to measure machine Theory of Mind in social deduction games like Mafia by privately probing agents after every utterance. It distinguishes between losing due to misreading the game versus wasting a correct read. The system reveals that stated confidence is poorly calibrated and belief trajectories show how agents converge on the Mafia role.

Key concepts

MafiaScope
A non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind in social deduction games. It works by having agents privately answer structured questions after every public statement, allowing researchers to score beliefs against known ground truth.
Probe Engine
The component that pauses the game after each public event to privately interrogate alive agents using structured probes. These probes ask specific questions about an agent's role beliefs, suspicion rankings, and planned actions without those answers re-entering the game.
Belief Trajectories
The measurable path of an agent's opinions over time. For Villager-side agents, this trajectory shows they start with no opinion and gradually converge on identifying the Mafia role as more evidence accumulates in the game.

Terminology used across episodes

This episode discusses

The paper

MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games · Read on arXiv

Ilia Karpov

HSE University

An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and its public claims may differ from privately elicited assessments. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for multi-agent reasoning. After each public utterance, living agents privately answer configurable structured questions on copies of their available conversations. Probing is non-invasive with respect to the acting context: neither the questions nor the answers enter the histories used to continue the game. Role assessments are scored against roles known to the engine; second-order predictions are compared with other agents' reports. An interactive visualizer renders the assessment trajectories: impersonate mode focuses on one agent's information and reports, panels chart timeline-aligned accuracy and calibration, and counterfactual replay branches from saved states. An offline heuristic links shifts in collective suspicion to episodes for inspection. Experiments span multiple models, languages, and probing configurations; the demonstration presents episode inspection, calibration, probe-repeatability and probe/no-probe checks, and utterance-replacement replay. Code: https://github.com/karpovilia/mafiascope Demo: https://karpovilia.github.io/mafiascope Video: https://vimeo.com/1208920221

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games".

Jane: Here's the gist of the paper. An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes <ref:2607.10645#pg2> Pruning redundant reasoning tokens MafiaScope is presented as a non-invasive, time-resolved belief probing system designed to measure machine Theory of Mind within social deduction games like Mafia <ref:2607.10645#pg4>.

Tom: It allows researchers to distinguish between an agent losing by misreading the game or by wasting a correct read, a distinction that is invisible through outcomes and transcripts <ref:2607.10645#pg3>.

Jane: The system works after every public utterance by having every agent privately answer structured probe questions whose answers never re-enter the game and are scored against the ground truth the engine already knows <ref:2607.10645#pg4>.

Tom: How the training signal is built The core of MafiaScope consists of three decoupled layers: a game engine, a probe engine, and a viewer connected by JSONL logs <ref:2607.10645#pg4>.

Title and authors: Tom: Moving on to the title and authors, we have "MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games," which sounds pretty descriptive of what they’re trying to do.

Jane: It’s a mouthful, but the core idea is that this isn't just about playing Mafia; it's about using that game as a way to measure how agents are reasoning socially > <ref:2607.10645#pg1>.

Lu: The authors are working on this by building a whole infrastructure—a game engine, a probe engine, and a viewer connected by JSONL logs > <ref:2607.10645#pg4>. They’re essentially creating an open testbed for studying these complex social deduction games with LLM agents.

Meng: So they've built the mechanics to turn the game into data, which is what I usually focus on when we think about deployment > <ref:2607.10645#pg4>. The infrastructure itself, the way they wrote those logs in append-only JSONL, is important for how you ingest and analyze this information later.

Tom: Right. It’s about building a structured way to capture every single step of the game so we can probe it later without disrupting the flow of the actual play > <ref:2607.10645#pg4>. It’s about turning an event into a data point that can be analyzed in depth > <ref:2607.10645#pg1>.

Jane: And what this means for us is that we have a standardized way to run these kinds of experiments, which is super valuable when we are trying to compare different agent architectures or model families > <ref:2607.10645#pg4>. It gives us a common language for studying theory of mind in AI.

Lu: I think the related work section points out that existing methods like DeepRole already maintained posterior beliefs over role assignments inside a purpose-built planner, but MafiaScope probes and scores those beliefs differently > <ref:2607.10645#pg2>. They’re not just tracking what the agent believes; they are actively testing that belief against something else.

Meng: So they're adding a layer of active questioning to what was already there, which is useful because it moves it from passive observation to active interrogation > <ref:2607.10645#pg2>. It’s about forcing the model to articulate its reasoning rather than just letting it run its course.

Tom: And that leads right into the practical improvements they propose, which are where the real utility for us comes in—how we actually use this tool beyond just reading the paper > <ref:2607.10645#pg4>.

The paper's summary: Jane: So, to recap what we've covered, MafiaScope is essentially a non-invasive belief probing workflow designed to measure machine theory of mind within social deduction games > <ref:2607.10645#pg1>.

Lu: That means they’ve built a probe engine that runs after every public event and privately asks agents structured questions—things like role beliefs and planned actions > <ref:2607.10645#pg4>.

Meng: And these answers are scored against the ground truth that the engine already knows, so it lets you distinguish between losing by misreading or wasting a correct read > <ref:2607.10645#pg1>.

Tom: That distinction is invisible through just looking at outcomes or transcripts, which is a huge gap they’re trying to fill > <ref:2607.10645#pg1>. They can tell if the agent made a mistake in reading the situation or if it just got stuck in an unproductive loop > <ref:2607.10645#pg1>.

Jane: And then they have this replayable inspection environment where you can look at beliefs per utterance, per game, and even across games with views for calibration and deception > <ref:2607.10645#pg4>. It lets you see the agent's internal state in real time as it plays > <ref:2607.10645#pg4>.

Lu: And a reusable benchmark is also a key contribution here: a cross-model corpus of probed games plus case studies showing what outcome-only evaluation misses, like confidence detached from accuracy > <ref:2607.10645#pg4>.

Meng: It sounds like they’ve formalized the gap between what we measure and what we actually care about, which is important because outcome-only evaluation is often misleading > <ref:2607.10645#pg4>. It’s about measuring the belief dynamics themselves > <ref:2607.10645#pg4>.

Tom: And that means if you’re trying to build a model that can reason socially, you need tools like this to check if it's reasoning correctly or just mimicking a correct final output > <ref:2607.10645#pg1>.

Jane: So, in short, they are giving us the infrastructure to inspect the internal reasoning of these agents during social deduction games without having to change how they fundamentally operate > <ref:2607.10645#pg4>.

The paper's improvements: Lu: The authors suggest a few key improvements focused on adding better analysis layers to what’s already there, like better ways to study belief dynamics > <ref:2607.10645#pg4>. They want to go beyond just showing the game and the basic probes > <ref:2607.10645#pg4>.

Meng: I think one of those is developing a dynamic belief graph visualization that lets researchers inspect the ground-truth graph, one agent’s subjective graph, and the filtered game log over a shared timeline > #six <ref:2607.10645#pg2>. That would be really powerful for tracking how Villager-side agents start with no opinion, which was seventy-four point nine percent "Unknown" in round zero > <ref:2607.10645#pg4>.

Tom: That visualization sounds like a way to visualize that continuous-time graph of timestamped, confidence-weighted belief edges > <ref:2607.10645#pg4>. It lets you scrub timelines and inspect agent divergence in a very intuitive way > #six <ref:2607.10645#pg2>.

Jane: And they also talk about probe chaining re-injecting previous assessments can strengthen commitment regardless of new evidence > <ref:2607.10645#pg4>. That’s a mechanism we can use to test if initial beliefs have lasting influence on subsequent decisions.

Lu: That suggests that even if the agent gets new data, its initial internal state still plays a role in how it processes that data > <ref:2607.10645#pg4>. It's about testing the persistence of belief > <ref:2607.10645#pg4>.

Meng: And they also mention a flip-rate floor of thirty-four point eight percent which comes from probing five times each point, reweighted to the corpus composition, which shapes the measurement > <ref:2607.10645#pg4>. That’s a concrete metric that shows how robust this measurement is across different setups > <ref:2607.10645#pg4>.

Tom: So they are focusing on making sure these metrics aren't just arbitrary numbers but actually reflect real dynamics in the system, which is what separates a good experiment from noise > #six <ref:2607.10645#pg2>. It’s about measuring the actual system behavior, not just some abstract concept of reasoning quality > #six <ref:2607.10645#pg2>.

Jane: And I also want to talk about how they grade beliefs, not wins, which is a big shift from just looking at the final outcome as they are making those points > #four. That means we can finally look at the nuance of their internal reasoning > #four.

Lu: So when you combine all these ideas—the dynamic graph visualization and the belief edge analysis and that’s what makes this paper interesting for researchers in representation learning > #six <ref:2607.10645#pg2>. It’s about seeing how agents build their understanding over time > #four.

Meng: And I just want to make sure we're talking about things that actually matter for deployment, like what breaks at scale or how much computation this probing adds to the system > #four.

Conclusion: Tom: Alright, wrapping up this look at the MafiaScope paper. The main implication is that we have tools to inspect agents' beliefs in social deduction games without changing their core operation > <ref:2607.10645#pg1>.

Jane: We’re gaining a new way to grade reasoning by looking at these belief trajectories, which are continuous graphs of timestamped, confidence-weighted belief edges > #four.

Lu: So we can track how those Villager-side agents start with no opinion and converge on the Mafia as more evidence accumulates > #four. It gives researchers a dynamic view of machine reasoning in these scenarios > #six <ref:2607.10645#pg2>.

Meng: For production systems, I just want to make sure we're not adding too much overhead when trying to run this level of probing against live traffic or complex workloads > #four.

Lalam: I’m pretty sure that with the MafiaScope method, we can finally see how agents build their understanding over time on the game board > #four. It's a huge step forward in understanding machine theory of mind > #four.

Tom: We have this non-invasive belief probing workflow that lets us score beliefs against ground truth, which tells us whether an agent lost by misreading or wasting a correct read > #one <ref:2607.10645#pg1,whether an agent lost by misreading>. That’s the core mechanism we need to be excited about for AI research > #one <ref:2607.10645#pg1>.

Jane: It shifts the focus from just checking the final result to understanding the actual process behind that result > #four. That’s a big shift in how we evaluate agents on social tasks, and it opens up a lot of avenues for future work > #four.

More episodes

← Home