Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions

The gist The authors introduce MIDA, a new task and dataset to benchmark Multimodal Large Language Models (MLLMs) in assessing deception in complex multi-party social interactions.

Introduction and Motivation

Deception profoundly impacts interpersonal relationships, economic activities, and public safety, making it crucial for building intelligent and secure AI systems like advanced conversational agents [8]. Existing research suffers from limitations such as a lack of interactional context, simplification of social complexity in multi-party networks, and a scarcity of verifiable ground truth [31]. To address these issues, the authors introduce the social deduction game Werewolf as a controlled environment that provides objective, deterministic ground truth for every statement [20]. They formalize this into the Multimodal Interactive Deception Assessment (MIDA) task and construct a novel multimodal dataset totaling 2,360 dialogues [18].

The MIDA Benchmark and Dataset

The MIDA dataset is built upon two corpora: MIDA-Ego4D, consisting of 40 game sessions recorded from a third-person perspective, and MIDA-YouTube, curated from 151 game videos collected from YouTube [3.1]. The annotation pipeline involves manually annotating crucial “night actions” to mitigate uncertainty and using an LLM assistant to parse events against the game rules and metadata [3.1]. Annotation principles dictate that deception is judged based on the speaker’s private knowledge state at the moment of utterance [3.1]. A human validation study achieved a high human-LLM agreement accuracy of 96.6% on a subset of MIDA-Ego4d data [3.1].

Failure Modes in Current MLLMs

The benchmark reveals significant performance gaps, indicating that state-of-the-art MLLMs lack core human intelligence, specifically the ability to ‘read the room’ and assess deception [Abstract]. The analysis categorizes two core failure modes: 1) Difficulty in distinguishing salient social signals from distracting noise and 2) A lack of functional “Theory of Mind” [19] — the ability to internally model what others know, believe, or intend [Abstract]. Current MLLMs function as powerful knowledge engines but fall short as competent social agents [Abstract].

Proposed Reasoning Modules

To improve performance on this challenging task, the authors propose two novel modules:

1. Social Chain-of-Thought (SoCoT): This pipeline enforces a structured reasoning process in three steps: Step 1 is Low-level Perception, where MLLMs analyze multimodal inputs to extract behavioral primitives (facial expressions, body gestures, and vocal cues) [4.1]. Step 2 involves High-level Social Reasoning, simulating Theory-of-Mind reasoning to predict goals and motives [4.1]. Step 3 is Decision and Rationale Generation, which produces a final judgment with an evidence-grounded justification [4.1].

2. Dynamic Social Epistemic Memory (DSEM): This module maintains a structured, time-evolving representation of each participant’s beliefs and knowledge through a JSON-formatted Memory Board Mt p [4.2]. The update process is modeled as a state transition function fDSEM: Mt+1 p = fDSEM(Mt p, Et+1, Ot+1) [4.2].

Experimental Results and Conclusion

The experimental results confirm that SoCoT and DSEM modules improve MLLM performance on the MIDA benchmark [Abstract]. For closed-source models, the DSEM module delivered the strongest improvements where it matters most for deception detection, achieving the best Binary accuracy (41.7; +2.3 over baseline) and the best Macro-F1 (47.8; +3.3) [5.5]. The analysis shows that while MLLMs can handle neutral or descriptive utterances, they struggle with the underlying epistemic reasoning that governs truth and belief in social interactions [6]. Ultimately, addressing these challenges requires developing context-adaptive alignment strategies less risk-averse in adversarial settings; new architectures with integrated Theory of Mind reasoning; and more robust methods for grounding language in nonverbal visual cues [6]. The findings underscore the urgent need for building more perceptive and trustworthy AI systems [Abstract].

--- Page 1 ---

Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Ruicong Liu, Yoichi Sato 1 The University of Tokyo. Abstract Despite their advanced reasoning capabilities, stateof-the-art Multimodal Large Language Models (MLLMs) demonstrably lack a core component of human intelligence: the ability to ‘read the room’ and assess deception in complex social interactions. To rigorously quantify this failure, we introduce a new task, Multimodal Interactive Deception Assessment (MIDA), and present a novel multimodal dataset providing synchronized video and text with verifiable ground-truth labels for every statement. We establish a comprehensive benchmark evaluating 12 state-of-the-art open- and closed-source MLLMs, revealing a significant performance gap: even powerful models like GPT-4o struggle to distinguish truth from falsehood reliably. Our analysis of failure modes indicates that these models fail to effectively ground language in multimodal social cues and lack the ability to model what others know, believe, or intend, highlighting the urgent need for novel approaches to building more perceptive and trustworthy AI systems. To take a step forward, we design a Social Chain-of-Thought (SoCoT) reasoning pipeline and a Dynamic Social Epistemic Memory (DSEM) module. Our framework yields performance improvement on this challenging task, demonstrating a promising new path toward building MLLMs capable of genuine human-like social reasoning. 1. Introduction Deception, a pervasive social phenomenon [8], profoundly impacts interpersonal relationships, economic activities, and even public safety. Accurately identifying deceptive intent in discourse is crucial for understanding the complexities of human interactions. It also presents a core challenge for building more intelligent [6, 7] and secure AI systems [17, 45, 52], such as advanced conversational agents and content moderation platforms [36]. To foster a future where AI and humans can collaborate seamlessly [16, 39], equipping AI with sophisticated social perception and reasoning capabilities is indispensable. While deception detection has garnered research attention, existing work largely suffers from three major limitations. (i) Lack of interactional context. Prior studies often operate in isolation, analyzing single text snippets [31], unidirectional speech videos [34, 41], or independent physiological signals [21, 42]. However, real-world deception is not a static act but a dynamic, interactive process. (ii) Simplification of social complexity. While prior work has made progress in two-person settings [38], their interaction patterns are relatively structured (one person describes, another guesses). This fails to capture the complexity of authentic social deception, which frequently unfolds in messy, multi-party networks of shifting alliances, rivalries, and group pressure common to human social dynamics [9]. (iii) Scarcity of verifiable ground truth. A pivotal obstacle is the lack of publicly available datasets with verifiable ground truth. In real-world scenarios, objectively annotating the precise moments of deception is often impossible, severely hampering the training and evaluation of predictive models [34]. To address these limitations, we introduce the social deduction game Werewolf, where players must conceal their secret roles, using deceptive communication to mislead others. By day, everyone votes to eliminate a player, while at night, Werewolves and other special roles secretly perform their night actions to achieve their hidden goals. This complex multi-party game serves as a controlled yet ecologically valid environment that elicits natural, high-stakes deception. Crucially, its rule-based nature provides objective, deterministic ground truth for every statement [20]. Building on this paradigm, we formalize a new task Multimodal Interactive Deception Assessment (MIDA). We construct a novel multimodal benchmark dataset totaling 2,360 dialogues. Our semi-automated pipeline involves manually annotating crucial “night actions” to mitigate the uncertainty, then using an LLM assistant to parse events, and rigorously verifying initial veracity labels against the game’s ground truth, ensuring high-fidelity, verifiable annotations [3.1]. We conduct a comprehensive benchmark of 12 state-of-the-art open- and closed-source MLLMs on MIDA, reveal arXiv:2511.16221v1 [cs.CV] 20 Nov 2025

--- Page 2 ---

Kevin is accurately reporting his role and night action of checking Daniel's cards. Let’s start the Werewolf Game. Daniel what were you? You are lying. I was the Seer, I’ve checked your card and I started out as the Seer and I looked at these two cards Human Annotation TRUE FALSE NEUTRAL A complete fabrication. Daniel knows he started as the Werewolf and is now the Minion. He is lying NEUTRAL This is a question. There is no further implication Task1 Strategy Classification (C.1) Social Chain-of-Thought (SoCoT) (B) Evaluated MLLMs (C.2) Dynamic Social Epistemic Memory (DSEM) Game Metadata Transcript Game Rules Closed-source Open-source Assigned Role Role Actions Known facts Connections Kaelan Kevin Jessica Daniel Face Body Voice Dominant emotion Emotion scores Facial cues Description • Identity Declaration • Accusation • Interrogation • Call for Action • Defense • Evidence • TRUE FALSE NEUTRAL Task2 Deception Assessment Additional Justification NEUTRAL FALSE TRUE Figure 1.

1. The MIDA task requires models to perform reasoning from Persuasive Strategy Classification to Deception Assessment with a supporting justification. This improves AI systems by shifting them from simple pattern matching to complex, multi-step inference, forcing them to generate a short, evidence-grounded justification for their final judgment.

2. Integrate the Social Chain-of-Thought (SoCoT) pipeline into the MLLM architecture by requiring explicit grounding in multimodal cues during Step 1: Multimodal inputs (Body, Face, and Voice) are prepared by using Game Metadata to crop visual bounding boxes (via MTCNN [51] for Face) and utilizing parselmouth [15]/librosa [26] to extract key acoustic features. This prevents models from failing to distinguish salient social signals from distracting noise.

3. Implement the Dynamic Social Epistemic Memory (DSEM) module, which maintains a structured, time-evolving representation of each participant’s beliefs and knowledge, modeled by the state transition function: Mt+1 p = fDSEM(Mt p, Et+1, Ot+1). This allows AI systems to move beyond static context to model what others know, believe, or intend, directly addressing the identified failure mode of lacking a functional “Theory of Mind.