Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs
cs.CV, cs.SD, eess.AS
Submitted: 2026-09-25
Updated: 2026-09-25
Comments: Accepted by NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues.
Terminology
Abstract
Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across four conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method.
Sources
- Qwen2.5-VL Technical Report
- Seeing to Generalize: How Visual Data Corrects Binding Shortcuts
- DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
- MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
- Discovering Variable Binding Circuitry with Desiderata
- Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding
- Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models
- ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting
- Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
- Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
- See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
- MUSAN: A Music, Speech, and Noise Corpus
- video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
- D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
- Visual Prompting in Multimodal Large Language Models: A Survey
- SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models