SciPhi-Chat: Training a Spatial Audio Agent for Question Answering, Dialogue, and Tool Use
cs.SD
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/microsoft/sciphi-chat
License: http://creativecommons.org/licenses/by/4.0/
The gist: Spatial audio question answering requires associating sound identities with locations and reasoning about their relations.
Abstract
Spatial audio question answering requires associating sound identities with locations and reasoning about their relations. This paper introduces SciPhi-Chat, a spatial audio agent that answers questions, conducts multi-turn dialogue, and calls tools on first-order ambisonic recordings. The accompanying MASU benchmark contains 12,000 samples across 42 task categories, the widest task coverage of any spatial model, with real recordings and simulated audio scenes. SciPhi-Chat achieves a score of 44.9%, 7.4 percentage points above the previous state-of-the-art. On the external ASA-Bench, SciPhi-Chat achieves 74.5% azimuth and 60.0% elevation accuracy, the highest among locally evaluated models. Removing reasoning supervision reduces ASA-Bench accuracy by 17.4 percentage points. All materials will be released via https://github.com/microsoft/sciphi-chat
Sources
- Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding
- Audio Dialogues: Dialogues dataset for audio and music understanding
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
- OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models
- SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models
- Step-Audio 2 Technical Report
- Qwen3-Omni Technical Report
- Spatial Audio Question Answering and Reasoning on Dynamic Source Movements
- Towards Spatial Audio Understanding via Question Answering
- Spatial Audio Motion Understanding and Reasoning
- Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
- MOSS-Audio Technical Report
- Qwen2.5-Omni Technical Report
- Qwen3 Technical Report
- FOA Tokenizer: Low-bitrate Neural Codec for First Order Ambisonics with Spatial Consistency Loss
- STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events
- Qwen3.5-Omni Technical Report
- BEATs: Audio Pre-Training with Acoustic Tokenizers
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems