TORUS: A Unified Closed-Loop Benchmark for Unified Audio Models
cs.SD, cs.AI, cs.CL
Submitted: 2026-07-30
Updated: 2026-10-21
Project page: https://torus-benchmark.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Unified audio models capable of audio understanding, audio generation and, increasingly, audio editing are proliferating rapidly.
Terminology
Abstract
Unified audio models capable of audio understanding, audio generation and, increasingly, audio editing are proliferating rapidly. Yet a basic question about them remains unanswered: do the two heads of a unified model agree about the same audio? Current practice evaluates each capability in isolation on specialized benchmarks, and never asks whether a model can make sense of its own generations. We present TORUS, the first self-coherence test for audio-native unified models. TORUS comprises 48 three-stage self-coherence tests carrying 432 six-option questions spanning speech, sound and music across five task families. We holistically evaluate five open unified models alongside a Cascaded Baseline that combines state-of-the-art specialized generation, editing and understanding models. The best unified model answers 50.5% of questions against the Cascaded Baseline's 63.2% and a 16.7% chance floor. Models struggle on audio editing. Among the evaluated audio models (specialized and unified), we observe limited self-coherence, and thus position self-coherence as an essential test for future audio systems.
Sources
- All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
- Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
- ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation
- Ming-Omni: A Unified Multimodal Model for Perception and Generation
- Unified Audio Intelligence Without Regressing on Text Intelligence
- AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering
- UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation
- UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark
- MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models
- MMAE: A Massive Multitask Audio Editing Benchmark
- MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model
- Quantifying the Gap between Understanding and Generation within Unified Multimodal Models
- Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
- Step-Audio-EditX Technical Report
- UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization
- SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
- Harmonic Representations in Symbolic Music Transformers are Localized and Controllable
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving