Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs
cs.CL
Submitted: 2026-07-07
Updated: 2026-10-29
Comments: 22 pages, 9 figures
Code: https://github.com/HITsz-TMG/Lychee-FDhttps:
Project page: https://hitsz-tmg.github.io/Lychee-FD
License: http://creativecommons.org/licenses/by/4.0/
The gist: Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community.
Terminology
Abstract
Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity -- ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.
Sources
- FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
- FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations
- Fun-Audio-Chat Technical Report
- Moshi: a speech-text foundation model for real-time dialogue
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
- VITA: Towards Open-Source Interactive Omni Multimodal LLM
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
- LUCY: Linguistic Understanding and Control Yielding Early Stage of Her
- WavChat: A Survey of Spoken Dialogue Models
- Scaling Laws for Neural Language Models
- MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free
- Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems
- Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
- FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems
- Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
- A Preliminary Exploration with GPT-4o Voice Mode
- An Overview of Multi-Task Learning in Deep Neural Networks
- Step-Audio 2 Technical Report
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data