TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving
summary
The gist
Full-Duplex Speech Language Models (FD-SLMs) enable realtime spoken interaction by modeling turn-taking behaviors such as interruptions and overlapping speech <ref:2508.07375#pg2> End-to-end (e2e)…
In short
Full-Duplex Speech Language Models (FD-SLMs) struggle with natural conversation flow due to long speech sequences and limited data. TurnGuide introduces a method that dynamically segments assistant speech into turns and interleaves text tokens with speech tokens for each turn. This approach significantly improves the semantic coherence of FD-SLMs, achieving state-of-the-art performance on complex turn-taking events.
Key concepts
- Full-Duplex Speech Language Models (FD-SLMs)
- These are specialized AI models designed for real-time spoken interaction where both speakers can talk simultaneously. They model complex conversational behaviors like interruptions and overlapping speech by processing audio from two channels at once, aiming to create human-like dialogue.
- TurnGuide
- This is a novel text-speech interleaved generation approach for FD-SLMs. It works by first dynamically segmenting the assistant's speech into distinct turns and then interleaving text tokens with corresponding speech tokens for each turn. This guides the model to generate semantically meaningful and coherent dialogue.
- Text-Speech Interleaving
- This strategy involves mixing text tokens and speech tokens during generation, specifically by applying text guidance on the assistant channel. The framework uses a 5:5 ratio to ensure that text generation precedes speech generation, effectively using the text information to improve the natural acoustic flow.
- Dynamic Turn Segmentation
- This part of the method identifies individual turns within an audio stream. It achieves this by applying Voice Activity Detection (VAD) to find speech segments and then merging these into Inter-Pausal Units (IPUs) if they meet a specific pause threshold, allowing the model to know exactly when one speaker's turn ends and another begins.
Terminology used across episodes
This episode discusses
- TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving · Paper Radio
- Recent Advances in Speech Language Models: A Survey
- GPT-4o System Card
- MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
- Moshi: a speech-text foundation model for real-time dialogue
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
- Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
- GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
- RoboEgo System Card: An Omnimodal Model with Native Full Duplexity
- GPT-4 Technical Report
- VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models
- VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
- SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model
- OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
- Aligning Spoken Dialogue Models from User Interactions
- Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
- Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS
- Reinforcement Learning from Human Feedback · Paper Radio
The paper
TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving · Read on arXiv
Chinese University of Hong Kong · Huawei Technologies
Full-Duplex Speech Language Models (FD-SLMs) are specialized foundation models designed to enable natural, real-time spoken interactions by modeling complex conversational turn-taking such as interruptions, backchannels, and overlapping speech. End-to-end (e2e) FD-SLMs leverage real-world double-channel conversational data to capture nuanced two-speaker dialogue patterns for human-like interactions, but their conversational abilities often degrade compared to pure-text conversation due to prolonged speech sequences and limited high-quality spoken dialogue data. Although interleaved text-speech generation could mitigate this degradation, integrating discrete text tokens into continuous double-channel audio streams could disrupt the precise time alignment required for fluid interaction. To address this, we propose TurnGuide, a novel text-speech interleaved generation approach for e2e FD-SLMs that dynamically segments assistant speech into dialogue turns and interleaves turn-level text and speech generation. This approach allows FD-SLMs to integrate the semantic intelligence of LLMs without compromising the natural acoustic flow. Extensive experiments show that TurnGuide not only significantly improves e2e FD-SLMs to produce semantically meaningful, coherent speech but also achieves state-of-the-art performance on various turn-taking events. Demos are available at https://dreamtheater123.github.io/TurnGuide-Demo/. Code is available at https://github.com/dreamtheater123/TurnGuide.
DOI: 10.21437/Interspeech.2026-1141
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving".
Jane: Here's the gist of the paper. Full-Duplex Speech Language Models (FD-SLMs) enable realtime spoken interaction by modeling turn-taking behaviors such as interruptions and overlapping speech <ref:2508.07375#pg2> End-to-end (e2e) FDSLMs learn from double-channel conversational audio to capture natural dialogue dynamics, but their conversational quality often degrades compared to text-only models due to long speech sequences and limited high-quality spoken data <ref:2508.07375#pg2> We propose TurnGuide, a turn-level text–speech interleaved framework that dynamically segments assistant speech and jointly generates text and speech <ref:2508.07375#pg2> Experiments show that TurnGuide significantly improves semantic coherence and achieves state-of-the-art performance on diverse turn-taking events <ref:2508.07375#pg2>.
Tom: Pruning redundant reasoning tokens Speech Language Models (SLMs) have attracted attention as foundation models for speech-based interactions with human users <ref:2508.07375#pg2> Full-Duplex Speech Language Models (FDSLMs), a specialized type of SLM, model user- and assistantchannel audio data simultaneously to support real-time communication with various natural turn-taking events such as interruptions
3: , backchannels
4: , and overlapping speech
5: <ref:2508.07375#pg2>.
Jane: Existing FD-SLMs can be categorized as cascaded and end-to-end (e2e) systems <ref:2508.07375#pg2>.
Tom: Cascaded FD-SLMs utilize a separate prediction module to classify spoken dialogues into predefined states, guiding SLM generation
3, 7: <ref:2508.07375#pg2>.
Paper summary: Tom: We’re talking about the paper "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving." The authors are Wenqian Cui, Lei Zhu, Xiao-Hui Li, Zhihan Guo, Haoli Bai, Lu Hou, and Irwin King. They start by saying that full duplex speech language models can model real-time interactions using double-channel audio data to capture things like interruptions and overlapping speech.
Jane: But they point out that end-to-end versions of these FDSLMs often struggle because they degrade compared to pure text conversation when the speeches get long or the spoken data isn't super high quality. They mention that while interleaving text and speech can help, putting discrete text tokens into continuous dual-channel audio messes up the timing for fluid interaction.
Lu: So TurnGuide proposes a new framework, which is a turn-level text-speech interleaved approach. The main claim is that by dynamically segmenting the assistant's speech into turns and then interleaving text and speech generation at the turn level, they can integrate semantic intelligence from large language models while keeping the natural acoustic flow intact.
Meng: What I’m focusing on is that this framework has two key components: a dynamic turn segmentation and alignment part, which uses Voice Activity Detection to find speech segments, and a modeling part that handles text-guided full duplex dialogue. It sounds like they are trying to solve the alignment problem first before they even start generating anything.
Lalam: I think the segmentation process is crucial because it breaks down the audio stream into units that make sense for conversation flow, and then they use those segments to apply channel-wise interleaving and text-speech interleaving strategies.
Tom: Exactly. They use VAD to identify speech segments, then merge them into Inter-Pausal Units if they have pauses within a threshold of zero point five seconds <ref:2508.07375#pg3>. That’s the first part of their dynamic segmentation framework.
Jane: And once those segments are defined, the second part involves modeling text-guided full duplex dialogue. They use channel-wise interleaving to alternate speech chunks between the two channels and then text-speech interleaving to apply text guidance on the assistant channel by mixing tokens for each turn.
Lu: I think what's compelling is how they structure this joint generation. They have a five to five ratio for text chunks versus speech chunks, which ensures that the text generation happens before the speech generation, which is key for effective guidance <ref:2508.07375#pg3>.
Meng: From a deployment view, if we can get this dynamic segmentation to work reliably on real traffic data, it solves the problem of unpredictable input length and timing that breaks current models when scaling up. That seems like a big win for robustness.
Lalam: The core idea is using text guidance on the assistant channel by interleaving text tokens with speech tokens for each turn, which allows the semantic intelligence from LLMs to be integrated without breaking the natural acoustic flow of the audio stream.
Jane: So it’s about making sure that every time there's a turn, there's a precise alignment between what’s being spoken and what text is being generated, which sounds like exactly what we need for smooth interactions.
Conclusion: Tom: So, to wrap up, the paper "TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving" is about using dynamic turn segmentation to make end-to-end FDSLMs much better at producing coherent speech. The authors are Cui, Zhu, Li, Guo, Bai, Hou, and King.
Jane: It boils down to this: they take the problem of long sequences in spoken dialogue and solve it by dynamically segmenting the speech into turns and then using turn-level text-speech interleaving to guide the generation process.
Lu: The implication is that we can finally use those powerful LLMs for spoken interaction more effectively, even in messy, real-time scenarios without completely sacrificing the natural sound of a conversation.
Meng: For deployment, it means moving away from systems that rely on simple state predictors toward models that can handle the complexity of actual turn dynamics more organically. It’s about building something that actually runs reliably when you put it under load.
Lalam: The paper shows significant improvements in semantic quality and various natural turn-taking events, which validates the entire framework they built for spoken dialogue systems.
Tom: So essentially, TurnGuide offers a novel way to improve the dialogue generation abilities of these models by structuring how text and speech are interleaved across channels dynamically. It achieves performance comparable to its upper bounds when trained with this proposed framework.
Jane: It’s about taking a complex problem in spoken dialogue and providing a concrete method for better, more coherent, and more natural interactions in the real world.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI