DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control
summary
The gist
DialInstruct-TTS proposes an interactive multi-turn expressive TTS framework that enables users to iteratively refine generated speech through natural-language dialogue until the desired style is…
In short
DialInstruct-TTS is an interactive framework that lets users refine generated speech through natural dialogue until it matches their desired style. It uses a Qwen planner to understand revisions and a flow-matching TTS backbone to apply fine-grained control over emotion, rate, volume, and phonation on specific parts of the speech while preserving other sections.
Key concepts
- Dialogue Revision Mechanism
- This mechanism allows users to iteratively improve speech using natural language. The Qwen planner understands context like "the same part" and identifies actions such as adding or weakening a control, generating only the changes needed for the current turn.
- Temporal Acoustic Control Module
- This module applies precise acoustic changes to specific time segments of the speech. It uses frame-local cross-attention and attribute controllers to modify emotion, speaking rate, volume, and phonation within designated temporal regions while keeping the overall style consistent.
- Non-Target Preservation Strategy
- To keep previously satisfactory parts of the speech intact during revisions, this strategy clamps the non-target latent trajectory at every step. This ensures that only the requested target regions are updated according to new instructions, preserving other sections of the audio.
- Delta Plan
- The planner generates a delta plan containing only the changes required for the current user instruction. This enables context-dependent and relative instructions, meaning users do not need to restate the entire control specification in every turn.
Terminology used across episodes
This episode discusses
- DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control · Paper Radio
- CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
- Qwen3-TTS Technical Report
- UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech
- Movie Gen: A Cast of Media Foundation Models
- JVS corpus: free Japanese multi-speaker voice corpus
- Qwen3 Technical Report
- Beyond One-Size-Fits-All: Personalized and Culturally Adaptive Emotional TTS via Interactive Optimization of Individual Emotion Perception Spaces
The paper
DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control · Read on arXiv
Wangzixi Zhou, Bagus Tris Atmaja Sakriani Sakti
Nara Institute of Science and Technology
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control".
Dev: Here's the gist of the paper. DialInstruct-TTS proposes an interactive multi-turn expressive TTS framework that enables users to iteratively refine generated speech through natural-language dialogue until the desired style is achieved The proposed system allows users to progressively refine generated speech through natural-language dialogue System Architecture The framework integrates a Qwen-based Dialogue Revision Planner and a Temporal Acoustic Control Module The planner receives the dialogue history, previous control state, and latest instruction into a resolved global caption, a delta plan, and an updated complete TemporalPlan The acoustic module realizes the TemporalPlan over a flow-matching TTS backbone that provides span-level control over emotion, speaking rate, volume, and phonation This module employs frame-local cross-attention and attribute-specific controllers to apply changes to designated temporal regions while maintaining the global speaking style Dialogue Revision Mechanism The Qwen3-based Dialogue Revision Planner resolves contextual expressions such as “the same part” and “the previously modified region” using the previous planner output as explicit revision memory The planner identifies revision operations including adding, strengthening, weakening, retaining, replacing, and removing a control based on user feedback It generates a delta plan containing only the changes made in the current turn This mechanism enables context-dependent and relative instructions without requiring users to restate the complete control specification Fine-Grained Control Implementation The Temporal Acoustic Control Module utilizes Token Encoding and Temporal Localization to map target spans to the acoustic latent timeline using MFA alignment during training Frame-local control is applied through a Control Cross-Attention Adapter that binds TemporalPlan semantics to target frames The Attribute Controller consists of independent branches for emotion, rate, volume, and phonation modifications These controllers inject residuals into the frozen RF-DiT backbone to produce the final hidden-state update Non-Target Preservation Strategy To preserve satisfactory regions across dialogue turns, the framework introduces reference-latent revision which explicitly clamps the non-target latent trajectory at every sampling step The target and transition regions are updated according to the current instruction, while non-target regions retain their previous acoustic realization This formulation ensures that the accepted speech is preserved across iterative refinement turns Experimental Validation Subjective evaluations demonstrate improved global instruction following, accurate local style control, and effective non-target preservation compared to conventional one-shot interaction The multi-turn evaluation shows that incremental instructions substantially improve perceived convenience and interaction comfort over full restatement Furthermore, the proposed system achieves a macro-average preference of 92.1% for multi-turn revision versus 3.3% for full restatement The component ablations reveal that non-target locking significantly improves joint success by 45 percentage points for rate, 15 for volume, and 35 for rate removal This indicates that the proposed method effectively controls expressive speech while maintaining competitive naturalness and linguistic accuracy Limitations The current framework has been trained and evaluated only on Japanese speech, and its cross-lingual generalization remains unexplored Emotion control is limited to four basic categories: anger, fear, happy, and sad Future work should extend the framework to additional languages, compound emotions, and more fine-grained or continuous affective states Temporal local control may still reduce overall speech naturalness Emotion revision can occasionally introduce speaker drift or abrupt local boundaries The supported attributes also differ in maturity, with volume and whisper-like phonation remaining less reliable Ethical Considerations The experiments involve general Japanese speech generation and expressive-style control and contain no sensitive, discriminatory, or harmful content The authors are responsible for the research design, experiments, analysis, and verification of manuscript content Expressive TTS technology may be misused for voice impersonation or misleading synthetic content The released demo is intended solely for academic evaluation This research study was reviewed and approved by the Institutional Review Board of
Institution Name Withheld for Double-Blind Review: Participants provided informed consent prior to participation, and no personal information unrelated to the study was collected The authors are responsible for the research design, experiments, analysis, and verification of manuscript content References Chihiro Arata.
Rosa: 2026a.
Dev: Irodori-tts: A flow matchingbased text-to-speech model with emoji-driven style control Chihiro Arata.
Rosa: 2026b.
Paper summary: Rosa: So wrapping up with DialInstruct-TTS, the authors are proposing this interactive multi-turn expressive TTS framework for progressive refinement through natural language dialogue until the desired style is achieved >
Dev: It’s essentially pairing a Qwen-based planner that understands context and dialogue history with a temporal acoustic control module that handles span level changes in real time >
Taro: What it means for us is that instead of fighting the system by rewriting long prompts, we can have a back-and-forth conversation with the AI to sculpt the voice exactly how we want >
Rosa: The title DialInstruct-TTS points directly to this iterative refinement process where you guide the speech creation through dialogue and control specific acoustic attributes >
Dev: It shows that combining a powerful language model for planning with fine-grained, temporally localized acoustic adjustments can lead to much better interactive control than previous one-shot methods >
Conclusion: Rosa: So, DialInstruct-TTS is this new system where you can actually talk to an AI to change your speech style turn by turn >
Dev: Yeah, they’ve built a framework that lets you iteratively refine generated speech using natural language dialogue until you get the exact vibe right >
Taro: It handles things like asking for "the same part" or "weakened" without you having to rewrite the whole prompt every single time >
Rosa: It’s about making this interaction fluid, so instead of a one-shot request that fails, you can just keep guiding the AI through a conversation >
Dev: The authors say this moves beyond just changing big settings like emotion or speed; they let you pinpoint exact moments in the speech timeline to tweak attributes >
Taro: That’s where the control module comes in, using things like frame local cross attention to target specific time segments for changes >
Rosa: And what I find interesting is how they handle keeping the parts you *don't* change—they have a strategy called reference latent revision that locks the non-target regions >
Dev: That’s smart because it prevents you from messing up a section of speech that already sounded good, which is something I worry about in real-time loops >
Taro: It addresses the failure mode where one small change accidentally ruins the rest of the sentence, which is a big problem when you're testing autonomy systems >
Rosa: So, for people who just listen to this show and want to know what this means, it’s about moving TTS from a static generator to something truly collaborative >
Dev: It’s about making expressive speech generation less about hitting the right button once, and more like directing an actor through a scene >
Taro: We'll be looking at how robust this iterative control is when you throw unexpected things at it in the next part >
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G