DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control".
Dev: Here's the gist of the paper. DialInstruct-TTS proposes an interactive multi-turn expressive TTS framework that enables users to iteratively refine generated speech through natural-language dialogue until the desired style is achieved The proposed system allows users to progressively refine generated speech through natural-language dialogue System Architecture The framework integrates a Qwen-based Dialogue Revision Planner and a Temporal Acoustic Control Module The planner receives the dialogue history, previous control state, and latest instruction into a resolved global caption, a delta plan, and an updated complete TemporalPlan The acoustic module realizes the TemporalPlan over a flow-matching TTS backbone that provides span-level control over emotion, speaking rate, volume, and phonation This module employs frame-local cross-attention and attribute-specific controllers to apply changes to designated temporal regions while maintaining the global speaking style Dialogue Revision Mechanism The Qwen3-based Dialogue Revision Planner resolves contextual expressions such as “the same part” and “the previously modified region” using the previous planner output as explicit revision memory The planner identifies revision operations including adding, strengthening, weakening, retaining, replacing, and removing a control based on user feedback It generates a delta plan containing only the changes made in the current turn This mechanism enables context-dependent and relative instructions without requiring users to restate the complete control specification Fine-Grained Control Implementation The Temporal Acoustic Control Module utilizes Token Encoding and Temporal Localization to map target spans to the acoustic latent timeline using MFA alignment during training Frame-local control is applied through a Control Cross-Attention Adapter that binds TemporalPlan semantics to target frames The Attribute Controller consists of independent branches for emotion, rate, volume, and phonation modifications These controllers inject residuals into the frozen RF-DiT backbone to produce the final hidden-state update Non-Target Preservation Strategy To preserve satisfactory regions across dialogue turns, the framework introduces reference-latent revision which explicitly clamps the non-target latent trajectory at every sampling step The target and transition regions are updated according to the current instruction, while non-target regions retain their previous acoustic realization This formulation ensures that the accepted speech is preserved across iterative refinement turns Experimental Validation Subjective evaluations demonstrate improved global instruction following, accurate local style control, and effective non-target preservation compared to conventional one-shot interaction The multi-turn evaluation shows that incremental instructions substantially improve perceived convenience and interaction comfort over full restatement Furthermore, the proposed system achieves a macro-average preference of 92.1% for multi-turn revision versus 3.3% for full restatement The component ablations reveal that non-target locking significantly improves joint success by 45 percentage points for rate, 15 for volume, and 35 for rate removal This indicates that the proposed method effectively controls expressive speech while maintaining competitive naturalness and linguistic accuracy Limitations The current framework has been trained and evaluated only on Japanese speech, and its cross-lingual generalization remains unexplored Emotion control is limited to four basic categories: anger, fear, happy, and sad Future work should extend the framework to additional languages, compound emotions, and more fine-grained or continuous affective states Temporal local control may still reduce overall speech naturalness Emotion revision can occasionally introduce speaker drift or abrupt local boundaries The supported attributes also differ in maturity, with volume and whisper-like phonation remaining less reliable Ethical Considerations The experiments involve general Japanese speech generation and expressive-style control and contain no sensitive, discriminatory, or harmful content The authors are responsible for the research design, experiments, analysis, and verification of manuscript content Expressive TTS technology may be misused for voice impersonation or misleading synthetic content The released demo is intended solely for academic evaluation This research study was reviewed and approved by the Institutional Review Board of
Institution Name Withheld for Double-Blind Review: Participants provided informed consent prior to participation, and no personal information unrelated to the study was collected The authors are responsible for the research design, experiments, analysis, and verification of manuscript content References Chihiro Arata.
Rosa: 2026a.
Dev: Irodori-tts: A flow matchingbased text-to-speech model with emoji-driven style control Chihiro Arata.
Rosa: 2026b.
Paper summary: Rosa: So wrapping up with DialInstruct-TTS, the authors are proposing this interactive multi-turn expressive TTS framework for progressive refinement through natural language dialogue until the desired style is achieved >
Dev: It’s essentially pairing a Qwen-based planner that understands context and dialogue history with a temporal acoustic control module that handles span level changes in real time >
Taro: What it means for us is that instead of fighting the system by rewriting long prompts, we can have a back-and-forth conversation with the AI to sculpt the voice exactly how we want >
Rosa: The title DialInstruct-TTS points directly to this iterative refinement process where you guide the speech creation through dialogue and control specific acoustic attributes >
Dev: It shows that combining a powerful language model for planning with fine-grained, temporally localized acoustic adjustments can lead to much better interactive control than previous one-shot methods >
Conclusion: Rosa: So, DialInstruct-TTS is this new system where you can actually talk to an AI to change your speech style turn by turn >
Dev: Yeah, they’ve built a framework that lets you iteratively refine generated speech using natural language dialogue until you get the exact vibe right >
Taro: It handles things like asking for "the same part" or "weakened" without you having to rewrite the whole prompt every single time >
Rosa: It’s about making this interaction fluid, so instead of a one-shot request that fails, you can just keep guiding the AI through a conversation >
Dev: The authors say this moves beyond just changing big settings like emotion or speed; they let you pinpoint exact moments in the speech timeline to tweak attributes >
Taro: That’s where the control module comes in, using things like frame local cross attention to target specific time segments for changes >
Rosa: And what I find interesting is how they handle keeping the parts you *don't* change—they have a strategy called reference latent revision that locks the non-target regions >
Dev: That’s smart because it prevents you from messing up a section of speech that already sounded good, which is something I worry about in real-time loops >
Taro: It addresses the failure mode where one small change accidentally ruins the rest of the sentence, which is a big problem when you're testing autonomy systems >
Rosa: So, for people who just listen to this show and want to know what this means, it’s about moving TTS from a static generator to something truly collaborative >
Dev: It’s about making expressive speech generation less about hitting the right button once, and more like directing an actor through a scene >
Taro: We'll be looking at how robust this iterative control is when you throw unexpected things at it in the next part >
Wangzixi Zhou, Bagus Tris Atmaja Sakriani Sakti
Nara Institute of Science and Technology
eess.AS
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/FunAudioLLM/CosyVoice
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 92/100
The gist: DialInstruct-TTS proposes an interactive multi-turn expressive TTS framework that enables users to iteratively refine generated speech through natural-language dialogue until the desired style is…
Key concepts
- Dialogue Revision Mechanism
- This mechanism allows users to iteratively improve speech using natural language. The Qwen planner understands context like "the same part" and identifies actions such as adding or weakening a control, generating only the changes needed for the current turn.
- Temporal Acoustic Control Module
- This module applies precise acoustic changes to specific time segments of the speech. It uses frame-local cross-attention and attribute controllers to modify emotion, speaking rate, volume, and phonation within designated temporal regions while keeping the overall style consistent.
- Non-Target Preservation Strategy
- To keep previously satisfactory parts of the speech intact during revisions, this strategy clamps the non-target latent trajectory at every step. This ensures that only the requested target regions are updated according to new instructions, preserving other sections of the audio.
- Delta Plan
- The planner generates a delta plan containing only the changes required for the current user instruction. This enables context-dependent and relative instructions, meaning users do not need to restate the entire control specification in every turn.
Terminology
Summary
DialInstruct-TTS proposes an interactive multi-turn expressive TTS framework that enables users to iteratively refine generated speech through natural-language dialogue until the desired style is achieved The proposed system allows users to progressively refine generated speech through natural-language dialogue
System Architecture
The framework integrates a Qwen-based Dialogue Revision Planner and a Temporal Acoustic Control Module The planner receives the dialogue history, previous control state, and latest instruction into a resolved global caption, a delta plan, and an updated complete TemporalPlan The acoustic module realizes the TemporalPlan over a flow-matching TTS backbone that provides span-level control over emotion, speaking rate, volume, and phonation This module employs frame-local cross-attention and attribute-specific controllers to apply changes to designated temporal regions while maintaining the global speaking style
Dialogue Revision Mechanism
The Qwen3-based Dialogue Revision Planner resolves contextual expressions such as “the same part” and “the previously modified region” using the previous planner output as explicit revision memory The planner identifies revision operations including adding, strengthening, weakening, retaining, replacing, and removing a control based on user feedback It generates a delta plan containing only the changes made in the current turn This mechanism enables context-dependent and relative instructions without requiring users to restate the complete control specification
Fine-Grained Control Implementation
The Temporal Acoustic Control Module utilizes Token Encoding and Temporal Localization to map target spans to the acoustic latent timeline using MFA alignment during training Frame-local control is applied through a Control Cross-Attention Adapter that binds TemporalPlan semantics to target frames The Attribute Controller consists of independent branches for emotion, rate, volume, and phonation modifications These controllers inject residuals into the frozen RF-DiT backbone to produce the final hidden-state update
Non-Target Preservation Strategy
To preserve satisfactory regions across dialogue turns, the framework introduces reference-latent revision which explicitly clamps the non-target latent trajectory at every sampling step The target and transition regions are updated according to the current instruction, while non-target regions retain their previous acoustic realization This formulation ensures that the accepted speech is preserved across iterative refinement turns
Experimental Validation
Subjective evaluations demonstrate improved global instruction following, accurate local style control, and effective non-target preservation compared to conventional one-shot interaction The multi-turn evaluation shows that incremental instructions substantially improve perceived convenience and interaction comfort over full restatement Furthermore, the proposed system achieves a macro-average preference of 92.1% for multi-turn revision versus 3. p. 63% for full restatement The component ablations reveal that non-target locking significantly improves joint success by 45 percentage points for rate, 15 for volume, and 35 for rate removal This indicates that the proposed method effectively controls expressive speech while maintaining competitive naturalness and linguistic accuracy
Limitations
The current framework has been trained and evaluated only on Japanese speech, and its cross-lingual generalization remains unexplored Emotion control is limited to four basic categories: anger, fear, happy, and sad Future work should extend the framework to additional languages, compound emotions, and more fine-grained or continuous affective states Temporal local control may still reduce overall speech naturalness Emotion revision can occasionally introduce speaker drift or abrupt local boundaries The supported attributes also differ in maturity, with volume and whisper-like phonation remaining less reliable
Ethical Considerations
The experiments involve general Japanese speech generation and expressive-style control and contain no sensitive, discriminatory, or harmful content The authors are responsible for the research design, experiments, analysis, and verification of manuscript content Expressive TTS technology may be misused for voice impersonation or misleading synthetic content The released demo is intended solely for academic evaluation This research study was reviewed and approved by the Institutional Review Board of [Institution Name Withheld for Double-Blind Review] Participants provided informed consent prior to participation, and no personal information unrelated to the study was collected The authors are responsible for the research design, experiments, analysis, and verification of manuscript content
References
Chihiro Arata. 2026a. Irodori-tts: A flow matchingbased text-to-speech model with emoji-driven style control Chihiro Arata. p. 8 2026b. SemanticDACVAE-Japanese-32dim: Lightweight audio vae for japanese speech Stephen Brade, Sam Anderson, Rithesh Kumar, Zeyu Jin, and Anh Truong. p. 8 2025. Speakeasy: Enhancing text-to-speech interactions for expressive content creation Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. p. 8 2025. F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, and Seong-Whan Lee. p. 8 2025. Emosphere++: Emotion9 controllable zero-shot text-to-speech via emotionadaptive spherical vector Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. p. 9 2019. Parameter-efficient transfer learning for NLP Edward J Hu, yelong shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. p. 9 2022. LoRA: Low-rank adaptation of large language models Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, Xinyu Zhang, Pei Zhang. p. 9 2026. Qwen3-tts technical report Shengpeng Ji et al. p. 9 2025. ControlSpeech: Towards simultaneous and independent zero-shot speaker cloning and zero-shot language style control Guanhou Liu et al. p. 9 2023. PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions Jiaxuan Liu et al. p. 9 2025. Uddetts: Unifying discrete and dimensional emotions for controllable emotional text-to-speech Michael McAuliffe et al. p. 9 2017. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi Shivam Mehta et al. p. 9 2024. Matcha-tts: A fast tts architecture with conditional flow matching Babak Naderi and Ross Cutler. p. 9 2020. An Open Source Implementation of ITU-T Recommendation P. p. 9808 with Validation William Peebles and Saining Xie. p. 9 2023. Scalable diffusion models with transformers Puyuan Peng et al. p. 9 2024. VoiceCraft: Zero-shot speech editing and text-to-speech in the wild Adam Polyak et al. p. 9 2025. Movie gen: A cast of media foundation models Alec Radford et al. p. 9 2023. Robust speech recognition via large-scale weak supervision Zheng-Yan Sheng et al. p. 9 2025. Voice attribute editing with text prompt Reo Shimizu et al. p. 9 2024. Prompttts++: Controlling speaker identity in prompt-based text-to-speech using natural language descriptions Shinnosuke Takamichi et al. p. 9 2019. Jvs corpus: free japanese multispeaker voice corpus Haobin Tang et al. 2023. EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis Qwen Team. p. 0 2025. Qwen3 technical report Dan Wells et al. p. 9 2024. Experimental evaluation of MOS, AB and BWS listening test designs Detai Xin et al. p. 4 2024. Jvnv: A corpus of japanese emotional speech with verbal content and nonverbal expressions Dongchao Yang et al. p. 9 2024. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt Lvmin Zhang et al. p. 0 2023. Adding conditional control to text-to-image diffusion models Wangzixi Zhou et al. p. 0 2025. Toward natural emotional text-to-speech system with fine-grained non-verbal expression control Wangzixi Zhou et al. p. 0 2026. Beyond one-size-fits-all: Personalized and culturally adaptive emotional tts via interactive optimization of individual emotion perception spaces Qwen Team. p. 0 2025. A Qwen Prompt The system prompt is divided into four blocks for readability You are a Dialogue Revision Planner for Japanese expressive text-to-speech synthesis All revisions must be derived from user feedback and its context The output must be valid JSON only You are an excellent, fastidious and diligent researcher who never introduces yourself or answers with only the requested text This paper is 2611. p. 213875. EVERY sentence must end with a reference token The paper's pages are labelled --- Page N --- counting from 1 — the same N as that page's --- Page N --- line The sentence's own full stop stays AFTER the closing > a token with nothing after the comma is wrong Copy a token exactly as it is printed for you. Never invent an id, never guess a page or a paragraph, and never change one Put the token at the END of the sentence it supports, and point at the passage the claim came from ANCHOR THE WORDS The words go INSIDE, in the same brackets: after the location, a comma with no space, then five or more words copied word for word from the passage the claim rests on No quotation marks around them, and shorter than five words loses the anchor The tokens printed for you above are the ids and pages to use The paper's pages are labelled --- Page N --- counting from 1 — the same N as that page's --- Page N --- line The sentence's own full stop stays AFTER the closing > a token with nothing after the comma is wrong
--- Page 4 ---
Make this part more angry Say it a bit slower Revise the Speech I really enjoyed the movie but the ending felt rushed. Target span Angry Slower Revised Speech Next turn Convert feedback into executable instructions one-shot, utterance-level control: when the result is unsatisfactory, users must restate the instruction and regenerate the entire utterance This interaction is poorly suited to fine-grained refinement Even when only a short span or a single attribute requires modification, full-utterance regeneration may alter regions that were already satisfactory Real-world voice production is instead iterative: after listening to a previous take, a director may ask that “this part” sound angrier, that “the same part” be weakened, or that one region be changed while the rest is retained An interactive TTS should also interpret such feedback in context, revise only the requested region, and preserve previously accepted content
--- Page 5 ---
Qwen-based Dialog Revision Planner User Feedback (New Caption) Text Previous Context Pretrained TTS Backbone Temporal Acoustic Control Module Previous Speech Speech Updated Temporal Plan Next turn Figure 2 Architecture of DialInstruct-TTS flow-matching TTS backbone providing spanlevel control over emotion, speaking rate, volume, and phonation
--- Page 7 ---
Temporal Acoustic Control Module Frame masks Target Transition No-target Reference Latent Revision The RF-DiT backbone first produces a freely updated latent zfree tau', after which masked recomposition is performed as ztau' = Mloc⊙zfree tau'+1−Mloc⊙(1−τ')r + τ'ϵ The target and transition regions follow the new instruction, while the non-target regions are projected onto the previous-turn same-noise trajectory after every flow step
--- Page 8 ---
Control token Frame-local Control Adapter Attribute Controller Emotion ControlNet Volume Controller Rate Controller Phonation Control Token Encoding Temporal Localization Frame masks Target Transition No-target Reference Latent Revision Previous Speech Speech Revised Speech Next turn Figure 3 Architecture of the Temporal Acoustic Control Module attribute-specific residuals and the final hidden-state update H't = Ht + R att t + R attr t
--- Page 9 ---
Token Encoding and Temporal Localization The Token Encoding module converts each TemporalPlan item pbar i = (ci, vi, si, τi, [ai, bi), ξi) into a control token: ki = Ec(ci) + Ev(vi) + Eτ (τi) + Wssi, where ci, vi, si, and τi denote the control type, value, strength and application type The resulting sequence ZP = [k1,..., kN] represents what should be controlled
--- Page 9 ---
B Structured Planner Output We serialize the planner output as a JSON object with three top-level fields: resolved caption, delta plan, and complete plan The input follows this JSON format For later turns, previous temporal plan contains the complete JSON output from the immediately preceding turn
--- Page 9 ---
C Software and Preprocessing We use Montreal Forced Aligner (MFA) version 3.3.9 with the japanese mfa v3. p. 20 acoustic model for phoneme-level alignment SoX version 14. p. 74.2 is used for local time-scale and gain modification Japanese CER is computed using kotoba-tech/kotoba-whisper-v2. p. 40
--- Page 9 ---
D Training Setting We adopt a modular stage-wise training strategy starting from the frozen Irodori-TTS backbone The complete stage-wise training requires approximately 20k optimization steps and 20 GPU-hours Each stage combines a target-weighted flowmatching objective with duration and non-target preservation losses
--- Page 9 ---
E Detailed Results Specific Global Style Control Performance As shown in Table 5, the variance statistics further indicate that our DialInstruct-TTS achieves not only high instruction-matching accuracy but also comparatively consistent ratings Its overall Match variance is 0.901, lower than those of CosyVoice3 (1. p. 6321), Irodori-TTS (1. p. 2802), and Qwen3-TTS (1. p. 1
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Interactive Multi-turn Refinement Capability: The system enables users to
iteratively refine generated speech through natural-language dialogue until the desired style is achieved,
moving beyond one-shot control by allowing users to modify only requested attributes or temporal spans while retaining previously satisfactory speech, as stated inReference-latent revision preserves non-target regions across turns.
-
Context-Aware Revision Planning: A Qwen3 planner integrates the
dialogue history and latest user instruction into a complete TemporalPlan,
resolving contextual expressions likethe same part
and identifying revision operations such as adding, strengthening, weakening, retaining, replacing, or removing a control. This allows for context-dependent and relative instructions without requiring users to restate the complete control specification. -
Fine-Grained Temporal Control: The system utilizes a Temporal Acoustic Control Module that applies changes frame-locally using
Frame masks
andAttribute Controller
branches to modify emotion, speaking rate, volume, and phonation independently while maintaining the global speaking style.
Sources
- CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
- Qwen3-TTS Technical Report
- UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech
- Movie Gen: A Cast of Media Foundation Models
- JVS corpus: free Japanese multi-speaker voice corpus
- Qwen3 Technical Report
- Beyond One-Size-Fits-All: Personalized and Culturally Adaptive Emotional TTS via Interactive Optimization of Individual Emotion Perception Spaces
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models