Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook".
Paper summary: Rosa: So we've talked about the Audiobook-CC framework, which has its context mechanism for consistency, its disentanglement for style control, and self-distillation for emotion. The authors are basically saying they built something that addresses the weaknesses of current text to speech systems when dealing with long audiobooks >
Dev: It’s definitely a system designed specifically around those challenges—contextual scenario adaptability, semantic prosodic consistency, and character persona stability and controllability >
Taro: For someone who just listens to the show, it means you should expect audiobooks that stay coherent over long periods and where you can actually dial in the intensity of the emotion or style you want from the character >
Rosa: That's right. The paper shows superior performance across narration, dialogue, and even whole chapters when compared to existing baseline models >
Dev: The authors are pointing out that their Infer-Ctx andInst configuration performed better than just using context or instruction modules alone, which suggests combining the context awareness with the instruction mechanism is key >
Taro: It’s interesting how they found that combination was most effective for dialogue generation specifically, even when comparing narration tests to dialogue tests >
Rosa: So in short, Audiobook-CC is a controllable system aimed at making long-form audiobooks more coherent and expressive by using those three core ideas >
Conclusion: Rosa: So we're wrapping up on Audiobook-CC, which is this new system for generating long audiobooks where you can actually control the style and emotion of what’s being read >
Dev: Yeah, it's about making sure that when an AI reads a whole book over time, the voice stays consistent and doesn't drift off into nonsense >
Taro: It addresses how those long sequences get messy. The authors propose three main things: a context mechanism for keeping track of what happened earlier in the story, a way to separate the style from just the words being read, and some self-distillation to get better emotional control >
Rosa: Exactly. They're trying to fix those old TTS systems that only handle one sentence well and totally lose their mind over long narratives >
Dev: The big point here is that they showed this framework works for narration, dialogue, and even whole chapters, and it outperforms the baseline models they tested >
Taro: And the most interesting part for us is how it handles that emotional control. They've got better discrimination between high intensity and low intensity on unrelated text compared to previous work >
Rosa: It means we could actually build audiobooks that sound like a real performance, not just a flat reading of text >
Dev: It makes me wonder how this translates to the real world, you know, for things where context matters over long periods rather than just short clips or single interactions >
Ximalaya Inc.
eess.AS
Submitted: 2025-09-22
Updated: 2026-11-18
Project page: https://everestai.github.io
Importance score: 90/100
The gist: Existing text-to-speech systems predominantly focus on single-sentence synthesis and lack adequate contextual modeling as well as fine-grained performance control capabilities for generating coherent…
Key concepts
- Contextual Consistency Mechanism
- This mechanism structures the entire audiobook sequence into a long context (seqC) by including pre-context and post-context information. This helps ensure that the synthesized speech maintains semantic consistency across long narrative segments, improving overall coherence in audiobooks.
- Disentanglement Paradigm
- This strategy decouples style control from the initial speech prompts. By training on discrete attribute labels (like 'angry' broken down into 'very angry, low volume'), the system achieves better semantic consistency between the desired style and the resulting speech, preventing prosodic interference.
- Self-Distillation
- To overcome data scarcity for high-intensity emotions, self-distillation is used. It synthesizes samples at various intensities using a pre-trained model, filters them for quality, and balances the intensity distribution. This process significantly enhances the emotional expressiveness and controllability of the final output.
Terminology
Summary
Existing text-to-speech systems predominantly focus on single-sentence synthesis and lack adequate contextual modeling as well as fine-grained performance control capabilities for generating coherent multicast audiobooks The authors propose a context-aware and emotion controllable speech synthesis framework specifically engineered for multicast audiobooks with three key innovations
Framework Innovations
The proposed Audiobook-CC framework introduces three primary innovations to address the limitations of existing TTS systems. These include a context mechanism for contextual consistency, a disentanglement paradigm to decouple style control from speech prompts for semantic consistency, and self-distillation to boost emotional expressiveness and instruction controllability. p. 0
Contextual Consistency Mechanism
The system incorporates a context modeling mechanism designed for long-form audiobook generation which dramatically improves semantic consistency across context. This is achieved by structuring the full contextual sequence as seqC = preS, pre context Cpre, preE, poS, post context Cpost, poE (2). p. 1 Furthermore, semantic consistency is enhanced through a decoupled training strategy that focuses on the adaptive relationship between target audio prosody and text semantics rather than traditional coupled “prompt-target” training models. p. 2
Controllability Enhancement Strategies
The controllability enhancement strategy focuses on both training and inference mechanisms to address character persona stability and controllability. During training, fiction or persona-derived instructions are decomposed into discrete attribute labels, such as decomposing ”shouting angrily” into ”very angry, low volume, slow speed” for an ill elderly person. p. 2 For inference, a third module converts user instructions into attribute combinations to mitigate linguistic ambiguity and enhance control precision. p. 2
Emotional Expressiveness Boost
To alleviate the scarcity of high-intensity emotional samples, a self-distillation strategy is employed to highly enhance emotional expressiveness and instruction controllability. This involves three steps: first, samples with varying intensity levels are synthesized using a pre-trained emotional TTS model; second, the generated samples are filtered through PER, speaker similarity and pitch to guarantee quality; third, the intensity distribution is balanced via targeted data augmentation. p. 2
Experimental Validation and Results
The experimental setup involved three stages of training using various datasets, including 1 million hours of audiobook data for finetuning CosyVoice2. The model was evaluated using subjective assessment methods like Single-sentence Mean Opinion Score (S-MOS) and Multi-sentence Mean Opinion Score (M-MOS) across narrative, dialogue, and chapter tests. p. 2 Results indicate that the proposed system largely exceeds both baseline models for all test setups. p. 2 The Infer-Ctx&Inst configuration achieved the most preferable result compared to just Infer-Ctx and Infer-inst, which is consistent with the results in Table 1. p. 2 The model exhibits stronger H-L (discriminability between “high-intensity” and “low-intensity”) emotion control on “Text-Unrelated” test set and outperforms baseline models [6] in high-intensity emotion control. p. 3
Conclusion
The paper proposes a controllable, context-aware TTS framework for multicast audiobooks, with 3 innovations: a context mechanism for contextual consistency, a disentanglement paradigm to decouple style control from speech prompts for semantic consistency, and self-distillation to boost emotional expressiveness and controllability. In future, the authors suggest expanding chapter data or selecting specific data to mitigate data sparsity, and explore reinforcement learning for performance improvement. p. 4
Acknowledgement
The authors acknowledge using Doubao [20] for this paper’s language polishing, including grammar checks, context-appropriate term selection, and text flow improvement. All theoretical frameworks, empirical data work, and key arguments are the authors’ independent efforts; they critically evaluated and integrated Doubao-generated suggestions to maintain academic authenticity and originality. p. 4
References
[1] Kyeongman Park, Seongho Joo, and Kyomin Jung. Multiactor-audiobook: Zero-shot audiobook generation with faces and voices of multiple speakers. p. 4 p. 4 arXiv preprint arXiv:2505. p. 4 p. 4 p. 4 p. 413082, 2025
[2] Yuxin Guo, Teng Wang, Yuying Ge, Shijie Ma, Yixiao Ge, Wei Zou, and Ying Shan. Audiostory: Generating long-form narrative audio with large language models. p. 4 p. 4 arXiv preprint arXiv:2508. p. 4 p. 420088, 2025
[3] Yan Rong, Shan Yang, Guangzhi Lei, and Li Liu. Dopamine audiobook: A training-free mllm agent for emotional and human-like audiobook generation. p. 4 p. 4 arXiv preprint arXiv:2504. p. 4 p. 411002, 2025
[4] Xuenan Xu, Jiahao Mei, Chenliang Li, Yuning Wu, Ming Yan, Shaopeng Lai, Ji Zhang, and Mengyue Wu. Mm-storyagent: Immersive narrated storybook video generation with a multi-agent paradigm across text, image and audio. p. 4 p. 4 arXiv preprint arXiv:2503. p. 4 p. 405242, 2025
[6] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. p. 4 p. 4 arXiv preprint arXiv:2412. p. 4 p. 410117
[8] OpenMOSS Team. Moss-ttsd: Text to spoken dialogue generation. p. 4 p. 4 https://www. p. 4 p. 4open-moss.com/en/moss-ttsd/, 2025
[9] Leying Zhang, Yao Qian, Long Zhou, Shujie Liu, Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He et al. Covomix: Advancing zero-shot speech generation for human-like multi-talker conversations. p. 4 p. 4 Advances in Neural Information Processing Systems 37:100291–100317, 2024
[14] Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. Cam++: A fast and efficient network for speaker verification using context-aware masking. p. 4 p. 4 arXiv preprint arXiv:2303. p. 4 p. 400332
[16] Shun Lei, Yixuan Zhou, Liyang Chen, Zhiyong Wu, Shiyin Kang, and Helen Meng. Context-aware coherent speaking style prediction with hierarchical transformers for audiobook speech synthesis, 2023
[17] Smith M and et al. Automating error frequency analysis via the phonemic edit distance ratio. p. 4 p. 4 J Speech Lang Hear Res 2019 Jun 19;62(6):1719-1723. p. 4 p. 4, 2019
[18] Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang et al. Cosyvoice 3: Towards in-thewild speech generation via scaling-up and post-training. p. 4 p. 4 arXiv preprint arXiv:2505.17589
[19] Ziyang Ma and et al. emotion2vec: Self-supervised pre-training for speech emotion representation. p. 4 p. 4 arXiv preprint arXiv:2312. p. 4 p. 415185
[20] Ziheng Jiang and et al. Megascale: Scaling large language model training to more than 10,000 gpus. p. 4 p. 4 arXiv preprint arXiv:2402. p. 4 p. 415627
--- Page 1 ---
AUDIOBOOK-CC: CONTROLLABLE LONG-CONTEXT SPEECH GENERATION FOR MULTICAST AUDIOBOOK Min Liu, JingJing Yin, Xiang Zhang, Siyu Hao, Yanni Hu, Bin Lin, Yuan Feng, Hongbin Zhou, Jianhao Ye⋆ Ximalaya Inc., China
--- Page 2 ---
Fig. 1. An overview of Audiobook-CC. p. 1 (a) The audiobook generation pipeline: long-form fiction is segmented into chapters, followed by textual and character persona analysis Narrations and dialogues are extracted, assigned to voices through casting, and synthesized using the proposed architecture, resulting in a neutral-style multicast audiobook (b) Detailed architecture of the proposed controllable long-context speech synthesis model
--- Page 3 ---
[V, seqC, S, seqE, T, Wi, Uj] (1) The symbols S and E denote the start and end of a sequence, while T marks the switch between text tokens and speech tokens The text tokens Wi are obtained by processing the input text with a text tokenizer Similarly, the speech tokens Uj are derived using the same speech tokenizer in cosyvoice2 A speaker embedding V, derived via Cam++ [14], is prepended to the input Contextual information is structured into pre-context Cpre and post-context Cpost, demarcated by preS / preE and poS / poE, respectively The full contextual sequence is formed as: seqC = preS, Cm pre, preE, poS, Cn post, poE (2) Control inputs (e.g., emotion or volume) are denoted El, with EM and /EM marking the boundaries The controllable sequence is constructed as: seqE = EM, El, /EM (3) We train an autoregressive (AR) speech language model conditioned on the speaker embedding V, contextual seqC, controllable seqE, text tokens Wi, and GT tokens Uj, to predict the shifted speech token sequence U˜j, formulated as: Y N j=1 p(U˜j V, seqC, seqE, W, U˜<j; θAR) (4)
--- Page 4 ---
While pre-context Cpre and post-context Cpost intuitively enhance contextual consistency, training solely on textual context without explicit speaker modeling still requires prompt speech for voice specification during inference Consequently, synthesized speech prosody remains constrained by the prompt speech, limiting semantic-prosodic alignment and narrative coherence in long-form synthesis To improve alignment between synthesized audio and text semantics while mitigating excessive prosodic interference from prompt audio, we adopt a decoupled training strategy. This approach reduces cross-audio prosodic interference at the data level and enhances consistency between synthesized audio and current text semantics. p. 2
Pruning redundant reasoning tokens
The authors detail the training and inference strategies of their system below. The training process involves three stages, with learning rates set to 1e-5 for the first two stages but 1e-6 for the last stage. p. 2
Where the gains come from
The results show that the dialogue tests are more advantageous than those of the narration tests for both MOS and ABX tests. The speech generation strategy of Infer-Ctx&Inst does benefit from combining context-aware narration generation and instructed dialogue generation. p. 2 The Ctx&Inst achieves the most preferable result compared to just Infer-Ctx and Infer-inst, which is consistent with the results in Table 1. p. 2 The non-decoupled model achieves excessively high SS [14], indicating over-similar timbre, prosody, and emotion—and thus impractical for audiobook due to insufficient character diversity. p. 3 The context-augmented model achieved a higher S-MOS, with listening tests confirming improved coherence. p. 3
Fine-Grained Emotional Control
The model exhibits stronger H-L (discriminability between “high-intensity” and “low-intensity”) emotion control on “Text-Unrelated” test set and outperforms baseline models [6] in high-intensity emotion control. Table 3 shows our model exhibits stronger H-L (discriminability between “high-intensity” and “low-intensity”) emotion control on “Text-Unrelated” test set and outperform baseline models [6] in high-intensity emotion control. p. 3 The model also outperforms the baseline model [6] in terms of mixed emotion performance. p. 3
The paper proposes a controllable, context-aware TTS framework for multicast audiobooks, with 3 innovations: a context mechanism for contextual consistency, a disentanglement paradigm to decouple style control from speech prompts for semantic consistency, and self-distillation to boost emotional expressiveness and controllability. Experiments show the framework outperforms baselines in narration, dialogue, and chapter generation, with ablation studies validating its key components. p. 3 In future, the authors can expand chapter data or select specific data to mitigate data sparsity, and explore reinforcement learning for performance improvement. p. 4
[1] Kyeongman Park, Seongho Joo, and Kyomin Jung. Multiactor-audiobook: Zero-shot audiobook generation with faces and voices of multiple speakers. arXiv preprint arXiv:2505.13082, 2025
[2] Yuxin Guo, Teng Wang, Yuying Ge, Shijie Ma, Yixiao Ge, Wei Zou, and Ying Shan. Audiostory: Generating long-form narrative audio with large language models. arXiv preprint arXiv:2508.20088, 2025
[3] Yan Rong, Shan Yang, Guangzhi Lei, and Li Liu. Dopamine audiobook: A training-free mllm agent for emotional and human-like audiobook generation. arXiv preprint arXiv:2504.11002, 2025
[4] Xuenan Xu, Jiahao Mei, Chenliang Li, Yuning Wu, Ming Yan, Shaopeng Lai, Ji Zhang, and Mengyue Wu. Mm-storyagent: Immersive narrated storybook video generation with a multi-agent paradigm across text, image and audio. arXiv preprint arXiv:2503.05242
[6] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117
[8] OpenMOSS Team. Moss-ttsd: Text to spoken dialogue generation. https://www.open-moss.com/en/moss-ttsd/, 2025
[9] Leying Zhang, Yao Qian, Long Zhou, Shujie Liu, Dongmei Wang, Xiaofei Wang et al. Covomix: Advancing zero-shot speech generation for human-like multi-talker conversations. Advances in Neural Information Processing Systems 37:100291–100317, 2024
[14] Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. Cam++: A fast and efficient network for speaker verification using context-aware masking. arXiv preprint arXiv:2303.00332
[16] Shun Lei, Yixuan Zhou, Liyang Chen, Zhiyong Wu, Shiyin Kang, and Helen Meng. Context-aware coherent speaking style prediction with hierarchical transformers for audiobook speech synthesis, 2023
[17] Smith M and et al. Automating error frequency analysis via the phonemic edit distance ratio. J Speech Lang Hear Res 2019 Jun 19;62(6):1719-1723., 2019
[18] Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang et al. Cosyvoice 3: Towards in-thewild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589
[19] Ziyang Ma and et al. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185
[20] Ziheng Jiang and et al. Megascale: Scaling large language model training to more than 10,000 gpus. arXiv preprint arXiv:2402.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Decoupled training strategy for semantic consistency: Employing independent modeling that focuses on
the adaptive relationship between target audio prosody and text semantics
instead of traditional coupledprompt-target
training reduces cross-audio prosodic interference at the data level and enhances consistency between synthesized audio and current text semantics. -
Persona stability via multi-constraint prompt selection: Implement a mechanism where prompts are selected based on three refinements:
prompts are selected within the same chapter with voiceprint similarity constraints [14] to reduce cross-chapter variance,
determining anoptimal similarity threshold to balance voice stability and prosodic diversity,
and incorporatinghigh-quality labeled multi-emotional speech data to alleviate sample sparsity.
-
Controllability enhancement via attribute decomposition: Decompose fiction or persona instructions into discrete attribute labels, such as decomposing
shouting angrily
into specific acoustic features likevery angry, low volume, slow speed
for a given persona. This facilitates an explicitattribute–acoustic feature mapping learning.
-
Emotional expressiveness boost through self-distillation: Use a self-distillation strategy where samples are first synthesized using a pre-trained emotional TTS model, then filtered through
PER, speaker similarity and pitch to guarantee quality,
and finally the intensity distribution is balanced via targeted data augmentation. -
Enhanced coherence with contextual text input: Feed the Text-Speech Language Model with context augmented by
preceding and subsequent sentence
of the target text alone, which has been shown to achieve a higher S-MOS, leading to improved coherence in long-form synthesis.
Sources
- MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers
- AudioStory: Generating Long-Form Narrative Audio with Large Language Models
- Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Immersive Audiobook Generation
- MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audio
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
- MoonCast: High-Quality Zero-Shot Podcast Generation
- Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
- Prosody Analysis of Audiobooks
- Text-aware and Context-aware Expressive Audiobook Speech Synthesis
- Long-Context Speech Synthesis with Context-Aware Memory
- CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
- Decoupled Weight Decay Regularization
- CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
- MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models