Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook
summary
The gist
Existing text-to-speech systems predominantly focus on single-sentence synthesis and lack adequate contextual modeling as well as fine-grained performance control capabilities for generating coherent…
In short
The Audiobook-CC framework addresses limitations in existing text-to-speech systems by introducing three innovations for generating coherent, controllable multicast audiobooks: a context mechanism for long-form consistency, a disentanglement paradigm to separate style from speech prompts, and self-distillation to boost emotional expressiveness. Experiments show the system outperforms baselines across narration and dialogue tests.
Key concepts
- Contextual Consistency Mechanism
- This mechanism structures the entire audiobook sequence into a long context (seqC) by including pre-context and post-context information. This helps ensure that the synthesized speech maintains semantic consistency across long narrative segments, improving overall coherence in audiobooks.
- Disentanglement Paradigm
- This strategy decouples style control from the initial speech prompts. By training on discrete attribute labels (like 'angry' broken down into 'very angry, low volume'), the system achieves better semantic consistency between the desired style and the resulting speech, preventing prosodic interference.
- Self-Distillation
- To overcome data scarcity for high-intensity emotions, self-distillation is used. It synthesizes samples at various intensities using a pre-trained model, filters them for quality, and balances the intensity distribution. This process significantly enhances the emotional expressiveness and controllability of the final output.
Terminology used across episodes
This episode discusses
- Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook · Paper Radio
- MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers
- AudioStory: Generating Long-Form Narrative Audio with Large Language Models
- Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Immersive Audiobook Generation
- MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audio
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
- MoonCast: High-Quality Zero-Shot Podcast Generation
- Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
- Prosody Analysis of Audiobooks
- Text-aware and Context-aware Expressive Audiobook Speech Synthesis
- Long-Context Speech Synthesis with Context-Aware Memory
- CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
- Decoupled Weight Decay Regularization
- CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
- MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
The paper
Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook · Read on arXiv
Ximalaya Inc.
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook".
Paper summary: Rosa: So we've talked about the Audiobook-CC framework, which has its context mechanism for consistency, its disentanglement for style control, and self-distillation for emotion. The authors are basically saying they built something that addresses the weaknesses of current text to speech systems when dealing with long audiobooks >
Dev: It’s definitely a system designed specifically around those challenges—contextual scenario adaptability, semantic prosodic consistency, and character persona stability and controllability >
Taro: For someone who just listens to the show, it means you should expect audiobooks that stay coherent over long periods and where you can actually dial in the intensity of the emotion or style you want from the character >
Rosa: That's right. The paper shows superior performance across narration, dialogue, and even whole chapters when compared to existing baseline models >
Dev: The authors are pointing out that their Infer-Ctx andInst configuration performed better than just using context or instruction modules alone, which suggests combining the context awareness with the instruction mechanism is key >
Taro: It’s interesting how they found that combination was most effective for dialogue generation specifically, even when comparing narration tests to dialogue tests >
Rosa: So in short, Audiobook-CC is a controllable system aimed at making long-form audiobooks more coherent and expressive by using those three core ideas >
Conclusion: Rosa: So we're wrapping up on Audiobook-CC, which is this new system for generating long audiobooks where you can actually control the style and emotion of what’s being read >
Dev: Yeah, it's about making sure that when an AI reads a whole book over time, the voice stays consistent and doesn't drift off into nonsense >
Taro: It addresses how those long sequences get messy. The authors propose three main things: a context mechanism for keeping track of what happened earlier in the story, a way to separate the style from just the words being read, and some self-distillation to get better emotional control >
Rosa: Exactly. They're trying to fix those old TTS systems that only handle one sentence well and totally lose their mind over long narratives >
Dev: The big point here is that they showed this framework works for narration, dialogue, and even whole chapters, and it outperforms the baseline models they tested >
Taro: And the most interesting part for us is how it handles that emotional control. They've got better discrimination between high intensity and low intensity on unrelated text compared to previous work >
Rosa: It means we could actually build audiobooks that sound like a real performance, not just a flat reading of text >
Dev: It makes me wonder how this translates to the real world, you know, for things where context matters over long periods rather than just short clips or single interactions >
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G