Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook

summary

The gist

Existing text-to-speech systems predominantly focus on single-sentence synthesis and lack adequate contextual modeling as well as fine-grained performance control capabilities for generating coherent…

In short

The Audiobook-CC framework addresses limitations in existing text-to-speech systems by introducing three innovations for generating coherent, controllable multicast audiobooks: a context mechanism for long-form consistency, a disentanglement paradigm to separate style from speech prompts, and self-distillation to boost emotional expressiveness. Experiments show the system outperforms baselines across narration and dialogue tests.

Key concepts

Contextual Consistency Mechanism
This mechanism structures the entire audiobook sequence into a long context (seqC) by including pre-context and post-context information. This helps ensure that the synthesized speech maintains semantic consistency across long narrative segments, improving overall coherence in audiobooks.
Disentanglement Paradigm
This strategy decouples style control from the initial speech prompts. By training on discrete attribute labels (like 'angry' broken down into 'very angry, low volume'), the system achieves better semantic consistency between the desired style and the resulting speech, preventing prosodic interference.
Self-Distillation
To overcome data scarcity for high-intensity emotions, self-distillation is used. It synthesizes samples at various intensities using a pre-trained model, filters them for quality, and balances the intensity distribution. This process significantly enhances the emotional expressiveness and controllability of the final output.

Terminology used across episodes

This episode discusses

The paper

Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook · Read on arXiv

Ximalaya Inc.

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Audiobook-CC: Controllable Long-context Speech Generation for Multicast Audiobook".

Paper summary: Rosa: So we've talked about the Audiobook-CC framework, which has its context mechanism for consistency, its disentanglement for style control, and self-distillation for emotion. The authors are basically saying they built something that addresses the weaknesses of current text to speech systems when dealing with long audiobooks >

Dev: It’s definitely a system designed specifically around those challenges—contextual scenario adaptability, semantic prosodic consistency, and character persona stability and controllability >

Taro: For someone who just listens to the show, it means you should expect audiobooks that stay coherent over long periods and where you can actually dial in the intensity of the emotion or style you want from the character >

Rosa: That's right. The paper shows superior performance across narration, dialogue, and even whole chapters when compared to existing baseline models >

Dev: The authors are pointing out that their Infer-Ctx andInst configuration performed better than just using context or instruction modules alone, which suggests combining the context awareness with the instruction mechanism is key >

Taro: It’s interesting how they found that combination was most effective for dialogue generation specifically, even when comparing narration tests to dialogue tests >

Rosa: So in short, Audiobook-CC is a controllable system aimed at making long-form audiobooks more coherent and expressive by using those three core ideas >

Conclusion: Rosa: So we're wrapping up on Audiobook-CC, which is this new system for generating long audiobooks where you can actually control the style and emotion of what’s being read >

Dev: Yeah, it's about making sure that when an AI reads a whole book over time, the voice stays consistent and doesn't drift off into nonsense >

Taro: It addresses how those long sequences get messy. The authors propose three main things: a context mechanism for keeping track of what happened earlier in the story, a way to separate the style from just the words being read, and some self-distillation to get better emotional control >

Rosa: Exactly. They're trying to fix those old TTS systems that only handle one sentence well and totally lose their mind over long narratives >

Dev: The big point here is that they showed this framework works for narration, dialogue, and even whole chapters, and it outperforms the baseline models they tested >

Taro: And the most interesting part for us is how it handles that emotional control. They've got better discrimination between high intensity and low intensity on unrelated text compared to previous work >

Rosa: It means we could actually build audiobooks that sound like a real performance, not just a flat reading of text >

Dev: It makes me wonder how this translates to the real world, you know, for things where context matters over long periods rather than just short clips or single interactions >

More episodes

← Home