Adaptive Duration Model for Text Speech Alignment

arXiv:2507.22612 · cs.SD, cs.AI, eess.AS · Submitted 2025-07-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Adaptive Duration Model for Text Speech Alignment".

Jane: The gist In this paper, we propose an adaptive duration modeling framework called DurFormer to predict phoneme-level duration distribution with given text,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to recap where we are, this paper proposes the Adaptive Duration Model for Text Speech Alignment. The thesis is that instead of just relying on fixed external sources for durations in non-autoregressive TTS, you can predict phoneme-level duration distribution directly from the text input itself >

Jane: Essentially, they claim their new framework offers a more precise way to align phonemes and mel spectrograms harmoniously by predicting those durations based on the text and context conditions >

Lu: The architecture is called DurFormer. It takes the text prompt, speed level, and speech scene as input. It uses an attribute encoder to fuse these features into enhanced tokens before feeding them into a prediction process >

Meng: I see they are using a semantic extractor block from Roberta for the text semantics, which means they’re trying to capture the meaning of the text deeply before predicting durations >

Tom: And then they incorporate that semantic vector through a cross-attention module and an MLP layer for domain adaptation, which feeds into their duration prediction mechanism >

Jane: The key claim is that this combination of linguistic features and contextual information allows the model to perform better than previous methods, specifically showing improvements on alignment accuracy and better robustness when dealing with audio mismatches >

Lalam: The objective is to find an optimal monotonic path s star of phi, which means searching through all possible alignment paths S(Phi) for the best fit against the target duration sequence >

Tom: So, if I'm driving and my voice sounds different than the training data, this model is supposed to be able to adapt its duration predictions better because it considers those conditions >

Lu: That’s right. And they quantify that improvement numerically, stating roughly an eleven point three percent improvement on alignment accuracy >

Meng: It's good that they gave a number like eleven point three percent, but I wonder if that number was tested on real-world traffic or just clean datasets > <ref:2507.22612#pg1>

Jane: That’s a valid concern, Meng. But the paper does claim it makes zero-shot TTS models more robust to the mismatch between prompt audio and input audio >

Tom: So they are showing that this adaptation capability isn't just theoretical; it translates into better performance when you try to use a model with minimal or no specific audio prompts >

Lalam: It’s all about learning how the linguistic characteristic of the text scripts affects the rhythm of the synthesized speech, which in turn dictates the duration of each phoneme >

Jane: So, if we're trying to understand this paper for someone who doesn't read it, it boils down to this: they built a system that uses text meaning and current speaking conditions to guess how long each part of a word should take when making speech from text >

Tom: It’s about moving beyond just using external sources for timing and embedding the actual text structure into the duration prediction itself. That's what they achieved in this paper.

Conclusion: Tom: So, looking at "Adaptive Duration Model for Text Speech Alignment" by Cao et al., the main point is that they’ve developed a novel duration prediction framework called DurFormer >

Jane: It moves beyond just using external sources for durations and instead learns to predict them directly from the text while incorporating contextual information like speaking speed level and scene >

Lu: The authors are trying to solve the alignment problem by searching for an optimal monotonic path s star of phi that aligns the phoneme and mel spectrogram precisely >

Meng: In simple terms, this means if you give it text and some conditions, it can give you a much better idea of how long each phoneme should be than before >

Tom: They show this translates to roughly an eleven point three percent improvement on alignment accuracy and better performance for zero-shot TTS models when there’s a mismatch in audio prompts >

Jane: So the implication is that for anyone using neural text to speech, especially those deploying systems where you can't rely on fixed timing data, having a model that adapts its duration based on context makes the system much more reliable.

Lu: They’re doing this by deeply fusing linguistic meaning into the prediction process through semantic fusion modules >

Lalam: It’s about learning how the text structure influences the rhythm of speech and adjusting those durations dynamically >

Tsinghua University

cs.SD, cs.AI, eess.AS

Submitted: 2025-07-30

Updated: 2026-10-15

Comments: the work is not complete, it is a bad work and outdated

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 76/100

The gist: The gist In this paper, we propose an adaptive duration modeling framework called DurFormer to predict phoneme-level duration distribution with given text, which has shown more precise prediction and

Key concepts

Duration Prediction Framework
This is the core idea of the paper: a system that predicts how long each phoneme in a text should last when spoken. It tries to find an optimal sequence of durations that best matches the text and speech features, serving as the foundation for accurate alignment.
Attribute Encoder
This component extracts and combines different types of information influencing speech duration. It separates factors into 'subjective' (like speaker characteristics) and 'objective' (like external speaking conditions) to create a richer understanding of how speech timing is affected.
Alignment Objective
The mathematical goal is to find the best mapping between phonemes and the resulting audio spectrogram. This objective is framed as a sequence prediction problem, where the model learns to predict the target duration sequence based on text and condition inputs.

Terminology

Summary

The gist

In this paper, we propose an adaptive duration modeling framework called DurFormer to predict phoneme-level duration distribution with given text, which has shown more precise prediction and condition adaptation ability compared to previous baseline models.

How it works

  1. The problem is formulated as searching for an optimal monotonic path s∗(Φ) that can align phoneme and mel-spectrogram precisely and harmoniously, where the alignment objective is formulated as a duration sequence prediction problem with the given text phoneme The mathematical formulation can be formulated as a duration sequence prediction problem with the given text phoneme The mathematical formulation can be formulated as a duration sequence prediction problem with the given text phoneme

The proposed DurFormer architecture receives text input, with speech speed level and speech scene as condition information The core components include an attribute encoder, semantic adapter, and a probability module

Key Components of DurFormer

** Attribute Encoder: This component is designed to extract and fuse heterogeneous features by categorizing influencing factors into objective and subjective types A dedicated factor encoder is designed to extract and fuse these heterogeneous features effectively The optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor should be speaker-specified and related to the speaker’s speaking characteristics The objective factor means the external speaking condition that will influence the speech duration This optimization problem can be formulated as min θ Lalign(θ) = EΦDtarget − f(Φ, E(τin, τex); θ)2 (3)> The subjective factor

Improvements for AI systems

  1. Improve alignment robustness by replacing unstable attention mechanisms with an adaptive duration modeling framework that treats duration prediction as a conditional regression problem. This allows for more precise phoneme-level duration distribution, leading to a roughly 11.3 percents immprovement on alignment accuracy and making the performance of zero-shot TTS models more robust to the mismatch between prompt audio and input audio.

  2. Enhance TTS quality by incorporating richer contextual features into duration prediction through a dedicated factor encoder that extracts and fuses heterogeneous features categorized as objective and subjective types. This fusion process allows for better capture of linguistic nuances, such as speaker characteristics (subjective) and speaking scenarios (objective).

  3. Increase prediction flexibility by adopting a distributional approach in the probability module, where the model predicts the mean and the variance of the duration sequence, rather than a single definite value. This probabilistic output enables more versatile duration information that is expected to enhance the performance of TTS systems.

Sources

Related papers