Predictive-Generative Drift Decomposition for Speech Enhancement and Separation

arXiv:2605.06189 · eess.AS, cs.LG · Submitted 2026-05-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Predictive-Generative Drift Decomposition for Speech Enhancement and Separation".

Dev: Here's the gist of the paper. We propose a plug-and-play framework for speech enhancement and separation that augments existing predictors with a generative speech prior Our approach, termed Stochastic Interpolant Prior for Speech (SIPS), builds on stochastic interpolants and leverages their flexibility to bridge predictive and generative modeling This results in a mathematically grounded framework for combining strong pretrained predictors with the expressive power of generative models The Core Methodology The SIPS framework decomposes the interpolation dynamics into a task-specific drift and a stochastic denoising component, allowing a predictive estimate to be integrated directly into the generative sampling process Specifically, during inference, the predictor provides a deterministic drift that steers the sampling process toward a task-consistent estimate Concurrently, the learned score function or denoiser estimates the Gaussian noise and steers the trajectories toward regions of higher likelihood This decomposition enables the predictor to maintain fidelity to the observation, while the generative prior mitigates artifacts and enhances naturalness The Training Signal is Built The score model is trained using only clean speech, yielding a degradation-agnostic prior that does not depend on a specific corruption process This training objective involves minimizing the loss E h∥Dθ t, S + (a + γ(t))Z − Z∥2 i, where t ∼ Uniform

0, 1: , S ∼ ρS, and Z ∼ N (0, I) is a standard Gaussian random variable with identity covariance matrix I While this training setup corresponds to so-called mirror interpolants, it also introduces a mismatch between training and inference because the denoiser is exposed only to clean speech corrupted by Gaussian noise, whereas at inference time Xt additionally contains environmental noise or interfering speakers inherited from the observation Y Where the Gains Come From The method demonstrates effectiveness for both speech enhancement and speech separation using recent predictors such as SEMamba and FlexIO Across tasks and models, our method consistently improves non-intrusive perceptual quality scores, indicating enhanced naturalness, while incurring only minor decreases in reference-based distortion metrics For speech separation on the WHAMR!

Paper summary: Rosa: We just touched on how this paper proposes a framework called Stochastic Interpolant Prior for Speech, or SIPS. The main thesis is that they can bridge the gap between predictive modeling and generative modeling by decomposing the interpolation dynamics into a task-specific drift and a stochastic denoising component.

Dev: They claim this decomposition allows you to integrate your existing strong pretrained predictors directly into the generative sampling process during inference, which results in a framework that is mathematically grounded. It lets you combine those strong predictors with the expressive power of generative models.

Taro: So, the paper suggests that instead of just using one approach for restoration, you can have two cooperating mechanisms working simultaneously: one steering the prediction based on what it thinks should happen, and another guiding the samples toward where they are most likely to be in terms of clean speech distribution.

Rosa: They achieve this by training a score model using only clean speech data. This is important because it creates a prior that isn't tied to any specific noise or corruption process during training, making it more general.

Dev: But there’s a caveat they admit: this setup means the denoiser only sees clean speech mixed with Gaussian noise when it's being trained. At inference time, that denoiser has to deal with the real messy observation Y, which includes things like environmental noise or other speakers.

Taro: That mismatch is where the complexity lies in making this work robustly outside of a perfect simulation setup. How do you handle that gap between training and using it on real-world data?

Rosa: They address this by framing restoration as an inference problem where samples are drawn from a posterior distribution based on an assumed observation model, rather than just learning a direct mapping. This is what allows the generative model to work with the noisy input Y.

Dev: It’s essentially combining a deterministic path from the predictor with a probabilistic sampling guided by the generative prior to approximate that posterior distribution pSY under some noise assumptions <ref:2605.06189#pg2>.

Taro: That means they aren't just doing one thing; they are using the predictor output as an intermediate step to guide a modified prior when sampling, which is a bit more involved than standard diffusion models.

Rosa: And the paper shows this approach is effective for both speech enhancement and speech separation tasks, specifically mentioning predictors like SEMamba and FlexIO. They claim it consistently improves non-intrusive perceptual quality scores across those different models.

Dev: It also notes that these improvements don't come with major hits to reference-based distortion metrics, which is a good balance for practical use cases.

Conclusion: Rosa: So, wrapping up the Predictive-Generative Drift Decomposition for Speech Enhancement and Separation paper by Richter et al., the core idea is using SIPS to combine predictive methods with generative priors by decomposing the dynamics into drift and denoising components.

Taro: It moves beyond just applying a single enhancement model; it’s about structuring how a predictor steers the process while a generative prior handles the naturalness of the output distribution.

Dev: The authors show that this framework is mathematically grounded because they explicitly define how to integrate the predictive estimate into the generative sampling step, which is what makes it plug-and-play.

Rosa: The title itself tells you a lot: "Predictive-Generative Drift Decomposition." It’s about systematically managing the movement of the signal estimation through these two interacting forces.

Taro: For someone who listens to the show, this means that instead of just swapping out one enhancement tool for another, you can leverage this structural decomposition to improve quality reliably across different tasks.

Dev: The real implication is that we can get better perceptual quality by ensuring the predictor remains faithful to what it sees, while the generative component cleans up the noise and adds naturalness.

Rosa: It’s about finding a way to make those powerful generative models more reliable when they are being guided by a strong predictive signal from an existing model.

Julius Richter, Yoshiki Masuyama, Christoph Boeddeker, Takahiro Edo, Gordon Wichern, Jonathan Le Roux

MERL

eess.AS, cs.LG

Submitted: 2026-05-07

Updated: 2026-11-06

Comments: Accepted at NeurIPS 2026

Code: https://github.com/merlresearch/sips-speech

Project page: https://sp-uhh.github.io/ears_dataset/3https://github.com/kaituoxu/Conv-TasNet

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: We propose a plug-and-play framework for speech enhancement and separation that augments existing predictors with a generative speech prior Our approach…

Key concepts

Stochastic Interpolant Prior for Speech (SIPS)
This is the proposed framework that augments existing speech predictors with a generative speech prior. It uses stochastic interpolants to bridge predictive and generative modeling, creating a mathematically grounded method for combining strong pre-trained predictors with generative models.
Task-Specific Drift
During inference, the predictor provides a deterministic component that steers the sampling process toward an estimate consistent with the specific task at hand. This drift helps maintain fidelity to the original observation by guiding where the generative model should focus its attention.
Stochastic Denoising Component
This part of SIPS is handled by a learned score function or denoiser. It estimates Gaussian noise and steers the sampling trajectories toward areas of higher likelihood, effectively mitigating artifacts introduced by the generative prior and enhancing naturalness.

Terminology

Summary

We propose a plug-and-play framework for speech enhancement and separation that augments existing predictors with a generative speech prior Our approach, termed Stochastic Interpolant Prior for Speech (SIPS), builds on stochastic interpolants and leverages their flexibility to bridge predictive and generative modeling This results in a mathematically grounded framework for combining strong pretrained predictors with the expressive power of generative models

The Core Methodology

The SIPS framework decomposes the interpolation dynamics into a task-specific drift and a stochastic denoising component, allowing a predictive estimate to be integrated directly into the generative sampling process Specifically, during inference, the predictor provides a deterministic drift that steers the sampling process toward a task-consistent estimate Concurrently, the learned score function or denoiser estimates the Gaussian noise and steers the trajectories toward regions of higher likelihood This decomposition enables the predictor to maintain fidelity to the observation, while the generative prior mitigates artifacts and enhances naturalness

The Training Signal is Built

The score model is trained using only clean speech, yielding a degradation-agnostic prior that does not depend on a specific corruption process This training objective involves minimizing the loss E h∥Dθ t, S + (a + γ(t))Z − Z∥2 i, where t ∼ Uniform[0, 1], S ∼ ρS, and Z ∼ N (0, I) is a standard Gaussian random variable with identity covariance matrix I While this training setup corresponds to so-called mirror interpolants, it also introduces a mismatch between training and inference because the denoiser is exposed only to clean speech corrupted by Gaussian noise, whereas at inference time Xt additionally contains environmental noise or interfering speakers inherited from the observation Y

Where the Gains Come From

The method demonstrates effectiveness for both speech enhancement and speech separation using recent predictors such as SEMamba and FlexIO Across tasks and models, our method consistently improves non-intrusive perceptual quality scores, indicating enhanced naturalness, while incurring only minor decreases in reference-based distortion metrics For speech separation on the WHAMR!

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Predictor-generative combination allows for a mathematically grounded framework for combining strong pretrained predictors with the expressive power of generative models. This enables a unified framework that generalizes across different predictors and additive degradation tasks, overcoming limitations of architecture-specific conditioning in prior hybrid approaches.

  2. The system can produce speech enhancement by using the output of a pretrained predictor Pphi to define a deterministic drift vˆ that steers the sampling process toward a task-consistent estimate. Simultaneously, the learned denoiser Dtheta guides this trajectory toward regions of higher likelihood, which enforces consistency with the distribution of natural speech and improving perceptual realism.

  3. The framework can be trained using only clean speech to yield a degradation-agnostic prior that can be reused across tasks, meaning the score model is trained independently of the specific corruption process, allowing it to generalize across speech enhancement and separation without retraining for each new degradation setting.

  4. The system achieves improved perceptual quality by consistently enhancing non-intrusive metrics like DNSMOS, NISQA, and UTMOS while incurring only minor decreases in reference-based distortion metrics, demonstrating that predictive and generative modeling need not be competing paradigms.

  5. For speech separation, the system can combine a predictor (like FlexIO) with the SIPS prior to achieve gains up to +1.0 NISQA, showing that the combination improves non-intrusive metrics (DNSMOS, NISQA and UTMOS) with comparable SI-SDR.

Sources

Related papers