Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement
summary
The gist
We introduce own-voice cancellation (OVC): removing a target (enrolled) speaker from a noisy multi-speaker mixture while preserving any remaining speech.
In short
The paper introduces Own-Voice Cancellation (OVC) to remove a target speaker from noisy multi-speaker audio while keeping other speech intact. It addresses latency issues in far-field streaming by using a time-domain model with only 2ms algorithmic latency. The proposed Mamba-MinGRU architecture outperforms existing methods like TD-SpeakerBeam by achieving competitive performance with significantly reduced computational cost, making it practical for low-latency denoising.
Key concepts
- Own-Voice Cancellation (OVC)
- OVC is a technique designed to isolate and remove the voice of the person using a device from a noisy recording. It works by treating the target speaker as something to be canceled out, ensuring that any other speech present in the mixture remains clear and undistorted.
- Mamba-MinGRU
- This is a specific time-domain model architecture used for speech enhancement. It replaces traditional masking networks with Mamba blocks and incorporates a MinGRU recurrence mechanism for temporal mixing. This structure allows the model to process audio efficiently while maintaining low algorithmic latency, crucial for real-time streaming applications.
- Algorithmic Latency
- This refers to the time delay introduced by the processing steps within a machine learning model. The paper focuses on achieving very low latency, specifically 2ms. This is vital for far-field devices where delays beyond this threshold cause noticeable echo artifacts in the user's own voice.
- SDR and PMOS
- SDR (Signal-to-Distortion Ratio) is a metric used to measure how clear the enhanced speech sounds compared to the original noisy signal. PMOS (Predicted Mean Opinion Score) is another metric that estimates how good the resulting audio quality will be perceived by humans, helping evaluate performance beyond just technical scores.
Terminology used across episodes
This episode discusses
- Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement · Paper Radio
- Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation
- SepMamba: State-space models for speaker separation using Mamba
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Were RNNs All We Needed?
- Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
- TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement
- Listen only to me! How well can target speech extraction handle false alarms?
- SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling
- Target Speech Extraction with Pre-trained Self-supervised Learning Models
- Diagonal State Spaces are as Effective as Structured State Spaces
- WHAM!: Extending Speech Separation to Noisy Environments
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation
- Layer Normalization
- Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
- Resurrecting Recurrent Neural Networks for Long Sequences
- Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers
- SDR - half-baked or well done?
- Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment
- Decoupled Weight Decay Regularization
- Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
The paper
Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement · Read on arXiv
WS Audiology · Technical University of Denmark
We introduce own-voice cancellation (OVC): removing a target (enrolled) speaker from a noisy multi-speaker mixture while preserving any remaining speech. Framed as the complement of target speaker extraction, OVC addresses latency-induced own-voice artifacts that arise when a far-field device streams enhanced audio back to the user, as the round-trip time easily exceeds the perceptual threshold for own-voice distortion. We condition a time-domain model with only 2 ms algorithmic latency on a short enrollment utterance and benchmark TD-SpeakerBeam alongside a lighter Mamba-MinGRU masker built from Mamba blocks with MinGRU temporal mixing. Replacing the ConvTasNet-based auxiliary network with a linear RNN encoder improves both signal-to-distortion ratio and predicted MOS while reducing compute. Results establish OVC as a practical, low-latency enhancement objective for far-field denoising.
DOI: 10.21437/Interspeech.2022-10674
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Don't Listen to Me".
Dev: Here's the gist of the paper. We introduce own-voice cancellation (OVC): removing a target (enrolled) speaker from a noisy multi-speaker mixture while preserving any remaining speech.
Rosa: This work establishes OVC as a practical, low-latency enhancement objective for far-field denoising.
Dev: Problem Context The problem of enhancing speech degraded by environmental noise, interfering speakers, or reverberant effects has been widely studied and remains a common obstacle in many realworld applications such as telecommunications, smart speakers, and conferencing devices.
Rosa: A particularly challenging scenario arises when a far-field device captures, enhances, and streams audio back to the user.
Paper summary: Rosa: So we're looking at this paper called "Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement." Basically it tackles that annoying echo problem when you use a far-field device streaming audio back to you.
Dev: Exactly. The core idea here is own-voice cancellation or OVC. It's about removing a specific target speaker from a noisy multi-speaker mixture while keeping any other speech that might be there, which is framed as the opposite of target speaker extraction.
Taro: And the authors are focused on solving those latency artifacts that happen when the round-trip time gets too long, easily exceeding what people can tolerate for own-voice distortion.
Rosa: They condition a time-domain model using only two milliseconds of algorithmic latency on a short enrollment utterance, and they benchmark this against something called TD-SpeakerBeam versus a lighter Mamba-MinGRU masker built with Mamba blocks and MinGRU temporal mixing.
Dev: That Mamba variant is introduced as a way to get better performance at much lower compute than TD-SpeakerBeam, while still maintaining that two millisecond algorithmic latency across all causal configurations.
Taro: The paper shows that even this smaller Mamba-MinGRU variant achieves competitive causal performance, specifically eleven point four seven dB SDR on F, at two milliseconds algorithmic latency, which suggests it scales well for resource-constrained streaming devices <ref:2606.23332#pg1>.
Rosa: Then they look at how to improve the auxiliary network, the part that conditions the model on speaker information. They compare using a ConvTasNet network with just one repetition against a bidirectional linear RNN network using only five blocks.
Dev: Replacing that ConvTasNet based auxiliary encoder with a linear RNN encoder actually cuts down the auxiliary compute significantly, from one point six seven GMAC/s to zero point two six GMAC/s, while also improving the SDR on the full mixture condition in all settings.
Taro: The loss function they use is a negative thresholded signaldistortion ratio loss, LSDR, which they extended to handle silence by defining an active case for when there's a speaker present apart from the enrolled speaker and an inactive case when there's no signal at all.
Rosa: So, what this means for us in the real world is that they've established OVC as a practical enhancement objective for far-field denoising, specifically targeting low latency.
Dev: It really boils down to making sure that when you get enhanced audio back from a device, your own voice doesn't sound delayed or distorted because of the processing time involved.
Taro: The main takeaway is that this approach allows for low-footprint own-voice suppression in streaming devices, and they are aiming for only two milliseconds of algorithmic latency.
Rosa: So we're talking about a system that can handle the delay issue without needing a huge computational budget or high latency, which is crucial for consumer devices.
Dev: The paper focuses heavily on how the architecture handles those time-domain aspects, showing that even with these lightweight modifications, you can achieve decent performance metrics like SDR and predicted mean opinion score.
Taro: They also flag a limitation: the work investigates two auxiliary networks and shows how changing that network affects performance, but they don't really explore training with more than two simultaneous speakers or evaluating under reverberant conditions yet.
Rosa: Right, so it’s a solid architecture for low-latency scenarios, but the authors are still looking at extending it to handle more complex real-world audio environments.
Dev: It sets a clear path forward by proving this method is feasible for deployment in situations where you can't afford heavy processing or long delays.
Taro: The paper on "Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement" shows us a way to address the echo artifacts that come from streaming enhanced audio back to the user by using a model with only two milliseconds of algorithmic latency.
Rosa: And it frames this as removing a target speaker from a noisy mixture, and it really highlights how important low latency is for maintaining natural-sounding communication in those far-field setups.
Dev: It’s about finding a balance between getting good separation performance and keeping the processing time extremely short, which is exactly what we need to think about when designing real-time loops.
Taro: The paper also shows how switching from a ConvTasNet based auxiliary network to a linear RNN encoder improves both the signal-to-distortion ratio and predicted mean opinion score while cutting down on compute.
Rosa: So it’s not just about latency anymore; it’s about optimizing the entire pipeline, from the initial masking network to how we condition that main model on speaker information.
Dev: The implication is that for devices streaming audio back, we can use these lightweight methods to suppress our own voice without introducing noticeable delay or significant quality loss.
Conclusion: Rosa: So we're wrapping up on this paper, "Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement." Essentially, they’ve shown a way to cut out your own voice when you’re listening to enhanced audio from a far-field device without making things lag too much.
Dev: Yeah, the authors focus heavily on this own-voice cancellation idea. They frame it as the opposite of target speaker extraction; instead of finding the target, they remove it from a noisy mixture while keeping everything else intact.
Taro: The main point is that this addresses those delay artifacts you get when audio goes back and forth, because the round-trip time is easily over ten milliseconds.
Rosa: Exactly. They condition their time-domain model with just two milliseconds of algorithmic latency on a very short enrollment utterance, which is key for streaming devices.
Dev: It’s about matching the performance of bigger models, like TD-SpeakerBeam, but using much less compute while keeping that low latency—specifically two milliseconds in all causal configurations.
Taro: Even this smaller Mamba-MinGRU variant gets competitive performance on its own, hitting eleven point four seven decibels SDR on F at those two milliseconds. That’s pretty solid for a resource-constrained device.
Rosa: They also tweaked the auxiliary network—the part that learns about the speaker—switching from a ConvTasNet to a linear RNN encoder helped cut down compute and actually improved the signal-to-distortion ratio when you test it on the full mixture.
Dev: From an engineering standpoint, cutting that auxiliary compute from one point six seven GMAC per second down to zero point two six GMAC per second is significant for real-time loops.
Taro: But they did flag a few things. They didn't really test this with more than two simultaneous speakers or under really noisy reverberant conditions yet. That’s where the real world gets messy, you know?
Rosa: Right, so the authors conclude that OVC is a practical enhancement objective for far-field denoising because it works well at low latency and low compute.
Dev: It paves the way for building these types of systems in streaming devices where you can't afford heavy processing or long delays.
Taro: So what this means for us is that we’re finally getting a path toward suppressing our own voice in those far-field setups without introducing noticeable delay or significant quality loss.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G