DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

Abstract

Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.

Introduction

Audio-visual speech enhancement (AVSE) aims to recover the speech signal of a target speaker from noisy and interfering speech by exploiting complementary acoustic and visual information. Compared with audio-only speech separation approaches, AVSE benefits from visual cues such as facial movements and speaker appearance, which provide valuable information for identifying and extracting the target speaker. However, deploying AVSE systems in real-world environments remains challenging. Unlike conventional benchmarks based on clean videos and artificially mixed speech, real-world scenarios often involve naturally occurring speech overlap, reverberation, environmental noise, and degraded visual observations. Existing AVSE approaches mainly focus on designing effective audio-visual fusion mechanisms, where audio and visual representations are extracted separately and then fused within the separation network, allowing visual information to directly guide speech reconstruction. Such approaches heavily rely on the quality and reliability of visual inputs. In real-world scenarios, visual signals may be affected by occlusion, low resolution, missing frames, blur, or far-field recording conditions. When degraded visual features are directly incorporated into the separation process, inaccurate visual information may interfere with speech reconstruction and reduce system robustness. Moreover, the lack of large-scale training data with realistic acoustic conditions further limits the generalization ability of existing AVSE systems.

DAVE separates audio reconstruction from visual speaker attribution, where the audio branch focuses on robust speech separation and the visual branch provides speaker-related information for target identification. The main contributions are: (1) construction of DAVE-Corpus, a large-scale AVSE training corpus with 219,411 realistic mixtures; (2) proposal of DAVE, a decoupled audio-visual enhancement framework with progressive multi-objective optimization and metric-preserving selective enhancement; (3) extensive experiments on the Real-World Audio-Visual Speech Enhancement Challenge demonstrating effectiveness under realistic mixed scenarios and degraded visual conditions.

Data Construction

DAVE-Corpus is constructed in three stages: synthetic mixture generation, official protocol remix, and EchoSet replay.

Synthetic Mixture Generation: Speech segments are collected from three public meeting corpora: AliMeeting, MISP, and AISHELL-4. Quality-based screening using frame-level energy statistics retains segments with estimated SNR above 25 dB, duration between 2.0 and 15.0 seconds, and silence ratio below 35%. After filtering, 4,431 speech segments are selected (1,469 from AliMeeting, 2,926 from MISP, 36 from AISHELL-4). A combinatorial augmentation pipeline generates diverse meeting-like mixtures. For each mixture, two speech segments are randomly sampled while avoiding highly similar speakers (rejected if cosine similarity of speaker embeddings exceeds 0.5). Mixture duration is randomly sampled between 4.0 and 8.0 seconds. The second source is randomly scaled with relative gain from [−3, 3] dB. A frozen FunASR-Nano-2512 model generates reference transcriptions. 50% of mixtures are augmented with simulated reverberation using pyroomacoustics (room dimensions 3–8 m × 3–7 m × 2.5–3.5 m, RT60 between 0.2 and 0.4 seconds). Environmental noise from MUSAN is added with SNR values between 12 and 30 dB. The synthetic subset contains 199,998 mixtures.

Official Protocol Remix: Using the official challenge protocol, 8,000 training mixtures are re-synthesized by randomly pairing de-duplicated single-speaker segments from the development set, excluding 150 held-out sessions for validation.

EchoSet Replay: To prevent catastrophic forgetting of the pre-training domain during fine-tuning, 11,413 replay pairs from the EchoSet corpus are included as an auxiliary subset.

The final DAVE-Corpus totals 219,411 training mixtures: 199,998 synthetic pairs, 8,000 official protocol remixes, and 11,413 EchoSet replay pairs.

Methodology

DAVE decouples audio reconstruction from visual speaker attribution, comprising three components: an audio-only separation module, a visual speaker attribution module, and a certified selective enhancement module.

Audio Separation: The TIGER architecture is scaled from 0.82M to 2.56M parameters (encoder channels from 256 to 512, feature channels from 128 to 256, repeating blocks from 8 to 12), termed TIGER-M, trained from scratch on DAVE-Corpus with multi-objective optimization:

• Permutation-invariant SI-SDR loss: LPIT = min over permutations of negative SI-SDR

• CER loss: minimizes teacher-forcing cross-entropy of a frozen FunASR-Nano-2512 ASR model on separated outputs, with gradients back-propagated through a differentiable log-mel filterbank

• Speaker fidelity loss: minimizes cosine distance between speaker embeddings from frozen WeSpeaker ResNet34

• Perceptual losses: differentiable implementations of STOI (Pearson correlation 1.0000), PESQ (Pearson correlation 0.9249), and UTMOS (using official UTMOSv2 weights), computed only on samples with clean references

Visual Speaker Attribution: Combines four complementary evidence sources into a weighted vote:

• Voiceprint similarity (weight 3.0) using WeSpeaker ResNet34 trained on CN-Celeb

• Audio-visual matching score from StableSyncNet (weight 1.5)

• Audio-visual synchrony score from pre-trained SyncNet fine-tuned with Perfect Match paradigm (weight 1.0)

• Lip-keypoint motion energy from FAN (weight 0.7)

Each vote is normalized and combined to produce a one-shot attribution decision.

Certified Selective Enhancement: Restricted to the no-reference partition:

• Scene Routing: A lightweight acoustic scene classifier distinguishes remix from real-recording samples using spectral statistics, achieving 98.13% accuracy on the development set. Only samples routed to the no-reference class enter enhancement steps.

• Layered GAN denoising: A layered GAN-based denoiser built on MossFormerGAN is applied only to routed no-reference samples.

• Loudness Normalization: Scales each output by 6× relative to original amplitude following ITU-R BS.1770, with peak-protection bound 0.99/peak to prevent clipping.

Experiments

TIGER-M is trained from scratch on DAVE-Corpus using 8×A800 GPUs with native PyTorch DDP. Each GPU processes a batch of 2 three-second segments. Adam optimizer with peak learning rate 1×10−3, linear warm-up of 1,000 steps, ReduceLROnPlateau scheduling with patience 3. Model evaluated every 4,000 steps on a held-out validation set of 149 sessions using permutation-invariant SI-SDR as selection criterion. Evaluation follows the official Real-World AVSE Challenge protocol reporting seven metrics: SI-SDR, PESQ, STOI, UTMOS, DNSMOS OVRL, CER, and speaker similarity.

Ablation Study: The baseline TIGER-M with only PIT SI-SDR loss achieves SI-SDR of 9.84 dB and CER of 21.9%. Adding CER loss improves CER from 21.9% to 19.3%, with side benefits of +0.47 dB in SI-SDR and +0.016 in speaker similarity. Adding speaker fidelity loss improves speaker similarity from 0.679 to 0.697. Perceptual losses bring PESQ increase of 0.09, UTMOS from 1.999 to 2.77, OVRL from 1.631 to 2.02, and CER drops to 17.1%. Certified selective enhancement improves DNSMOS OVRL by 0.10 and UTMOS by 0.04, with remaining five metrics unchanged by construction.

Official Challenge Results: On Track 1 (real-world mixed), DAVE substantially outperforms the official baseline across all seven metrics, raising SI-SDR by over 16 dB, reducing CER from 1.025 to 0.171, and more than doubling speaker similarity to 0.726. On Track 2 (visual degradation), DAVE maintains strong performance with SI-SDR of 8.93 dB and CER of 0.220, confirming robustness of the decoupled design. The separation backbone operates without visual inputs, making it immune to degraded visual conditions.

Conclusion

DAVE is a decoupled audio-visual enhancement framework that reserves visual information for reliable decision-making instead of direct feature fusion. The DAVE-Corpus with 219,411 mixtures addresses data scarcity via combinatorial acoustic augmentation and room impulse response simulation. Progressive multi-objective optimization jointly improves separation, intelligibility, speaker identity, and perceptual quality. The visual speaker attribution module and certified selective enhancement chain add robustness without risking metric degradation. Experiments on the Real-World AVSE Challenge confirm strong performance under both real-world mixed and degraded visual conditions. Future work will explore more adaptive decoupled architectures for complex real-world audio-visual environments.

Improvements to AI Systems:

1. Decoupled Multi-Modal Architecture for Robustness: Instead of fusing visual features directly into the main processing network (which makes the system fragile to degraded inputs), AI systems can adopt a decoupled design where one branch (e.g., audio) handles core signal reconstruction independently, and another branch (e.g., visual) only provides high-level attribution or routing decisions. This makes the system immune to low-quality or missing modalities.

2. Progressive Multi-Objective Optimization with Differentiable Perceptual Metrics: AI systems can be trained with a staged loss function that combines task-specific losses (e.g., SI-SDR for separation) with intelligibility (CER), speaker identity (cosine similarity), and perceptual quality (differentiable STOI, PESQ, UTMOS). This allows joint optimization of multiple user-facing metrics without sacrificing one for another, and the differentiable implementations enable end-to-end training.

3. Certified Selective Enhancement with No-Reference Partitioning: AI systems can include a post-processing chain (e.g., denoising, loudness normalization) that is only applied to samples lacking clean references, using a scene router (e.g., a lightweight classifier) to decide which samples qualify. This guarantees that reference-based metrics (like PESQ or SI-SDR) are never degraded by enhancement, while no-reference metrics (like DNSMOS) are improved—providing a certified non-degradation guarantee.

4. Combinatorial Acoustic Augmentation for Data Scarcity: AI systems can generate large-scale, realistic training corpora by combinatorially pairing speech segments from public meeting corpora, applying random relative gains, simulated reverberation (with varied room dimensions and RT60), and environmental noise (at controlled SNR ranges). This addresses data scarcity and improves generalization to real-world acoustic conditions.

5. Speaker Attribution via Weighted Multi-Evidence Voting: Instead of relying on a single visual cue, AI systems can combine multiple complementary evidence sources (e.g., voiceprint similarity, audio-visual matching, synchrony scores, lip-motion energy) with learned or fixed weights to make robust one-shot speaker attribution decisions. This improves target identification accuracy even when individual cues are unreliable.

6. Replay-Based Continual Learning to Prevent Catastrophic Forgetting: When fine-tuning on a new domain (e.g., official challenge protocol), AI systems can include a small replay subset from the pre-training domain (e.g., EchoSet) in the training batch. This maintains performance on the original domain while adapting to the new one, improving stability and generalization.

7. Quality-Based Data Filtering for Training Purity: AI systems can pre-filter training segments using frame-level energy statistics (e.g., SNR > 25 dB, duration 2–15 s, silence ratio 0.5) to avoid ambiguous or low-quality training examples, leading to faster convergence and better final performance.

What the Improved AI System Can Do:

• Robust Speech Separation in Real-World Conditions: It can separate a target speaker’s voice from overlapping speech, reverberation, and environmental noise, even when the visual input is occluded, blurry, low-resolution, or missing frames—because the core separation does not depend on visual features.

• Simultaneously Optimize Multiple Quality Metrics: It can improve intelligibility (lower CER), speaker identity preservation (higher speaker similarity), and perceptual quality (higher PESQ, STOI, UTMOS, DNSMOS) in a single training run, rather than trading off one metric for another.

• Guarantee Non-Degradation of Reference-Based Metrics: It can apply enhancement (e.g., denoising, loudness normalization) only when no clean reference exists, ensuring that metrics like SI-SDR and PESQ are never worsened by post-processing, while still improving no-reference metrics like DNSMOS.

• Scale Training Data Efficiently: It can generate 219,000+ realistic mixtures from a small set of public meeting recordings, enabling training of larger models (e.g., 2.56M parameters) without needing expensive new data collection.

• Accurately Identify Target Speakers from Degraded Visuals: It can fuse voiceprint, lip motion, and audio-visual synchrony cues to robustly attribute speech to the correct speaker, even when visual quality is poor.

• Adapt to New Domains Without Forgetting: It can fine-tune on a new challenge protocol while retaining performance on the original pre-training domain, thanks to replay-based continual learning.

• Operate in Real-Time with High Accuracy: It can achieve state-of-the-art results (e.g., SI-SDR improvement of 16 dB over baseline, CER reduction from 1.025 to 0.171) on real-world mixed and visually degraded scenarios, making it suitable for hearing aids, video conferencing, and human-robot interaction.

Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.