REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction

arXiv:2608.03964 · eess.AS · Submitted 2026-08-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction".

Dev: Here's the gist of the paper. Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice <ref:2608.03964#pg2> The paper introduces REAL-2MIX, a Mandarin AVTSE benchmark of jointly recorded real twospeaker mixtures with synchronized multi-view video <ref:2608.03964#pg2> This work matters because it tests identity faithfulness in audio-visual target speaker extraction by evaluating whether a model follows the requested lip cue during joint recordings <ref:2608.03964#pg2> Benchmark Construction and Data Curation REAL-2MIX is introduced to benchmark identity-faithful AV-TSE on purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg2> The task definition involves predicting the target speaker waveform sˆq = fθ(x, vq) given an overlap recording x(t) = sA(t)+sB(t)+n(t) <ref:2608.03964#pg2> Each scene records two stationary participants in a shared environment and includes preceding A-only and B-only stages to provide in-scene speaker references <ref:2608.03964#pg2> The recording protocol uses distances of 0.5, 1.5, and 3 m and side-view angles of 30◦, 45◦, and 60◦ <ref:2608.03964#pg2> The complete processed release contains 77 scene directories from 14 unique speaker dyads with a total of 7,598 mono clips <ref:2608.03964#pg2> Training Signal Architecture The extractor uses frozen, 1,280-dimensional projected AV-HuBERT features and target-conditioned training <ref:2608.03964#pg2> The architecture employs a single-output TF-GridNet where the visual stream is processed through input-level fusion and layer-wise FiLM modulation <ref:2608.03964#pg2> Specifically, the frozen visual branch produces a 1,280-dimensional visual sequence which is linearly projected to 1,280 dimensions and rescaled by a learned scalar <ref:2608.03964#pg2> The separator contains 6 GridNet blocks with 128 channels, where the visual sequence is supplied directly to the layer-specific FiLM generators <ref:2608.03964#pg2> Evaluation Metrics and Analysis The performance is jointly evaluated using Qwen3-ASR-1.7B CER for content and WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD) for target identity <ref:2608.03964#pg2> Speaker similarity is reported as SpkSim, which compares the embedding of the output sˆi,q with the corresponding solo-reference mean e r i,q <ref:2608.03964#pg2> The OSD-first strict identity evaluation uses an automatic residualoverlap proxy before speaker identification to classify outputs into mutually exclusive classes <ref:2608.03964#pg2> Ablation results show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608.03964#pg2> Corpus Complementarity The work complements REAL-2MIX with VOXBLINK2-AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training <ref:2608.03964#pg2> VOXBLINK2-AVSE is constructed from the in-thewild VoxBlink2 collection but undergoes audio-visual validity checks and DNSMOS filtering to ensure quality <ref:2608.03964#pg2> This curated corpus emphasizes reliable visual target selection rather than raw hours alone, providing a distinct operating point for AVSE training <ref:2608.03964#pg2> The final evaluation tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608.03964#pg2> The conclusion is that REAL-2MIX turns audio-visual TSE into a paired identity test where the same jointly recorded mixture must yield A under A’s lip cue and B under B’s lip cue <ref:2608.03964#pg2> The combination of speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608.03964#pg2> The auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion <ref:2608.03964#pg2> The paper concludes that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608.03964#pg2> References Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.

Dev: LRS3-TED: A large-scale dataset for visual speech recognition <ref:2608.03964#pg2> Adrian Bulat and Georgios Tzimiropoulos.

Paper summary: Rosa: So we're looking at this paper called REAL-2MIX, which is about identity faithfulness in audio-visual target speaker extraction <ref:2608.03964#pg1,audio-visual target speaker extraction>. Essentially, the main problem they tackle is that sometimes an audio visual target speaker extractor should pull the right person based on what you see in the video, but a separator can totally ignore that visual cue and just keep outputting whatever voice sounds loudest.

Dev: Right. It sets up this benchmark called REAL-2MIX to test exactly how faithful the identity extraction is when you have these real mixtures recorded with synchronized multi-view video <ref:2608.03964#pg1,with synchronized multi-view video>. The paper claims this work matters because it checks whether a model actually follows the requested lip cue during those joint recordings <ref:2608.03964#pg2>.

Taro: So, they’re not just looking at isolated audio or video; they are using joint recordings where you have both speakers present and synchronized video, which is a pretty complex setup to get right <ref:2608.03964#pg1>.

Rosa: Exactly. They build this benchmark using purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg1>. The task is to predict the target speaker waveform given an overlap recording, which is basically trying to figure out who is talking in that noisy mix <ref:2608.03964#pg1>.

Dev: And they set up the scenes with two stationary participants in a shared environment, and they include those preceding A only and B only stages to give you references for each speaker right there on the recording <ref:2608.03964#pg1>.

Taro: They also specify distances like zero point five meters, one point five meters, and three meters away, and side-view angles of thirty degrees up to sixty degrees for the cameras <ref:2608.03964#pg1>.

Rosa: After processing all that, they have seventy-seven scene directories from fourteen different speaker pairs with a total of seven thousand five hundred ninety-eight mono clips, including six thousand forty-two dual annotated mixtures <ref:2608.03964#pg1>. That’s the dataset size for this benchmark.

Dev: Now, looking at how they train their extractor, they use frozen one thousand two hundred eighty dimensional projected audio visual HuBERT features and target conditioned training <ref:2608.03964#pg2>. The architecture is a single output TF-GridNet where the visual stream goes through input level fusion and layer wise FiLM modulation <ref:2608.03964#pg2>.

Taro: That means the visual branch produces a one thousand two hundred eighty dimensional sequence, which they linearly project to one thousand two hundred eighty dimensions and then rescale it by a learned scalar <ref:2608.03964#pg2>. The separator itself has six GridNet blocks with one hundred twenty eight channels, where the visual sequence is supplied directly to the layer specific FiLM generators <ref:2608.03964#pg2>.

Paper summary: Rosa: They use frozen features, but they condition the training using this visual input in a very specific way, which is interesting for how much reliance a model has on what it sees versus just what it hears <ref:2608.03964#pg2>.

Dev: And when we look at evaluation, they use Qwen three ASR one point seven billion CER for the content and WeSpeaker ResNet thirty four plus Overlapped Speech Detection for the target identity <ref:2608.03964#pg2>. Speaker similarity is reported as SpkSim, which compares the embedding of their output speaker with the corresponding solo reference mean <ref:2608.03964#pg2>.

Taro: They also use an OSD first strict identity evaluation that uses an automatic residual overlap proxy before identifying speakers to sort outputs into mutually exclusive classes <ref:2608.03964#pg2>. The ablation results show that having matched training data and suppressing interferer sound matters more than just making the GridNet back end bigger <ref:2608.03964#pg2>.

Rosa: So, what this means for us is that they are testing if an extractor can follow the visual intent while also suppressing overlap, and preserving the exact utterance requested <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s a pretty rigorous test for identity faithfulness.

Dev: It really is. And to complement this work, they released something called VOXBLINK two AVSE, which is a large identity disjoint audio lip corpus that they curated for training <ref:2608.03964#pg2>. This corpus comes from the in the wild VoxBlink two collection but they ran it through audio visual validity checks and DNSMOS filtering to ensure quality <ref:2608.03964#pg2>.

Taro: That curated corpus is important because it puts a focus on reliable visual target selection instead of just having raw hours, which gives a different operating point for training the AVSE model <ref:2608.03964#pg2>.

Rosa: So the final evaluation tests whether an extractor follows the visual intent, suppresses overlap, and preserves the requested utterance in this new context <ref:two thousand six hundred eight point zero three nine six four#pg1. It seems like they've put together a pretty comprehensive test for this kind of extraction task.

Dev: The conclusion is that REAL-2MIX turns audio visual target speaker extraction into a paired identity test where the same jointly recorded mixture must yield speaker A under A's lip cue and speaker B under B's lip cue <ref:two thousand six hundred eight point zero three nine six four#pg1. The combination of speaker margin, OSD first strict identity rates, and Qwen three ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:two thousand six hundred eight point zero three nine six four#pg1.

Paper summary: Taro: But I gotta point out a limitation mentioned in the paper. They state that the auxiliary speaker loss requires a controlled rerun before they can draw a causal conclusion about its effectiveness <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s not an immediate proof of concept for that specific loss term.

Rosa: That's fair. So, while they show the potential, they are being cautious about drawing definitive conclusions about every single component in the training setup <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s a careful look at how these components interact when you test for identity faithfulness.

Dev: So we've seen that this benchmark is built on real mixtures with synchronized video, and that they use a specific TF-GridNet architecture conditioned by visual features <ref:two thousand six hundred eight point zero three nine six four#pg2. It’s testing the core idea of whether the model respects the visual instruction in those complex audio visual scenarios.

Taro: And it seems like they emphasize that having good training data and suppressing interferer sound is more important than just having a bigger GridNet back end to get better results <ref:two thousand six hundred eight point zero three nine six four#pg2. That tells us where the real bottleneck might be in building these systems.

Rosa: Exactly. And they also look at how different methods compare, like Conv-TasNet versus TF-GridNet, showing that the time frequency modeling in the GridNet is adding something concrete to speech separation <ref:two thousand six hundred eight point zero three nine six four#pg2.

Dev: And they are using metrics like Qwen three ASR one point seven billion CER for content and WeSpeaker ResNet thirty four plus Overlapped Speech Detection for identity, which gives a solid way to measure both the accuracy of what was extracted and the faithfulness of that extraction <ref:two thousand six hundred eight point zero three nine six four#pg2.

Taro: So, when you look at this whole piece, it’s about how we train these models to handle the messy reality of real-world audio visual data, where things can get overlapped or misdirected <ref:two thousand six hundred eight point zero three nine six four#pg1. It sets a standard for what we consider identity faithfulness in this area.

Rosa: And to wrap up, the work on REAL-2MIX gives us a clear way to test if an audio visual target speaker extractor is actually doing what it's supposed to do when things get complicated <ref:two thousand six hundred eight point zero three nine six four#pg1. We'll be looking at how these challenges are handled in the next part of the show.

Conclusion: Rosa: So we’re wrapping up with some thoughts on this paper, REAL-2MIX, by Afouras, Chung, and Zisserman <ref:2608.03964#pg1>. Basically they’re testing if these models actually follow the visual cues when trying to pick a target speaker from a mix of audio and video.

Dev: Right. It’s about checking identity faithfulness in audio visual target speaker extraction, which is a pretty specific test because you want the model to listen to what it sees, not just what it hears <ref:2608.03964#pg2>.

Taro: The whole point of REAL-2MIX is this paired identity test where the same mixture has to yield speaker A under A's lip cue and speaker B under B's lip cue <ref:2608.03964#pg1>. It’s a very strict way to measure if the model is actually doing what it’s supposed to be doing <ref:2608.03964#pg1>.

Rosa: And they use speaker margin, overlap detection, and some specific metrics like Qwen three ASR CER to see if the whole system works together <ref:two thousand six hundred eight point zero three nine six four#pg2. It’s a pretty comprehensive way to check if an extractor follows visual intent while suppressing overlap and preserving the requested utterance <ref:two thousand six hundred eight point zero three nine six four#pg1.

Dev: I think what this means for us is that when we build these systems, we can't just look at audio quality or video resolution anymore; we have to test if the model respects those visual instructions under realistic, messy conditions <ref:two thousand six hundred eight point zero three nine six four#pg1.

Taro: Exactly. The limitation they mentioned is that they need a controlled rerun for the auxiliary speaker loss before they can really claim anything causal about it <ref:two thousand six hundred eight point zero three nine six four#pg1. So we have to be careful not to draw conclusions too fast <ref:two thousand six hundred eight point zero three nine six four#pg1.

Rosa: True. It’s a careful look at how these different parts interact when you test for that kind of fidelity <ref:two thousand six hundred eight point zero three nine six four#pg1. But this benchmark gives us a solid standard for what identity faithfulness actually looks like in the audio visual space <ref:two thousand six hundred eight point zero three nine six four#pg1.

Dev: So, we’ve seen how they build the test, what the results were with those specific metrics, and where they might be cautious about their conclusions <ref:two thousand six hundred eight point zero three nine six four#pg2. Now we need to see how this idea of identity faithfulness plays out when you move beyond these controlled mixed recordings <ref:two thousand six hundred eight point zero three nine six four#pg1.

School of Cyber Science and Engineering, Wuhan University · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen

eess.AS

Submitted: 2026-08-04

Updated: 2026-11-06

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 91/100

The gist: Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice…

Key concepts

REAL-2MIX
This is a benchmark created for testing how well models can perform audio-visual target speaker extraction when the audio comes from real mixtures recorded with synchronized multi-view video. It specifically checks if the model follows visual lip cues during these joint recordings.
Audio-Visual Target Speaker Extraction (AVTSE)
This is a task where a system takes an overlapping recording of two speakers and the video, and must identify which speaker's voice is the target speaker. The goal is to find the target speaker's waveform based on both audio and visual information.
Identity Faithfulness
This refers to how accurately a model preserves the true identity of a person when performing a task. In this context, it means checking if an extractor correctly identifies the intended target speaker even when visual cues (like lip movements) are present in the joint recording.

Terminology

Summary

Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice <ref:2608. p. 0

Where the paper says it

cn, mingli369@cuhk. cn Abstract Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice.

03964#pg2> The paper introduces REAL-2MIX, a Mandarin AVTSE benchmark of jointly recorded real twospeaker mixtures with synchronized multi-view video <ref:2608. p. 0
Where the paper says it

cn Abstract Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin AVTSE benchmark of jointly recorded real twospeaker mixtures with synchronized multi-view video.

03964#pg2> This work matters because it tests identity faithfulness in audio-visual target speaker extraction by evaluating whether a model follows the requested lip cue during joint recordings <ref:2608. p. 4
Where the paper says it

Together with speaker-disjoint manifests, the corpus emphasizes reliable visual target selection rather than raw hours alone. REAL2MIX then provides the complementary real-scene test: whether a model trained on these curated face– voice pairs follows the requested lip cue during jointly recorded Mandarin overlap.

03964#pg2>

Benchmark Construction and Data Curation

REAL-2MIX is introduced to benchmark identity-faithful AV-TSE on purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg2> The task definition involves predicting the target speaker waveform sˆq = fθ(x, vq) given an overlap recording x(t) = sA(t)+sB(t)+n(t) <ref:2608. p. 1

Where the paper says it

1 Task Definition and Recording Protocol Let an overlap recording be x(t) = sA(t)+sB(t)+ n(t), where n includes room acoustics and background noise. Given the visual stream vq for target q ∈ {A, B}, the extractor predicts sˆq = fθ(x, vq).

03964#pg2> Each scene records two stationary participants in a shared environment and includes preceding A-only and B-only stages to provide in-scene speaker references <ref:2608. p. 1
Where the paper says it

This requirement rules out visual bypass, in which both prompts return the same dominant source. Each scene records two stationary participants in a shared environment.

03964#pg2> The recording protocol uses distances of 0. p. 2
Where the paper says it

2 Geometry and Acquisition Figure 1 summarizes the recording geometry and shows a representative frontal-view frame. The protocol uses distances of 0.

5, 1.5, and 3 m and side-view angles of 30◦, 45◦, and 60◦ <ref:2608. p. 2
Where the paper says it

The protocol uses distances of 0. 5, and 3 m and side-view angles of 30◦ , 45◦ , and 60◦ .

03964#pg2> The complete processed release contains 77 scene directories from 14 unique speaker dyads with a total of 7,598 mono clips <ref:2608. p. 2
Where the paper says it

3 Audited Corpus Statistics The complete processed release contains 77 scene directories from 14 unique speaker dyads. Its 7,598 mono 16-kHz clips total 11.

03964#pg2>

Training Signal Architecture

The extractor uses frozen, 1,280-dimensional projected AV-HuBERT features and target-conditioned training <ref:2608.03964#pg2> The architecture employs a single-output TF-GridNet where the visual stream is processed through input-level fusion and layer-wise FiLM modulation <ref:2608. p. 4

Where the paper says it

5 AV-HuBERT-Conditioned TF-GridNet Figure 2 summarizes the two complementary roles of the visual stream. Input-level fusion aligns the target cue with the mixture spectrogram before the first GridNet block, while layer-wise FiLM reinjects target information throughout the separator rather than relying on a single early fusion.

03964#pg2> Specifically, the frozen visual branch produces a 1,280-dimensional visual sequence which is linearly projected to 1,280 dimensions and rescaled by a learned scalar <ref:2608. p. 4
Where the paper says it

A large AV-HuBERT backbone produces a 1,024-dimensional visual sequence, which a learned linear layer projects to 1,280 dimensions and rescales by a learned scalar. Both the backbone and projection are frozen for separator training.

03964#pg2> The separator contains 6 GridNet blocks with 128 channels, where the visual sequence is supplied directly to the layer-specific FiLM generators <ref:2608. p. 4
Where the paper says it

The resulting visual map has shape B × 1 × T × F and is concatenated with the real and imaginary mixture components before the twodimensional input convolution. In parallel, the resampled visual sequence is supplied directly to the layer-specific FiLM generators described below.

03964#pg2>

Evaluation Metrics and Analysis

The performance is jointly evaluated using Qwen3-ASR-1.7B CER for content and WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD) for target identity <ref:2608. p. 0

Where the paper says it

We jointly evaluate content with Qwen3-ASR-1. 7B CER and target identity with WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD).

03964#pg2> Speaker similarity is reported as SpkSim, which compares the embedding of the output sˆi,q with the corresponding solo-reference mean e r i,q <ref:2608. p. 6
Where the paper says it

If ei,q is the embedding of output sˆi,q and e r i,q is the corresponding solo-reference mean, we report SpkSim = 1 |P| X (i,q)∈P e ⊤ i,qe r i,q ∥ei,q∥2∥e r i,q∥2 . (5) We report target similarity, interferer similarity, and their mean difference (speaker margin).

03964#pg2> The OSD-first strict identity evaluation uses an automatic residualoverlap proxy before speaker identification to classify outputs into mutually exclusive classes <ref:2608. p. 6
Where the paper says it

4 OSD-First Strict Identity Evaluation The real overlap recordings have no synchronized isolated physical sources, so target SI-SDR is not defined. We instead use an automatic residualoverlap proxy before speaker identification.

03964#pg2> Ablation results show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608. p. 7
Where the paper says it

VOXBLINK2-AVSE supplies 766 hours of identity-diverse aligned training data, while frozen AV-HuBERT features, speaker margin training, and layer-wise FiLM make the target cue explicit throughout the separator. The ablations show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion.

03964#pg2>

Corpus Complementarity

The work complements REAL-2MIX with VOXBLINK2-AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training <ref:2608.03964#pg2> VOXBLINK2-AVSE is constructed from the in-thewild VoxBlink2 collection but undergoes audio-visual validity checks and DNSMOS filtering to ensure quality <ref:2608. p. 3

Where the paper says it

4 The VOXBLINK2-AVSE Training Corpus 4. 1 Source Corpus and Audio-Visual Validation We construct VOXBLINK2-AVSE from the in-thewild VoxBlink2 collection (Lin et al.

03964#pg2> This curated corpus emphasizes reliable visual target selection rather than raw hours alone, providing a distinct operating point for AVSE training <ref:2608. p. 4
Where the paper says it

This task-ready representation avoids repeating decoding, synchronization, ROI validation, and acoustic screening inside each AVSE training pipeline. Together with speaker-disjoint manifests, the corpus emphasizes reliable visual target selection rather than raw hours alone.

03964#pg2> The final evaluation tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608. p. 7
Where the paper says it

The ablations show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion. Speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly test whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance.

03964#pg2>

Conclusion on Extractor Design

The conclusion is that REAL-2MIX turns audio-visual TSE into a paired identity test where the same jointly recorded mixture must yield A under A’s lip cue and B under B’s lip cue <ref:2608.03964#pg2> The combination of speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608. p. 7

Where the paper says it

The ablations show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion. Speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly test whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance.

03964#pg2> The auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion <ref:2608. p. 7
Where the paper says it

VOXBLINK2-AVSE supplies 766 hours of identity-diverse aligned training data, while frozen AV-HuBERT features, speaker margin training, and layer-wise FiLM make the target cue explicit throughout the separator. The ablations show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion.

03964#pg2> The paper concludes that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608. p. 7
Where the paper says it

VOXBLINK2-AVSE supplies 766 hours of identity-diverse aligned training data, while frozen AV-HuBERT features, speaker margin training, and layer-wise FiLM make the target cue explicit throughout the separator. The ablations show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion.

03964#pg2>

References

Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. LRS3-TED: A large-scale dataset for visual speech recognition <ref:2608. p. 8

Where the paper says it

References Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. LRS3-TED: A large-scale dataset for visual speech recognition.

03964#pg2> Adrian Bulat and Georgios Tzimiropoulos. p. 8
Where the paper says it

arXiv preprint. Adrian Bulat and Georgios Tzimiropoulos.

2017. How far are we from solving the 2d & 3d face alignment problem? p. 8
Where the paper says it

Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?

(and a dataset of 230,000 3d facial landmarks). p. 8
Where the paper says it

How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks).

In IEEE International Conference on Computer Vision, pages 1021–1030 <ref:2608. p. 8
Where the paper says it

(and a dataset of 230,000 3d facial landmarks). In IEEE International Conference on Computer Vision, pages 1021–1030.

03964#pg2> Qiong Cao, Li Shen, Weidi Xie, Omkar M. p. 8
Where the paper says it

In IEEE International Conference on Computer Vision, pages 1021–1030. Qiong Cao, Li Shen, Weidi Xie, Omkar M.

Parkhi, and Andrew Zisserman. p. 8
Where the paper says it

Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman.

2018. VGGFace2: A dataset for recognising faces across pose and age. p. 8
Where the paper says it

Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age.

In IEEE International Conference on Automatic Face & Gesture Recognition, pages 67–74 <ref:2608. p. 8
Where the paper says it

VGGFace2: A dataset for recognising faces across pose and age. In IEEE International Conference on Automatic Face & Gesture Recognition, pages 67–74.

03964#pg2> Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. p. 8
Where the paper says it

In IEEE International Conference on Automatic Face & Gesture Recognition, pages 67–74. Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman.

2017. Lip reading sentences in the wild. p. 8
Where the paper says it

Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild.

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3444–3453 <ref:2608. p. 8
Where the paper says it

Lip reading sentences in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3444–3453.

03964#pg2> Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. p. 8
Where the paper says it

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3444–3453. Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.

2020. RetinaFace: Single-shot multi-level face localisation in the wild. p. 8
Where the paper says it

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. RetinaFace: Single-shot multi-level face localisation in the wild.

In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5202–5211 <ref:2608. p. 8
Where the paper says it

RetinaFace: Single-shot multi-level face localisation in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5202–5211.

03964#pg2> Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. p. 8
Where the paper says it

In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5202–5211. Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck.

2020. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification. p. 8
Where the paper says it

Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification.

In Interspeech, pages 3830–3834 <ref:2608. p. 8
Where the paper says it

ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Interspeech, pages 3830–3834.

03964#pg2> Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. p. 8
Where the paper says it

The Annals of Statistics, 7(1):1–26. Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T.

Freeman, and Michael Rubinstein. p. 8
Where the paper says it

Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein.

2018. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. p. 8
Where the paper says it

Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.

ACM Transactions on Graphics, 37(4):1–11 <ref:2608. p. 8
Where the paper says it

Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics, 37(4):1–11.

03964#pg2> Ruohan Gao and Kristen Grauman. p. 8
Where the paper says it

ACM Transactions on Graphics, 37(4):1–11. Ruohan Gao and Kristen Grauman.

2021. Visualvoice: Audio-visual speech separation with cross-modal consistency. p. 8
Where the paper says it

Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15490–15500 <ref:2608. p. 8
Where the paper says it

Visualvoice: Audio-visual speech separation with cross-modal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15490–15500.

03964#pg2> Vladimir I. p. 8
Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15490–15500. Vladimir I.

Levenshtein. p. 6
Where the paper says it

Before scoring, the reference and hypothesis are stripped of Chinese and ASCII punctuation, consecutive whitespace is collapsed, and whitespace is excluded from character tokens. The micro-averaged CER is CER = P (i,q)∈P ED(A(ˆsi,q), yi,q) P (i,q)∈P |yi,q| , (6) where ED is Levenshtein distance (Levenshtein, 1966).

1966. Binary codes capable of correcting deletions, insertions, and reversals. p. 8
Where the paper says it

Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals.

Soviet Physics Doklady, 10(8):707–710 <ref:2608. p. 8
Where the paper says it

Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710.

03964#pg2> Shaole Li, Shuai Wang, Jiangyu Han, Ke Zhang, Wupeng Wang, and Haizhou Li. p. 8
Where the paper says it

Soviet Physics Doklady, 10(8):707–710. Shaole Li, Shuai Wang, Jiangyu Han, Ke Zhang, Wupeng Wang, and Haizhou Li.

2025. REAL-T: Real conversational mixtures for target speaker extraction. p. 8
Where the paper says it

Shaole Li, Shuai Wang, Jiangyu Han, Ke Zhang, Wupeng Wang, and Haizhou Li. REAL-T: Real conversational mixtures for target speaker extraction.

In Interspeech 2025, pages 1923–1927 <ref:2608. p. 8
Where the paper says it

REAL-T: Real conversational mixtures for target speaker extraction. In Interspeech 2025, pages 1923–1927.

03964#pg2> Yuke Lin, Ming Cheng, Fulin Zhang, Yingying Gao, Shilei Zhang, and Ming Li. p. 8
Where the paper says it

In Interspeech 2025, pages 1923–1927. Yuke Lin, Ming Cheng, Fulin Zhang, Yingying Gao, Shilei Zhang, and Ming Li.

2024. Voxblink2: A 100k+ speaker recognition corpus and the open-set speaker-identification benchmark. p. 8
Where the paper says it

Yuke Lin, Ming Cheng, Fulin Zhang, Yingying Gao, Shilei Zhang, and Ming Li. Voxblink2: A 100k+ speaker recognition corpus and the open-set speaker-identification benchmark.

In Proc. Interspeech 2024, pages 4263–4267 <ref:2608. p. 8
Where the paper says it

Voxblink2: A 100k+ speaker recognition corpus and the open-set speaker-identification benchmark. Interspeech 2024, pages 4263–4267.

03964#pg2> Yi Luo and Nima Mesgarani. p. 8
Where the paper says it

Interspeech 2024, pages 4263–4267. Yi Luo and Nima Mesgarani.

2019. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation. p. 8
Where the paper says it

Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation.

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–1266 <ref:2608. p. 8
Where the paper says it

Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256– 1266.

03964#pg2> Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. p. 8
Where the paper says it

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256– 1266. Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.

2017. Voxceleb: A large-scale speaker identification dataset. p. 8
Where the paper says it

Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker identification dataset.

In Interspeech, pages 2616–2620 <ref:2608. p. 8
Where the paper says it

Voxceleb: A large-scale speaker identification dataset. In Interspeech, pages 2616–2620.

03964#pg2> Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Atsunori Ogawa, and Tomohiro Nakatani. p. 8
Where the paper says it

In Interspeech, pages 2616–2620. Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Atsunori Ogawa, and Tomohiro Nakatani.

2019. Multimodal speakerbeam: Single channel target speech extraction with audio-visual speaker clues. p. 8
Where the paper says it

Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Atsunori Ogawa, and Tomohiro Nakatani. Multimodal speakerbeam: Single channel target speech extraction with audio-visual speaker clues.

In Interspeech, pages 2718–2722 <ref:2608. p. 8
Where the paper says it

Multimodal speakerbeam: Single channel target speech extraction with audio-visual speaker clues. In Interspeech, pages 2718–2722.

03964#pg2> Zexu Pan, Meng Ge, and Haizhou Li. p. 8
Where the paper says it

In Interspeech, pages 2718–2722. Zexu Pan, Meng Ge, and Haizhou Li.

2022. USEV: Universal speaker extraction with visual cue. p. 8
Where the paper says it

Zexu Pan, Meng Ge, and Haizhou Li. USEV: Universal speaker extraction with visual cue.

IEEE/ACM Transactions on Audio, Speech and Language Processing, 30:3032–3045 <ref:2608. p. 8
Where the paper says it

USEV: Universal speaker extraction with visual cue. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:3032–3045.

03964#pg2> Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. p. 8
Where the paper says it

In IEEE Automatic Speech Recognition and Understanding Workshop, pages 1–8. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville.

2018. FiLM: Visual reasoning with a general conditioning layer. p. 8
Where the paper says it

Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer.

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32 <ref:2608. p. 8
Where the paper says it

FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.

03964#pg2> Chandan KA Reddy, Vishak Gopal, and Ross Cutler. p. 8
Where the paper says it

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. Chandan KA Reddy, Vishak Gopal, and Ross Cutler.

2021. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. p. 8
Where the paper says it

Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors.

In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493–6497 <ref:2608. p. 8
Where the paper says it

Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493–6497.

03964#pg2> Florian Schroff, Dmitry Kalenichenko, and James Philbin. p. 8
Where the paper says it

In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493–6497. Florian Schroff, Dmitry Kalenichenko, and James Philbin.

2015. FaceNet: A unified embedding for face recognition and clustering. p. 8
Where the paper says it

Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clustering.

In IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823 <ref:2608. p. 8
Where the paper says it

FaceNet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823.

03964#pg2> Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. p. 8
Where the paper says it

In IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823. Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed.

2022. Learning audio-visual speech representation by masked multimodal cluster prediction. p. 8
Where the paper says it

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction.

In International Conference on Learning Representations <ref:2608. p. 8
Where the paper says it

Learning audio-visual speech representation by masked multimodal cluster prediction. In International Conference on Learning Representations.

03964#pg2> Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. p. 8
Where the paper says it

In International Conference on Learning Representations. Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin.

2026. Qwen3-ASR technical report. p. 8
Where the paper says it

Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-ASR technical report.

arXiv preprint <ref:2608. p. 8
Where the paper says it

LRS3-TED: A large-scale dataset for visual speech recognition. arXiv preprint.

03964#pg2> Fei Su, Cancan Li, Ming Li, and Juan Liu. p. 8
Where the paper says it

arXiv preprint. Fei Su, Cancan Li, Ming Li, and Juan Liu.

2026. M2savsr: Modality-aware multi-view self-supervised representation for robust audio-visual speech recognition. p. 8
Where the paper says it

Fei Su, Cancan Li, Ming Li, and Juan Liu. M2savsr: Modality-aware multi-view self-supervised representation for robust audio-visual speech recognition.

arXiv preprint arXiv:2606. p. 8
Where the paper says it

M2savsr: Modality-aware multi-view self-supervised representation for robust audio-visual speech recognition. arXiv preprint arXiv:2606.

05763 <ref:2608. p. 0
Where the paper says it

We complement this benchmark with VOXBLINK2- AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training, and with an AV-HuBERTconditioned TF-GridNet. Our contributions are threefold: • We release REAL-2MIX, a controlled realscene test set benchmark with jointly recorded overlap, scene-matched solo references, multiple distances, and three synchronized viewarXiv:2608.

03964#pg2> Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. p. 9
Where the paper says it

Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking? p. 9

Where the paper says it

Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking?

2021. Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection. p. 9
Where the paper says it

Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection.

In ACM Multimedia, pages 3927–3935 <ref:2608. p. 9
Where the paper says it

exploring long-term temporal features for audio-visual active speaker detection. In ACM Multimedia, pages 3927–3935.

03964#pg2> Andrew J. p. 8
Where the paper says it

References Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. LRS3-TED: A large-scale dataset for visual speech recognition.

Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. p. 9
Where the paper says it

In ACM Multimedia, pages 3927–3935. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm.

IEEE Transactions on Information Theory, 13(2):260–269 <ref:2608. p. 9
Where the paper says it

Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):260–269.

03964#pg2> Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe. p. 9
Where the paper says it

IEEE Transactions on Information Theory, 13(2):260–269. Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe.

2023. TF-GridNet: Integrating full- and subband modeling for speech separation. p. 9
Where the paper says it

Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe. TF-GridNet: Integrating full- and subband modeling for speech separation.

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:3221–3236 <ref:2608. p. 9
Where the paper says it

TF-GridNet: Integrating full- and subband modeling for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:3221–3236.

03964#pg2> Bang Zeng and Ming Li. p. 9
Where the paper says it

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:3221–3236. Bang Zeng and Ming Li.

2025. USEF-TSE: Universal speaker embedding free target speaker extraction. p. 9
Where the paper says it

Bang Zeng and Ming Li. USEF-TSE: Universal speaker embedding free target speaker extraction.

IEEE Transactions on Audio, Speech and Language Processing, 33:2110–2124 <ref:2608. p. 9
Where the paper says it

USEF-TSE: Universal speaker embedding free target speaker extraction. IEEE Transactions on Audio, Speech and Language Processing, 33:2110–2124.

03964#pg2> Kateˇrina Žmolíková, Marc Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget, and Jan Cernock ˇ y. p. 9
Where the paper says it

IEEE Transactions on Audio, Speech and Language Processing, 33:2110–2124. Kateˇrina Žmolíková, Marc Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget, and Jan Cernock ˇ y.

2019. SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures. p. 9
Where the paper says it

Kateˇrina Žmolíková, Marc Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget, and Jan Cernock ˇ y. ` SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures.

IEEE Journal of Selected Topics in Signal Processing, 13(4):800–814 <ref:2608. p. 9
Where the paper says it

` SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures. IEEE Journal of Selected Topics in Signal Processing, 13(4):800–814.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Visual target extraction can be made identity faithful by conditioning a separator on visual cues using frozen, 1280-dimensional projected AV-HuBERT features and layer-wise feature modulation. This allows for explicit target information to be injected throughout the separator rather than relying on single early fusion points.

  2. The system can be trained with an auxiliary speaker margin loss that encourages the output waveform to be closer to the target than the interferer in a frozen 192-dimensional ECAPA-TDNN speaker encoder space. This supervision directly addresses whether changing the visual target changes the extracted identity, rather than only whether one prompted output sounds clean.

  3. The improved system can perform strict identity evaluation using an OSD-first approach where outputs below a certain residual overlap threshold are classified as wrong speaker based on a positive or non-positive speaker margin. This provides strict correct and wrong speaker classifications, offering a more robust measure than just content character error rate.

Abstract

Audio-visual target speaker extraction should recover the person indicated by the video, rather than repeatedly return the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin benchmark of jointly recorded two-speaker mixtures with synchronized multi-view video and scene-matched solo references. Its 77 scenes contain 7,598 clips; filtering yields 6,038 evaluable mixtures, each queried for both speakers, giving 12,076 paired target queries. We also curate VOXBLINK2-AVSE, comprising 250,828 synchronized audio--lip pairs. A seven-system TF-GridNet study examines visual representations, training corpora, and conditioning strategies. We evaluate transcription accuracy, speaker identity, residual overlap, and perceptual quality. The AVSR early-fusion system achieves 22.99% CER, and 93.38% of its outputs are more similar to the target speaker than to the interferer. Yet only 42.49% satisfy both this identity criterion and the overlap detector's single-speaker criterion; both target queries satisfy these criteria for just 20.77% of mixtures. A separate Real-AVSE challenge study examines development-remix adaptation. The benchmark reveals that favorable content and identity scores need not coincide with low detected overlap, motivating paired-target evaluation.

Sources

Related papers