REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction

summary

The gist

Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice…

In short

REAL-2MIX benchmarks identity-faithful audio-visual target speaker extraction using real, jointly recorded mixtures with synchronized multi-view video. It tests if a model correctly identifies the target speaker based on visual cues during joint recordings, proving that simply following lip cues is insufficient.

Key concepts

REAL-2MIX
This is a benchmark created for testing how well models can perform audio-visual target speaker extraction when the audio comes from real mixtures recorded with synchronized multi-view video. It specifically checks if the model follows visual lip cues during these joint recordings.
Audio-Visual Target Speaker Extraction (AVTSE)
This is a task where a system takes an overlapping recording of two speakers and the video, and must identify which speaker's voice is the target speaker. The goal is to find the target speaker's waveform based on both audio and visual information.
Identity Faithfulness
This refers to how accurately a model preserves the true identity of a person when performing a task. In this context, it means checking if an extractor correctly identifies the intended target speaker even when visual cues (like lip movements) are present in the joint recording.

Terminology used across episodes

This episode discusses

The paper

REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction · Read on arXiv

School of Cyber Science and Engineering, Wuhan University · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen

Audio-visual target speaker extraction should recover the person indicated by the video, rather than repeatedly return the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin benchmark of jointly recorded two-speaker mixtures with synchronized multi-view video and scene-matched solo references. Its 77 scenes contain 7,598 clips; filtering yields 6,038 evaluable mixtures, each queried for both speakers, giving 12,076 paired target queries. We also curate VOXBLINK2-AVSE, comprising 250,828 synchronized audio--lip pairs. A seven-system TF-GridNet study examines visual representations, training corpora, and conditioning strategies. We evaluate transcription accuracy, speaker identity, residual overlap, and perceptual quality. The AVSR early-fusion system achieves 22.99% CER, and 93.38% of its outputs are more similar to the target speaker than to the interferer. Yet only 42.49% satisfy both this identity criterion and the overlap detector's single-speaker criterion; both target queries satisfy these criteria for just 20.77% of mixtures. A separate Real-AVSE challenge study examines development-remix adaptation. The benchmark reveals that favorable content and identity scores need not coincide with low detected overlap, motivating paired-target evaluation.

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction".

Dev: Here's the gist of the paper. Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice <ref:2608.03964#pg2> The paper introduces REAL-2MIX, a Mandarin AVTSE benchmark of jointly recorded real twospeaker mixtures with synchronized multi-view video <ref:2608.03964#pg2> This work matters because it tests identity faithfulness in audio-visual target speaker extraction by evaluating whether a model follows the requested lip cue during joint recordings <ref:2608.03964#pg2> Benchmark Construction and Data Curation REAL-2MIX is introduced to benchmark identity-faithful AV-TSE on purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg2> The task definition involves predicting the target speaker waveform sˆq = fθ(x, vq) given an overlap recording x(t) = sA(t)+sB(t)+n(t) <ref:2608.03964#pg2> Each scene records two stationary participants in a shared environment and includes preceding A-only and B-only stages to provide in-scene speaker references <ref:2608.03964#pg2> The recording protocol uses distances of 0.5, 1.5, and 3 m and side-view angles of 30◦, 45◦, and 60◦ <ref:2608.03964#pg2> The complete processed release contains 77 scene directories from 14 unique speaker dyads with a total of 7,598 mono clips <ref:2608.03964#pg2> Training Signal Architecture The extractor uses frozen, 1,280-dimensional projected AV-HuBERT features and target-conditioned training <ref:2608.03964#pg2> The architecture employs a single-output TF-GridNet where the visual stream is processed through input-level fusion and layer-wise FiLM modulation <ref:2608.03964#pg2> Specifically, the frozen visual branch produces a 1,280-dimensional visual sequence which is linearly projected to 1,280 dimensions and rescaled by a learned scalar <ref:2608.03964#pg2> The separator contains 6 GridNet blocks with 128 channels, where the visual sequence is supplied directly to the layer-specific FiLM generators <ref:2608.03964#pg2> Evaluation Metrics and Analysis The performance is jointly evaluated using Qwen3-ASR-1.7B CER for content and WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD) for target identity <ref:2608.03964#pg2> Speaker similarity is reported as SpkSim, which compares the embedding of the output sˆi,q with the corresponding solo-reference mean e r i,q <ref:2608.03964#pg2> The OSD-first strict identity evaluation uses an automatic residualoverlap proxy before speaker identification to classify outputs into mutually exclusive classes <ref:2608.03964#pg2> Ablation results show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608.03964#pg2> Corpus Complementarity The work complements REAL-2MIX with VOXBLINK2-AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training <ref:2608.03964#pg2> VOXBLINK2-AVSE is constructed from the in-thewild VoxBlink2 collection but undergoes audio-visual validity checks and DNSMOS filtering to ensure quality <ref:2608.03964#pg2> This curated corpus emphasizes reliable visual target selection rather than raw hours alone, providing a distinct operating point for AVSE training <ref:2608.03964#pg2> The final evaluation tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608.03964#pg2> The conclusion is that REAL-2MIX turns audio-visual TSE into a paired identity test where the same jointly recorded mixture must yield A under A’s lip cue and B under B’s lip cue <ref:2608.03964#pg2> The combination of speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608.03964#pg2> The auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion <ref:2608.03964#pg2> The paper concludes that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608.03964#pg2> References Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.

Dev: LRS3-TED: A large-scale dataset for visual speech recognition <ref:2608.03964#pg2> Adrian Bulat and Georgios Tzimiropoulos.

Paper summary: Rosa: So we're looking at this paper called REAL-2MIX, which is about identity faithfulness in audio-visual target speaker extraction <ref:2608.03964#pg1,audio-visual target speaker extraction>. Essentially, the main problem they tackle is that sometimes an audio visual target speaker extractor should pull the right person based on what you see in the video, but a separator can totally ignore that visual cue and just keep outputting whatever voice sounds loudest.

Dev: Right. It sets up this benchmark called REAL-2MIX to test exactly how faithful the identity extraction is when you have these real mixtures recorded with synchronized multi-view video <ref:2608.03964#pg1,with synchronized multi-view video>. The paper claims this work matters because it checks whether a model actually follows the requested lip cue during those joint recordings <ref:2608.03964#pg2>.

Taro: So, they’re not just looking at isolated audio or video; they are using joint recordings where you have both speakers present and synchronized video, which is a pretty complex setup to get right <ref:2608.03964#pg1>.

Rosa: Exactly. They build this benchmark using purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg1>. The task is to predict the target speaker waveform given an overlap recording, which is basically trying to figure out who is talking in that noisy mix <ref:2608.03964#pg1>.

Dev: And they set up the scenes with two stationary participants in a shared environment, and they include those preceding A only and B only stages to give you references for each speaker right there on the recording <ref:2608.03964#pg1>.

Taro: They also specify distances like zero point five meters, one point five meters, and three meters away, and side-view angles of thirty degrees up to sixty degrees for the cameras <ref:2608.03964#pg1>.

Rosa: After processing all that, they have seventy-seven scene directories from fourteen different speaker pairs with a total of seven thousand five hundred ninety-eight mono clips, including six thousand forty-two dual annotated mixtures <ref:2608.03964#pg1>. That’s the dataset size for this benchmark.

Dev: Now, looking at how they train their extractor, they use frozen one thousand two hundred eighty dimensional projected audio visual HuBERT features and target conditioned training <ref:2608.03964#pg2>. The architecture is a single output TF-GridNet where the visual stream goes through input level fusion and layer wise FiLM modulation <ref:2608.03964#pg2>.

Taro: That means the visual branch produces a one thousand two hundred eighty dimensional sequence, which they linearly project to one thousand two hundred eighty dimensions and then rescale it by a learned scalar <ref:2608.03964#pg2>. The separator itself has six GridNet blocks with one hundred twenty eight channels, where the visual sequence is supplied directly to the layer specific FiLM generators <ref:2608.03964#pg2>.

Paper summary: Rosa: They use frozen features, but they condition the training using this visual input in a very specific way, which is interesting for how much reliance a model has on what it sees versus just what it hears <ref:2608.03964#pg2>.

Dev: And when we look at evaluation, they use Qwen three ASR one point seven billion CER for the content and WeSpeaker ResNet thirty four plus Overlapped Speech Detection for the target identity <ref:2608.03964#pg2>. Speaker similarity is reported as SpkSim, which compares the embedding of their output speaker with the corresponding solo reference mean <ref:2608.03964#pg2>.

Taro: They also use an OSD first strict identity evaluation that uses an automatic residual overlap proxy before identifying speakers to sort outputs into mutually exclusive classes <ref:2608.03964#pg2>. The ablation results show that having matched training data and suppressing interferer sound matters more than just making the GridNet back end bigger <ref:2608.03964#pg2>.

Rosa: So, what this means for us is that they are testing if an extractor can follow the visual intent while also suppressing overlap, and preserving the exact utterance requested <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s a pretty rigorous test for identity faithfulness.

Dev: It really is. And to complement this work, they released something called VOXBLINK two AVSE, which is a large identity disjoint audio lip corpus that they curated for training <ref:2608.03964#pg2>. This corpus comes from the in the wild VoxBlink two collection but they ran it through audio visual validity checks and DNSMOS filtering to ensure quality <ref:2608.03964#pg2>.

Taro: That curated corpus is important because it puts a focus on reliable visual target selection instead of just having raw hours, which gives a different operating point for training the AVSE model <ref:2608.03964#pg2>.

Rosa: So the final evaluation tests whether an extractor follows the visual intent, suppresses overlap, and preserves the requested utterance in this new context <ref:two thousand six hundred eight point zero three nine six four#pg1. It seems like they've put together a pretty comprehensive test for this kind of extraction task.

Dev: The conclusion is that REAL-2MIX turns audio visual target speaker extraction into a paired identity test where the same jointly recorded mixture must yield speaker A under A's lip cue and speaker B under B's lip cue <ref:two thousand six hundred eight point zero three nine six four#pg1. The combination of speaker margin, OSD first strict identity rates, and Qwen three ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:two thousand six hundred eight point zero three nine six four#pg1.

Paper summary: Taro: But I gotta point out a limitation mentioned in the paper. They state that the auxiliary speaker loss requires a controlled rerun before they can draw a causal conclusion about its effectiveness <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s not an immediate proof of concept for that specific loss term.

Rosa: That's fair. So, while they show the potential, they are being cautious about drawing definitive conclusions about every single component in the training setup <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s a careful look at how these components interact when you test for identity faithfulness.

Dev: So we've seen that this benchmark is built on real mixtures with synchronized video, and that they use a specific TF-GridNet architecture conditioned by visual features <ref:two thousand six hundred eight point zero three nine six four#pg2. It’s testing the core idea of whether the model respects the visual instruction in those complex audio visual scenarios.

Taro: And it seems like they emphasize that having good training data and suppressing interferer sound is more important than just having a bigger GridNet back end to get better results <ref:two thousand six hundred eight point zero three nine six four#pg2. That tells us where the real bottleneck might be in building these systems.

Rosa: Exactly. And they also look at how different methods compare, like Conv-TasNet versus TF-GridNet, showing that the time frequency modeling in the GridNet is adding something concrete to speech separation <ref:two thousand six hundred eight point zero three nine six four#pg2.

Dev: And they are using metrics like Qwen three ASR one point seven billion CER for content and WeSpeaker ResNet thirty four plus Overlapped Speech Detection for identity, which gives a solid way to measure both the accuracy of what was extracted and the faithfulness of that extraction <ref:two thousand six hundred eight point zero three nine six four#pg2.

Taro: So, when you look at this whole piece, it’s about how we train these models to handle the messy reality of real-world audio visual data, where things can get overlapped or misdirected <ref:two thousand six hundred eight point zero three nine six four#pg1. It sets a standard for what we consider identity faithfulness in this area.

Rosa: And to wrap up, the work on REAL-2MIX gives us a clear way to test if an audio visual target speaker extractor is actually doing what it's supposed to do when things get complicated <ref:two thousand six hundred eight point zero three nine six four#pg1. We'll be looking at how these challenges are handled in the next part of the show.

Conclusion: Rosa: So we’re wrapping up with some thoughts on this paper, REAL-2MIX, by Afouras, Chung, and Zisserman <ref:2608.03964#pg1>. Basically they’re testing if these models actually follow the visual cues when trying to pick a target speaker from a mix of audio and video.

Dev: Right. It’s about checking identity faithfulness in audio visual target speaker extraction, which is a pretty specific test because you want the model to listen to what it sees, not just what it hears <ref:2608.03964#pg2>.

Taro: The whole point of REAL-2MIX is this paired identity test where the same mixture has to yield speaker A under A's lip cue and speaker B under B's lip cue <ref:2608.03964#pg1>. It’s a very strict way to measure if the model is actually doing what it’s supposed to be doing <ref:2608.03964#pg1>.

Rosa: And they use speaker margin, overlap detection, and some specific metrics like Qwen three ASR CER to see if the whole system works together <ref:two thousand six hundred eight point zero three nine six four#pg2. It’s a pretty comprehensive way to check if an extractor follows visual intent while suppressing overlap and preserving the requested utterance <ref:two thousand six hundred eight point zero three nine six four#pg1.

Dev: I think what this means for us is that when we build these systems, we can't just look at audio quality or video resolution anymore; we have to test if the model respects those visual instructions under realistic, messy conditions <ref:two thousand six hundred eight point zero three nine six four#pg1.

Taro: Exactly. The limitation they mentioned is that they need a controlled rerun for the auxiliary speaker loss before they can really claim anything causal about it <ref:two thousand six hundred eight point zero three nine six four#pg1. So we have to be careful not to draw conclusions too fast <ref:two thousand six hundred eight point zero three nine six four#pg1.

Rosa: True. It’s a careful look at how these different parts interact when you test for that kind of fidelity <ref:two thousand six hundred eight point zero three nine six four#pg1. But this benchmark gives us a solid standard for what identity faithfulness actually looks like in the audio visual space <ref:two thousand six hundred eight point zero three nine six four#pg1.

Dev: So, we’ve seen how they build the test, what the results were with those specific metrics, and where they might be cautious about their conclusions <ref:two thousand six hundred eight point zero three nine six four#pg2. Now we need to see how this idea of identity faithfulness plays out when you move beyond these controlled mixed recordings <ref:two thousand six hundred eight point zero three nine six four#pg1.

More episodes

← Home