REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction".
Dev: Here's the gist of the paper. Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice <ref:2608.03964#pg2> The paper introduces REAL-2MIX, a Mandarin AVTSE benchmark of jointly recorded real twospeaker mixtures with synchronized multi-view video <ref:2608.03964#pg2> This work matters because it tests identity faithfulness in audio-visual target speaker extraction by evaluating whether a model follows the requested lip cue during joint recordings <ref:2608.03964#pg2> Benchmark Construction and Data Curation REAL-2MIX is introduced to benchmark identity-faithful AV-TSE on purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg2> The task definition involves predicting the target speaker waveform sˆq = fθ(x, vq) given an overlap recording x(t) = sA(t)+sB(t)+n(t) <ref:2608.03964#pg2> Each scene records two stationary participants in a shared environment and includes preceding A-only and B-only stages to provide in-scene speaker references <ref:2608.03964#pg2> The recording protocol uses distances of 0.5, 1.5, and 3 m and side-view angles of 30◦, 45◦, and 60◦ <ref:2608.03964#pg2> The complete processed release contains 77 scene directories from 14 unique speaker dyads with a total of 7,598 mono clips <ref:2608.03964#pg2> Training Signal Architecture The extractor uses frozen, 1,280-dimensional projected AV-HuBERT features and target-conditioned training <ref:2608.03964#pg2> The architecture employs a single-output TF-GridNet where the visual stream is processed through input-level fusion and layer-wise FiLM modulation <ref:2608.03964#pg2> Specifically, the frozen visual branch produces a 1,280-dimensional visual sequence which is linearly projected to 1,280 dimensions and rescaled by a learned scalar <ref:2608.03964#pg2> The separator contains 6 GridNet blocks with 128 channels, where the visual sequence is supplied directly to the layer-specific FiLM generators <ref:2608.03964#pg2> Evaluation Metrics and Analysis The performance is jointly evaluated using Qwen3-ASR-1.7B CER for content and WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD) for target identity <ref:2608.03964#pg2> Speaker similarity is reported as SpkSim, which compares the embedding of the output sˆi,q with the corresponding solo-reference mean e r i,q <ref:2608.03964#pg2> The OSD-first strict identity evaluation uses an automatic residualoverlap proxy before speaker identification to classify outputs into mutually exclusive classes <ref:2608.03964#pg2> Ablation results show that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608.03964#pg2> Corpus Complementarity The work complements REAL-2MIX with VOXBLINK2-AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training <ref:2608.03964#pg2> VOXBLINK2-AVSE is constructed from the in-thewild VoxBlink2 collection but undergoes audio-visual validity checks and DNSMOS filtering to ensure quality <ref:2608.03964#pg2> This curated corpus emphasizes reliable visual target selection rather than raw hours alone, providing a distinct operating point for AVSE training <ref:2608.03964#pg2> The final evaluation tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608.03964#pg2> The conclusion is that REAL-2MIX turns audio-visual TSE into a paired identity test where the same jointly recorded mixture must yield A under A’s lip cue and B under B’s lip cue <ref:2608.03964#pg2> The combination of speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608.03964#pg2> The auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion <ref:2608.03964#pg2> The paper concludes that matched training data and interferer suppression matter more than simply enlarging the GridNet back end <ref:2608.03964#pg2> References Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.
Dev: LRS3-TED: A large-scale dataset for visual speech recognition <ref:2608.03964#pg2> Adrian Bulat and Georgios Tzimiropoulos.
Paper summary: Rosa: So we're looking at this paper called REAL-2MIX, which is about identity faithfulness in audio-visual target speaker extraction <ref:2608.03964#pg1,audio-visual target speaker extraction>. Essentially, the main problem they tackle is that sometimes an audio visual target speaker extractor should pull the right person based on what you see in the video, but a separator can totally ignore that visual cue and just keep outputting whatever voice sounds loudest.
Dev: Right. It sets up this benchmark called REAL-2MIX to test exactly how faithful the identity extraction is when you have these real mixtures recorded with synchronized multi-view video <ref:2608.03964#pg1,with synchronized multi-view video>. The paper claims this work matters because it checks whether a model actually follows the requested lip cue during those joint recordings <ref:2608.03964#pg2>.
Taro: So, they’re not just looking at isolated audio or video; they are using joint recordings where you have both speakers present and synchronized video, which is a pretty complex setup to get right <ref:2608.03964#pg1>.
Rosa: Exactly. They build this benchmark using purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg1>. The task is to predict the target speaker waveform given an overlap recording, which is basically trying to figure out who is talking in that noisy mix <ref:2608.03964#pg1>.
Dev: And they set up the scenes with two stationary participants in a shared environment, and they include those preceding A only and B only stages to give you references for each speaker right there on the recording <ref:2608.03964#pg1>.
Taro: They also specify distances like zero point five meters, one point five meters, and three meters away, and side-view angles of thirty degrees up to sixty degrees for the cameras <ref:2608.03964#pg1>.
Rosa: After processing all that, they have seventy-seven scene directories from fourteen different speaker pairs with a total of seven thousand five hundred ninety-eight mono clips, including six thousand forty-two dual annotated mixtures <ref:2608.03964#pg1>. That’s the dataset size for this benchmark.
Dev: Now, looking at how they train their extractor, they use frozen one thousand two hundred eighty dimensional projected audio visual HuBERT features and target conditioned training <ref:2608.03964#pg2>. The architecture is a single output TF-GridNet where the visual stream goes through input level fusion and layer wise FiLM modulation <ref:2608.03964#pg2>.
Taro: That means the visual branch produces a one thousand two hundred eighty dimensional sequence, which they linearly project to one thousand two hundred eighty dimensions and then rescale it by a learned scalar <ref:2608.03964#pg2>. The separator itself has six GridNet blocks with one hundred twenty eight channels, where the visual sequence is supplied directly to the layer specific FiLM generators <ref:2608.03964#pg2>.
Paper summary: Rosa: They use frozen features, but they condition the training using this visual input in a very specific way, which is interesting for how much reliance a model has on what it sees versus just what it hears <ref:2608.03964#pg2>.
Dev: And when we look at evaluation, they use Qwen three ASR one point seven billion CER for the content and WeSpeaker ResNet thirty four plus Overlapped Speech Detection for the target identity <ref:2608.03964#pg2>. Speaker similarity is reported as SpkSim, which compares the embedding of their output speaker with the corresponding solo reference mean <ref:2608.03964#pg2>.
Taro: They also use an OSD first strict identity evaluation that uses an automatic residual overlap proxy before identifying speakers to sort outputs into mutually exclusive classes <ref:2608.03964#pg2>. The ablation results show that having matched training data and suppressing interferer sound matters more than just making the GridNet back end bigger <ref:2608.03964#pg2>.
Rosa: So, what this means for us is that they are testing if an extractor can follow the visual intent while also suppressing overlap, and preserving the exact utterance requested <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s a pretty rigorous test for identity faithfulness.
Dev: It really is. And to complement this work, they released something called VOXBLINK two AVSE, which is a large identity disjoint audio lip corpus that they curated for training <ref:2608.03964#pg2>. This corpus comes from the in the wild VoxBlink two collection but they ran it through audio visual validity checks and DNSMOS filtering to ensure quality <ref:2608.03964#pg2>.
Taro: That curated corpus is important because it puts a focus on reliable visual target selection instead of just having raw hours, which gives a different operating point for training the AVSE model <ref:2608.03964#pg2>.
Rosa: So the final evaluation tests whether an extractor follows the visual intent, suppresses overlap, and preserves the requested utterance in this new context <ref:two thousand six hundred eight point zero three nine six four#pg1. It seems like they've put together a pretty comprehensive test for this kind of extraction task.
Dev: The conclusion is that REAL-2MIX turns audio visual target speaker extraction into a paired identity test where the same jointly recorded mixture must yield speaker A under A's lip cue and speaker B under B's lip cue <ref:two thousand six hundred eight point zero three nine six four#pg1. The combination of speaker margin, OSD first strict identity rates, and Qwen three ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:two thousand six hundred eight point zero three nine six four#pg1.
Paper summary: Taro: But I gotta point out a limitation mentioned in the paper. They state that the auxiliary speaker loss requires a controlled rerun before they can draw a causal conclusion about its effectiveness <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s not an immediate proof of concept for that specific loss term.
Rosa: That's fair. So, while they show the potential, they are being cautious about drawing definitive conclusions about every single component in the training setup <ref:two thousand six hundred eight point zero three nine six four#pg1. It’s a careful look at how these components interact when you test for identity faithfulness.
Dev: So we've seen that this benchmark is built on real mixtures with synchronized video, and that they use a specific TF-GridNet architecture conditioned by visual features <ref:two thousand six hundred eight point zero three nine six four#pg2. It’s testing the core idea of whether the model respects the visual instruction in those complex audio visual scenarios.
Taro: And it seems like they emphasize that having good training data and suppressing interferer sound is more important than just having a bigger GridNet back end to get better results <ref:two thousand six hundred eight point zero three nine six four#pg2. That tells us where the real bottleneck might be in building these systems.
Rosa: Exactly. And they also look at how different methods compare, like Conv-TasNet versus TF-GridNet, showing that the time frequency modeling in the GridNet is adding something concrete to speech separation <ref:two thousand six hundred eight point zero three nine six four#pg2.
Dev: And they are using metrics like Qwen three ASR one point seven billion CER for content and WeSpeaker ResNet thirty four plus Overlapped Speech Detection for identity, which gives a solid way to measure both the accuracy of what was extracted and the faithfulness of that extraction <ref:two thousand six hundred eight point zero three nine six four#pg2.
Taro: So, when you look at this whole piece, it’s about how we train these models to handle the messy reality of real-world audio visual data, where things can get overlapped or misdirected <ref:two thousand six hundred eight point zero three nine six four#pg1. It sets a standard for what we consider identity faithfulness in this area.
Rosa: And to wrap up, the work on REAL-2MIX gives us a clear way to test if an audio visual target speaker extractor is actually doing what it's supposed to do when things get complicated <ref:two thousand six hundred eight point zero three nine six four#pg1. We'll be looking at how these challenges are handled in the next part of the show.
Conclusion: Rosa: So we’re wrapping up with some thoughts on this paper, REAL-2MIX, by Afouras, Chung, and Zisserman <ref:2608.03964#pg1>. Basically they’re testing if these models actually follow the visual cues when trying to pick a target speaker from a mix of audio and video.
Dev: Right. It’s about checking identity faithfulness in audio visual target speaker extraction, which is a pretty specific test because you want the model to listen to what it sees, not just what it hears <ref:2608.03964#pg2>.
Taro: The whole point of REAL-2MIX is this paired identity test where the same mixture has to yield speaker A under A's lip cue and speaker B under B's lip cue <ref:2608.03964#pg1>. It’s a very strict way to measure if the model is actually doing what it’s supposed to be doing <ref:2608.03964#pg1>.
Rosa: And they use speaker margin, overlap detection, and some specific metrics like Qwen three ASR CER to see if the whole system works together <ref:two thousand six hundred eight point zero three nine six four#pg2. It’s a pretty comprehensive way to check if an extractor follows visual intent while suppressing overlap and preserving the requested utterance <ref:two thousand six hundred eight point zero three nine six four#pg1.
Dev: I think what this means for us is that when we build these systems, we can't just look at audio quality or video resolution anymore; we have to test if the model respects those visual instructions under realistic, messy conditions <ref:two thousand six hundred eight point zero three nine six four#pg1.
Taro: Exactly. The limitation they mentioned is that they need a controlled rerun for the auxiliary speaker loss before they can really claim anything causal about it <ref:two thousand six hundred eight point zero three nine six four#pg1. So we have to be careful not to draw conclusions too fast <ref:two thousand six hundred eight point zero three nine six four#pg1.
Rosa: True. It’s a careful look at how these different parts interact when you test for that kind of fidelity <ref:two thousand six hundred eight point zero three nine six four#pg1. But this benchmark gives us a solid standard for what identity faithfulness actually looks like in the audio visual space <ref:two thousand six hundred eight point zero three nine six four#pg1.
Dev: So, we’ve seen how they build the test, what the results were with those specific metrics, and where they might be cautious about their conclusions <ref:two thousand six hundred eight point zero three nine six four#pg2. Now we need to see how this idea of identity faithfulness plays out when you move beyond these controlled mixed recordings <ref:two thousand six hundred eight point zero three nine six four#pg1.
School of Cyber Science and Engineering, Wuhan University · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen
eess.AS
Submitted: 2026-08-04
Updated: 2026-11-06
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 91/100
The gist: Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice…
Key concepts
- REAL-2MIX
- This is a benchmark created for testing how well models can perform audio-visual target speaker extraction when the audio comes from real mixtures recorded with synchronized multi-view video. It specifically checks if the model follows visual lip cues during these joint recordings.
- Audio-Visual Target Speaker Extraction (AVTSE)
- This is a task where a system takes an overlapping recording of two speakers and the video, and must identify which speaker's voice is the target speaker. The goal is to find the target speaker's waveform based on both audio and visual information.
- Identity Faithfulness
- This refers to how accurately a model preserves the true identity of a person when performing a task. In this context, it means checking if an extractor correctly identifies the intended target speaker even when visual cues (like lip movements) are present in the joint recording.
Terminology
Summary
Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice <ref:2608. p. 0 cn, mingli369@cuhk. cn
Abstract
Audio-visual target speaker extraction should
return the speaker indicated by the video, yet a
separator can ignore the visual cue and repeatedly output the acoustically dominant voice. cn
Abstract
Audio-visual target speaker extraction should
return the speaker indicated by the video, yet a
separator can ignore the visual cue and repeatedly output the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin AVTSE benchmark of jointly recorded real twospeaker mixtures with synchronized multi-view
video. Together with speaker-disjoint manifests, the corpus emphasizes reliable visual target selection rather than raw hours alone. REAL2MIX then provides the complementary real-scene
test: whether a model trained on these curated face–
voice pairs follows the requested lip cue during
jointly recorded Mandarin overlap.Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Construction and Data Curation
REAL-2MIX is introduced to benchmark identity-faithful AV-TSE on purpose-recorded real mixtures with synchronized audio and multi-view video <ref:2608.03964#pg2> The task definition involves predicting the target speaker waveform sˆq = fθ(x, vq) given an overlap recording x(t) = sA(t)+sB(t)+n(t) <ref:2608. p. 1 1 Task Definition and Recording Protocol
Let an overlap recording be x(t) = sA(t)+sB(t)+
n(t), where n includes room acoustics and background noise. Given the visual stream vq for target
q ∈ {A, B}, the extractor predicts
sˆq = fθ(x, vq). This requirement rules out visual
bypass, in which both prompts return the same
dominant source. Each scene records two stationary participants
in a shared environment. 2 Geometry and Acquisition
Figure 1 summarizes the recording geometry and
shows a representative frontal-view frame. The
protocol uses distances of 0. The
protocol uses distances of 0. 5, and 3 m and
side-view angles of 30◦
, 45◦
, and 60◦
. 3 Audited Corpus Statistics
The complete processed release contains 77 scene
directories from 14 unique speaker dyads. Its 7,598
mono 16-kHz clips total 11.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Training Signal Architecture
The extractor uses frozen, 1,280-dimensional projected AV-HuBERT features and target-conditioned training <ref:2608.03964#pg2> The architecture employs a single-output TF-GridNet where the visual stream is processed through input-level fusion and layer-wise FiLM modulation <ref:2608. p. 4 5 AV-HuBERT-Conditioned TF-GridNet
Figure 2 summarizes the two complementary roles
of the visual stream. Input-level fusion aligns the
target cue with the mixture spectrogram before the
first GridNet block, while layer-wise FiLM reinjects target information throughout the separator
rather than relying on a single early fusion. A large AV-HuBERT backbone produces a 1,024-dimensional visual sequence, which
a learned linear layer projects to 1,280 dimensions
and rescales by a learned scalar. Both the backbone
and projection are frozen for separator training. The resulting visual map has shape
B × 1 × T × F and is concatenated with the real
and imaginary mixture components before the twodimensional input convolution. In parallel, the resampled visual sequence is supplied directly to the
layer-specific FiLM generators described below.Where the paper says it
Where the paper says it
Where the paper says it
Evaluation Metrics and Analysis
The performance is jointly evaluated using Qwen3-ASR-1.7B CER for content and WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD) for target identity <ref:2608. p. 0 We jointly evaluate content
with Qwen3-ASR-1. 7B CER and target identity with WeSpeaker ResNet34 plus Overlapped
Speech Detection (OSD). If ei,q is the embedding of output sˆi,q and e
r
i,q is the
corresponding solo-reference mean, we report
SpkSim = 1
|P|
X
(i,q)∈P
e
⊤
i,qe
r
i,q
∥ei,q∥2∥e
r
i,q∥2
. (5)
We report target similarity, interferer similarity, and
their mean difference (speaker margin). 4 OSD-First Strict Identity Evaluation
The real overlap recordings have no synchronized
isolated physical sources, so target SI-SDR is not
defined. We instead use an automatic residualoverlap proxy before speaker identification. VOXBLINK2-AVSE supplies 766 hours of
identity-diverse aligned training data, while frozen
AV-HuBERT features, speaker margin training,
and layer-wise FiLM make the target cue explicit
throughout the separator. The ablations show that
matched training data and interferer suppression
matter more than simply enlarging the GridNet
back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Corpus Complementarity
The work complements REAL-2MIX with VOXBLINK2-AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training <ref:2608.03964#pg2> VOXBLINK2-AVSE is constructed from the in-thewild VoxBlink2 collection but undergoes audio-visual validity checks and DNSMOS filtering to ensure quality <ref:2608. p. 3 4 The VOXBLINK2-AVSE Training
Corpus
4. 1 Source Corpus and Audio-Visual
Validation
We construct VOXBLINK2-AVSE from the in-thewild VoxBlink2 collection (Lin et al. This task-ready representation avoids repeating decoding, synchronization, ROI validation,
and acoustic screening inside each AVSE training pipeline. Together with speaker-disjoint manifests, the corpus emphasizes reliable visual target selection rather than raw hours alone. The ablations show that
matched training data and interferer suppression
matter more than simply enlarging the GridNet
back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion. Speaker margin, OSD-first strict identity rates, and
Qwen3-ASR CER jointly test whether an extractor follows visual intent, suppresses overlap, and
preserves the requested utterance.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Extractor Design
The conclusion is that REAL-2MIX turns audio-visual TSE into a paired identity test where the same jointly recorded mixture must yield A under A’s lip cue and B under B’s lip cue <ref:2608.03964#pg2> The combination of speaker margin, OSD-first strict identity rates, and Qwen3-ASR CER jointly tests whether an extractor follows visual intent, suppresses overlap, and preserves the requested utterance <ref:2608. p. 7 The ablations show that
matched training data and interferer suppression
matter more than simply enlarging the GridNet
back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion. Speaker margin, OSD-first strict identity rates, and
Qwen3-ASR CER jointly test whether an extractor follows visual intent, suppresses overlap, and
preserves the requested utterance. VOXBLINK2-AVSE supplies 766 hours of
identity-diverse aligned training data, while frozen
AV-HuBERT features, speaker margin training,
and layer-wise FiLM make the target cue explicit
throughout the separator. The ablations show that
matched training data and interferer suppression
matter more than simply enlarging the GridNet
back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion. VOXBLINK2-AVSE supplies 766 hours of
identity-diverse aligned training data, while frozen
AV-HuBERT features, speaker margin training,
and layer-wise FiLM make the target cue explicit
throughout the separator. The ablations show that
matched training data and interferer suppression
matter more than simply enlarging the GridNet
back end; the auxiliary speaker loss requires a controlled rerun before drawing a causal conclusion.Where the paper says it
Where the paper says it
Where the paper says it
References
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. LRS3-TED: A large-scale dataset for visual speech recognition <ref:2608. p. 8 References
Triantafyllos Afouras, Joon Son Chung, and Andrew
Zisserman. LRS3-TED: A large-scale dataset
for visual speech recognition. arXiv preprint. Adrian Bulat and Georgios Tzimiropoulos. Adrian Bulat and Georgios Tzimiropoulos. How
far are we from solving the 2d & 3d face alignment
problem? How
far are we from solving the 2d & 3d face alignment
problem? (and a dataset of 230,000 3d facial landmarks). (and a dataset of 230,000 3d facial landmarks). In IEEE International Conference on Computer Vision, pages 1021–1030. In IEEE International Conference on Computer Vision, pages 1021–1030. Qiong Cao, Li Shen, Weidi Xie, Omkar M. Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and
Andrew Zisserman. Parkhi, and
Andrew Zisserman. VGGFace2: A dataset for
recognising faces across pose and age. VGGFace2: A dataset for
recognising faces across pose and age. In IEEE International Conference on Automatic Face & Gesture
Recognition, pages 67–74. In IEEE International Conference on Automatic Face & Gesture
Recognition, pages 67–74. Joon Son Chung, Andrew Senior, Oriol Vinyals, and
Andrew Zisserman. Joon Son Chung, Andrew Senior, Oriol Vinyals, and
Andrew Zisserman. Lip reading sentences in
the wild. Lip reading sentences in
the wild. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages
3444–3453. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages
3444–3453. Jiankang Deng, Jia Guo, Evangelos Ververas, Irene
Kotsia, and Stefanos Zafeiriou. Jiankang Deng, Jia Guo, Evangelos Ververas, Irene
Kotsia, and Stefanos Zafeiriou. RetinaFace:
Single-shot multi-level face localisation in the wild. RetinaFace:
Single-shot multi-level face localisation in the wild. In IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 5202–5211. In IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 5202–5211. Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN
based speaker verification. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN
based speaker verification. In Interspeech, pages
3830–3834. The Annals of Statistics, 7(1):1–26. Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel,
Kevin Wilson, Avinatan Hassidim, William T. Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel,
Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Freeman, and Michael Rubinstein. Looking to
listen at the cocktail party: A speaker-independent
audio-visual model for speech separation. Looking to
listen at the cocktail party: A speaker-independent
audio-visual model for speech separation. ACM
Transactions on Graphics, 37(4):1–11. ACM
Transactions on Graphics, 37(4):1–11. Ruohan Gao and Kristen Grauman. Ruohan Gao and Kristen Grauman. Visualvoice:
Audio-visual speech separation with cross-modal
consistency. Visualvoice:
Audio-visual speech separation with cross-modal
consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 15490–15500. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 15490–15500. Vladimir I. Before scoring, the reference and hypothesis are
stripped of Chinese and ASCII punctuation, consecutive whitespace is collapsed, and whitespace is excluded from character tokens. The micro-averaged
CER is
CER =
P
(i,q)∈P ED(A(ˆsi,q), yi,q)
P
(i,q)∈P |yi,q|
, (6)
where ED is Levenshtein distance (Levenshtein,
1966). Levenshtein. Binary codes capable of
correcting deletions, insertions, and reversals. Binary codes capable of
correcting deletions, insertions, and reversals. Soviet
Physics Doklady, 10(8):707–710. Soviet
Physics Doklady, 10(8):707–710. Shaole Li, Shuai Wang, Jiangyu Han, Ke Zhang, Wupeng Wang, and Haizhou Li. Shaole Li, Shuai Wang, Jiangyu Han, Ke Zhang, Wupeng Wang, and Haizhou Li. REAL-T: Real
conversational mixtures for target speaker extraction. REAL-T: Real
conversational mixtures for target speaker extraction. In Interspeech 2025, pages 1923–1927. In Interspeech 2025, pages 1923–1927. Yuke Lin, Ming Cheng, Fulin Zhang, Yingying Gao,
Shilei Zhang, and Ming Li. Yuke Lin, Ming Cheng, Fulin Zhang, Yingying Gao,
Shilei Zhang, and Ming Li. Voxblink2: A
100k+ speaker recognition corpus and the open-set
speaker-identification benchmark. Voxblink2: A
100k+ speaker recognition corpus and the open-set
speaker-identification benchmark. Interspeech 2024, pages 4263–4267. Interspeech 2024, pages 4263–4267. Yi Luo and Nima Mesgarani. Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for
speech separation. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for
speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–
1266. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–
1266. Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker identification dataset. Voxceleb: A large-scale speaker identification dataset. In Interspeech, pages 2616–2620. In Interspeech, pages 2616–2620. Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Atsunori Ogawa, and Tomohiro Nakatani. Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Atsunori Ogawa, and Tomohiro Nakatani. Multimodal speakerbeam: Single channel target speech
extraction with audio-visual speaker clues. Multimodal speakerbeam: Single channel target speech
extraction with audio-visual speaker clues. In Interspeech, pages 2718–2722. In Interspeech, pages 2718–2722. Zexu Pan, Meng Ge, and Haizhou Li. Zexu Pan, Meng Ge, and Haizhou Li. USEV: Universal speaker extraction with visual cue. USEV: Universal speaker extraction with visual cue. IEEE/ACM
Transactions on Audio, Speech, and Language Processing, 30:3032–3045. In
IEEE Automatic Speech Recognition and Understanding Workshop, pages 1–8. Ethan Perez, Florian Strub, Harm de Vries, Vincent
Dumoulin, and Aaron Courville. Ethan Perez, Florian Strub, Harm de Vries, Vincent
Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. FiLM: Visual reasoning with a general conditioning layer. In
Proceedings of the AAAI Conference on Artificial
Intelligence, volume 32. In
Proceedings of the AAAI Conference on Artificial
Intelligence, volume 32. Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective
speech quality metric to evaluate noise suppressors. Dnsmos: A non-intrusive perceptual objective
speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), pages 6493–6497. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), pages 6493–6497. Florian Schroff, Dmitry Kalenichenko, and James
Philbin. Florian Schroff, Dmitry Kalenichenko, and James
Philbin. FaceNet: A unified embedding for
face recognition and clustering. FaceNet: A unified embedding for
face recognition and clustering. In IEEE Conference
on Computer Vision and Pattern Recognition, pages
815–823. In IEEE Conference
on Computer Vision and Pattern Recognition, pages
815–823. Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual
speech representation by masked multimodal cluster
prediction. Learning audio-visual
speech representation by masked multimodal cluster
prediction. In International Conference on Learning
Representations. In International Conference on Learning
Representations. Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei
Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao,
Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and
Junyang Lin. Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei
Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao,
Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and
Junyang Lin. Qwen3-ASR technical report. LRS3-TED: A large-scale dataset
for visual speech recognition. arXiv preprint. arXiv preprint. Fei Su, Cancan Li, Ming Li, and Juan Liu. Fei Su, Cancan Li, Ming Li, and Juan Liu. M2savsr: Modality-aware multi-view self-supervised representation for robust audio-visual speech recognition. M2savsr: Modality-aware multi-view self-supervised representation for robust audio-visual speech recognition. arXiv preprint arXiv:2606. We complement this benchmark with VOXBLINK2-
AVSE, a large identity-disjoint audio-lip corpus curated for AVSE training, and with an AV-HuBERTconditioned TF-GridNet. Our contributions are threefold:
• We release REAL-2MIX, a controlled realscene test set benchmark with jointly recorded
overlap, scene-matched solo references, multiple distances, and three synchronized viewarXiv:2608. Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian,
Mike Zheng Shou, and Haizhou Li. Is someone speaking? p. 9 Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian,
Mike Zheng Shou, and Haizhou Li. Is someone speaking? Is someone speaking? exploring long-term temporal features
for audio-visual active speaker detection. exploring long-term temporal features
for audio-visual active speaker detection. In ACM
Multimedia, pages 3927–3935. References
Triantafyllos Afouras, Joon Son Chung, and Andrew
Zisserman. LRS3-TED: A large-scale dataset
for visual speech recognition. In ACM
Multimedia, pages 3927–3935. Error bounds for convolutional
codes and an asymptotically optimum decoding algorithm. Error bounds for convolutional
codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory,
13(2):260–269. IEEE Transactions on Information Theory,
13(2):260–269. Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi,
Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe. Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi,
Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe. TF-GridNet: Integrating full- and subband modeling for speech separation. TF-GridNet: Integrating full- and subband modeling for speech separation. IEEE/ACM
Transactions on Audio, Speech, and Language Processing, 31:3221–3236. IEEE/ACM
Transactions on Audio, Speech, and Language Processing, 31:3221–3236. Bang Zeng and Ming Li. Bang Zeng and Ming Li. USEF-TSE: Universal speaker embedding free target speaker extraction. USEF-TSE: Universal speaker embedding free target speaker extraction. IEEE Transactions on Audio, Speech and Language
Processing, 33:2110–2124. IEEE Transactions on Audio, Speech and Language
Processing, 33:2110–2124. Kateˇrina Žmolíková, Marc Delcroix, Keisuke Kinoshita,
Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget,
and Jan Cernock ˇ y. Kateˇrina Žmolíková, Marc Delcroix, Keisuke Kinoshita,
Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget,
and Jan Cernock ˇ y. ` SpeakerBeam: Speaker
aware neural network for target speaker extraction in
speech mixtures. ` SpeakerBeam: Speaker
aware neural network for target speaker extraction in
speech mixtures. IEEE Journal of Selected Topics in
Signal Processing, 13(4):800–814.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Visual target extraction can be made identity faithful by conditioning a separator on visual cues using frozen, 1280-dimensional projected AV-HuBERT features and layer-wise feature modulation. This allows for explicit target information to be injected throughout the separator rather than relying on single early fusion points.
-
The system can be trained with an auxiliary speaker margin loss that encourages the output waveform to be closer to the target than the interferer in a frozen 192-dimensional ECAPA-TDNN speaker encoder space. This supervision directly addresses
whether changing the visual target changes the extracted identity, rather than only whether one prompted output sounds clean.
-
The improved system can perform strict identity evaluation using an OSD-first approach where outputs below a certain residual overlap threshold are classified as wrong speaker based on a positive or non-positive speaker margin. This provides
strict correct
andwrong speaker
classifications, offering a more robust measure than just content character error rate.
Abstract
Audio-visual target speaker extraction should recover the person indicated by the video, rather than repeatedly return the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin benchmark of jointly recorded two-speaker mixtures with synchronized multi-view video and scene-matched solo references. Its 77 scenes contain 7,598 clips; filtering yields 6,038 evaluable mixtures, each queried for both speakers, giving 12,076 paired target queries. We also curate VOXBLINK2-AVSE, comprising 250,828 synchronized audio--lip pairs. A seven-system TF-GridNet study examines visual representations, training corpora, and conditioning strategies. We evaluate transcription accuracy, speaker identity, residual overlap, and perceptual quality. The AVSR early-fusion system achieves 22.99% CER, and 93.38% of its outputs are more similar to the target speaker than to the interferer. Yet only 42.49% satisfy both this identity criterion and the overlap detector's single-speaker criterion; both target queries satisfy these criteria for just 20.77% of mixtures. A separate Real-AVSE challenge study examines development-remix adaptation. The benchmark reveals that favorable content and identity scores need not coincide with low detected overlap, motivating paired-target evaluation.
Sources
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models