Sequence-to-Sequence Refinement for EEND-VC Diarization with a Shared Encoder
eess.AS
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/liyunlongaaa/NSD-MS2S
License: http://creativecommons.org/licenses/by/4.0/
The gist: End-to-end neural diarization with vector clustering (EEND-VC) separates local segmentation from global speaker assignment, leaving speaker confusion (SC) despite strong segmentation.
Terminology
Abstract
End-to-end neural diarization with vector clustering (EEND-VC) separates local segmentation from global speaker assignment, leaving speaker confusion (SC) despite strong segmentation. We propose shared-encoder sequence-to-sequence (S2S) refinement, which reuses DiariZen's self-supervised learning (SSL) representations for speaker-conditioned decoding and prior-constrained assignment. We study frozen-feature refinement with WavLM and structurally pruned w2v-BERT 2.0, and extend the latter to joint training of local segmentation and refinement heads. The heads learn separate layer mixtures while their losses update one acoustic encoder. Across eight domains, frozen-feature refinement reduces macro diarization error rate (DER) from 12.95% to 12.40% with WavLM and from 12.35% to 12.08% with w2v-BERT 2.0, with relative SC reductions of 17.4% and 14.6%. It adds approximately 15% to system real-time factor. The joint system further lowers macro DER to 11.90%, improving seven of eight domains over non-joint refinement. Error decomposition separates SC-driven refinement gains from detection-driven joint-system gains.
Sources
- Seamless: Multilingual Expressive and Streaming Speech Translation
- Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning
- Learning Sparse Neural Networks through $L_0$ Regularization
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models