DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models
summary
The gist
Multi-speaker automatic speech recognition aims to transcribe conversational speech involving multiple speakers by capturing what was said, who said it…
In short
DM-ASR reframes multi-speaker ASR as a dialogue generation task guided by speaker and time cues. It uses a compact model, speech encoder, and LLM decoder to combine acoustic evidence with diarization information. The method shows that explicit diarization provides a strong structural prior for small models, achieving high performance even with limited training data.
Key concepts
- Framework Decomposition
- DM-ASR breaks transcription into sequential queries, where each query targets one specific speaker within a time segment. This separation allows the system to treat 'who' (speaker) and 'when' (time) distinctly from 'what' (linguistic content), enabling better integration of diarization data with language modeling.
- Special Token Design
- The model uses three types of special tokens: speaker index tokens to encode who is speaking, timestamp tokens to discretize continuous time into fixed intervals, and control tokens for structural elements like audio delimiters. These help the model explicitly understand the speaker and temporal context during generation.
- Label Perturbation Strategy
- During training, the system randomly changes speaker labels and timestamps with a small probability. This forces the model to learn robust features that allow it to recover from imperfect or noisy diarization cues by relying on acoustic evidence and dialogue context instead of just perfect external data.
Terminology used across episodes
This episode discusses
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models · Paper Radio
- Seamless: Multilingual Expressive and Streaming Speech Translation
- The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition
- TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding
- Diarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models
- Summary on The Multilingual Conversational Speech Language Model Challenge: Datasets, Tasks, Baselines, and Methods
- VIBEVOICE-ASR Technical Report
- Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio
- DiarizationLM: Speaker Diarization Post-Processing with Large Language Models · Paper Radio
- The DKU-DukeECE-Lenovo System for the Diarization Task of the 2021 VoxCeleb Speaker Recognition Challenge
- Qwen2.5-Omni Technical Report
- SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models
- MOSS Transcribe Diarize Technical Report
The paper
DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models · Read on arXiv
School of Artifcial Intelligence, Wuhan University · School of Computer Science, Wuhan University · Tencent Ethereal Audio Lab, Tencent
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models".
Dev: Here's the gist of the paper. Multi-speaker automatic speech recognition aims to transcribe conversational speech involving multiple speakers by capturing what was said, who said it, and sometimes when it was spoken <ref:2604.22467#pg2> The proposed DM-ASR framework reformulates multi-speaker transcription as a multi-turn dialogue generation process guided by diarization cues <ref:2604.22467#pg4> This approach enables a compact model to perform speaker-aware and temporally grounded transcription more effectively, achieving strong performance with relatively small models and limited training data <ref:2604.22467#pg2> Framework Decomposition DM-ASR decomposes the task into a sequence of speaker-and time-conditioned queries, where each query corresponds to one speaker in one time segment <ref:2604.22467#pg2> This formulation explicitly decouples speaker-temporal structure (who and when) from linguistic content (what), enabling effective integration of diarization cues with the reasoning capability of large language models <ref:2604.22467#pg2> The framework consists of four main components: a speech encoder, a projector, a large language model decoder, and a special-token discretization mechanism <ref:2604.22467#pg2> Special Token Design DM-ASR introduces three types of special tokens to encode speaker and temporal information <ref:2604.22467#pg2> Speaker tokens are a set of speaker index tokens, which are remapped to local speaker indices based on their order of appearance within each audio chunk to reduce vocabulary sparsity <ref:2604.22467#pg2> Timestamp tokens discretize continuous timestamps into integer indices using a fixed temporal resolution Δt = 0.1 s <ref:2604.22467#pg2> Control tokens include audio delimiters, speaker delimiters, time delimiters, and word-level timestamp triggers such as <ref:2604.22467#pg2> Training Strategy for Robustness During training, the model uses teacher forcing to concatenate all dialogue turns within a chunk into a single sequence, computing the cross-entropy loss only on the response tokens <ref:2604.22467#pg2> To improve robustness to imperfect diarization cues, DM-ASR introduces a label perturbation strategy where speaker labels and start/end timestamps are randomly perturbed with probability p = 0.1 <ref:2604.22467#pg2> This prevents the model from over-relying on perfectly correct diarization cues and encourages it to recover from noisy conditions using both acoustic evidence and dialogue context <ref:2604.22467#pg2> Performance Insights Experiments show that diarization systems provide more reliable speaker identities and segment-level boundaries, while LLMs excel at modeling linguistic content and long-range dependencies, demonstrating their complementary strengths <ref:2604.22467#pg2> The model can effectively exploit reliable diarization cues and gradually learn to correct imperfect ones as model size and training data increase <ref:2604.22467#pg2> Word-level timestamp prediction is shown to improve both output structure and text accuracy by encouraging the model to better associate lexical units with local acoustic evidence <ref:2604.22467#pg2> Inference Evaluation At inference time, DM-ASR follows the same multi-turn formulation, performing autoregressive generation turn by turn <ref:2604.22467#pg2> Multiple evaluation setups are defined to analyze interaction with external cues, including settings where both speaker labels and timestamps are predicted by the LLM <ref:2604.22467#pg2> Results indicate that using diarization-provided speaker and time information consistently yields the best performance on DER, showing that reliable external diarization cues remain highly beneficial for speaker attribution <ref:2604.22467#pg2> The findings suggest that explicit diarization information remains a valuable structural prior for small-size multi-speaker ASR models <ref:2604.22467#pg2> Overall, the findings suggest that explicit diarization information remains a valuable structural prior for small-size multi-speaker ASR models, even in the era of unified Speech-LLMs <ref:2604.22467#pg2> The results also reveal an important limitation: under relatively small model and data scales, fully LLM-predicted speaker labels and timestamps still do not consistently outperform strong diarization front-ends <ref:2604.22467#pg2> This gap can be further reduced by scaling up both model capacity and training data <ref:2604.22467#pg2> The paper validates the effectiveness of DM-ASR across both Mandarin and English benchmarks, demonstrating its generalizability <ref:2604.22467#pg2> The proposed method achieves strong multi-speaker ASR performance with a relatively small model and limited training data <ref:2604.22467#pg2> The results also demonstrate that word-level timestamp prediction not only provides richer structured outputs, but also improves transcription quality <ref:2604.
Title and authors: Rosa: Let's talk about who wrote this, and what that title really means for us as we look at this DM-ASR paper. The authors are Li Li, Ming Cheng, Weixin Zhu, Yannan Wang, and Juan Liu.
Dev: They're coming from different places—Wuhan University School of Artificial Intelligence, Tencent Ethereal Audio Lab, and The Chinese University of Hong Kong Shenzhen—so you get a mix of deep research and practical audio engineering insights here.
Taro: I'm interested in what the title implies about their approach: "Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models." It suggests they aren't just tweaking one part of the pipeline; they are creating a whole unified system that works both before and after initial processing.
Rosa: Right. The core idea is that this framework treats diarization not as a separate pre-processing step, but as an explicit structural prior that directly guides the LLM decoding process throughout the entire generation.
Dev: So it's like the diarization information tells the model exactly who is talking and when, so it doesn't have to guess those things from scratch during transcription.
Taro: It moves away from just using diarization to split audio into single-speaker chunks for a separate ASR backend; instead, they use that cue inside the unified system itself.
The paper's summary: Rosa: Now, looking at the actual summary of "DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models," it explains how they achieve this unification. They decompose the task into a sequence of speaker-and time-conditioned queries.
Dev: That's a key technical move, breaking down what they do into one speaker in one time segment at a time. It explicitly decouples the speaker-temporal structure, which is who and when, from the linguistic content, which is what was said.
Taro: That decoupling is important because it lets them integrate those diarization cues directly with the reasoning capabilities of large language models during transcription generation.
Rosa: They use four main components for this framework: a speech encoder, a projector, a large language model decoder, and a special-token discretization mechanism to handle the speaker and time information.
Dev: The special tokens they introduce are pretty clever; they have speaker index tokens that map to local indices based on appearance order in an audio chunk to keep the vocabulary manageable.
Taro: And then they have timestamp tokens that discretize continuous timestamps into integer indices using a fixed temporal resolution of zero point one seconds, or zero point one s.
Rosa: Plus, they have control tokens like audio and speaker delimiters, time delimiters, and word-level timestamp triggers to give the model fine-grained structural information.
The paper's improvements: Dev: Moving into what they actually improved in "DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models," they focus on how they handle the learning process to make it robust.
Rosa: They use teacher forcing during training, where all dialogue turns within a chunk are concatenated into one sequence, but they only compute the cross-entropy loss on the response tokens.
Taro: That's a specific strategy for training; they aren't just looking at the final output accuracy; they're focusing their learning signal on generating the actual speech content correctly.
Dev: But to make it more robust against imperfect diarization cues, they introduce a label perturbation strategy where speaker labels and start or end timestamps are randomly perturbed with a probability of zero point one.
Rosa: That perturbation prevents the model from over-relying on perfectly correct diarization cues, forcing it to recover from noisy conditions by using both the acoustic evidence and that dialogue context.
Taro: So, they're not just trusting the initial diarization output; they’re training the model to be resilient when those cues are slightly off, which is a big step for real-world deployment.
Conclusion: Rosa: So wrapping up with "DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models," the main conclusion is that explicit diarization information remains a valuable structural prior for small-size multi-speaker ASR models, even in this new era of unified Speech LLMs.
Dev: They found that while LLMs are great at modeling the linguistic content and long-range dependencies, using reliable external diarization provides more consistent speaker identities and segment boundaries when you're working with smaller models.
Taro: They also showed that word-level timestamp prediction actually helps improve both the output structure and the text accuracy by encouraging the model to better associate each lexical unit with its local acoustic evidence.
Rosa: And we saw this in inference too; using diarization provided speaker and time information consistently yielded the best performance on DER, showing that reliable external diarization cues are still highly beneficial for speaker attribution.
Dev: But they also found a limitation: under relatively small model and data scales, fully LLM-predicted speaker labels and timestamps still don't consistently outperform strong diarization front-ends.
Taro: That gap can definitely be narrowed if you scale up both the model capacity and the training data.
Rosa: It’s important to remember that this paper validates the effectiveness of DM-ASR across both Mandarin and English benchmarks, which shows it has good generalizability.
Dev: Overall, this work demonstrates strong multi-speaker ASR performance with a relatively small model and limited training data, which is a significant achievement for efficiency.
Taro: It proves that you can get high performance without needing massive amounts of perfectly labeled data if you have the right structural guidance.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G