Optimal transport meets speech: a tutorial review
eess.AS, cs.AI, cs.CL, cs.LG
Submitted: 2026-09-24
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Optimal Transport (OT) provides a principled framework for comparing and transforming probability distributions while preserving geometric structure.
Terminology
Abstract
Optimal Transport (OT) provides a principled framework for comparing and transforming probability distributions while preserving geometric structure. Recently, OT has gained significant attention in machine learning due to its ability to measure discrepancies between distributions, even when their supports do not overlap, making it effective for tasks such as generative modeling, domain adaptation, and transfer learning. Despite its success in fields such as computer vision and natural language processing, OT remains relatively underexplored in speech research. Speech signals present unique challenges, including temporal dynamics, speaker variability, noise, reverberation, and heterogeneous multimodal representations involving audio, text, and visual information. These factors often lead to distribution mismatches, where OT offers a natural framework for alignment and interpretation. This work aims to promote broader adoption of OT in speech processing by: (1) reviewing OT foundations through intuitive physical interpretations and highlighting connections to modern generative models; (2) presenting computational algorithms suitable for deep learning frameworks; and (3) demonstrating OT applications in cross-domain and cross-modal speech tasks, including speech enhancement, automatic speech recognition, language and speaker recognition, and audio spoof detection. We highlight OT's strong potential for addressing distributional variations in real-world speech applications.
Sources
- Improving Sequence-to-Sequence Learning via Optimal Transport
- Neural Ordinary Differential Equations
- Scaling Algorithms for Unbalanced Transport Problems
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Distilling the Knowledge of BERT for CTC-based ASR
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Sequence Transduction with Recurrent Neural Networks
- Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
- Distilling the Knowledge in a Neural Network
- An Introduction to Flow Matching and Diffusion Models
- Linguistic Knowledge Transfer Learning for Speech Enhancement
- Auto-Encoding Variational Bayes
- Transport-based analysis, modeling, and learning from signal and data distributions
- Neural Optimal Transport
- Pre-training for Speech Translation: CTC Meets Optimal Transport
- Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement
- Flow Matching for Generative Modeling
- Flow Matching Guide and Code
- Optimal Transport for Machine Learners
- Generative Modeling by Estimating Gradients of the Data Distribution
Related papers
- Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
- X-VC: Zero-shot Streaming Voice Conversion in Codec Space
- Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
- VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation
- Conditional Flow Matching for Visually-Guided Acoustic Highlighting
- Anonymization, Not Elimination: Utility-Preserved Speech Anonymization