Discrete Optimal Transport and Voice Conversion

arXiv:2505.04382 · eess.AS, cs.LG, cs.SD · Submitted 2025-05-07 · Read on arXiv

eess.AS, cs.LG, cs.SD

Submitted: 2025-05-07

Updated: 2026-06-15

Comments: 5 pages, 1 figure, 7 tables. 11th International Conference on Machine Learning Technologies (ICMLT), Berlin, Germany, May 2026

DOI: 10.1109/ICMLT69916.2026.11688511

Code: https://github.com/openai/whisper

Project page: https://anton-selitskiy.github.io/dotvc

License: http://creativecommons.org/licenses/by/4.0/

The gist: We propose kDOT, a discrete optimal transport (OT) framework for voice conversion (VC) operating in a pretrained speech embedding space.

Terminology

Abstract

We propose kDOT, a discrete optimal transport (OT) framework for voice conversion (VC) operating in a pretrained speech embedding space. In contrast to the averaging strategies used in kNN-VC and SinkVC, and the independence assumption adopted in MKL, our method employs the barycentric projection of the discrete OT plan to construct a transport map between source and target speaker embedding distributions. We conduct a comprehensive ablation study over the number of transported embeddings and systematically analyze the impact of source and target utterance duration. Experiments on LibriSpeech demonstrate that OT with barycentric projection consistently improves distribution alignment and often outperforms averaging-based approaches in terms of WER, MOS, and FAD. Furthermore, we show that applying discrete OT as a post-processing step can transform spoofed speech into samples that are misclassified as bona fide by a state-of-the-art spoofing detector. This demonstrates the strong domain adaptation capability of OT in embedding space, while also revealing important security implications for spoof detection systems.

Sources

Related papers