VOSSA: Voiceprint Optimization for Streaming Speech Architectures

arXiv:2609.38887 · eess.AS, cs.CL, cs.LG · Submitted 2026-09-30 · Read on arXiv

eess.AS, cs.CL, cs.LG

Submitted: 2026-09-30

Updated: 2026-10-04

Comments: Published in Proceedings of Interspeech 2026. Best Student Paper Award

Journal ref: Tseng, M.-R., Quamer, W., Nasrallah, G., Gutierrez-Osuna, R. (2026) VOSSA: Voiceprint Optimization for Streaming Speech Architectures. Proc. Interspeech 2026, 4716-4720

DOI: 10.21437/Interspeech.2026-2763

Project page: https://morris88826.github.io/VOSSA

License: http://creativecommons.org/licenses/by/4.0/

The gist: Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models.

Terminology

Embeddings — 17× in this paper · explained in 2 episodes
Representation Learning — 5× in this paper · explained in 2 episodes
Verification — 3× in this paper · explained in 2 episodes
Alignment — 2× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Cosine Similarity — 2× in this paper · explained in 3 episodes
IEEE Automatic Speech Recognition and Understanding Workshop — 2× in this paper
Mean Opinion Score — 2× in this paper
Pseudo-labels — 2× in this paper · explained in 2 episodes
Regularization — 2× in this paper · explained in 3 episodes
Wasserstein Distance — 2× in this paper · explained in 4 episodes
Accuracy — 1× in this paper · explained in 1 episode
Aggregation — 1× in this paper · explained in 1 episode
Centre for Speech Technology Research — 1× in this paper
Contrastive Learning — 1× in this paper · explained in 5 episodes
Generative AI — 1× in this paper · explained in 2 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Intelligence Advanced Research Projects Activity — 1× in this paper
Transformer — 1× in this paper · explained in 2 episodes

Abstract

Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effective for speaker discrimination, these embeddings are trained to remain stable across phonetic and prosodic variations within-speaker, which may conflict with frame-level acoustic generation in streaming constraints. To address this issue, we propose VOSSA (Voiceprint Optimization for Streaming Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling. The embedding is trained jointly with VC objectives, removing the need for a separate speaker encoder. Across six datasets, VOSSA improves F0 dynamics and vowel-discriminative acoustic cues while maintaining comparable NISQA-MOS, WER, and speaker similarity. Perceptual tests further indicate improvements in naturalness, speaker similarity, intelligibility, and vibrancy.

Terminology

Abstract

Related papers