UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models

arXiv:2510.22588 · eess.AS, cs.CL · Submitted 2025-10-26 · Read on arXiv

eess.AS, cs.CL

Submitted: 2025-10-26

Updated: 2026-08-29

Comments: EMNLP 2026 Findings

Code: https://github.com/bigai-nlco/UltraVoice

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Fine-Tuning — 14× in this paper · explained in 5 episodes
• Consistency — 10× in this paper · explained in 3 episodes
• Core — 10× in this paper · explained in 2 episodes
• Benchmark — 5× in this paper · explained in 2 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Generalization — 4× in this paper · explained in 13 episodes
• Supervised Fine-Tuning — 4× in this paper
• Accuracy — 3× in this paper · explained in 1 episode
• Adam — 3× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 3× in this paper · explained in 8 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Instruction Following Rate — 2× in this paper
• Lexical Diversity — 2× in this paper · explained in 2 episodes
• Mean Opinion Score — 2× in this paper
• Robustness — 2× in this paper · explained in 5 episodes
• Word Error Rate — 2× in this paper
• Adaptation — 1× in this paper · explained in 1 episode
• Audio-Language Model — 1× in this paper
• Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 3 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Words Per Minute — 1× in this paper

Sources

• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
• Audio-Aware Large Language Models as Judges for Speaking Styles
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Recent Advances in Speech Language Models: A Survey
• Moshi: a speech-text foundation model for real-time dialogue
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
• LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis
• OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue
• Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
• InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
• GPT-4o System Card
• WavChat: A Survey of Spoken Dialogue Models
• emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

Related papers

• Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models
• EarLog: Opportunistic Ear Physiology Sensing toward Personalized Hearing Risk Awareness
• VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation