Marco-Voice Technical Report
Fengping Tian, Chenyang Lyu, Xuanfan Ni, Haoqin Sun, Qingjuan Li, Zhiqiang Qian, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, Kaifu Zhang
cs.CL, cs.SD, eess.AS
Submitted: 2026-08-18
Updated: 2026-08-19
Comments: Technical Report. Our code and dataset are publicly available at https://github.com/AIDC-AI/Marco-Voice and https://huggingface.co/datasets/AIDC-AI/CSEMOTIONS respectively for non-commercial use only
Code: https://github.com/AIDC-AI/Marco-Voice
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: This paper presents a multifunctional speech synthesis system that integrates voice cloning and emotion control speech synthesis within a unified framework.
Terminology
Abstract
This paper presents a multifunctional speech synthesis system that integrates voice cloning and emotion control speech synthesis within a unified framework. The goal of this work is to address longstanding challenges in achieving highly expressive, controllable, and natural speech generation that faithfully preserves speaker identity across diverse linguistic and emotional contexts. Our approach introduces an effective speaker-emotion disentanglement mechanism with in-batch contrastive learning, enabling independent manipulation of speaker identity and eemotional style, as well as rotational emotional embedding integration method for smooth emotion control. To support comprehensive training and evaluation, we construct CSEMOTIONS, a high-quality emotional speech dataset containing 10 hours of Mandarin speech from ten professional speakers across seven emotional categories. Extensive experiments demonstrate that our system, Marco-Voice, achieves substantial improvements in both objective and subjective metrics. Comprehensive evaluations and analysis were conducted, results show that MarcoVoice delivers competitive performance in terms of speech clarity and emotional richness, representing a substantial advance in the field of expressive neural speech synthesis. Our code and dataset are publicly available at https://github.com/AIDC-AI/Marco-Voice and https://huggingface.co/datasets/AIDC-AI/CSEMOTIONS respectively.
Sources
- EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
- Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesis
- Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models
- CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models
- UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts
- EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark
- DS-TTS: Zero-Shot Speaker Style Adaptation from Voice Clips via Dynamic Dual-Style Feature Modulation
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
- A Survey on Neural Speech Synthesis
- Improving and generalizing flow-based generative models with minibatch optimal transport
- SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement
- A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering