MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
summary
The gist
Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation…
In short
MINT-Bench is a comprehensive multilingual benchmark designed to systematically evaluate instruction-following Text-to-Speech (TTS) systems across ten languages. It uses a hierarchical taxonomy and a multi-stage data construction pipeline to test controllability, assessing content consistency, instruction following, and perceptual quality using hybrid evaluation methods.
Key concepts
- Hierarchical Multi-axis Taxonomy
- This is the core structure of the benchmark, organizing TTS attributes into four axes: Difficulty Level, Control Domain (like Age or Pitch), Control Specification (how precise the control is), and Fine-grained Control Pattern. It uses ten atomic acoustic attributes to define a vast inventory for testing.
- Multi-stage Data Construction Pipeline
- Instead of creating instruction pairs directly, this pipeline builds them in three stages: first defining valid cases based on the taxonomy (Node Specification), second creating a structured plan scaffold, and finally realizing that plan into natural language instructions paired with synthesis text to ensure high control.
- Hierarchical Hybrid Evaluation Protocol
- This protocol assesses TTS quality progressively. It first checks content consistency via ASR error rates, then uses a Large Audio Language Model (LALM) like Gemini to score instruction following, and finally applies perceptual bonuses only for samples that pass the instruction-following test.
Terminology used across episodes
This episode discusses
- MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech · Paper Radio
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
- IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
- CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
- FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech System
- VoiceSculptor: Your Voice, Designed By You
- MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions
- InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
- S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
- FleSpeech: Flexibly Controllable Speech Generation with Various Prompts
- IndexTTS 2.5 Technical Report
- Fish Audio S2 Technical Report
- Natural language guidance of high-fidelity text-to-speech with synthetic annotations
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
- VibeVoice Technical Report
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play · Paper Radio
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Qwen3-TTS Technical Report
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
- Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
The paper
MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech · Read on arXiv
Huakang Chen, Jingbin Hu, Liumeng Xue, Qirui Zhan, Wenhao Li, Guobin Ma, Hanke Xie, Dake Guo, Linhan Ma, Yuepeng Jiang, Bengu Wu
Audio, Speech and Language Processing Lab (ASLP@NPU) · School of Intelligence Science and Technology, Nanjing University School of Intelligence Science and Technology, Suzhou China · Yutu Zhineng Beijing China · Lingguang Zhaxian Technology Shanghai China
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech".
Dev: Here's the gist of the paper. Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation, yet its evaluation remains underdeveloped due to limited benchmark coverage, weak diagnostic granularity, and insufficient multilingual support MINT-Bench is a comprehensive multilingual benchmark for instruction-following TTS built upon a hierarchical multi-axis taxonomy, a scalable multi-stage data construction pipeline, and a hierarchical hybrid evaluation protocol that jointly assesses content consistency, instruction following, and perceptual quality Experiments across ten languages show that current systems remain far from solved: frontier commercial systems lead overall, while leading open-source models become highly competitive and can even outperform commercial counterparts in localized settings such as Chinese The benchmark further reveals that harder compositional and paralinguistic controls remain major bottlenecks for current systems Benchmark Structure and Taxonomy MINT-Bench formulates instruction-following TTS evaluation as a structured and extensible benchmark construction problem rather than treating it as a flat collection of prompts At its core is a Hierarchical Multi-axis Taxonomy grounded in ten atomic timbre and style attributes that define the core benchmark inventory This taxonomy organizes benchmark coverage across four axes: Difficulty Level, Control Domain, Control Specification, and Fine-grained Control Pattern The core of the taxonomy is ten atomic acoustic attributes summarized in Table 1 which include Age, Gender, Pitch, Texture (timbre), Speed, Volume (style), Emotion (style), Accent (style/language-specific), Personality (style/persona), Tone (style/tone), and Extra Vocal control Data Construction Pipeline MINT-Bench constructs benchmark instances through a controllable multi-stage pipeline rather than generating final instruction-text pairs in a single step The construction pipeline consists of three stages: node specification, structured label planning, and instruction-text realization Stage 1 involves Taxonomy Node Specification to define valid cases under the taxonomy and prepare internal inventory for coverage control Stage 2 is Structured Label Planning which converts node specifications into an intermediate structured plan that serves as a normalized control scaffold Stage 3 is Instruction-Text Pair Construction where the structured plan is realized as a natural-language instruction paired with a synthesis text, designed to minimize direct lexical leakage of the control target Hierarchical Evaluation Protocol MINT-Bench introduces a hierarchical hybrid evaluation protocol that progressively evaluates content consistency, instruction following, and perceptual quality by combining objective tools with Large Audio Language Model (LALM)-based judgments Stage 1 assesses Content Consistency using ASR transcription and sample-level Word Error Rate (WER) checking against language-specific thresholds to calculate a content consistency coefficient Stage 2 evaluates Instruction Following by using a LALM, such as Gemini, assigning a discrete instruction-following score based on task-aware judging prompts derived from the taxonomy Stage 3 assesses Perceptual Quality and Timbre Diversity only for samples with strong instruction-following performance (score = 3) by applying binary perceptual bonuses and calculating a Timbre Diversity Score (TDS) Experimental Findings Experiments across ten languages show that instruction-following TTS remains far from solved, with frontier commercial systems leading overall performance, while leading open-source systems become highly competitive in localized settings such as Chinese The benchmark reveals that strong content fidelity does not necessarily imply strong controllability, and harder compositional and Special extra-vocal cases remain major bottlenecks for current systems Timbre diversity should be interpreted as a separate conditional diagnostic of oneto-many generation ability rather than as a surrogate for overall controllability Difficulty-sensitive results expose a clear drop from easy control to structured and extra-vocal cases, with Disfluency Control consistently being the hardest subset Key Contributions The key contributions of the work are summarized as follows: • We present MINT-Bench, a comprehensive multilingual benchmark for instruction-following TTS designed to support systematic, fine-grained, and scalable evaluation of controllability and generation quality • We introduce a hierarchical multi-axis taxonomy together with a scalable multi-stage data construction pipeline enabling semantically clear, structurally controlled, and extensible benchmark construction across diverse control settings and languages • We design a hierarchical hybrid evaluation protocol that progressively assesses content consistency, instruction following, and perceptual quality by combining objective tools with LALM-based judgments Limitations and Future Directions Although MINT-Bench provides a structured and extensible benchmark for instruction-following TTS, several limitations remain First, MINT-Bench covers a bounded benchmark space rather than the full range of real-world instructions Second, the current multilingual setting is not fully balanced as English and Chinese use large splits while others use mini splits Third, the evaluation pipeline relies partly on LALM-based judging which may introduce bias and instability Future work should focus on expanding coverage, improving multilingual balance, and reducing dependence on proprietary evaluators References
1: Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang et al.
Dev: Seed-TTS: A Family of High-Quality Versatile Speech Generation Models.
Rosa: CoRR abs/2406.02430 (2024).
Paper summary: Rosa: To wrap up, MINT-Bench is this comprehensive multilingual benchmark for instruction-following text-to-speech that uses a hierarchical multi-axis taxonomy, a scalable three stage data construction pipeline, and that hierarchical hybrid evaluation protocol.
Dev: The authors present this to systematically support fine grained and scalable evaluation of controllability and generation quality across diverse control settings and languages.
Taro: What this means is we have a better way to measure how well these instruction following systems actually work when they are being controlled by complex, nuanced prompts.
Rosa: It moves the conversation from just checking if the voice sounds good to systematically testing if it can follow a wide variety of human instructions consistently in many languages.
Dev: The authors show that while commercial systems still lead on English, open source models are getting competitive in localized areas like Chinese, but they highlight that compositional and extra vocal cases are where the real struggle is right now.
Taro: So the implication for us is that we need benchmarks that test those hard compositional and extra vocal controls to really push these systems further into useful, controllable speech generation.
Conclusion: Rosa: So we’re wrapping up our look at MINT-Bench, this new benchmark for instruction following text to speech.
Dev: Yeah, it’s essentially this big framework they put together to systematically test how well these models actually follow complex voice commands across different languages and styles.
Taro: It sounds like they aren't just testing if the voice sounds good, but if it can reliably do specific things you ask for, like changing the emotion or the accent exactly how you want.
Rosa: Exactly. The authors are building this structure to make sure we aren't just looking at a few examples, but getting a really clear picture of where these systems are actually succeeding and where they’re struggling.
Dev: They did this by creating a way to build the benchmark in stages, starting with defining the different types of speech control you can ask for—like age or texture—and then realizing those definitions into actual text prompts.
Taro: That construction pipeline sounds like it's designed to make sure the tests aren't just random prompts, but structured challenges that actually probe the limits of what the AI can do.
Rosa: And they’re using this hierarchical evaluation protocol, which means they test things in order, starting with checking if the speech content itself makes sense and then moving on to how well it followed your instruction.
Dev: It’s a multi-step check—first content consistency, then instruction following judged by a large language model like Gemini, and finally perceptual quality only for the samples that actually passed the instruction test.
Taro: That sounds like a good way to isolate where the failure modes are happening. They aren't just looking at one big score; they’re breaking it down into pieces so we can see exactly which kind of control is hard.
Rosa: And the results show that even with this comprehensive setup, we still see a gap between the best commercial systems on English and what the leading open source models can do in more localized settings like Chinese.
Dev: It confirms that while the tech is getting sophisticated, there’s still a big difference in how well they handle those harder tasks—the compositional ones and the extra vocal controls.
Taro: So it tells us that just making the voice sound natural isn't enough; we need to focus our research on making these systems robust when they have to do multiple, specific things at once.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G