MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech

arXiv:2604.17958 · eess.AS, cs.SD · Submitted 2026-04-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech".

Dev: Here's the gist of the paper. Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation, yet its evaluation remains underdeveloped due to limited benchmark coverage, weak diagnostic granularity, and insufficient multilingual support MINT-Bench is a comprehensive multilingual benchmark for instruction-following TTS built upon a hierarchical multi-axis taxonomy, a scalable multi-stage data construction pipeline, and a hierarchical hybrid evaluation protocol that jointly assesses content consistency, instruction following, and perceptual quality Experiments across ten languages show that current systems remain far from solved: frontier commercial systems lead overall, while leading open-source models become highly competitive and can even outperform commercial counterparts in localized settings such as Chinese The benchmark further reveals that harder compositional and paralinguistic controls remain major bottlenecks for current systems Benchmark Structure and Taxonomy MINT-Bench formulates instruction-following TTS evaluation as a structured and extensible benchmark construction problem rather than treating it as a flat collection of prompts At its core is a Hierarchical Multi-axis Taxonomy grounded in ten atomic timbre and style attributes that define the core benchmark inventory This taxonomy organizes benchmark coverage across four axes: Difficulty Level, Control Domain, Control Specification, and Fine-grained Control Pattern The core of the taxonomy is ten atomic acoustic attributes summarized in Table 1 which include Age, Gender, Pitch, Texture (timbre), Speed, Volume (style), Emotion (style), Accent (style/language-specific), Personality (style/persona), Tone (style/tone), and Extra Vocal control Data Construction Pipeline MINT-Bench constructs benchmark instances through a controllable multi-stage pipeline rather than generating final instruction-text pairs in a single step The construction pipeline consists of three stages: node specification, structured label planning, and instruction-text realization Stage 1 involves Taxonomy Node Specification to define valid cases under the taxonomy and prepare internal inventory for coverage control Stage 2 is Structured Label Planning which converts node specifications into an intermediate structured plan that serves as a normalized control scaffold Stage 3 is Instruction-Text Pair Construction where the structured plan is realized as a natural-language instruction paired with a synthesis text, designed to minimize direct lexical leakage of the control target Hierarchical Evaluation Protocol MINT-Bench introduces a hierarchical hybrid evaluation protocol that progressively evaluates content consistency, instruction following, and perceptual quality by combining objective tools with Large Audio Language Model (LALM)-based judgments Stage 1 assesses Content Consistency using ASR transcription and sample-level Word Error Rate (WER) checking against language-specific thresholds to calculate a content consistency coefficient Stage 2 evaluates Instruction Following by using a LALM, such as Gemini, assigning a discrete instruction-following score based on task-aware judging prompts derived from the taxonomy Stage 3 assesses Perceptual Quality and Timbre Diversity only for samples with strong instruction-following performance (score = 3) by applying binary perceptual bonuses and calculating a Timbre Diversity Score (TDS) Experimental Findings Experiments across ten languages show that instruction-following TTS remains far from solved, with frontier commercial systems leading overall performance, while leading open-source systems become highly competitive in localized settings such as Chinese The benchmark reveals that strong content fidelity does not necessarily imply strong controllability, and harder compositional and Special extra-vocal cases remain major bottlenecks for current systems Timbre diversity should be interpreted as a separate conditional diagnostic of oneto-many generation ability rather than as a surrogate for overall controllability Difficulty-sensitive results expose a clear drop from easy control to structured and extra-vocal cases, with Disfluency Control consistently being the hardest subset Key Contributions The key contributions of the work are summarized as follows: • We present MINT-Bench, a comprehensive multilingual benchmark for instruction-following TTS designed to support systematic, fine-grained, and scalable evaluation of controllability and generation quality • We introduce a hierarchical multi-axis taxonomy together with a scalable multi-stage data construction pipeline enabling semantically clear, structurally controlled, and extensible benchmark construction across diverse control settings and languages • We design a hierarchical hybrid evaluation protocol that progressively assesses content consistency, instruction following, and perceptual quality by combining objective tools with LALM-based judgments Limitations and Future Directions Although MINT-Bench provides a structured and extensible benchmark for instruction-following TTS, several limitations remain First, MINT-Bench covers a bounded benchmark space rather than the full range of real-world instructions Second, the current multilingual setting is not fully balanced as English and Chinese use large splits while others use mini splits Third, the evaluation pipeline relies partly on LALM-based judging which may introduce bias and instability Future work should focus on expanding coverage, improving multilingual balance, and reducing dependence on proprietary evaluators References

1: Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang et al.

Dev: Seed-TTS: A Family of High-Quality Versatile Speech Generation Models.

Rosa: CoRR abs/2406.02430 (2024).

Paper summary: Rosa: To wrap up, MINT-Bench is this comprehensive multilingual benchmark for instruction-following text-to-speech that uses a hierarchical multi-axis taxonomy, a scalable three stage data construction pipeline, and that hierarchical hybrid evaluation protocol.

Dev: The authors present this to systematically support fine grained and scalable evaluation of controllability and generation quality across diverse control settings and languages.

Taro: What this means is we have a better way to measure how well these instruction following systems actually work when they are being controlled by complex, nuanced prompts.

Rosa: It moves the conversation from just checking if the voice sounds good to systematically testing if it can follow a wide variety of human instructions consistently in many languages.

Dev: The authors show that while commercial systems still lead on English, open source models are getting competitive in localized areas like Chinese, but they highlight that compositional and extra vocal cases are where the real struggle is right now.

Taro: So the implication for us is that we need benchmarks that test those hard compositional and extra vocal controls to really push these systems further into useful, controllable speech generation.

Conclusion: Rosa: So we’re wrapping up our look at MINT-Bench, this new benchmark for instruction following text to speech.

Dev: Yeah, it’s essentially this big framework they put together to systematically test how well these models actually follow complex voice commands across different languages and styles.

Taro: It sounds like they aren't just testing if the voice sounds good, but if it can reliably do specific things you ask for, like changing the emotion or the accent exactly how you want.

Rosa: Exactly. The authors are building this structure to make sure we aren't just looking at a few examples, but getting a really clear picture of where these systems are actually succeeding and where they’re struggling.

Dev: They did this by creating a way to build the benchmark in stages, starting with defining the different types of speech control you can ask for—like age or texture—and then realizing those definitions into actual text prompts.

Taro: That construction pipeline sounds like it's designed to make sure the tests aren't just random prompts, but structured challenges that actually probe the limits of what the AI can do.

Rosa: And they’re using this hierarchical evaluation protocol, which means they test things in order, starting with checking if the speech content itself makes sense and then moving on to how well it followed your instruction.

Dev: It’s a multi-step check—first content consistency, then instruction following judged by a large language model like Gemini, and finally perceptual quality only for the samples that actually passed the instruction test.

Taro: That sounds like a good way to isolate where the failure modes are happening. They aren't just looking at one big score; they’re breaking it down into pieces so we can see exactly which kind of control is hard.

Rosa: And the results show that even with this comprehensive setup, we still see a gap between the best commercial systems on English and what the leading open source models can do in more localized settings like Chinese.

Dev: It confirms that while the tech is getting sophisticated, there’s still a big difference in how well they handle those harder tasks—the compositional ones and the extra vocal controls.

Taro: So it tells us that just making the voice sound natural isn't enough; we need to focus our research on making these systems robust when they have to do multiple, specific things at once.

Huakang Chen, Jingbin Hu, Liumeng Xue, Qirui Zhan, Wenhao Li, Guobin Ma, Hanke Xie, Dake Guo, Linhan Ma, Yuepeng Jiang, Bengu Wu

Audio, Speech and Language Processing Lab (ASLP@NPU) · School of Intelligence Science and Technology, Nanjing University School of Intelligence Science and Technology, Suzhou China · Yutu Zhineng Beijing China · Lingguang Zhaxian Technology Shanghai China

eess.AS, cs.SD

Submitted: 2026-04-20

Updated: 2026-11-09

Code: https://github.com/inclusionAI/Ming-omni-tts2https:

Project page: https://aslp-lab.github.io/MINT-Bench-Demo

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation…

Key concepts

Hierarchical Multi-axis Taxonomy
This is the core structure of the benchmark, organizing TTS attributes into four axes: Difficulty Level, Control Domain (like Age or Pitch), Control Specification (how precise the control is), and Fine-grained Control Pattern. It uses ten atomic acoustic attributes to define a vast inventory for testing.
Multi-stage Data Construction Pipeline
Instead of creating instruction pairs directly, this pipeline builds them in three stages: first defining valid cases based on the taxonomy (Node Specification), second creating a structured plan scaffold, and finally realizing that plan into natural language instructions paired with synthesis text to ensure high control.
Hierarchical Hybrid Evaluation Protocol
This protocol assesses TTS quality progressively. It first checks content consistency via ASR error rates, then uses a Large Audio Language Model (LALM) like Gemini to score instruction following, and finally applies perceptual bonuses only for samples that pass the instruction-following test.

Terminology

Summary

Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation, yet its evaluation remains underdeveloped due to limited benchmark coverage, weak diagnostic granularity, and insufficient multilingual support MINT-Bench is a comprehensive multilingual benchmark for instruction-following TTS built upon a hierarchical multi-axis taxonomy, a scalable multi-stage data construction pipeline, and a hierarchical hybrid evaluation protocol that jointly assesses content consistency, instruction following, and perceptual quality Experiments across ten languages show that current systems remain far from solved: frontier commercial systems lead overall, while leading open-source models become highly competitive and can even outperform commercial counterparts in localized settings such as Chinese The benchmark further reveals that harder compositional and paralinguistic controls remain major bottlenecks for current systems

Benchmark Structure and Taxonomy

MINT-Bench formulates instruction-following TTS evaluation as a structured and extensible benchmark construction problem rather than treating it as a flat collection of prompts At its core is a Hierarchical Multi-axis Taxonomy grounded in ten atomic timbre and style attributes that define the core benchmark inventory This taxonomy organizes benchmark coverage across four axes: Difficulty Level, Control Domain, Control Specification, and Fine-grained Control Pattern The core of the taxonomy is ten atomic acoustic attributes summarized in Table 1 which include Age, Gender, Pitch, Texture (timbre), Speed, Volume (style), Emotion (style), Accent (style/language-specific), Personality (style/persona), Tone (style/tone), and Extra Vocal control

Data Construction Pipeline

MINT-Bench constructs benchmark instances through a controllable multi-stage pipeline rather than generating final instruction-text pairs in a single step The construction pipeline consists of three stages: node specification, structured label planning, and instruction-text realization Stage 1 involves Taxonomy Node Specification to define valid cases under the taxonomy and prepare internal inventory for coverage control Stage 2 is Structured Label Planning which converts node specifications into an intermediate structured plan that serves as a normalized control scaffold Stage 3 is Instruction-Text Pair Construction where the structured plan is realized as a natural-language instruction paired with a synthesis text, designed to minimize direct lexical leakage of the control target

Hierarchical Evaluation Protocol

MINT-Bench introduces a hierarchical hybrid evaluation protocol that progressively evaluates content consistency, instruction following, and perceptual quality by combining objective tools with Large Audio Language Model (LALM)-based judgments Stage 1 assesses Content Consistency using ASR transcription and sample-level Word Error Rate (WER) checking against language-specific thresholds to calculate a content consistency coefficient Stage 2 evaluates Instruction Following by using a LALM, such as Gemini, assigning a discrete instruction-following score based on task-aware judging prompts derived from the taxonomy Stage 3 assesses Perceptual Quality and Timbre Diversity only for samples with strong instruction-following performance (score = 3) by applying binary perceptual bonuses and calculating a Timbre Diversity Score (TDS)

Experimental Findings

Experiments across ten languages show that instruction-following TTS remains far from solved, with frontier commercial systems leading overall performance, while leading open-source systems become highly competitive in localized settings such as Chinese The benchmark reveals that strong content fidelity does not necessarily imply strong controllability, and harder compositional and Special extra-vocal cases remain major bottlenecks for current systems Timbre diversity should be interpreted as a separate conditional diagnostic of oneto-many generation ability rather than as a surrogate for overall controllability Difficulty-sensitive results expose a clear drop from easy control to structured and extra-vocal cases, with Disfluency Control consistently being the hardest subset

Key Contributions

The key contributions of the work are summarized as follows:

• We present MINT-Bench, a comprehensive multilingual benchmark for instruction-following TTS designed to support systematic, fine-grained, and scalable evaluation of controllability and generation quality

• We introduce a hierarchical multi-axis taxonomy together with a scalable multi-stage data construction pipeline enabling semantically clear, structurally controlled, and extensible benchmark construction across diverse control settings and languages

• We design a hierarchical hybrid evaluation protocol that progressively assesses content consistency, instruction following, and perceptual quality by combining objective tools with LALM-based judgments

Limitations and Future Directions

Although MINT-Bench provides a structured and extensible benchmark for instruction-following TTS, several limitations remain First, MINT-Bench covers a bounded benchmark space rather than the full range of real-world instructions Second, the current multilingual setting is not fully balanced as English and Chinese use large splits while others use mini splits Third, the evaluation pipeline relies partly on LALM-based judging which may introduce bias and instability Future work should focus on expanding coverage, improving multilingual balance, and reducing dependence on proprietary evaluators

References

[1] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang et al. 2024. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. p. 9 CoRR abs/2406. p. 902430 (2024). p. 9 arXiv:2406. p. 902430 doi:10. p. 948550/ARXIV. p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 92406.02430

[1] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu et al. 2022. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. p. 9 IEEE J. Sel. Top. Signal Process. p. 9 16, 6 (2022), 1505–1518 doi:10. p. 91109/JSTSP. p. 92022.3188113

[3] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng et al. 2025. F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. p. 9 In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025 doi:10. p. 948550/ARXIV.2506.16381

[4] Wei Deng, Siyi Zhou, Jingchen Shu et al. 2025. IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System. p. 9 CoRR abs/2502. p. 9 p. 905512 (2025). p. 9 arXiv:2502. p. 9 p. 905512 doi:10. p. 948550/ARXIV.2503.

[6] Zhihao Du, Qian Chen, Shiliang Zhang et al. p. 9 2024. CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. p. 9 CoRR abs/2407. p. 905407 (2024). p. 9 arXiv:2407. p. 905407 doi:10. p. 948550/ARXIV.2412

[13] Kexin Huang, Qian Tu, Liwei Fan et al. 2025. InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems. p. 9 p. 9 CoRR abs/2506. p. 9 p. 916381 (2025). p. 9 p. 9 arXiv:2506. p. 9 p. 916381 doi:10. p. 9 p. 948550/ARXIV.2506

[37] Dongchao Yang, Songxiang Liu, Rongjie Huang et al. 2024. InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt. p. 1 p. 1 IEEE ACM Trans. p. 9 p. 9 Audio Speech Lang. p. 0 p. 0 Process. 32 (2024), 2913–2925 doi:10. p. 9 p. 91109/TASLP. p. 9 p. 9

[44] Yixuan Zhou, Xiaoyu Qin, Zeyu Jin et al. p. 9 2024. VoxInstruct: Expressive Human Instructionto-Speech Generation with Unified Multilingual Codec Language Modelling. p. 1 In Proceedings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne, VIC, Australia doi:10. p. 9 p. 91145/3664647.3681680

[5] Anuj Diwan, Zhisheng Zheng, David Harwath et al. 2025. Scaling Rich Style-Prompted Text-to-Speech Datasets. p. 9 In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China doi:10. p. 918653/V1/2025. p. 9EMNLP-MAIN. p. 9180

[7] Zhihao Du, Yuxuan Wang et al. 2024. CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. p. 9 CoRR abs/2412. p. 910117 (2024). p. 9 arXiv:2412. p. 910117 doi:10. p. 948550/ARXIV.

[8] Zhifu Gao, Shiliang Zhang et al. p. 9 2023. Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition. p. 9 In 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022 doi:10. p. 921437/INTERSPEECH. p. 9

[9] Haohan Guo, Kun Xie et al. p. 9 2025. FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech System. p. 9 CoRR abs/2503. p. 920499 (2025). p. 9 arXiv:2503. p. 9

[11] Jingbin Hu, Huakang Chen et al. p. 9 2026. VoiceSculptor: Your Voice, Designed By You. p. 9 arXiv:2601. p. 910629 [eess. p. 9AS] https://arxiv. p. 9 p. 9 p. 9org/abs/2601. p. 910629

[13] Kexin Huang, Qian Tu, Liwei Fan et al. 2025. InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems. CoRR abs/2506.16381 (2025). arXiv:2506.16381 doi:10.48550/ARXIV.

[40] Jun Zhan, Mingyang Han et al. p. 9 2025. VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions. p. 2 CoRR abs/2509. p. 9 p. 9 p. 9 p. 909716 (2025). p. 9 arXiv:2509. p. 0 p. 0 p. 009716 doi:10. p. 948550/ARXIV.

[43] Siyi Zhou, Yiquan Zhou, Yi He et al. p. 9 2026. IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech. p. 9 In Fortieth AAAI Conference on Artificial Intelligence, Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20-27, 2026 doi:10. p. 01609/AAAI.V40I41.40820

[35] LLM-Core Xiaomi. 2025. MiMo-Audio: Audio Language Models are Few-Shot Learners. p. 6 CoRR abs/2512. p. 923808 (2025). p. 9 arXiv:2512. p. 0

[36] Hanke Xie, Haopeng Lin et al. p. 0 p. 0 2025. SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity. p. 8 p. 8 arXiv:2510. p. 0 p. 023541 [eess. p. 0 p. 0AS] https://arxiv.org/abs/2510. p. 9 p. 923541

[38] Guanrou Yang, Chen Yang et al. 2025. EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting. p. 1 In Proceedings of the 33rd ACM International Conference on Multimedia, MM 2025, Dublin, Ireland, October 27-31, 2025 doi:10. p. 91145/3746027.3754829

[46] Xinfa Zhu, Wenjie Tian et al. 2024. UniStyle: Unified Style Modeling for Speaking Style Captioning and Stylistic Speech Synthesis. p. 0 In Proceedings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne, VIC, Australia doi:10.1145/3664647.3681465

[29] Gemini Team. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. p. 9 CoRR abs/2507. p. 906261 (2025). p. 9 arXiv:2507. p. 906261 doi:10. p. 948550/ARXIV.

[33] Yuancheng Wang, Haoyue Zhan et al. p. 9 2025. MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer. p. 9 In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 OpenReview. p. 9net doi:10. p. 948550/ARXIV.

[39] Zhen Ye, Xinfa Zhu et al. p. 9 2025. Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis. p. 8 CoRR abs/2502.04128 (2025). p. 9 arXiv:2502.04128 doi:10. p. 948550/ARXIV.

[31] Chengyi Wang, Sanyuan Chen et al. p. 9 p. 9 p. 9 2023. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. p. 9 p. 9 p. 9 CoRR abs/2301. p. 9 p. 9 p. 902111 (2023). p. 9 p. 9 p. 9 arXiv:2301. p. 9 p. 9 p. 902111 doi:10. p. 948550/ARXIV.

[36] Hanke Xie, Haopeng Lin et al. 2025. SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity. arXiv:2510.23541 [eess.AS] https://arxiv.org/abs/2510.23541

[37] Dongchao Yang, Songxiang Liu et al. 2024. InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt. IEEE ACM Trans. Audio Speech Lang. Process. 32 (2024), 2913–2925 doi:10.1109/TASLP.

[45] Yixuan Zhou, Guoyang Zeng et al. p. 9 p. 9 p. 9 2025. VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning. CoRR abs/2509.24650 (2025). p. 9 p. 9 arXiv:2509.

[31] Chengyi Wang, Sanyuan Chen et al. 2023. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. CoRR abs/2301.02111 (2023). arXiv:2301.

[45] Yixuan Zhou, Guoyang Zeng et al. 2025. VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning. CoRR abs/2509.24650 (2025). arXiv:2509.

[31] Chengyi Wang, Sanyuan Chen et al. 2023. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. CoRR abs/2301.02111 (2023). arXiv:2301.

[45] Yixuan Zhou, Guoyang Zeng et al. 2025. VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning. CoRR abs/2509.

Appendix Details

The Appendix provides extended details of the hierarchical multi-axis taxonomy, including expanded definitions of attribute values, full taxonomy nodes, control patterns, and item budgets for both Large and Mini splits It also includes detailed prompts for the Structured Label Planning stage and Instruction-Text Pair Construction stage to ensure reproducibility Finally, it details the LALM evaluation prompts for Easy, Hard, and Special nodes as well as language-specific WER thresholds The Appendix also includes a human annotation setup to validate the reliability of the LALM-based judging pipeline

Table Summaries

Table 3 reports detailed results on English (en) and Chinese (zh) large splits, showing that strong commercial systems define the leading frontier on English, but leading open-source systems can already become highly competitive in localized bilingual settings Table 4 summarizes multilingual evaluation results across ten languages, showing that Gemini 2.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement a hierarchical multi-axis taxonomy to structure benchmark coverage around ten atomic acoustic attributes, allowing for systematic comparison across timbre and style domains. This allows researchers to move beyond flat prompt sets toward a more structured and diagnostically informative evaluation framework.

  2. Develop a scalable multi-stage data construction pipeline consisting of node specification, structured label planning, and instruction-text realization. This pipeline ensures that the final instruction-text pairs are semantically clear, structurally controlled, and extensible across diverse control settings and languages.

  3. Design a hierarchical hybrid evaluation protocol that progressively assesses content consistency, instruction following (using LALM scores), and perceptual quality using objective tools combined with Large Audio Language Model judgments. This allows for fine-grained diagnosis of where systems succeed or fail by combining Stage 1, 2, and 3 metrics.

  4. Create a conditional perceptual quality stage that is applied only to samples with strong instruction-following performance (level 3), refining the final score through naturalness and expressiveness bonuses. This ensures that high-quality instruction following is measured against high perceptual quality without penalizing clearly failed or partially realized samples.

  5. Establish language-specific Word Error Rate thresholds based on a trimmed mean of overall system WERs for each language, ensuring that content consistency checks are fair across different linguistic resources and not disproportionately penalizing low-resource languages.

Sources

Related papers