GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark

arXiv:2606.28884 · eess.AS · Submitted 2026-06-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark".

Dev: Here's the gist of the paper. GigaSpeechBench introduces a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech to address limitations in existing evaluations that often overestimate real-world robustness <ref:2606.28884#pg2>.

Rosa: Benchmark Composition GigaSpeechBench features five distinct modules designed to capture diverse real-world challenges <ref:2606.28884#pg6>.

Dev: These modules include: 1.

Rosa: 14 languages and regions, such as 7 Arabic-speaking regions, 5 Southeast Asian languages, and 2 East Asian languages with challenging speech like Japanese and Korean <ref:2606.28884#pg7>.

Paper summary: Rosa: So, looking at the GigaSpeechBench paper by Yujie Tu and his team, what they’re really pushing is the need for a unified testbed that reflects real-world complexity. It claims that standard benchmarks don't tell you how robust an AI system actually is when it faces the diverse acoustic and linguistic challenges people encounter every day <ref:2606.28884#pg1> and this benchmark aims to provide that comprehensive view <ref:2606.28884#pg1>.

Dev: They are presenting six hundred eighty hours of data covering everything from different accents and dialects to domain-specific terms, suggesting that the performance degradation you see on standard tests doesn't always translate when you introduce these real-world variables <ref:2606.28884#pg10>.

Taro: The implication for us, as people building these systems, is that relying on high scores on existing benchmarks isn't enough; we need to test the system’s failure modes when the world misbehaves in a complex way <ref:2606.28884#pg2>.

Rosa: They emphasize that systems nearing saturation on those standard tests degrade substantially when they hit low-resource or dialectal speech, and accent robustness varies sharply across English varieties, which is a big point for anyone working on multilingual ASR <ref:2606.28884#pg10>.

Dev: And they point out that the aggregate word error rate can hide significant struggles with entity-rich utterances in those vertical domains, showing you need to look at performance granularly rather than just a single number <ref:2606.28884#pg15>.

Taro: It seems like this benchmark is designed to be a diagnostic tool, giving us reproducible results for tracking progress on these underrepresented and practically critical dimensions of robust speech recognition <ref:2606.28884#pg10>.

Rosa: So, in short, GigaSpeechBench gives us a common testbed to measure performance across all those long-tail evaluation axes we talked about, making it clear where the gaps are between lab results and actual usage <ref:2606.28884#pg10>.

Conclusion: Rosa: So, we’re wrapping up our look at GigaSpeechBench, which is this massive multilingual speech testbed.

Dev: Yeah, it’s about taking all this data and showing how much real-world complexity breaks the models we usually test them on in a lab.

Rosa: The authors built over six hundred hours of human-annotated speech to cover languages, dialects, accents, and even speech from older adults or children.

Dev: It's not just about collecting more data; it’s about making sure the evaluation measures actually reflect how these systems fail when they hit those specific real-world conditions.

Rosa: They found that strong performance on standard tests doesn't reliably transfer to this messy, in-the-wild speech you hear every day.

Dev: Exactly. The results show that models getting close to perfect on existing benchmarks drop off a lot when they encounter low-resource languages or specific accents.

Rosa: And it’s not just about the overall error rate; they looked at how the systems handle specific words in different subject areas, like those vertical domains.

Dev: That’s a key part of it. You can have a good number but still fail miserably on something critical, like recognizing a specific term in a technical setting.

Rosa: So, this benchmark is set up to be that diagnostic tool we talked about—a way to track progress on those hard areas where the current models are weakest.

Dev: It really gives us a better picture of where the real safety risks are for deploying these AI systems in the wild.

Rosa: Because if you want to know if an AI can actually handle a diverse, messy conversation, this is one way to see how it stacks up against what’s possible outside the lab.

SJTU · SII · Alibaba · THU · ASLP@NPU · NTU · CASIA · UIUC-SZ · CUHK-SZ · Fudan University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Audio Speech and Language Processing Group, School of Computer Science, Northwestern Polytechnical University, Tianjin University, Nanyang Technological University, Institute of Automation Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of Illinois Urbana-Champaign · The Chinese University of Hong Kong Shenzhen · Fudan University, Shanghai, China. · State Key Laboratory of Complex & Critical Software Environment, Seattle.

eess.AS

Submitted: 2026-06-27

Updated: 2026-11-12

Code: https://github.com/SpeechColab/GigaSpeechBenchhttps:

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 98/100

The gist: GigaSpeechBench introduces a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech to address limitations in existing…

Key concepts

In-the-Wild Speech
This refers to spontaneous, unscripted conversations collected from real life, not clean studio recordings. The data was gathered using heuristic screening to prioritize natural speech over read or scripted content. This makes the benchmark more representative of how ASR systems perform in actual use.
Vertical Domains
These are specific areas of specialized vocabulary across 12 vertical domains for both Chinese and English. Evaluating performance here tests an ASR system's ability to recognize entity-aware terminology, which is crucial for real-world applications like medical or legal transcription.
Accent Robustness
This concept measures how well a speech recognition model handles different ways of pronouncing English, such as Chinese, Indian, Japanese, and Scottish English. The benchmark shows that performance varies sharply across these varieties, indicating that models trained on standard accents may struggle with others.
Long-Tail Evaluation Axes
The goal is to create a unified testbed for evaluating speech recognition systems across many diverse and underrepresented dimensions. This allows researchers to track progress on critical areas like low-resource languages and dialectal speech, which are often overlooked by existing benchmarks.

Terminology

Summary

GigaSpeechBench introduces a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech to address limitations in existing evaluations that often overestimate real-world robustness <ref:2606. p. 0

Where the paper says it

Existing evaluations address challenges in isolation, lacking a unified benchmark for domain terminology, age variation, dialects, accents, and low-resource languages, particularly across the Middle East and Southeast Asia, representing over one billion under-evaluated speakers. To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech.

28884#pg2>.

Benchmark Composition

GigaSpeechBench features five distinct modules designed to capture diverse real-world challenges <ref:2606.28884#pg6>. These modules include:

  1. 14 languages and regions, such as 7 Arabic-speaking regions, 5 Southeast Asian languages, and 2 East Asian languages with challenging speech like Japanese and Korean <ref:2606. p. 1

    Where the paper says it

    Collectively, these limitations create a gap between benchmark performance and real-world ASR reliability, making it unclear whether recent systems are genuinely robust or merely well-adapted to existing benchmark distributions. To address this gap, we introduce GigaSpeechBench, a 680-hour human-annotated ASR benchmark built from in-the-wild speech under complex acoustic conditions, organized into five modules: • 14 languages and regions: 7 Arabic-speaking regions, including Iraq, Algeria, the United Arab Emirates, Egypt, Morocco, Saudi Arabia, and Syria; 5 Southeast Asian languages, including Indonesian, Malay, Filipino (Tagalog), Vietnamese, and Thai; and 2 East Asian languages with challenging speech, Japanese and Korean, with 20 hours each.

    28884#pg7>.

  2. 6 Chinese dialects, with each dialect having a dedicated set of hours for evaluation <ref:2606. p. 9

    Where the paper says it

    Child speech is typically characterized by a higher fundamental frequency, developing vocal-tract characteristics, and less stable pronunciation, whereas older-adult speech may exhibit reduced volume, slower articulation, and age-related voice variation. To evaluate whether current ASR systems can handle such demographic differences, we construct dedicated child and older-adult speech evaluation sets for both English and Chinese, with 10 hours of speech for each age group in each language.

    28884#pg8>.

  3. 6 English accents, covering Chinese, Indian, Japanese, Filipino, Scottish, and Singaporean English <ref:2606. p. 1

    Where the paper says it

    • 6 Chinese dialects: Xiang, Jin, Gan, Min, Yue, and Wu, with 10 hours each. • 6 English accents: Chinese, Indian, Japanese, Filipino, Scottish, and Singaporean English, with 10 hours each.

    28884#pg8>.

  4. Dense terminology across 12 vertical domains for both Chinese and English <ref:2606. p. 0

    Where the paper says it

    To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech. It features five modules: (1) 12 low-resource Middle Eastern and Southeast Asian languages, plus challenging Japanese and Korean; (2) 6 Chinese dialects; (3) 6 English accents; (4) dense terminology across 12 vertical domains for Chinese and English; and (5) older adult and child speech.

    28884#pg8>.

  5. Speech from older adults and children, with dedicated evaluation sets for both age groups in Mandarin and English <ref:2606. p. 1

    Where the paper says it

    In addition, current ASR evaluations remain limited in their coverage of terminology-dense speech, despite its importance in professional domains such as medicine, law, finance, and technology Wang et al. Older adult and child speech requires dedicated ASR evaluation as age-related speech variation introduces unique challenges, including vocal tract development, higher pitch, and inconsistent pronunciation in children, as well as reduced volume, slower articulation, and tremors in older adults Zhou et al.

    28884#pg8>.

Data Curation Pipeline

The construction of GigaSpeechBench involved a rigorous curation pipeline aimed at collecting spontaneous speech under complex conditions <ref:2606.28884#pg10>. This process included:

  1. Source Discovery and Video Selection, which used a heuristic screening procedure based on multiple evidence sources to prioritize spontaneous conversational speech and exclude read or scripted content. p. 4

    Where the paper says it

    We first build candidate source pools from YouTube. Since the language or regional variety of a video cannot be determined from metadata alone, we use a heuristic screening procedure based on multiple evidence sources, including channel descriptions, video titles, comments, uploader information, and other publicly available metadata.

  2. Video-level audio screening, where videos longer than one hour were removed to prevent a few speakers from dominating the benchmark. p. 4

    Where the paper says it

    We then perform video-level audio screening before annotation. Videos longer than one hour are removed to prevent a few speakers or sources from dominating the benchmark and to limit within5

  3. Segmentation and Manual Transcription, where retained segments were transcribed in the native writing form of the target language or regional variety by professional annotators. p. 6

    Where the paper says it

    Each retained segment is transcribed in the native writing form of the target language or regional variety. Annotators are instructed to transcribe only target-language speech.

  4. Quality Control and Test Set Construction, which involved manual quality inspection and post-processing to remove invalid segments, non-target-language speech, and audio with low intelligibility. p. 7

    Where the paper says it

    We further conduct post-processing before forming the final benchmark. This step removes invalid segments missed during annotation, segments dominated by non-target-language speech, incomplete or clearly mismatched transcriptions, and audio whose intelligibility is too low for stable evaluation.

Evaluation Metrics Focus

The benchmark evaluates performance across various dimensions to expose critical evaluation blind spots <ref:2606.28884#pg10>. Key evaluation aspects include:

  1. Multilingual and Low-Resource Languages, targeting underrepresented populations in the Middle East and Southeast Asia. p. 8

    Where the paper says it

    For each subsequent module, the same set of systems is evaluated wherever the language or domain is supported by the system; cells marked “–” in the result tables indicate that the corresponding system does not officially support that language or returned no usable output. 2 Low Resource Language We construct a multilingual ASR benchmark targeting low-resource languages, with a particular focus on underrepresented regions across the Middle East, Southeast Asia, and East Asia.

  2. Chinese Dialects and Accented English, using a unified 10-hour-per-variety protocol for direct comparison. p. 2

    Where the paper says it

    (2023) reveal severe performance drops across diverse accents. GigaSpeechBench enables direct comparison by evaluating six Chinese dialects and six English accents under a unified 10-hour-per-variety protocol.

  3. Domain-Specific Terminology, utilizing paired hotword lists across twelve vertical domains to measure entity-aware ASR. p. 2

    Where the paper says it

    (2023) and audio-LM-based systems show little to no average-WER change and only modest, model-dependent gains on entity-rich tokens. In contrast, GigaSpeechBench provides paired hotword lists across twelve vertical domains for both Chinese and English on genuine human speech, enabling controlled measurement of entity-aware ASR rather than relying on synthetic audio.

  4. Speaker Demographics, unifying evaluation of older adult and child speech in both Mandarin and English. p. 3

    Where the paper says it

    (2025), or on seniors, such as SeniorTalk yang et al. GigaSpeechBench unifies both, offering 10 hours of child and older-adult speech in both Mandarin and English.

  5. Speech-to-Text Translation, providing human-translated references for 11 underrepresented source languages under noisy conditions. p. 3

    Where the paper says it

    (2025)) are largely confined to European languages, formal settings, or synthetic/read audio. GigaSpeechBench advances ST evaluation by providing human-translated references on the same in-the-wild audio used for ASR, covering 11 underrepresented Middle Eastern and Southeast Asian languages under noisy, multi-speaker conditions.

Key Findings on Model Performance

Extensive evaluations of leading foundation models and commercial APIs reveal significant performance degradation when tested on GigaSpeechBench compared to existing benchmarks <ref:2606.28884#pg10>. Specifically, the results demonstrate that strong performance on standard benchmarks does not reliably transfer to these challenging settings. p. 2

Where the paper says it

(2021) ✗ ✓ ✗ ✗ ✗ 8 - Read ContextASR-Bench Wang et al. (2025a) ✗ ✗ ✗ ✓ ✗ 10+ - Synthetic GigaSpeechBench (Ours) ✓ ✓ ✓ ✓ ✓ 14/6/6/12/2 20/10/10/20/10 Spontaneous Evaluating leading foundation models and commercial APIs on GigaSpeechBench reveals that strong performance on existing benchmarks does not reliably transfer to these challenging settings, exposing critical evaluation blind spots.

The study shows that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech. p. 9
Where the paper says it

Our evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions. We demonstrate that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech.

Furthermore, accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains. p. 9
Where the paper says it

We demonstrate that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains.

Limitations of the Benchmark

Several limitations are acknowledged regarding the benchmark construction and evaluation methods. These include:

  1. Text normalization for low-resource languages still requires refinement by native-speaking linguistic experts. p. 8

    Where the paper says it

    Table 2 presents the performance of representative state-of-the-art ASR systems on our benchmark. Overall, these results show that current ASR systems still struggle substantially on newly collected in-the-wild low-resource and regional speech, even when they perform well on standard benchmarks such as Common Voice and FLEURS.

  2. Some Chinese dialects lack unified standard writing systems, leading to potential inconsistencies in character-level outputs that make CER insufficient for full assessment. Future work should explore evaluation criteria better suited to low-resource languages and dialectal scenarios than WER/CER. p. 3

    Where the paper says it

    Table 2: Low-resource languages benchmark results. WER (%) is reported for Arabic and Southeast Asian languages, while CER (%) is reported for East Asian languages.

Ethical Considerations

The ethical framework for the dataset ensures responsible data handling. All collected audio is sourced from materials released under a Creative Commons license, and personally identifiable information (PII) samples have been manually removed during annotation. p. 4

Where the paper says it

We then perform video-level audio screening before annotation. Videos longer than one hour are removed to prevent a few speakers or sources from dominating the benchmark and to limit within5

Annotators are compensated fairly through a professional data annotation company, and the team is committed to ongoing maintenance of the dataset. p. 6
Where the paper says it

The screened videos are sent to a professional annotation company for voice activity detection and manual transcription. Annotators first segment the continuous audio into utterance-level speech segments.

Conclusion

In this work, GigaSpeechBench provides a unified multilingual ASR benchmark designed to place long-tail evaluation axes on a common testbed. The construction of this benchmark involved curating over 680 hours of newly collected, manually transcribed in-the-wild speech covering 12 underrepresented languages from the Middle East and Southeast Asia, six Chinese dialects, six English accents, and 12 hotword-rich vertical domains in both Chinese and English. p. 9

Where the paper says it

5 Conclusion In this work, we present GigaSpeechBench, a unified multilingual ASR benchmark designed to place long-tail evaluation axes on a common testbed. To construct this benchmark, we curated over 680 hours of newly collected, manually transcribed in-the-wild speech covering 12 underrepresented languages from the Middle East and Southeast Asia, six Chinese dialects, six English accents, and 12 hotword-rich vertical domains in both Chinese and English.

The evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions. p. 9
Where the paper says it

Across this testbed, we conducted a large-scale comparison of representative commercial APIs, closed-source ASR-specialized models, and open-source foundation systems. Our evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions.

p. 9
Where the paper says it

Across this testbed, we conducted a large-scale comparison of representative commercial APIs, closed-source ASR-specialized models, and open-source foundation systems. Our evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions.

Systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech. p. 9
Where the paper says it

Our evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions. We demonstrate that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech.

The study demonstrates that accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains. p. 9
Where the paper says it

We demonstrate that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains.

p. 9
Where the paper says it

We demonstrate that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains.

We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts to serve as a reproducible diagnostic tool for tracking progress on these underrepresented, yet practically critical, dimensions of robust speech recognition. p. 9
Where the paper says it

We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts. We hope this benchmark serves as a reproducible diagnostic tool for tracking progress on these underrepresented, yet practically critical, dimensions of robust speech recognition.

p. 9
Where the paper says it

We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts. We hope this benchmark serves as a reproducible diagnostic tool for tracking progress on these underrepresented, yet practically critical, dimensions of robust speech recognition.

References

The paper cites numerous foundational works across ASR and MT research. Notable references include Conneau et al. p. 7

Where the paper says it

1 Evaluated Systems We evaluate a comprehensive set of state-of-the-art ASR systems spanning both commercial APIs and open-source models. Specifically, the commercial APIs include Microsoft Azure Speech1 , Google Chirp32 , OpenAI GPT-4o Transcribe3 , Gemini 3.

(2022) on FLEURS, Ardila et al. p. 2
Where the paper says it

/Age Speech Type FLEURS Conneau et al. (2022) ✓ ✗ ✗ ✗ ✗ 102 <3 Read Common Voice Ardila et al.

(2014) on BABEL <ref:2606. p. 2
Where the paper says it

3 Read BABEL Gales et al. (2014) ✓ ✗ ✗ ✗ ✗ 25 - Spontaneous MLC-SLM Mu et al.

28884#pg15>, and various works related to Chinese dialects like Wenetspeech-YUE Li et al. p. 8
Where the paper says it

All dialects are provided with phonetic transcriptions, while Min is instead provided with Mandarin translations due to the extreme complexity of its character-topronunciation mapping. It is worth noting that the Yue and Wu dialect evaluation sets are sourced from WenetSpeech-YUE Li et al.

(2026) <ref:2606. p. 0
Where the paper says it

(2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read or prompted speech collected under relatively clean conditions. Such settings underrepresent key Technical Report arXiv:2606.

p. 0
Where the paper says it

(2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read or prompted speech collected under relatively clean conditions. Such settings underrepresent key Technical Report arXiv:2606.

28884#pg14>. The paper also references work on speaker demographics such as SeniorTalk yang et al. p. 3
Where the paper says it

(2000), and ChildMandarin Zhou et al. (2025), or on seniors, such as SeniorTalk yang et al.

(2025) <ref:2606. p. 0
Where the paper says it

(2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read or prompted speech collected under relatively clean conditions. Such settings underrepresent key Technical Report arXiv:2606.

28884#pg14>.

Appendix Details

Segment duration and text length statistics show that the vast majority of audio segments fall between 0.5 and 10 seconds, consistent with spontaneous conversational speech <ref:2606. p. 4

Where the paper says it

A channel is retained only when these signals consistently suggest that it contains speech from the target language or regional variety. From the selected channels, we prioritize videos containing spontaneous conversational speech and exclude recordings dominated by read, scripted, or narration-style speech.

28884#pg15>. The reference text length distribution indicates most references fall within the 10–30 word range <ref:2606.28884#pg15>. Entity-level evaluation in vertical domains offers a more fine-grained diagnostic perspective than aggregate metrics, which can be dominated by non-terminological words <ref:2606. p. 9
Where the paper says it

Furthermore, accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains. We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts.

28884#pg15>. The final benchmark consists of utterance-level audio segments paired with native manual transcripts for native ASR evaluation <ref:2606. p. 7
Where the paper says it

The final benchmark consists of utterance-level audio segments paired with native manual transcripts. These test sets are used for native ASR evaluation under diverse but transcribable acoustic conditions.

28884#pg15>.

Table Summaries

Table 2 presents the performance of representative state-of-the-art ASR systems on the benchmark, showing struggles on low-resource languages. Table 3 reports results for Common Voice benchmark across various models and languages. p. 3

Where the paper says it

76 Table 3: Common Voice benchmark results. Only languages with available evaluation data are reported.

Table 4 shows FLEURS benchmark results across different systems and languages. p. 8
Where the paper says it

Table 2 presents the performance of representative state-of-the-art ASR systems on our benchmark. Overall, these results show that current ASR systems still struggle substantially on newly collected in-the-wild low-resource and regional speech, even when they perform well on standard benchmarks such as Common Voice and FLEURS.

Tables 6, 7, and 8 provide detailed CER and WER results for Chinese dialects, vertical domains, and English accents respectively. p. 9
Where the paper says it

Entity tokens that do not appear in the reference transcription are excluded from the B-WER denominator. Due to space limitations, the standard CER and WER tables together with their detailed discussion are deferred to Appendix C, and the B-CER results for Chinese and B-WER results for English are shown in Table 7 and Table 8.

Table 9 presents child and elderly speech recognition benchmark results. p. 6
Where the paper says it

Table 9: Child and elderly speech recognition benchmark results. WER (%) is used for English speech, and CER (%) is used for Chinese speech.

Table 10 shows English translation benchmark results using metrics like sacreBLEU and COMET. p. 8
Where the paper says it

These systems are evaluated using the OpenSTBench An et al. (2026) toolkit with standard translation-quality metrics, including sacreBLEU Post (2018), chrF++ Popovic´ (2017), COMET Rei et al.

Experimental Setup

The evaluation involves testing a comprehensive set of state-of-the-art ASR systems, including commercial APIs like Microsoft Azure Speech and OpenAI GPT-4o Transcribe, alongside open-source models such as Whisper Large v3 and Meta OmniASR 3B. For speech translation, systems like Azure Translate and Gemini 3. p. 9

Where the paper says it

7 Speech Translation GigaSpeechBench further supports speech-to-text translation evaluation by providing humantranslated English and Chinese references for 11 underrepresented source languages. We evaluate Azure Translate, Gemini-3-Flash-Preview, SeamlessM4T-v2-Large, and Qwen3.

0 Flash are evaluated using the OpenSTBench toolkit with standard translation-quality metrics. p. 8
Where the paper says it

These systems are evaluated using the OpenSTBench An et al. (2026) toolkit with standard translation-quality metrics, including sacreBLEU Post (2018), chrF++ Popovic´ (2017), COMET Rei et al.

The evaluation methodology ensures that for each subsequent module, the same set of systems is tested wherever the language or domain is supported by the system. p. 8
Where the paper says it

(2020), and BLEURT Sellam et al. For each subsequent module, the same set of systems is evaluated wherever the language or domain is supported by the system; cells marked “–” in the result tables indicate that the corresponding system does not officially support that language or returned no usable output.

Conclusion Summary

The work successfully presents GigaSpeechBench, a unified multilingual ASR benchmark designed to place long-tail evaluation axes on a common testbed. The evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions. Systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech <ref:2606. p. 9

Where the paper says it

Our evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions. We demonstrate that systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech.

28884#pg10>. The study demonstrates that accent robustness varies sharply across English varieties, and aggregate WER often masks sizeable degradation on entity-rich utterances in vertical domains. We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts to serve as a reproducible diagnostic tool for tracking progress on these underrepresented, yet practically critical, dimensions of robust speech recognition.

Author Affiliations

The authors are affiliated with institutions including X-LANCE Lab at MoE Key Lab of Artificial Intelligence and Alibaba Group <ref:2606.28884#pg15>. Other affiliations include Tianjin University, Tsinghua University, Nanyang Technological University, the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, the University of Illinois Urbana-Champaign, The Chinese University of Hong Kong-SZ, Fudan University in Shanghai and Seasalt.ai <ref:2606. p. 0

Where the paper says it

(2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read or prompted speech collected under relatively clean conditions. Such settings underrepresent key Technical Report arXiv:2606.

28884#pg15>.

References List

The paper includes a comprehensive list of references supporting its methodology and context. These include foundational work on multilingual speech corpora like Common Voice Ardila et al. p. 2

Where the paper says it

1 Multilingual and Low Resource Benchmarks Multilingual evaluations such as FLEURS Conneau et al. (2022) and Common Voice Ardila et al.

(2020) <ref:2606. p. 0
Where the paper says it

(2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read or prompted speech collected under relatively clean conditions. Such settings underrepresent key Technical Report arXiv:2606.

28884#pg14>, works on Chinese dialect resources such as Wenetspeech-chuan Li et al. p. 2
Where the paper says it

2 Chinese Dialects and Accented English Chinese dialect resources such as KeSpeech Tang et al. (2021), the WenetSpeech series Li et al.

(2026) <ref:2606.28884#pg14>, and papers detailing metrics like chrF++ Popovic´ (2017). p. 8
Where the paper says it

These systems are evaluated using the OpenSTBench An et al. (2026) toolkit with standard translation-quality metrics, including sacreBLEU Post (2018), chrF++ Popovic´ (2017), COMET Rei et al.

The list also includes recent advancements in large language models such as GPT-4o system card. p. 8
Where the paper says it

As shown in Table 5, despite recent advancements in foundation models, substantial performance gaps remain between standard and accented English, underscoring the persistent challenge of geographical robustness. 4 Chinese Dialects Our benchmark includes 10 hours of speech for each of the six major Chinese dialects: Xiang, Jin, Min, Yue, Gan and Wu.

Author List

The authors listed are Yujie Tu, Yifan Yang, Tianrui Wang, Yanqiao Zhu, Guodong Lin, Mingchen Shao, Haoran Wang, Junzhe Liu, Yuxiang Fu, Yizhou Peng, Changsong Liu7 (multiple references) <ref:2606.28884#pg15>. Other contributing authors include Kele Xu (multiple references) <ref:2606.28884#pg15>.

Contact Information

The paper provides links to the GitHub repository and Hugging Face dataset for GigaSpeechBench. These resources facilitate reproducible, real-world ASR evaluation. p. 2

Where the paper says it

(2025a) ✗ ✗ ✗ ✓ ✗ 10+ - Synthetic GigaSpeechBench (Ours) ✓ ✓ ✓ ✓ ✓ 14/6/6/12/2 20/10/10/20/10 Spontaneous Evaluating leading foundation models and commercial APIs on GigaSpeechBench reveals that strong performance on existing benchmarks does not reliably transfer to these challenging settings, exposing critical evaluation blind spots. All resources will be released to facilitate reproducible, real-world ASR evaluation.

The authors also provide links to the Azure Speech Services, Google Speech-to-Text, and OpenAI API documentation. p. 7
Where the paper says it

com/speech-to-text 3https://developers. com/api/docs/models/gpt-4o-transcribe 4https://docs.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Increase robustness against linguistic variability by training models specifically on GigaSpeechBench modules including 12 low-resource Middle Eastern and Southeast Asian languages, 6 Chinese dialects, and 6 English accents. This allows systems to perform reliably in settings where existing benchmarks overestimate real-world robustness.

  2. Enhance domain-specific recognition accuracy by utilizing the paired hotword lists across twelve vertical domains for both Chinese and English on genuine human speech to enable controlled measurement of entity-aware ASR rather than relying on synthetic audio.

  3. Improve performance in challenging acoustic environments by training models on spontaneous conversations, overlapping speech, background noise, far-field recordings, and device-dependent recordings as illustrated in Figure 1. This addresses the limitations where strong performance on existing benchmarks does not reliably transfer to these challenging settings.

  4. Enhance recognition of specialized vocabulary by employing a Biased Word Error Rate (B-WER) calculation that applies standard WER only to reference tokens belonging to annotated entity lists, providing a more fine-grained diagnostic perspective than aggregate metrics.

  5. Improve accuracy in demographic variations by including dedicated evaluation sets for older adult and child speech in both English and Chinese, which addresses the finding that demographic acoustic variation remains under-addressed by current ASR systems.

  6. Enhance speech translation capabilities by providing human-annotated Chinese and English translations for 11 languages to support AST evaluation on in-the-wild audio, specifically for underrepresented Middle Eastern and Southeast Asian languages.

Abstract

While modern ASR systems achieve low error rates on high-resource benchmarks, such performance often overestimates real-world robustness. Existing evaluations address challenges in isolation, lacking a unified benchmark for domain terminology, age variation, dialects, accents, and low-resource languages, particularly across the Middle East and Southeast Asia, representing over one billion under-evaluated speakers. To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 690 hours of human-annotated speech. It features five modules: (1) 12 low-resource Middle Eastern and Southeast Asian languages, plus challenging Japanese and Korean; (2) 6 Chinese dialects; (3) 6 English accents; (4) dense terminology across 12 vertical domains for Chinese and English; and (5) older adult and child speech. We further provide human-annotated Chinese and English translations for 11 languages to support AST evaluation. Extensive evaluations of leading foundation models and commercial APIs reveal significant performance degradation in these challenging settings, exposing critical evaluation blind spots.

Sources

Related papers