GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark".
Dev: Here's the gist of the paper. GigaSpeechBench introduces a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech to address limitations in existing evaluations that often overestimate real-world robustness <ref:2606.28884#pg2>.
Rosa: Benchmark Composition GigaSpeechBench features five distinct modules designed to capture diverse real-world challenges <ref:2606.28884#pg6>.
Dev: These modules include: 1.
Rosa: 14 languages and regions, such as 7 Arabic-speaking regions, 5 Southeast Asian languages, and 2 East Asian languages with challenging speech like Japanese and Korean <ref:2606.28884#pg7>.
Paper summary: Rosa: So, looking at the GigaSpeechBench paper by Yujie Tu and his team, what they’re really pushing is the need for a unified testbed that reflects real-world complexity. It claims that standard benchmarks don't tell you how robust an AI system actually is when it faces the diverse acoustic and linguistic challenges people encounter every day <ref:2606.28884#pg1> and this benchmark aims to provide that comprehensive view <ref:2606.28884#pg1>.
Dev: They are presenting six hundred eighty hours of data covering everything from different accents and dialects to domain-specific terms, suggesting that the performance degradation you see on standard tests doesn't always translate when you introduce these real-world variables <ref:2606.28884#pg10>.
Taro: The implication for us, as people building these systems, is that relying on high scores on existing benchmarks isn't enough; we need to test the system’s failure modes when the world misbehaves in a complex way <ref:2606.28884#pg2>.
Rosa: They emphasize that systems nearing saturation on those standard tests degrade substantially when they hit low-resource or dialectal speech, and accent robustness varies sharply across English varieties, which is a big point for anyone working on multilingual ASR <ref:2606.28884#pg10>.
Dev: And they point out that the aggregate word error rate can hide significant struggles with entity-rich utterances in those vertical domains, showing you need to look at performance granularly rather than just a single number <ref:2606.28884#pg15>.
Taro: It seems like this benchmark is designed to be a diagnostic tool, giving us reproducible results for tracking progress on these underrepresented and practically critical dimensions of robust speech recognition <ref:2606.28884#pg10>.
Rosa: So, in short, GigaSpeechBench gives us a common testbed to measure performance across all those long-tail evaluation axes we talked about, making it clear where the gaps are between lab results and actual usage <ref:2606.28884#pg10>.
Conclusion: Rosa: So, we’re wrapping up our look at GigaSpeechBench, which is this massive multilingual speech testbed.
Dev: Yeah, it’s about taking all this data and showing how much real-world complexity breaks the models we usually test them on in a lab.
Rosa: The authors built over six hundred hours of human-annotated speech to cover languages, dialects, accents, and even speech from older adults or children.
Dev: It's not just about collecting more data; it’s about making sure the evaluation measures actually reflect how these systems fail when they hit those specific real-world conditions.
Rosa: They found that strong performance on standard tests doesn't reliably transfer to this messy, in-the-wild speech you hear every day.
Dev: Exactly. The results show that models getting close to perfect on existing benchmarks drop off a lot when they encounter low-resource languages or specific accents.
Rosa: And it’s not just about the overall error rate; they looked at how the systems handle specific words in different subject areas, like those vertical domains.
Dev: That’s a key part of it. You can have a good number but still fail miserably on something critical, like recognizing a specific term in a technical setting.
Rosa: So, this benchmark is set up to be that diagnostic tool we talked about—a way to track progress on those hard areas where the current models are weakest.
Dev: It really gives us a better picture of where the real safety risks are for deploying these AI systems in the wild.
Rosa: Because if you want to know if an AI can actually handle a diverse, messy conversation, this is one way to see how it stacks up against what’s possible outside the lab.
SJTU · SII · Alibaba · THU · ASLP@NPU · NTU · CASIA · UIUC-SZ · CUHK-SZ · Fudan University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Audio Speech and Language Processing Group, School of Computer Science, Northwestern Polytechnical University, Tianjin University, Nanyang Technological University, Institute of Automation Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of Illinois Urbana-Champaign · The Chinese University of Hong Kong Shenzhen · Fudan University, Shanghai, China. · State Key Laboratory of Complex & Critical Software Environment, Seattle.
eess.AS
Submitted: 2026-06-27
Updated: 2026-11-12
Code: https://github.com/SpeechColab/GigaSpeechBenchhttps:
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: GigaSpeechBench introduces a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech to address limitations in existing…
Key concepts
- In-the-Wild Speech
- This refers to spontaneous, unscripted conversations collected from real life, not clean studio recordings. The data was gathered using heuristic screening to prioritize natural speech over read or scripted content. This makes the benchmark more representative of how ASR systems perform in actual use.
- Vertical Domains
- These are specific areas of specialized vocabulary across 12 vertical domains for both Chinese and English. Evaluating performance here tests an ASR system's ability to recognize entity-aware terminology, which is crucial for real-world applications like medical or legal transcription.
- Accent Robustness
- This concept measures how well a speech recognition model handles different ways of pronouncing English, such as Chinese, Indian, Japanese, and Scottish English. The benchmark shows that performance varies sharply across these varieties, indicating that models trained on standard accents may struggle with others.
- Long-Tail Evaluation Axes
- The goal is to create a unified testbed for evaluating speech recognition systems across many diverse and underrepresented dimensions. This allows researchers to track progress on critical areas like low-resource languages and dialectal speech, which are often overlooked by existing benchmarks.
Terminology
Summary
GigaSpeechBench introduces a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech to address limitations in existing evaluations that often overestimate real-world robustness <ref:2606. p. 0 Existing evaluations
address challenges in isolation, lacking a unified benchmark for domain terminology, age variation, dialects, accents, and low-resource languages, particularly across
the Middle East and Southeast Asia, representing over one billion under-evaluated
speakers. To address this gap, we introduce GigaSpeechBench, a comprehensive
multilingual and multidimensional in-the-wild ASR & AST benchmark comprising
680 hours of human-annotated speech.Where the paper says it
Benchmark Composition
GigaSpeechBench features five distinct modules designed to capture diverse real-world challenges <ref:2606.28884#pg6>. These modules include:
-
14 languages and regions, such as 7 Arabic-speaking regions, 5 Southeast Asian languages, and 2 East Asian languages with challenging speech like Japanese and Korean <ref:2606. p. 1
28884#pg7>.Where the paper says it
Collectively, these limitations create a gap between benchmark performance and real-world ASR reliability, making it unclear whether recent systems are genuinely robust or merely well-adapted to existing benchmark distributions. To address this gap, we introduce GigaSpeechBench, a 680-hour human-annotated ASR benchmark built from in-the-wild speech under complex acoustic conditions, organized into five modules: • 14 languages and regions: 7 Arabic-speaking regions, including Iraq, Algeria, the United Arab Emirates, Egypt, Morocco, Saudi Arabia, and Syria; 5 Southeast Asian languages, including Indonesian, Malay, Filipino (Tagalog), Vietnamese, and Thai; and 2 East Asian languages with challenging speech, Japanese and Korean, with 20 hours each.
-
6 Chinese dialects, with each dialect having a dedicated set of hours for evaluation <ref:2606. p. 9
28884#pg8>.Where the paper says it
Child speech is typically characterized by a higher fundamental frequency, developing vocal-tract characteristics, and less stable pronunciation, whereas older-adult speech may exhibit reduced volume, slower articulation, and age-related voice variation. To evaluate whether current ASR systems can handle such demographic differences, we construct dedicated child and older-adult speech evaluation sets for both English and Chinese, with 10 hours of speech for each age group in each language.
-
6 English accents, covering Chinese, Indian, Japanese, Filipino, Scottish, and Singaporean English <ref:2606. p. 1
28884#pg8>.Where the paper says it
• 6 Chinese dialects: Xiang, Jin, Gan, Min, Yue, and Wu, with 10 hours each. • 6 English accents: Chinese, Indian, Japanese, Filipino, Scottish, and Singaporean English, with 10 hours each.
-
Dense terminology across 12 vertical domains for both Chinese and English <ref:2606. p. 0
28884#pg8>.Where the paper says it
To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech. It features five modules: (1) 12 low-resource Middle Eastern and Southeast Asian languages, plus challenging Japanese and Korean; (2) 6 Chinese dialects; (3) 6 English accents; (4) dense terminology across 12 vertical domains for Chinese and English; and (5) older adult and child speech.
-
Speech from older adults and children, with dedicated evaluation sets for both age groups in Mandarin and English <ref:2606. p. 1
28884#pg8>.Where the paper says it
In addition, current ASR evaluations remain limited in their coverage of terminology-dense speech, despite its importance in professional domains such as medicine, law, finance, and technology Wang et al. Older adult and child speech requires dedicated ASR evaluation as age-related speech variation introduces unique challenges, including vocal tract development, higher pitch, and inconsistent pronunciation in children, as well as reduced volume, slower articulation, and tremors in older adults Zhou et al.
Data Curation Pipeline
The construction of GigaSpeechBench involved a rigorous curation pipeline aimed at collecting spontaneous speech under complex conditions <ref:2606.28884#pg10>. This process included:
-
Source Discovery and Video Selection, which used a heuristic screening procedure based on multiple evidence sources to prioritize spontaneous conversational speech and exclude read or scripted content. p. 4
Where the paper says it
We first build candidate source pools from YouTube. Since the language or regional variety of a video cannot be determined from metadata alone, we use a heuristic screening procedure based on multiple evidence sources, including channel descriptions, video titles, comments, uploader information, and other publicly available metadata.
-
Video-level audio screening, where videos longer than one hour were removed to prevent a few speakers from dominating the benchmark. p. 4
Where the paper says it
We then perform video-level audio screening before annotation. Videos longer than one hour are removed to prevent a few speakers or sources from dominating the benchmark and to limit within5
-
Segmentation and Manual Transcription, where retained segments were transcribed in the native writing form of the target language or regional variety by professional annotators. p. 6
Where the paper says it
Each retained segment is transcribed in the native writing form of the target language or regional variety. Annotators are instructed to transcribe only target-language speech.
-
Quality Control and Test Set Construction, which involved manual quality inspection and post-processing to remove invalid segments, non-target-language speech, and audio with low intelligibility. p. 7
Where the paper says it
We further conduct post-processing before forming the final benchmark. This step removes invalid segments missed during annotation, segments dominated by non-target-language speech, incomplete or clearly mismatched transcriptions, and audio whose intelligibility is too low for stable evaluation.
Evaluation Metrics Focus
The benchmark evaluates performance across various dimensions to expose critical evaluation blind spots <ref:2606.28884#pg10>. Key evaluation aspects include:
-
Multilingual and Low-Resource Languages, targeting underrepresented populations in the Middle East and Southeast Asia. p. 8
Where the paper says it
For each subsequent module, the same set of systems is evaluated wherever the language or domain is supported by the system; cells marked “–” in the result tables indicate that the corresponding system does not officially support that language or returned no usable output. 2 Low Resource Language We construct a multilingual ASR benchmark targeting low-resource languages, with a particular focus on underrepresented regions across the Middle East, Southeast Asia, and East Asia.
-
Chinese Dialects and Accented English, using a unified 10-hour-per-variety protocol for direct comparison. p. 2
Where the paper says it
(2023) reveal severe performance drops across diverse accents. GigaSpeechBench enables direct comparison by evaluating six Chinese dialects and six English accents under a unified 10-hour-per-variety protocol.
-
Domain-Specific Terminology, utilizing paired hotword lists across twelve vertical domains to measure entity-aware ASR. p. 2
Where the paper says it
(2023) and audio-LM-based systems show little to no average-WER change and only modest, model-dependent gains on entity-rich tokens. In contrast, GigaSpeechBench provides paired hotword lists across twelve vertical domains for both Chinese and English on genuine human speech, enabling controlled measurement of entity-aware ASR rather than relying on synthetic audio.
-
Speaker Demographics, unifying evaluation of older adult and child speech in both Mandarin and English. p. 3
Where the paper says it
(2025), or on seniors, such as SeniorTalk yang et al. GigaSpeechBench unifies both, offering 10 hours of child and older-adult speech in both Mandarin and English.
-
Speech-to-Text Translation, providing human-translated references for 11 underrepresented source languages under noisy conditions. p. 3
Where the paper says it
(2025)) are largely confined to European languages, formal settings, or synthetic/read audio. GigaSpeechBench advances ST evaluation by providing human-translated references on the same in-the-wild audio used for ASR, covering 11 underrepresented Middle Eastern and Southeast Asian languages under noisy, multi-speaker conditions.
Key Findings on Model Performance
Extensive evaluations of leading foundation models and commercial APIs reveal significant performance degradation when tested on GigaSpeechBench compared to existing benchmarks <ref:2606.28884#pg10>. Specifically, the results demonstrate that strong performance on standard benchmarks does not reliably transfer to these challenging settings. p. 2 (2021) ✗ ✓ ✗ ✗ ✗ 8 - Read
ContextASR-Bench Wang et al. (2025a) ✗ ✗ ✗ ✓ ✗ 10+ - Synthetic
GigaSpeechBench (Ours) ✓ ✓ ✓ ✓ ✓ 14/6/6/12/2 20/10/10/20/10 Spontaneous
Evaluating leading foundation models and commercial APIs on GigaSpeechBench reveals that strong
performance on existing benchmarks does not reliably transfer to these challenging settings, exposing
critical evaluation blind spots. Our evaluation reveals consistent performance gaps between
standard academic datasets and more realistic conditions. We demonstrate that systems nearing
saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and
dialectal speech. We demonstrate that systems nearing
saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and
dialectal speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate
WER often masks sizeable degradation on entity-rich utterances in vertical domains.Where the paper says it
Where the paper says it
Where the paper says it
Limitations of the Benchmark
Several limitations are acknowledged regarding the benchmark construction and evaluation methods. These include:
-
Text normalization for low-resource languages still requires refinement by native-speaking linguistic experts. p. 8
Where the paper says it
Table 2 presents the performance of representative state-of-the-art ASR systems on our benchmark. Overall, these results show that current ASR systems still struggle substantially on newly collected in-the-wild low-resource and regional speech, even when they perform well on standard benchmarks such as Common Voice and FLEURS.
-
Some Chinese dialects lack unified standard writing systems, leading to potential inconsistencies in character-level outputs that make CER insufficient for full assessment. Future work should explore evaluation criteria better suited to low-resource languages and dialectal scenarios than WER/CER. p. 3
Where the paper says it
Table 2: Low-resource languages benchmark results. WER (%) is reported for Arabic and Southeast Asian languages, while CER (%) is reported for East Asian languages.
Ethical Considerations
The ethical framework for the dataset ensures responsible data handling. All collected audio is sourced from materials released under a Creative Commons license, and personally identifiable information (PII) samples have been manually removed during annotation. p. 4 We then perform video-level audio screening before annotation. Videos longer than one hour are
removed to prevent a few speakers or sources from dominating the benchmark and to limit within5 The screened videos are sent to a professional annotation company for voice activity detection and manual transcription. Annotators first segment
the continuous audio into utterance-level speech segments.Where the paper says it
Where the paper says it
Conclusion
In this work, GigaSpeechBench provides a unified multilingual ASR benchmark designed to place long-tail evaluation axes on a common testbed. The construction of this benchmark involved curating over 680 hours of newly collected, manually transcribed in-the-wild speech covering 12 underrepresented languages from the Middle East and Southeast Asia, six Chinese dialects, six English accents, and 12 hotword-rich vertical domains in both Chinese and English. p. 9 5 Conclusion
In this work, we present GigaSpeechBench, a unified multilingual ASR benchmark designed to place
long-tail evaluation axes on a common testbed. To construct this benchmark, we curated over 680
hours of newly collected, manually transcribed in-the-wild speech covering 12 underrepresented
languages from the Middle East and Southeast Asia, six Chinese dialects, six English accents, and
12 hotword-rich vertical domains in both Chinese and English. Across this testbed, we conducted a
large-scale comparison of representative commercial APIs, closed-source ASR-specialized models,
and open-source foundation systems. Our evaluation reveals consistent performance gaps between
standard academic datasets and more realistic conditions. Across this testbed, we conducted a
large-scale comparison of representative commercial APIs, closed-source ASR-specialized models,
and open-source foundation systems. Our evaluation reveals consistent performance gaps between
standard academic datasets and more realistic conditions. Our evaluation reveals consistent performance gaps between
standard academic datasets and more realistic conditions. We demonstrate that systems nearing
saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and
dialectal speech. We demonstrate that systems nearing
saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and
dialectal speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate
WER often masks sizeable degradation on entity-rich utterances in vertical domains. We demonstrate that systems nearing
saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and
dialectal speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate
WER often masks sizeable degradation on entity-rich utterances in vertical domains. We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts. We hope this benchmark serves as a reproducible diagnostic tool for tracking progress on these
underrepresented, yet practically critical, dimensions of robust speech recognition. We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts. We hope this benchmark serves as a reproducible diagnostic tool for tracking progress on these
underrepresented, yet practically critical, dimensions of robust speech recognition.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
The paper cites numerous foundational works across ASR and MT research. Notable references include Conneau et al. p. 7 1 Evaluated Systems
We evaluate a comprehensive set of state-of-the-art ASR systems spanning both commercial APIs
and open-source models. Specifically, the commercial APIs include Microsoft Azure Speech1
,
Google Chirp32
, OpenAI GPT-4o Transcribe3
, Gemini 3. /Age
Speech Type
FLEURS Conneau et al. (2022) ✓ ✗ ✗ ✗ ✗ 102 <3 Read
Common Voice Ardila et al. 3 Read
BABEL Gales et al. (2014) ✓ ✗ ✗ ✗ ✗ 25 - Spontaneous
MLC-SLM Mu et al. All dialects are provided with phonetic transcriptions, while Min
is instead provided with Mandarin translations due to the extreme complexity of its character-topronunciation mapping. It is worth noting that the Yue and Wu dialect evaluation sets are sourced
from WenetSpeech-YUE Li et al. (2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read
or prompted speech collected under relatively clean conditions. Such settings underrepresent key
Technical Report
arXiv:2606. (2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read
or prompted speech collected under relatively clean conditions. Such settings underrepresent key
Technical Report
arXiv:2606. (2000), and ChildMandarin Zhou et al. (2025), or on seniors, such
as SeniorTalk yang et al. (2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read
or prompted speech collected under relatively clean conditions. Such settings underrepresent key
Technical Report
arXiv:2606.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Appendix Details
Segment duration and text length statistics show that the vast majority of audio segments fall between 0.5 and 10 seconds, consistent with spontaneous conversational speech <ref:2606. p. 4 A channel is
retained only when these signals consistently suggest that it contains speech from the target language
or regional variety. From the selected channels, we prioritize videos containing spontaneous
conversational speech and exclude recordings dominated by read, scripted, or narration-style speech. Furthermore, accent robustness varies sharply across English varieties, and aggregate
WER often masks sizeable degradation on entity-rich utterances in vertical domains. We release GigaSpeechBench alongside its annotation protocols, hotword lists, and evaluation scripts. The final benchmark consists of utterance-level audio segments paired with native manual transcripts. These test sets are used for native ASR evaluation under diverse but transcribable acoustic conditions.Where the paper says it
Where the paper says it
Where the paper says it
Table Summaries
Table 2 presents the performance of representative state-of-the-art ASR systems on the benchmark, showing struggles on low-resource languages. Table 3 reports results for Common Voice benchmark across various models and languages. p. 3 76
Table 3: Common Voice benchmark results. Only languages with available evaluation data are
reported. Table 2 presents the performance of representative state-of-the-art ASR systems on our benchmark. Overall, these results show that current ASR systems still struggle substantially on newly collected
in-the-wild low-resource and regional speech, even when they perform well on standard benchmarks
such as Common Voice and FLEURS. Entity tokens that do not appear in the reference transcription are excluded from
the B-WER denominator. Due to space limitations, the standard CER and WER tables together with
their detailed discussion are deferred to Appendix C, and the B-CER results for Chinese and B-WER
results for English are shown in Table 7 and Table 8. Table 9: Child and elderly speech recognition benchmark results. WER (%) is used for English
speech, and CER (%) is used for Chinese speech. These systems are evaluated using the OpenSTBench An
et al. (2026) toolkit with standard translation-quality metrics, including sacreBLEU Post (2018),
chrF++ Popovic´ (2017), COMET Rei et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Experimental Setup
The evaluation involves testing a comprehensive set of state-of-the-art ASR systems, including commercial APIs like Microsoft Azure Speech and OpenAI GPT-4o Transcribe, alongside open-source models such as Whisper Large v3 and Meta OmniASR 3B. For speech translation, systems like Azure Translate and Gemini 3. p. 9 7 Speech Translation
GigaSpeechBench further supports speech-to-text translation evaluation by providing humantranslated English and Chinese references for 11 underrepresented source languages. We evaluate
Azure Translate, Gemini-3-Flash-Preview, SeamlessM4T-v2-Large, and Qwen3. These systems are evaluated using the OpenSTBench An
et al. (2026) toolkit with standard translation-quality metrics, including sacreBLEU Post (2018),
chrF++ Popovic´ (2017), COMET Rei et al. (2020), and BLEURT Sellam et al. For each
subsequent module, the same set of systems is evaluated wherever the language or domain is supported
by the system; cells marked “–” in the result tables indicate that the corresponding system does not
officially support that language or returned no usable output.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion Summary
The work successfully presents GigaSpeechBench, a unified multilingual ASR benchmark designed to place long-tail evaluation axes on a common testbed. The evaluation reveals consistent performance gaps between standard academic datasets and more realistic conditions. Systems nearing saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and dialectal speech <ref:2606. p. 9 Our evaluation reveals consistent performance gaps between
standard academic datasets and more realistic conditions. We demonstrate that systems nearing
saturation on existing standard benchmarks degrade substantially on in-the-wild low-resource and
dialectal speech.Where the paper says it
Author Affiliations
The authors are affiliated with institutions including X-LANCE Lab at MoE Key Lab of Artificial Intelligence and Alibaba Group <ref:2606.28884#pg15>. Other affiliations include Tianjin University, Tsinghua University, Nanyang Technological University, the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, the University of Illinois Urbana-Champaign, The Chinese University of Hong Kong-SZ, Fudan University in Shanghai and Seasalt.ai <ref:2606. p. 0 (2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read
or prompted speech collected under relatively clean conditions. Such settings underrepresent key
Technical Report
arXiv:2606.Where the paper says it
References List
The paper includes a comprehensive list of references supporting its methodology and context. These include foundational work on multilingual speech corpora like Common Voice Ardila et al. p. 2 1 Multilingual and Low Resource Benchmarks
Multilingual evaluations such as FLEURS Conneau et al. (2022) and Common Voice Ardila et al. (2020) typically prioritize language coverage over acoustic diversity, with test sets dominated by read
or prompted speech collected under relatively clean conditions. Such settings underrepresent key
Technical Report
arXiv:2606. 2 Chinese Dialects and Accented English
Chinese dialect resources such as KeSpeech Tang et al. (2021), the WenetSpeech series Li et al. These systems are evaluated using the OpenSTBench An
et al. (2026) toolkit with standard translation-quality metrics, including sacreBLEU Post (2018),
chrF++ Popovic´ (2017), COMET Rei et al. As shown in Table 5,
despite recent advancements in foundation models, substantial performance gaps remain between
standard and accented English, underscoring the persistent challenge of geographical robustness. 4 Chinese Dialects
Our benchmark includes 10 hours of speech for each of the six major Chinese dialects: Xiang,
Jin, Min, Yue, Gan and Wu.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Author List
The authors listed are Yujie Tu, Yifan Yang, Tianrui Wang, Yanqiao Zhu, Guodong Lin, Mingchen Shao, Haoran Wang, Junzhe Liu, Yuxiang Fu, Yizhou Peng, Changsong Liu7 (multiple references) <ref:2606.28884#pg15>. Other contributing authors include Kele Xu (multiple references) <ref:2606.28884#pg15>.
Contact Information
The paper provides links to the GitHub repository and Hugging Face dataset for GigaSpeechBench. These resources facilitate reproducible, real-world ASR evaluation. p. 2 (2025a) ✗ ✗ ✗ ✓ ✗ 10+ - Synthetic
GigaSpeechBench (Ours) ✓ ✓ ✓ ✓ ✓ 14/6/6/12/2 20/10/10/20/10 Spontaneous
Evaluating leading foundation models and commercial APIs on GigaSpeechBench reveals that strong
performance on existing benchmarks does not reliably transfer to these challenging settings, exposing
critical evaluation blind spots. All resources will be released to facilitate reproducible, real-world
ASR evaluation. com/speech-to-text
3https://developers. com/api/docs/models/gpt-4o-transcribe
4https://docs.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Increase robustness against linguistic variability by training models specifically on GigaSpeechBench modules including 12 low-resource Middle Eastern and Southeast Asian languages, 6 Chinese dialects, and 6 English accents. This allows systems to perform reliably in settings where existing benchmarks
overestimate real-world robustness.
-
Enhance domain-specific recognition accuracy by utilizing the
paired hotword lists across twelve vertical domains for both Chinese and English on genuine human speech
to enablecontrolled measurement of entity-aware ASR rather than relying on synthetic audio.
-
Improve performance in challenging acoustic environments by training models on spontaneous conversations, overlapping speech, background noise, far-field recordings, and device-dependent recordings as illustrated in Figure 1. This addresses the limitations where
strong performance on existing benchmarks does not reliably transfer to these challenging settings.
-
Enhance recognition of specialized vocabulary by employing a Biased Word Error Rate (B-WER) calculation that applies standard WER only to reference tokens belonging to annotated entity lists, providing a
more fine-grained diagnostic perspective
than aggregate metrics. -
Improve accuracy in demographic variations by including dedicated evaluation sets for older adult and child speech in both English and Chinese, which addresses the finding that
demographic acoustic variation remains under-addressed by current ASR systems.
-
Enhance speech translation capabilities by providing human-annotated Chinese and English translations for 11 languages to support AST evaluation on in-the-wild audio, specifically for underrepresented Middle Eastern and Southeast Asian languages.
Abstract
While modern ASR systems achieve low error rates on high-resource benchmarks, such performance often overestimates real-world robustness. Existing evaluations address challenges in isolation, lacking a unified benchmark for domain terminology, age variation, dialects, accents, and low-resource languages, particularly across the Middle East and Southeast Asia, representing over one billion under-evaluated speakers. To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 690 hours of human-annotated speech. It features five modules: (1) 12 low-resource Middle Eastern and Southeast Asian languages, plus challenging Japanese and Korean; (2) 6 Chinese dialects; (3) 6 English accents; (4) dense terminology across 12 vertical domains for Chinese and English; and (5) older adult and child speech. We further provide human-annotated Chinese and English translations for 11 languages to support AST evaluation. Extensive evaluations of leading foundation models and commercial APIs reveal significant performance degradation in these challenging settings, exposing critical evaluation blind spots.
Sources
- OpenSTBench: Beyond Semantic Evaluation for Speech Translation
- Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- GPT-4o System Card
- PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech
- COMET: A Neural Framework for MT Evaluation
- Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation
- Qwen3.5-Omni Technical Report
- WenetSpeech-Wu: Datasets, Benchmarks, and Models for a Unified Chinese Wu Dialect Speech Processing Ecosystem
- ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark
- WildElder: A Chinese Elderly Speech Dataset from the Wild with Fine-Grained Manual Annotations
- CS-FLEURS: A Massively Multilingual and Code-Switched Speech Dataset
- Qwen3 Technical Report
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models
- RealComm: Benchmarking and Adapting Audio Deepfake Detection over Real Communication Channels