Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Sherzod Hakimov, Karl Osswald, Jelle Psurek, Eszter Bukovszky, A. Altar Lüser, David Schlangen
cs.CL, cs.AI
Submitted: 2026-08-02
Comments: Source code: https://github.com/clembench/multilingual
Code: https://github.com/clembench/multilingual
Project page: https://clembench.github.io/leaderboard.html
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others.
Terminology
Abstract
We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model's home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese-developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.
Sources
- Multilingual European Language Models: Benchmarking Approaches and Challenges
- TALES: Text Adventure Learning Environment Suite
- TextArena
- MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
- One ruler to measure them all: Benchmarking multilingual long-context language models
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following
- Humanity's Last Exam
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
- BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
- The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
- Towards Multilingual LLM Evaluation for European Languages
- PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts
- Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering