Skill Issue: Are Skills Language-Invariant in LLMs?
cs.CL, cs.AI, cs.GT, cs.LG
Submitted: 2026-08-26
Updated: 2026-08-26
Code: https://github.com/TextArena/TextArena
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies
Terminology
Abstract
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance. We do this via multilingual self-play: two instances of the same model compete in a text-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model's realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures in spatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.
Sources
- OpenAI Gym
- XOR QA: Cross-lingual Open-Retrieval Question Answering
- Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models
- No Language Left Behind: Scaling Human-Centered Machine Translation
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
- Cross-Lingual Exploration for Parametric Knowledge
- Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environments
- MindAgent: Emergent Gaming Interaction
- The Llama 3 Herd of Models
- GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
- TextArena
- Toucan: Many-to-Many Translation for 150 African Language Pairs
- Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
- IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer
- Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs
- Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languages
- Ministral 3
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering