Rethinking the Multilingual Reasoning Gap with Layer Swap
cs.CL
Submitted: 2026-05-26
Updated: 2026-08-26
Comments: Accepted to Findings of EMNLP 2026
Code: https://github.com/huggingface/trl
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages.
Terminology
Abstract
Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language (native reasoning) substantially degrades performance relative to allowing the model to reason in English before answering in the input language (English-pivoted reasoning). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of Qwen/Qwen3-8B-Base, and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5% across the five non-English languages, considerably smaller than previously reported. Weight-space analysis of the native specialists reveals aligned fine-tuning updates in the middle layers and divergence in the outer layers. This points to a largely language-agnostic reasoning core surrounded by language-specific layers. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist's stronger reasoning mid-layers into each native specialist, closing most of the native reasoning gap across the five non-English languages while preserving CoT in the target language. We release all models and datasets.
Sources
- Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models
- Long Chain-of-Thought Reasoning Across Languages
- Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?
- Olmo 3
- A Survey of Multilingual Reasoning in Language Models
- Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models
- Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy
- ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
- Do Multilingual LLMs Think In English?
- OpenAI o1 System Card
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Language Models are Multilingual Chain-of-Thought Reasoners
- Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
- Qwen3 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering