RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
Xinye Wang, Junxiao Liu, Shujian Huang
cs.CL
Submitted: 2026-08-06
Comments: 16 pages. Under review
Code: https://github.com/NJUNLP/RP-OPSD
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages.
Terminology
Abstract
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.
Sources
- IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- Long Chain-of-Thought Reasoning Across Languages
- Thought Anchors: Which LLM Reasoning Steps Matter?
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- OpenThoughts: Data Recipes for Reasoning Models
- Skill-Conditioned Gated Self-Distillation for LLM Reasoning
- Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
- TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning
- Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
- Reinforcement Learning via Self-Distillation
- Empowering Multi-step Reasoning across Languages via Tree-of-Thoughts
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Stable On-Policy Distillation through Adaptive Target Reformulation
- Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
- MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
- Language Models are Multilingual Chain-of-Thought Reasoners
- R3S: Refining and Recovering Reinforcement Signals for Multilingual Understanding and Reasoning
- Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?
- Crosslingual On-Policy Self-Distillation for Multilingual Reasoning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering