CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
cs.CL, cs.AI
Submitted: 2026-05-25
Updated: 2026-09-02
Comments: Findings of EMNLP 2026
Code: https://github.com/jjzha/CroCo
Project page: https://mlco2.github.io
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English.
Terminology
Abstract
Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, evaluated by two judges, both tuned models win 28/30 times against their respective base across 15 evaluated languages (high and low-resource). Overall, we show promising directions for multilingual preference tuning using self-generations.
Sources
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
- MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
- TranslateGemma Technical Report
- Direct Language Model Alignment from Online AI Feedback
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
- Crosslingual On-Policy Self-Distillation for Multilingual Reasoning
- RewardBench 2: Advancing Reward Model Evaluation
- EuroLLM-22B: Technical Report
- Olmo 3
- Self-Distillation Enables Continual Learning
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering