Tandem Reinforcement Learning with Verifiable Rewards
cs.AI
Submitted: 2026-06-26
Updated: 2026-09-27
Comments: 22 pages,6 figures,8 tables
Code: https://github.com/CSSLab/Tandem-RLVR
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Reinforcement learning with verifiable rewards (RLVR) has driven substantial gains in language-model reasoning.
Terminology
Abstract
Reinforcement learning with verifiable rewards (RLVR) has driven substantial gains in language-model reasoning. However, models fine-tuned with RLVR need not remain compatible with weaker partners: gains achieved alone may erode under collaboration, while policy drift may place reasoning beyond a weaker model's predictive reach. To address both problems, we introduce Tandem RLVR (TRLVR), bringing the recently proposed tandem-training paradigm beyond proof-of-concept settings and into modern RLVR. In TRLVR, a senior learns from rewarded trajectories co-generated with a frozen junior, while the standard GRPO objective is applied only to senior-emitted tokens. Training Qwen3-4B-Instruct on competition math, TRLVR performs on par with a matched GRPO senior when reasoning alone. We characterize compatibility through the complementary lenses of handoff robustness and junior legibility. Under reasoning-step handoffs, the TRLVR senior retains nearly all of its solo performance with the junior, reducing the communication tax to a negligible loss. The senior's reasoning is also more legible to the junior: its token distribution stays more closely anchored to the junior's, and the junior predicts the senior's chain-of-thought more readily token by token. An ablation that explicitly regularizes GRPO toward the junior does not recover the same combination of capability and compatibility. These results provide initial evidence that tandem rollout structure is a promising direction for keeping RLVR reasoning gains accessible to weaker partners.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Evaluating Large Language Models Trained on Code
- GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
- SSR: Speculative Parallel Scaling Reasoning in Test-time
- The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Designing Skill-Compatible AI: Methodologies and Frameworks in Chess
- Measuring Mathematical Problem Solving With the MATH Dataset
- Distilling the Knowledge in a Neural Network
- The Steganographic Potentials of Language Models
- Prover-Verifier Games improve legibility of LLM outputs
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Understanding R1-Zero-Like Training: A Critical Perspective
- Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
- Training language models to follow instructions with human feedback
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Large language models can learn and generalize steganographic chain-of-thought under process supervision
- Solving math word problems with process- and outcome-based feedback
- MARS: toward more efficient multi-agent collaboration for LLM reasoning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems