Selecting Diverse SFT Traces Improves Post-RL Generalization
cs.LG, cs.AI
Submitted: 2026-09-27
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL).
Terminology
Abstract
Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL). We present a comprehensive study of route diversity, the variation in the sequences of reasoning steps in supervised fine-tuning (SFT) data, and propose a lightweight, rule-based fingerprint to select for it. From one pool at one budget, with matched training recipes and checkpoints, selecting diverse rather than similar routes improves post-RL problem coverage across puzzles and mathematics, including on problems harder than those seen in either training stage. In synthetic experiments, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 points on environments held out from SFT. In a single-model condition, where one model writes every candidate, diverse selection gains up to 6.2 points of mean pass@8 across 10 mathematics benchmarks. Pre-RL diagnostics suggest why: diverse SFT can produce both successful and failed attempts on more prompts despite slightly lower mean accuracy, giving group-relative RL more prompts with a learning signal. On 3 open-source corpora, our CPU-only selector, without model calls, outperforms more expensive alternatives in every comparison of mean post-RL performance. These results identify reasoning-route diversity as a practical criterion for selecting SFT data that better prepares models for RL.
Sources
- Phi-4-reasoning Technical Report
- Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
- Llama-Nemotron: Efficient Reasoning Models
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models
- Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
- Behavior Injection: Preparing Language Models for Reinforcement Learning
- Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Process Reinforcement through Implicit Rewards
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Chasing Random: Instruction Selection Strategies Fail to Generalize
- Stream of Search (SoS): Learning to Search in Language
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
- OpenThoughts: Data Recipes for Reasoning Models
- Reinforced Self-Training (ReST) for Language Modeling
- Reasoning with Language Model is Planning with World Model
- OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks