Strategically Diverse Sampling for Self-Training
cs.CL
Submitted: 2026-09-25
Updated: 2026-09-25
License: http://creativecommons.org/licenses/by/4.0/
The gist: Many LLM training and inference methods, including RL and test-time scaling, depend on repeated sampling, but benefit only when the responses meaningfully differ.
Terminology
Abstract
Many LLM training and inference methods, including RL and test-time scaling, depend on repeated sampling, but benefit only when the responses meaningfully differ. Self-training faces the same challenge: training data is typically constructed by sampling IID responses and filtering primarily for correctness, thereby overrepresenting strategies a model already favours. We investigate strategic diversity, or substantive variation among approaches to a problem, as an alternative principle for constructing self-training data. We generate strategically diverse data with two sampling methods: GROOT, a new method which constructs a hierarchical tree of approaches and samples distinct paths, and Verbalized Sampling (VS), adapted to produce an unstructured set of approaches. Across competitive programming and Next-Chapter Prediction domains, models trained on strategically sampled data outperform IID-trained counterparts on difficult tasks and provide strong initializations for RL and test-time scaling. Most strikingly, self-training on strategically diverse but incorrect traces from Qwen3-4B outperforms IID distillation from a 235B teacher. These results challenge prevailing assumptions about what makes useful self-training data and show that diversity of approaches can matter more than correctness or teacher scale.
Sources
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing
- Evaluating Large Language Models Trained on Code
- Post-training Large Language Models for Diverse High-Quality Responses
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- COBALT: Contextual Bandit Learning with Offline Trajectories for Multi-Turn Code Generation
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- GLM-5: from Vibe Coding to Agentic Engineering
- OpenThoughts: Data Recipes for Reasoning Models
- Skywork Open Reasoner 1 Technical Report
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Task-Dependent Evaluation of LLM Output Homogenization: A Taxonomy-Guided Framework
- Where does output diversity collapse in post-training?
- Kimi K2: Open Agentic Intelligence
- Diversity in Large Language Models under Supervised Fine-Tuning
- Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning
- SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
- Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering