Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
Jim Dilkes, Vahid Yazdanpanah, Sebastian Stein
cs.AI, cs.CL, cs.LG
Submitted: 2026-08-03
Comments: Submitted to ACL Rolling Review (ARR) May 2026 cycle. OpenReview submission record at https://openreview.net/forum?id=PV945lekMa
License: http://creativecommons.org/licenses/by/4.0/
The gist: Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges
Terminology
Abstract
Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted. To facilitate ICE, we combine RL on the instruction-conditioned policy with self-distillation of its correct rollouts into the unconditioned test-time policy. ICE with this objective improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical reasoning tasks by 5.0% relative to training with DAPO, with improvement persisting at a longer 8K context. The improvement does not appear for Qwen3-4B at 4K, where the instructions do not expand base-model coverage.
Sources
- A General Language Assistant as a Laboratory for Alignment
- Nudging the Boundaries of LLM Reasoning
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions
- Prompt Baking
- Evaluating Large Language Models Trained on Code
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Measuring Mathematical Problem Solving With the MATH Dataset
- Self-Hinting Language Models Enhance Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning
- Self-Refine: Iterative Refinement with Self-Feedback
- Diversity-Incentivized Exploration for Versatile Reasoning
- Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence
- Reinforcement Learning via Self-Distillation
- Privileged Information Distillation for Language Models
- Improving Generative Ad Text on Facebook using Reinforcement Learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Training Language Models to Self-Correct via Reinforcement Learning
- On Entropy Control in LLM-RL Algorithms
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems