StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning
cs.AI
Submitted: 2026-05-26
Updated: 2026-08-26
Comments: Already been accepted as an EMNLP2026 Findings paper
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions.
Terminology
Abstract
Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.
Sources
- A General Theoretical Paradigm to Understand Learning from Human Preferences
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- Self-Distilled Agentic Reinforcement Learning
- Privileged Information Distillation for Language Models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Policy Distillation
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Proximal Policy Optimization Algorithms
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
- Self-Distilled RLVR
- ReAct: Synergizing Reasoning and Acting in Language Models
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems