AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

The paper addresses credit assignment in long-horizon multi-turn agentic reinforcement learning. The authors state: Reinforcement learning(RL) with verifiable rewards constructs trajectory-level advantage, yet often fails to credit the few pivotal decisions that drive outcomes in long-horizon multi-turn agentic RL. They note that many interactive environments provide verifiable rewards only after an entire trajectory terminates, forcing training algorithms to infer the contribution of each intermediate decision from a single sparse outcome. Furthermore, even successful trajectories may contain spurious, redundant, or misleading actions, whereas failed trajectories may still include useful reasoning.

Standard GRPO and its agentic variants construct a trajectory-level advantage from outcome rewards and broadcast it uniformly across the trajectory. Such uniform credit cannot distinguish a few pivotal decisions from routine operations. The paper also identifies two mismatches with existing self-distillation approaches (OPSD): "First, OPSD's token-level signals are not naturally aligned with agentic interaction, where multiple tokens jointly form an action and the environment responds only at turn boundaries. Second, even existing step-aware methods consider each turn in isolation, without accounting for the evidence accumulated through preceding interactions."

The central insight is that the credit of a turn should be determined not by its local signal in isolation, but by how much that signal changes the estimated probability of eventual success. Formally, they interpret the per-turn self-distillation gap as new evidence that induces a Bayesian belief update and define the corresponding belief state as the probability that the trajectory will ultimately succeed given the interaction history.

AgentOPSD is a turn-level credit-assignment method for long-horizon agents that "aggregates token-level teacher–student log-probability gaps into turn-level evidence. Starting from the average group success rate, it then recursively updates Bayesian belief state at each turn in log-odds space without additional rollouts or a learned critic. The outcome verifier determines the global direction of optimization, while the bayesian belief updates redistribute the trajectory-level learning signal across turns."

The paper formalizes turn-level credit via a Bayes-factor perspective: Let C denote the event that the trajectory eventually succeeds. If ak supports success, it should be more characteristic of successful behavior than of unsuccessful behavior. They derive: logit p(C sk, ak) − logit p(C sk) = log p(ak sk, C) / p(ak sk, ¬C), where the right-hand side is the ideal Bayes factor between the success-conditional and failure-conditional likelihoods of ak.

Because these outcome-conditional distributions are unavailable, they estimate the evidence using a privileged, success-associated self-teacher and the standard student policy. The student and teacher share parameters θ; the teacher conditions on a training-only retrieved skill c+: "h+ k,t = (sk, c+, yk,<t). The per-token contrast is δk,t = log πθ(yk,t h+ k,t) − log πθ(yk,t hk,t), and summing over the Lk tokens of a turn gives turn-level evidence ek = log πθ(ak sk, c+) / πθ(ak sk). This provides a tractable hindsight approximation to the ideal Bayesian turn evidence in Eq. (3), under the conditions detailed in Appendix A.1, and ek can be interpreted as an evidential score whose sign indicates whether ak raises or lowers support for eventual success."

The local score ek alone does not indicate whether the same evidence is pivotal or redundant given earlier turns. AgentOPSD therefore maintains a decaying evidence accumulator:

B0 = clip(R̄, ϵ0, 1 − ϵ0), c0 = 0,

ck = γ ck−1 + ek,

lk = logit(B0) + ck = logit(B0) + Σ j≤k γ k−j ej,

Bk = σ(lk).

Here R̄ = S/G is the fraction of successful trajectories in the group of size G—the standard GRPO group mean and serves as the Bayesian prior. γ ∈ (0,1] is a decay factor that down-weights older turns geometrically. The marginal revision of turn k is:

∆Bk = Bk − Bk−1 ≈ Bk−1(1 − Bk−1)[ek − (1 − γ)ck−1],

so evidence has greatest effect under uncertainty and is suppressed once support saturates.

They then align the revision with the verifier outcome: qk = sign(Aseq) ∆Bk, where The magnitude ∆Bk measures how much support the turn revises, while its sign records whether that revision agrees with the verifier's outcome signal.

The raw credit qk is normalized within trajectory and turned into a bounded multiplier: z k = (q k − µ q)/(σ q + ϵ0), w k = clip(1 + b z k, 1 − b, 1 + b), and the reshaped advantage is:

à k = A seq[(1 − λ) + λ w k], λ ∈ [0,1], b ∈ (0,1).

The objective is the clipped GRPO-style loss with the per-turn reshaped advantages, with No separate distillation loss is introduced; the detached self-teacher signal acts only through Ã. AgentOPSD requires only a single teacher forward pass per trajectory and adds no rollouts and no learned parameters over GRPO.

The method is evaluated on three environments: ALFWorld (text embodied benchmark with six household task categories), Search-QA following Search-R1 setup (single-hop NQ, TriviaQA, PopQA and multi-hop HotpotQA, 2Wiki, MuSiQue, Bamboogle), and WebShop (interactive online shopping, 128 fixed validation tasks). Models: Qwen2.5-3B and Qwen2.5-7B-Instruct trained on 8×H800 GPUs. Privileged skills are retrieved from the SkillBank of SkillRL... by keyword matching and are used only during training; inference uses no external skills.

Baselines include: Vanilla, Skill-Prompt, GRPO, Skill-GRPO, OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR, and StepOPSD — all of which use the teacher–student gap but inject it as a gate, magnitude, or auxiliary loss.

AgentOPSD consistently outperforms GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with Qwen2.5-7B (vs. 81.2 for GRPO). On WebShop Qwen2.5-7B, AgentOPSD achieves 90.2 Score and 79.7 Acc, the best among all methods. The authors note: The gain comes from credit construction, not privileged access... AgentOPSD outperforms GRPO+OPSD, Skill-SD, and RLSD on all eight aggregate comparisons across the two model scales, and exceeds SDAR on six of eight.

They also report horizon-robustness: "Figure 1(b) regresses per-sub-task success on the measured mean number of turns of successful episodes on ALFWorld (Qwen2.5-7B), reporting the success points lost per additional turn. The uniform-credit methods degrade fastest (−3.59 for RLSD and −2.91 for GRPO points per turn), whereas AgentOPSD is the flattest at −0.54."

The full method achieves 89.1 on ALFWorld with Qwen2.5-7B. Ablations:

• Per-token accumulation instead of turn-level: 85.9 — environment feedback is associated with a complete action rather than an individual token.

• Raw local gap ek instead of recursive revision ∆Bk: 82.8 — A raw ek scores each turn in isolation, whereas ∆Bk measures how that gap revises the belief state accumulated over the preceding history.

• Magnitude ∆Bk only (dropping outcome sign): 80.5 — The magnitude identifies where the belief state changes, but cannot determine whether that change agrees with the verifier outcome.

• Dropping empirical-rate initialization B0: 78.9 — The group success rate R̄ provides a verifier-grounded estimate of task difficulty before the trajectory-specific gap is accumulated.

Sweeping λ (reshaping weight), γ (evidence decay), and ϵ high (policy clip) shows only λ produces a systematic effect — λ=0.5 is best, and the method is largely unaffected by the other knobs. The spread across all knobs shrinks sharply on the four-turn Search-QA task—the settings that matter on long-horizon ALFWorld are largely inert when little history accumulates.

Appendix A proves several propositions: Prop. 1 (Boundedness): Ã k − A seq ≤ λbA seq; Prop. 2 (Sign Preservation): sign(Ã k) = sign(A seq); Prop. 3 (Recovery of GRPO): at λ=0, the gradient equals GRPO's; Prop. 4 (First-Order Decomposition): the belief revision equals B k−1(1−B k−1)∆l k to first order; Prop. 5 (Exact Budget): Σ k ∆B k = B K − B 0; Prop. 6 (Non-Identifiability of Per-Turn Contribution): per-turn credit is not identifiable from the trajectory return alone; Prop. 7: B0 is the MLE of the task success probability.

The approximation analysis shows the self-teacher contrast recovers the Bayes factor in the limit ρ k → 0 (when success is rare), and under (A1) alone, ek is the pointwise mutual information... positive iff ak raises the posterior success probability, preserving sign and ranking: sign(ek) = sign(Bk) and ek preserves the ranking of turns by evidential strength.

The paper situates AgentOPSD among (1) agentic post-training with verifiable rewards (GRPO, GiGPO), (2) on-policy (self-)distillation (OPSD, Skill-SD, RLSD, SDAR, StepOPSD), and (3) long-horizon credit assignment (PPO/GAE, RUDDER, VinePPO). It contrasts AgentOPSD with these: GiGPO and AgentOPSD therefore operate on complementary signal sources—environment rewards and self-distillation evidence, respectively; AgentOPSD first aggregates token-level gaps within each turn and then recursively accumulates the resulting evidence into a running support state; and The belief state plays the role of GAE's value baseline and its per-turn revision the role of the TD signal, but without a learned value network cost.

The paper concludes: "We studied credit assignment for long-horizon language agents, where trajectory-level rewards provide limited supervision for distinguishing pivotal decisions from routine or redundant actions. Our key insight is that turn-level credit should depend not only on a local signal, but also on how that signal revises the accumulated belief in eventual trajectory success... Experiments across three interactive environments and two model scales show that AgentOPSD consistently outperforms GRPO and strong self-distillation baselines. Ablations further confirm the importance of both turn-level signal aggregation and history-dependent belief revision. Overall, our results suggest that recursive belief updating provides a simple and effective approach to restoring fine-grained temporal credit in critic-free agentic reinforcement learning."

Improvements to AI systems from AgentOPSD:

• Accurate credit assignment in long-horizon agentic RL: Instead of broadcasting one trajectory-level reward uniformly to every action, the AI system assigns per-turn credit by recursively updating a Bayesian belief about eventual success. This lets it distinguish the few pivotal decisions from routine, redundant, or misleading actions.

• Turn-level evidence aggregation: The system aggregates token-level teacher–student log-probability gaps into a single per-turn evidence score, aligned with the fact that environment feedback arrives at turn boundaries, not per token.

• History-dependent credit revision: Each turn’s credit depends on how much its evidence changes the accumulated belief given previous turns, not just its local quality. A decaying evidence accumulator prevents redundant evidence from being over-counted and suppresses credit when success support is already saturated.

• Critic-free, rollout-free advantage reshaping: The system improves credit without a learned value network, additional rollouts, or extra trainable parameters—only a single self-teacher forward pass is added to GRPO.

• Bounded, sign-preserving optimization signal: The reshaped per-turn advantage is bounded and preserves the sign of the trajectory-level verifier outcome, so training remains stable and the global outcome direction is never reversed.

• Better long-horizon robustness: Performance degrades far less as task horizons grow (about −0.54 points per extra turn versus −2.91 for GRPO and −3.59 for RLSD on ALFWorld), making the system more reliable on long multi-turn tasks.

• Strong empirical performance: The improved agent achieves 89.1% success on ALFWorld with Qwen2.5-7B and 90.2 score / 79.7 accuracy on WebShop, outperforming GRPO, OPSD, Skill-SD, RLSD, and SDAR.

• Reduced hyperparameter sensitivity: The system is largely insensitive to evidence decay and clipping hyperparameters, with only the reshaping weight requiring tuning, simplifying deployment.

• Principled theoretical guarantees: The system preserves boundedness, sign consistency, reduction to GRPO at λ=0, and exact belief-budget accounting, making it safer to apply in production agentic systems.