Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

BCSD (Bidirectional Context Self-Distillation) is a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views.

The paper identifies improving LLM agents' use of external skills, rather than internalizing skill knowledge into model parameters, as an important direction for skill-based agents. BCSD derives complementary token-level guidance from augmented and reduced skill contexts and integrates it into GRPO advantage to improve external skill utilization for LLM agents. Experiments on ALFWorld and WebShop show that BCSD achieves the strongest overall performance across model scales, validating its effectiveness for skill-based LLM agents training.

BCSD retains external skills during both training and inference and improves their utilization through two complementary context views. An augmented view provides policy-dependent Meta-Skill guidance, while a pruned view offers a focused reference for task-specific skill usage. Their token-level gaps jointly rescale the GRPO advantage without reversing the reward-determined update direction. Experiments on ALFWorld and WebShop across model scales demonstrate the effectiveness of BCSD over skill-based RL and self-distillation baselines. Ablation studies validate the contributions of both context views, Meta-Skill refresh, and coefficient decay, while further analyses show improved training dynamics and greater reliance on external skills than on memorized task knowledge.

The method works as follows: For each query Q of task type c, the SkillBank provides a general skill G and a retrieved task-specific skill Sc. The base context is defined as C0 = (Q, G, Sc). For each trajectory τi sampled under C0, BCSD keeps the realized response sequence yi fixed and uses the same policy to re-score its tokens under C1 and C2. The augmented context extends C0 with the Meta-Skill Mc associated with the current task type c: C1 = (Q, G, Sc, Mc). The pruned context replaces G with Gp (pruned general skill) while preserving Sc: C2 = (Q, Gp, Sc). This yields two token-level log-probability gaps: ∆aug i,t = log πθ(yi,t C1, yi,<t) − log πθ(yi,t C0, yi,<t) and ∆pru i,t = log πθ(yi,t C2, yi,<t) − log πθ(yi,t C0, yi,<t). The two gaps are combined as ∆i,t = α∆aug i,t + (1−α)∆pru i,t, where α determines the relative weights of the augmented and pruned views. The bidirectional gap ∆i,t is used to modulate the token-level advantage without replacing it: wi,t = clip(exp(sign(AGRPO i)∆i,t), 1−c, 1+c), ABCSD i,t = AGRPO i[(1−λn) + λn wi,t]. BCSD changes only the magnitude of the advantage while preserving the update direction determined by the trajectory-level reward. The coefficient λn ∈ [0,1] controls the rescaling strength at training step n and is linearly decayed from λ0 to zero.

Meta-Skills are defined as high-level textual guidance that summarizes effective and ineffective ways of applying skills across rollout trajectories. For each task-specific skill Sc, a corresponding Meta-Skill Mc is maintained. Mc captures policy-dependent guidance on how Sc should be used and is provided only to the teacher as privileged context during training, not included in the student context C0 or used during inference. Before training, rollout trajectories from multiple queries associated with each skill Sc are collected and divided into successful and failed evidence sets. A frozen external LLM contrasts the two sets and initializes the meta-skill. Meta-Skills are refreshed after every validation window using trajectories collected during that window to track the policy's evolving skill-utilization patterns.

The final policy loss is LBCSD(θ) = −Ei,t[min(ρi,t(θ)ABCSD i,t, clip(ρi,t(θ), 1−ϵ, 1+ϵ)ABCSD i,t)] + βLKL, where ϵ is the clipping coefficient, β controls the strength of KL regularization, and LKL denotes the standard KL loss.

Experiments were conducted on ALFWorld and WebShop benchmarks with Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-1.7B models. Each model was trained for 150 steps with a rollout group size of K = 8 and evaluated every 15 steps. For BCSD, α = 0.9 and λn was linearly decayed from 0.1 to zero. Meta-Skills were refreshed at each evaluation interval.

Main results show BCSD achieves the strongest aggregate performance across all three backbones. With Qwen2.5-7B, BCSD reaches an average success rate of 83.6 on ALFWorld, outperforming the best baseline by 3.1 points, and achieves the highest WebShop success rate of 78.1. With Qwen2.5-3B, BCSD surpasses the strongest baselines by 5.4 points on ALFWorld and by 4.4 and 9.4 points in WebShop score and success rate, respectively. On Qwen3-1.7B, BCSD achieves the highest ALFWorld average success rate of 46.9, outperforming the best baseline by 3.9 points, and obtains the highest WebShop score of 76.8.

Ablation studies show that removing the Meta-Skill view causes the largest degradation among the two context-view ablations, reducing performance from 82.0 to 71.1 on ALFWorld and from 66.4 to 50.0 on WebShop. Removing the Pruned-Skill view also lowers performance to 75.0 and 57.0, respectively. Keeping λn constant reduces performance to 76.6 on ALFWorld and 52.3 on WebShop. Disabling Meta-Skill updates decreases performance to 78.1 on ALFWorld and 50.0 on WebShop. Setting λ0 = 0.1 achieves the best performance on both benchmarks, while increasing it to 0.3 or 0.5, or decreasing it to 0.05, consistently reduces performance. The (0.9, 0.1) configuration for (α, 1−α) sustains stronger late-stage improvement and achieves the highest final success rate.

Further analysis shows that the token-level sign agreement ratio between the two teacher views remains close to 50% throughout training on both benchmarks, indicating the two context views provide substantially different rather than redundant token-level guidance. Task-skill utilization analysis shows that BCSD's success rate decreases more than that of Skill GRPO* when the task-specific skill Sc is removed, dropping by 2.3 points on ALFWorld and 7.8 points on WebShop, compared with 0.8 and 6.2 points for Skill GRPO*. This greater sensitivity indicates that BCSD derives more of its gains from using the provided Task Skill without memorizing the task, internalizing how to use skills while keeping task-specific knowledge external and editable during inference.

Improvements to AI systems:

1. Dual-Context Advantage Rescaling for Skill Utilization

• Implement a training loop where each trajectory is re-scored under two modified contexts: one augmented with high-level meta-skill guidance and one pruned of general guidance to emphasize task-specific skills.

• Compute token-level log-probability gaps between these contexts and the base context, then combine them (weighted) to rescale the RL advantage magnitude without flipping its sign.

• This allows the AI to learn how to use external skills (e.g., tool APIs, domain procedures) more precisely, rather than memorizing task outcomes.

2. Policy-Dependent Meta-Skill Generation and Refresh

• Maintain a meta-skill text for each task-specific skill, generated by contrasting successful vs. failed rollout trajectories using a frozen external LLM.

• Refresh meta-skills periodically (e.g., every validation window) using new trajectories, so the guidance tracks the policy’s evolving skill-usage patterns.

• Use meta-skills only as privileged teacher context during training—never during inference—so the student learns to apply skills without relying on extra hints.

3. Complementary View Combination with Decaying Influence

• Use two teacher views (augmented and pruned) that provide token-level signals with near-50% sign disagreement, ensuring they are non-redundant.

• Combine them with a fixed weight (e.g., α=0.9 for augmented, 0.1 for pruned) and apply a clipped multiplicative factor to the advantage.

• Linearly decay the rescaling coefficient λn from 0.1 to zero over training, so early training benefits from strong skill-guidance shaping, while later training relies more on reward-driven optimization.

4. Reward-Direction-Preserving Advantage Modulation

• Modify only the magnitude of the advantage (not its sign) using the bidirectional gap, ensuring the update direction is always determined by the actual reward.

• This prevents the skill-guidance from overriding correct reward signals, reducing the risk of reward hacking or misaligned updates.

5. External Skill Retention and Editable Task Knowledge

• Keep task-specific skills (e.g., retrieval results, procedural instructions) external to model parameters during both training and inference.

• Train the model to become more sensitive to the presence of the task-specific skill (as shown by larger performance drops when it’s removed), meaning the model learns to use the skill rather than memorize the task.

• This enables runtime editing of skills (e.g., updating a tool’s API docs) without retraining.

What the improved AI system can do:

• More effective tool use in agentic tasks: The AI can follow external skill instructions (e.g., “how to book a flight” or “how to query a database”) with higher success rates, because it learns to align its token-level decisions with the skill’s intent, not just its surface text.

• Adapt to new or updated skills without retraining: Since skills remain external and the model learns to use them dynamically, swapping in a revised skill description (e.g., a new API version) improves performance immediately, without fine-tuning.

• Better generalization across model scales: The method works on small (1.7B) to medium (7B) models, improving success rates by 3–9 points over strong baselines on benchmarks like ALFWorld (household tasks) and WebShop (online shopping).

• Stable and interpretable training: The advantage modulation preserves reward-driven updates, avoids reward reversal, and shows improved training dynamics (e.g., smoother convergence and higher final performance) compared to constant or single-view guidance.

• Reduced reliance on memorized task knowledge: The AI relies more on the provided task-specific skill than on internalized task patterns, making it more robust to distribution shifts and more controllable via external knowledge injection.