From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation
summary
The gist
Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets…
In short
Small language models benefit from teacher trajectories, but direct imitation is inefficient for multi-turn behavior. Prefix-GRPO reuses these trajectories by decomposing them into replayable 'prefix queries' and online continuations. This allows teacher knowledge to be used multiple times, treating each prefix as a decision-aware training query, leading to improved performance.
Key concepts
- Prefix-GRPO
- A reinforcement learning framework that breaks down teacher agent paths into two parts: replayable prefixes and online continuation. It replays the prefix in the environment to find a valid intermediate state before letting the student continue interacting online. This unifies prefix and continuation learning under one policy optimization method.
- Group Queries
- The process of selecting specific cut-state queries from teacher trajectories. These queries are scored using teacher-forcing entropy under a policy-distilled SFT model to pinpoint the most informative points in the trajectory where a decision change occurs. This helps identify useful segments for prefix learning.
- Old-Logit Estimation
- A method used to place historical assistant tokens (prefixes) into the current training format. It uses a policy-distilled SFT checkpoint as a fixed 'old policy' to recompute teacher-forcing log-probabilities on the prompt history. This provides necessary annotations for applying clipped ratio training to these pre-existing prefix spans.
Terminology used across episodes
This episode discusses
- From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation · Paper Radio
- Small Language Models are the Future of Agentic AI · Paper Radio
- Distilling LLM Agent into Small Models with Retrieval and Code Tools
- The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks
- Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
- Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes
- AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
- AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
- Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineering
- Fast and Effective On-policy Distillation from Reasoning Prefixes
- AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
- A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
- AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
- RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use
- Agentic Reinforced Policy Optimization
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
The paper
From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation · Read on arXiv
Tianjin University · Peking University · Tianjin University of Technology
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "From Trajectories to Prefixes".
Jane: Here's the gist of the paper. Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets The paper proposes Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations This method allows teacher trajectories to be reused beyond one-shot distillation by treating them as multiple decision-aware training queries Framework Overview Prefix-GRPO introduces a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities This unifies prefix learning and continuation learning within the same policy-optimization form Prefix Construction with Group Queries The group-query construction selects a set of cut-state queries Qi = q i,1,..., q i,J i from each trajectory τ i These queries are scored using teacher-forcing entropy under the policy-distilled SFT model to identify informative cut points The construction involves several stages, including query selection, replay validation, and canonicalization of the prompt at the cut state Old-Logit Estimation for Distilled Prefixes Prefix tokens are different because they are teacher-derived assistant tokens already embedded in the prompt history To place them under the same clipped-ratio training form, Prefix-GRPO uses a policy-distilled SFT checkpoint as a fixed old policy for prefixes and recomputes teacher-forcing log-probabilities on the canonicalized prompt This involves storing three prompt-space annotations: an assistant-prefix span
b i, e i), a binary prefix mask m i over that span, and cached old log-probabilities l old i stored densely over the same span Unified Prefix-Continuation Optimization Prefix-GRPO writes continuation learning and prefix learning in a unified clipped-ratio form The training objective is given by L = − Σ t∈Mcont i g(r cont i, t, Aˆcont i,t; ϵ, ϵ) − λpre Σ u∈Mpre i g(r pre i, u, Aˆpre i; ϵ−pre, ϵ+) This form puts prefix learning and continuation learning under the same clipped surrogate while keeping their loss terms, old-policy sources, and advantage assignments separate Experimental Effectiveness Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines Specifically, Prefix-GRPO achieves the best overall Pass@8 in TextCraft and BabyAI The ablation studies indicate that explicit prefix optimization is most useful for recovering successful continuations from difficult intermediate states In this paper, Prefix-GRPO turns teacher trajectory prefixes into explicit optimization targets rather than treating them as passive context The experiments show that small-model agents benefit from combining distillation with reinforcement learning Future work should extend matched ablations across more environments and analyze alternative prefix-side objectives and old-policy anchors References [1: Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S., Lin, Y.
Tom: C., and Molchanov, P.
Jane: (2025).
Tom: Small Language Models are the Future of Agentic AI.
Paper summary: Tom: So we're looking at a paper called "From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation".
Jane: It sounds like they're tackling the problem where you have these really strong teacher interactions, but just directly copying them doesn't work well for getting small models to perform well over many turns.
Tom: Right, so the main idea is that instead of treating a whole long conversation as one big imitation target, they decompose those teacher interactions into smaller pieces.
Jane: Specifically, they propose this framework called Prefix-GRPO that splits the teacher's trajectory into two parts: replay-aligned prefix queries and online continuations.
Tom: And what makes it interesting is that for each prefix query, they actually replay it in the environment to get a valid intermediate state before letting the student continue interacting online and getting a task reward.
Jane: Plus, they’re not just looking at the text; they're applying clipped policy updates to those historical assistant tokens inside that replayed prefix using a policy-distilled SFT checkpoint to estimate their old log probabilities.
Tom: It seems like they’re unifying the learning of what comes before and what comes after in one single optimization form, which is a big thing for efficiency.
Lu: What I find interesting about this is how they tackle the core issue that many existing methods leave unsolved, which is how to split one long teacher trajectory into multiple valid training states <ref:2607.19395#pg2>. They aren't just using the whole thing as a supervised target; instead, their group-query construction selects informative cut states along the teacher trajectory.
Jane: So they score these cut points using teacher-forcing entropy under a policy-distilled SFT model to find where the information is most useful for different learning queries.
Lu: And this decomposition means each prefix exposes different partial progress, remaining subgoals, and recovery opportunities <ref:2607.19395#pg2>. It’s like they are creating differentiated learning queries from a single teacher rollout.
Meng: From an engineering standpoint, I'm thinking about the cost of this. If you're replaying prefixes in the environment for every query, that sounds computationally expensive at scale <ref:2607.19395#pg1>. I need to know if this approach is something we can actually deploy efficiently without massive overhead.
Tom: That’s a fair point, Meng. The paper does mention applying clipped policy updates to historical tokens inside the replayed prefix using a policy-distilled SFT checkpoint to estimate their old log probabilities.
Jane: So they are trying to reuse that decision information from the past without needing the full high-cost interaction every time.
Lalam: I've looked at this paper and I'm confident in its direction because it directly addresses the inefficiency of one-shot imitation targets for long-horizon environments <ref:2607.19395#pg1>. They show that this method improves small model agents over both distillation and standard reinforcement learning baselines <ref:2607.19395#pg1>.
Tom: Exactly, it’s not just a small improvement; the experiments on TextCraft and BabyAI show that Prefix-GRPO achieves the best overall Pass@eight in those environments <ref:2607.19395#pg1>. It seems like explicit prefix optimization is most useful when recovering successful continuations from difficult intermediate states, which is a concrete finding.
Paper summary: Jane: So, for someone who just listens to the show and doesn't read the paper, what does that actually mean for an agent?
Tom: It means instead of just learning the final answer from a teacher’s whole chat, you get a set of smaller training problems based on key moments in that chat.
Jane: And for each moment, you test how well your model can recover and continue from that exact point, using the context of what came before it.
Lu: The core idea here is treating teacher trajectory prefixes as explicit optimization targets instead of just passive context <ref:2607.19395#pg1>. This shifts the focus from compressing behavior in one shot to decomposing that behavior into reusable decision-aware training queries.
Jane: It’s about making sure the model learns *how* to recover, not just *what* the final response looks like <ref:2607.19395#pg1>.
Meng: I worry about the complexity of setting up all those components—the group-query construction and the old-logit estimation—does that setup become a huge bottleneck when we try to scale this to production traffic?
Tom: That’s a real concern, Meng. But they are trying to unify prefix learning and continuation learning under the same clipped surrogate loss function, which simplifies the optimization side.
Lalam: I'm pretty sure the unification is key here because it keeps the prefix learning and continuation learning under the same clipped ratio training form while keeping their loss terms and old-policy sources separate <ref:2607.19395#pg1>.
Jane: So, if we boil it down, this paper suggests that combining distillation with reinforcement learning can actually help small models perform better than just using standard distillation methods <ref:2607.19395#pg1>. It points toward a path where we use the structure of good interactions to guide learning in a more targeted way.
Tom: And it shows that the mechanism for doing this isn't just passive context; they turned teacher trajectory prefixes into explicit optimization targets <ref:2607.19395#pg1>. The experiments on ALFWorld and others show tangible benefits, which is what we need to see when we talk about practical applications.
Lu: Future work needs to extend these matched ablations across more environments, and they also plan to analyze alternative prefix-side objectives and old-policy anchors <ref:2607.19395#pg1>. This tells us the current method is a strong starting point, but there’s definitely room for deeper investigation into how this decomposition works across different types of interaction histories.
Meng: For deployment, we need to see if these decomposed queries can be generated fast enough to keep up with real-time agent interactions <ref:2607.19395#pg1>. Right now, the focus is on showing the concept works well in controlled settings like TextCraft and BabyAI.
Jane: So, this paper gives us a new way to think about reusing strong teacher data by breaking it down into replayable steps <ref:2607.19395#pg1>. It seems like the implication is that for complex tasks, we should focus on learning effective recovery from intermediate states rather than just memorizing the final output.
Conclusion: Tom: So we're wrapping up our look at "From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation".
Jane: It sounds like the authors are taking those long teacher interactions, which are usually just one big block of text, and they’re turning them into a bunch of smaller training steps.
Lu: Right, they decompose those teacher trajectories into replay-aligned prefix queries and online continuations. That means instead of treating the whole conversation as one thing to copy, you treat it like a set of specific questions to ask during training.
Meng: I'm wondering how much work that decomposition actually adds compared to just using the full trajectory for distillation. Is it worth the complexity?
Lalam: The core idea is making those prefixes explicit optimization targets instead of just passive context <ref:2607.19395#pg1>. It’s about reusing that teacher decision information in a smarter way.
Tom: So, the main implication is that small models can benefit by combining distillation with reinforcement learning <ref:2607.19395#pg1>. They're showing this method improves small model agents over standard RL baselines like we see on TextCraft and BabyAI <ref:2607.19395#pg1>.
Jane: That’s good because it suggests that for complex agent tasks, we should focus less on just getting the final answer right and more on learning how to recover from those tricky middle steps.
Lu: Exactly, they found that explicit prefix optimization is most useful when recovering successful continuations from difficult intermediate states <ref:2607.19395#pg1>. It turns a passive context into an active training target.
Meng: But what about the actual cost of replaying those prefixes in the environment? I need to know if this runs on real traffic without blowing the budget <ref:2607.19395#pg1>.
Lalam: They're trying to unify prefix learning and continuation learning under one clipped ratio training form, which simplifies the optimization side of things <ref:2607.19395#pg1>.
Tom: So, yeah, they’ve shown that this approach allows teacher trajectories to be reused beyond just one-shot distillation by treating them as multiple decision-aware training queries <ref:2607.19395#pg1>.
Jane: It moves the goal from imitating a single perfect answer to learning robust recovery strategies across many different points in time.
Lu: The paper also flags that future work should analyze alternative prefix side objectives and old policy anchors <ref:2607.19395#pg1>. That means there’s still a lot more to discover about how these prefixes work across different types of interactions.
Meng: So, the current win is in controlled environments like TextCraft and BabyAI where we can measure those Pass@eight improvements <ref:2607.19395#pg1>. The real question is if that translates when we deploy this to something unpredictable.
Tom: It does suggest combining distillation with reinforcement learning can help small models perform better, but the next step is seeing how scalable this decomposition really is.
Jane: Next time, we'll dig into the specific group-query construction they use to select those important cut points in the teacher trajectory <ref:2607.19395#pg2>.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI