PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

arXiv:2606.16215 · cs.CL, cs.AI, cs.LG · Submitted 2026-06-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents".

Tom: The gist The PACT framework proposes a privileged trace co-training method for multi-turn tool-use agents that keeps rollout generation prompt-only while using expert traces only as training-time optimization signals to…

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents. It’s about how to make these agents, the ones that reason and use tools across several turns, better after training.

Jane: Exactly. The main problem they’re tackling is that standard reinforcement learning struggles with sparse rewards and figuring out which part of the long sequence caused success, even when you match it to just using prompts for inference #pg1.

Lu: They are proposing PACT as a Privileged Trace Co-Training framework to fix this because supervised fine-tuning on expert traces gives you dense supervision but it can actually overconstrain the model to one specific path #pg2.

Meng: So, what’s the big idea behind PACT? Is it just another way to use traces?

Tom: Not quite. The core idea is to use those expert traces only as training-time optimization signals rather than hints during the actual rollout generation phase #pg1. They keep making the rollout generation prompt-only and then guide optimization with two specific signals.

Jane: That sounds like a smart way to avoid the issues of getting stuck on fixed trajectories from standard fine-tuning #pg2. How do they actually guide things once they have that prompt-only rollout?

Lu: They use a trace-conditioned reinforcement learning surrogate that evaluates those prompt-only rollouts under expert trace context #pg1. This lets them use the expert traces as privileged context for RL updates without ever letting them influence how the rollout is generated.

Meng: Okay, so you're looking at the prompt-only generation, but then checking if that path makes sense using the expert trace info? That’s interesting from an engineering standpoint.

Tom: Right. And they combine that with a component-aware supervised fine-tuning loss #pg1. This loss supervises only the model controllable parts, like the reasoning tokens and the tool-call tokens, instead of supervising the whole sequence #pg5.

Jane: So it’s not just optimizing the final answer; they are specifically paying attention to how those intermediate decisions—the reasoning steps and when to call a tool—are made #pg5. That sounds like a targeted way to improve process supervision.

Lu: They make this component-aware supervision stronger through something called Annealed Reasoning Supervision, where they supervise the semantic reasoning segments instead of just raw token counts, moving from L4 down to L0 #pg5.

Meng: That makes sense for practical application because it means you’re tuning the critical thinking parts of the model rather than just tweaking every single word #pg5. So what’s the overall objective they are trying to achieve?

Tom: The overall objective combines these two things: "L trace i = L h RL(i) + beta SFT L SFT(i; s)" #pg4. It’s blending the trace-conditioned RL update with that component-aware SFT loss #pg4.

Paper summary: Jane: And to stop the model from relying too much on that privileged context, they use something called prompt-only anchoring #pg4. This samples a gate for each prompt, and you switch between the trace-guided update and standard prompt-only RL based on that gate value #pg4.

Lu: From a latent-trace view, they decompose the likelihood of a rollout into two parts: the trace likelihood term which uses component-aware SFT to learn structure from expert traces #pg5, and the privileged rollout-likelihood term which uses trace-conditioned RL under expert context #pg5.

Meng: So it sounds like they’ve created a system where you get the structural knowledge from the traces while still keeping the generation process clean and prompt-only for inference #pg1. That’s a solid design for long interactions.

Tom: The results on FTRL, BFCL, and ToolHop show that PACT consistently outperforms strong SFT- and RL-based baselines #pg1. For example, on Qwen3-4B for FTRL, the Solve-R score went up from thirty-seven point one seven to forty-two point four one #pg6.

Jane: And the ablation study confirms that combining the trace-conditioned RL with that component-aware SFT actually gives them the best performance, hitting a Solve-R of forty point four five and a SolveF1 of thirty-three point two one #pg6.

Lu: They also found that prompt-only anchoring works well too; a moderate gate value seems to give you the best balance between getting better credit assignment from the traces and maintaining good prompt-only generalization #pg6.

Meng: So what’s the main thing this means for practical AI development? It suggests that you don't necessarily need to alter how you sample rollouts just to get better results; you can use existing traces in a smarter, more focused way during optimization #pg1.

Tom: It really does. The PACT framework shows the value of using expert traces purely as training signals during the optimization phase, not as hints that change the generation condition itself #pg1.

Jane: So to wrap up on this PACT paper, it’s about separating trace-based optimization from rollout generation while using two complementary signals—trace-conditioned RL and component-aware SFT—to get better results on multi-turn tool use agents #pg1.

Lu: It sets a path for combining imitation losses with reinforcement learning in a way that respects the rich structure of reasoning, tool calls, and observations in long interactions #pg2.

Meng: I think the practical implication is that we can get better performance on complex tasks without having to fundamentally change our inference pipeline for every new model iteration #pg1.

Tom: That’s what it points toward. We can refine the training objectives for these agents without messing with how they generate text in real-time #pg1.

Jane: So we’ve seen how PACT uses privileged trace co-training to outperform other strong SFT and RL methods on multi-turn benchmarks #pg6. That’s what this paper is all about.

Conclusion: Tom: So we’re wrapping up PACT, that paper about Privileged Trace Co-Training for Multi-Turn Tool-Use Agents by those authors at Zhenbangdu. It boils down to them figuring out how to train these complex agents better after they've already been trained on something.

Jane: Right. They basically found a way to use expert traces—those perfect sequences of reasoning and tool calls—not as direct training hints during the learning process, but more like special optimization signals for later.

Lu: I think what’s really clever about it is how they keep the rollout generation prompt-only, meaning it still looks like normal inference when it’s being created.

Meng: So they aren't changing how the AI actually generates its answers in real time, which is important for deployment. What does this mean for us on the ground?

Tom: It means we can use those high-quality expert traces to guide improvement without messing up the generation process itself, which solves a big problem with standard reinforcement learning setups.

Jane: They combine two things to do that: they have this trace-conditioned reinforcement learning surrogate and a component-aware supervised fine-tuning loss. That’s how they get both the structure and the specific tool calls right.

Lalam: From my side, this means when we tune these agents, we can focus on improving the logic of their reasoning steps rather than just tweaking their final output tokens randomly. It helps build a more robust cultural understanding within our systems.

Meng: So if I’m building a system that needs to handle long conversations, PACT suggests focusing my training on those critical decision points—the reasoning prefixes and the tool calls—instead of trying to supervise every single word in the whole chat history.

Tom: Exactly. And they found that this approach consistently beats other strong SFT and RL methods on benchmarks like FTRL and ToolHop. The results show a real lift in performance, specifically in how well the agents solve multi-turn problems.

Jane: It really proves that using expert traces as privileged signals during training is a way to get better credit assignment without getting stuck in those over-constrained loops that standard methods can fall into.

Meng: So it’s less about brute force training and more about intelligently leveraging existing high-quality data structures for targeted refinement. That’s a practical shift for building reliable agents on complex tasks.

Tom: It is a shift, definitely. Now, we need to see how this specific co-training setup handles even longer interactions, which is where the next part of the paper dives in.

Georgia Institute of Technology, Ohio State University, University of Pennsylvania, Arizona State University

cs.CL, cs.AI, cs.LG

Submitted: 2026-06-15

Updated: 2026-10-09

Comments: Project page: https://zhenbangdu.github.io/pact-project-page/

Project page: https://zhenbangdu.github.io/pactproject-page

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist The PACT framework proposes a privileged trace co-training method for multi-turn tool-use agents that keeps rollout generation prompt-only while using expert traces only as training-time

Key concepts

Prompt-Only Rollout Generation: This technique ensures that the agent generates its responses based solely on the input prompt, mimicking how it behaves during actual inference. This keeps the training process grounded in prompt-only behavior, preventing overfitting to specific trace sequences and ensuring generalization.
Trace-Conditioned RL Surrogate: This component uses expert traces as privileged context to guide reinforcement learning updates without using them to generate new rollouts. It evaluates the quality of prompt-only rollouts by considering the context provided by expert traces, allowing for targeted policy improvements based on trace information.
Component-Aware SFT Loss: Instead of supervising the entire trajectory, this loss only supervises model-controllable parts like reasoning tokens and tool-call tokens. It uses a component-aware mask to assign supervision weights to these specific parts, allowing for fine-grained process supervision over reasoning and tool use.

Terminology

Trace — 12× in this paper · explained in 2 episodes
Credit assignment — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist The PACT framework proposes a privileged trace co-training method for multi-turn tool-use agents that keeps rollout generation prompt-only while using expert traces only as training-time optimization signals to improve performance.

How it works

PACT addresses the challenge of post-training multi-turn tool-use agents by separating rollout generation from trace-based optimization (Page 1). The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints (Page 1). This is achieved by keeping the rollout generation prompt-only, and then using two complementary signals to guide optimization: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context and a component-aware SFT loss that supervises reasoning prefixes and tool-calls with annealed strength (Page 1).

Core Components of PACT

The framework introduces several key designs to achieve its objective (Page 2). First, the prompt-only rollout generation ensures that the policy samples are rolled out from the prompt alone, matching inference-time behavior (Page 2). Second, it utilizes a trace-conditioned RL surrogate where expert traces are used as privileged context to guide trace-conditioned RL updates without entering rollout generation (Page 4). This is defined by computing a policy ratio using the context c h = [xi; h∗i] (Page 4).

Component-Aware SFT

To provide direct process supervision, PACT adds an auxiliary component-aware SFT loss (Page 5). This loss supervises only model-controllable components: reasoning tokens and tool-call tokens rather than the full trajectory (Page 5). The supervision is controlled by a component-aware supervision mask m i,n(s) which selects the active reasoning-prefix positions through Ri(s) and assigns weight λa(s) to complete tool-call spans in Ai (Page 5). Furthermore, PACT employs Annealed Reasoning Supervision, where supervision is done over semantic reasoning segments rather than raw token counts, following the order L4 → L3 → · · · → L0 (Page 5).

Overall Objective and Anchoring

The overall objective for a prompt i combines these two trace-based updates: L trace i = L h RL(i) + βSFTLSFT(i; s) (Page 4). To prevent over-reliance on the privileged context, PACT incorporates prompt-only anchoring, which samples a prompt-level gate z (e) ∼ Bernoulli(q) for each prompt xi (Page 4). The final per-prompt objective is Li = (1 − z(e)i)L trace i + z(e)i L x RL(i), allowing the model to switch between trace-guided and standard prompt-only RL based on the gate value (Page 4).

Latent-Trace View

A latent-trace view connects these two objectives by decomposing the likelihood of a prompt-only rollout as marginalizing over possible latent traces: πθ(τ x) = X h pθ(h x)πθ(τ x, h) (Page 5). This decomposition shows two roles for expert traces: the trace-likelihood term, which encourages learning useful structure from the expert trace via component-aware SFT, and the privileged rollout-likelihood term, which scores prompt-only rollouts under the expert-trace context using trace-conditioned RL (Page 5).

Experimental Results

Experiments on FTRL, BFCL, and ToolHop consistently show that PACT consistently improves over strong SFT- and RL-based baselines (Page 1). For instance, on Qwen3-4B for FTRL, PACT improved Solve-R from 37.17 to 42.41 (Page 6). The ablation study confirms that trace-conditioned RL alone improves Solve-P over prompt-only RL, but combining it with component-aware SFT yields the best performance, reaching 40.45 for Solve-R and 33.21 for SolveF1 (Page 6). The effect of prompt-only anchoring shows that a moderate q provides a better trade-off: trace-based co-training improves credit assignment, while lightweight prompt-only anchoring improves prompt-only generalization (Page 6).

Conclusion

PACT successfully demonstrates the value of using expert traces as privileged optimization signals without exposing them during rollout generation (Page 1). The framework combines trace-conditioned RL and component-aware SFT, supported by a latent-trace view that explains how these objectives are complementary (Page 7). PACT consistently outperforms strong SFT- and RL-based baselines on multi-turn tool-use benchmarks (Page 6).

Conclusion

Improvements for AI systems

Bold header: Trace-Conditioned RL Surrogate Implementation

This improves agents by using expert traces as privileged context for policy updates, defined as c h = [xi; h∗i], which allows the model to learn from expert trajectories without altering rollout generation, as described in Section 4.2.

Bold header: Component-Aware SFT Loss

This enhances process supervision by implementing a component-aware SFT loss that supervises only model-controllable components: reasoning tokens and tool-call tokens, providing guidance through annealing schedules for reasoning prefixes and tool-call scales, as detailed in Section 4.3.

Bold header: Prompt-Only Anchoring Mechanism

This reduces over-reliance on training data by introducing a prompt-only anchoring mechanism, where the final objective is defined as Li = (1 − z(e)i)L trace i + z(e)i L x RL(i), ensuring the policy remains effective under the inference-time condition.

Bold header: Latent-Trace View for Objective Motivation

The paper provides a latent-trace view that connects the objectives, explaining how expert traces guide optimization without being used during rollout generation, linking the trace-likelihood term (Component-aware SFT) and the privileged rollout-likelihood term (Trace-conditioned RL), as shown in Section 5.

Bold header: Dynamic SFT Schedule Selection

The system can utilize segmentlevel annealing for reasoning and loss-scale annealing for tool-calls to achieve optimal performance, as supported by Table 6, which shows this combination outperforms token-count annealing or fixed supervision.