When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

arXiv:2608.05219 · cs.AI · Submitted 2026-08-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "When Privileged Guidance Misaligns".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents". It’s by Junzhuo Liu, Weiwei Li, Jun Ling, and Peng Wang from the University of Electronic Science and Technology of China.

Jane: It’s about using privileged on-policy distillation for multi-turn agents. The main idea is that when you have a synchronized teacher re-scoring a student's response at every turn with references like successful trajectories, it can be very dense supervision.

Lu: But the paper points out that in interactive environments, the student’s actions keep changing the execution state. If the student takes different actions or completes subgoals in a different order, its rollout might reach states not covered by the reference. This makes those references an unreliable source of guidance for where you are actually at.

Meng: So what's this mismatch?

Jane: The paper calls this failure mode state–reference mismatch. It happens when the successful trajectory reaches Product A from a results page, but the student is on Product B’s detail page and samples an action like "Back to Search". That reference is no longer applicable for the state the student is in now.

Tom: So what does this paper claim they are doing about it?

Lu: They introduce State-Matched Routing and Contextualized Self-Distillation, or SMRC-SD. The thesis is to provide privileged reference guidance that stays compatible with the student’s current execution state.

Jane: How do they manage that compatibility? They use the student's reached state to decide when and how a reference should guide the student. At each turn, they check state compatibility with the reference trajectory, and only distill on those matched turns.

Meng: What does that actually look like in practice?

Lu: State-Matched Routing determines if the reference supports the reached state and selects its latest compatible position. They check if "the current signature satisfies position k’s task-relevant requirements" and if "the action must also ground to an action in the current admissible-action list". If it doesn't match, that turn gets no distillation signal.

Tom: That sounds like they are filtering out bad guidance based on where you actually are.

Jane: Exactly. Then Contextualized Self-Distillation takes that matched position and constructs a privileged teacher context from the successful trajectory, a current-state summary, and a compatible continuation. This gives the teacher global structure along with localized guidance for the specific state.

Meng: So it’s not just matching; they are building this contextual package for distillation. How much does that add?

Lu: The experiments show gains over unconditional full-path self-distillation in embodied interaction, web navigation, and tool use. For example, with the Qwen3-1 point 7B model on ALFWorld, they improved task success from zero point seven four six to zero point eight six five and on WebShop from zero point five seven four to zero point six nine three >

Tom: That’s a solid lift when you look at those numbers against the unconditional approach, which is what we usually compare it to. What about the transferability?

Lu: They also found that policies trained with this method transfer to BFCL and ToolHop without needing any additional training. That suggests the guidance mechanism is robust across different tasks.

Jane: It sounds like they are showing that you don't need a perfect, static reference path to get good results in these multi-turn scenarios. So, what’s the bigger picture implication here?

Tom: The core idea is that executable references should be treated as conditional plans whose local validity needs to be established before they supervise a multi-turn policy. That means the reference isn't just a fixed script; it has to work for the state you are in right now.

Jane: It shifts the focus from just having a good trajectory to having a reliable way to use that trajectory when things get messy and dynamic during interaction.

Lu: The study also looked at some specific interventions, like fixed-state teacher interventions, which showed a contrast of +zero point zero seven zero versus +zero point one five three in the A–C comparison >

Meng: That suggests that compatibility—figuring out where the successful path provides a reliable basis for distillation—is really important here.

Tom: And they did some rigorous checks on their matching system, recovering ninety-eight point eight percent of the turns found by a history matcher on ALFWorld and succeeding on all seven hundred eighty-one audited structured-state matches >

Jane: So, to summarize this paper, "When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents" is about fixing the state–reference mismatch in privileged on-policy distillation. It shows how to use State-Matched Routing and Contextualized Self-Distillation to provide state-compatible reference guidance for multi-turn agents.

Lu: It’s important to remember their limitation too. They establish that "executable references should be treated as conditional plans whose local validity is established before they supervise a multi-turn policy". This means the method relies on you having a way to check that local validity first, which is an extra step for setting up your system >

Tom: So, for people building these systems, the lesson is that if you just feed in a reference trajectory without checking its relevance to the current state, it can actually hurt performance.

Jane: It shows that context matters immensely when you are dealing with sequences of actions where the environment keeps changing underneath your policy. We’ll have to keep an eye on how these routing mechanisms scale in even more complex, real-world interactions.

Conclusion: Tom: So, we’ve been talking about how privileged guidance can actually cause problems when agents are doing multi-turn tasks and references don't line up with where they are right now.

Jane: Right, so we’re looking at this paper called "When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents." The authors are Liu, Li, Ling, and Wang.

Lu: They’re focused on fixing that state–reference mismatch in agent training by using a specific routing method to decide when to use those successful trajectories as guidance.

Meng: It sounds like they're trying to build a smarter filter for when the teacher's reference actually makes sense for the student's current situation.

Tom: Exactly, Meng. They introduce State-Matched Routing and Contextualized Self-Distillation, or SMRC-SD, which explicitly checks if the reference trajectory matches where the agent is currently located.

Jane: So instead of just giving the teacher a full path every time, SMRC-SD figures out *when* to guide based on that state compatibility.

Lu: It involves two main parts: first, State-Matched Routing selects the most relevant position along that reference trajectory for the current state.

Tom: And second, Contextualized Self-Distillation takes that matched position and builds a rich context package—including a summary of what's happening now—to give the teacher better instructions.

Meng: From my side, it’s about making sure we aren't wasting compute on distillation signals that are just irrelevant because the student is somewhere else entirely.

Jane: And the results show this works. They saw gains over unconditional full-path self-distillation in tasks like ALFWorld and WebShop, improving task success rates by around twelve to eighteen percent on those benchmarks.

Tom: Those are solid numbers, Jane. It’s not just theoretical; they’re showing a tangible lift in performance when you have that state awareness built into the distillation process.

Lu: What this really means is that executable references shouldn't be treated as fixed scripts for every single turn; they need to be conditional plans where their local validity is checked first.

Meng: So, it’s about building a system that can dynamically assess the reliability of past successful actions before letting them dictate the next move.

Jane: And for anyone listening, it suggests that in complex interactions, having a reference path isn't enough; you need a mechanism to verify if that path is still relevant in the moment.

Tom: So, if you’re training agents with trajectory references and your agent keeps getting confused by those mismatches, SMRC-SD offers a way to guide the system more intelligently by matching states before guiding actions.

Junzhuo Liu, Weiwei Li, Jun Ling, Peng Wang

University of Electronic Science and Technology of China

cs.AI

Submitted: 2026-08-05

Updated: 2026-10-14

Comments: Updated the paper with Qwen3-4B experiments and FTRL tool-use benchmark

Code: https://github.com/liujunzhuo/SMRC-SD

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist: SMRC-SD introduces State-Matched Routing and Contextualized Self-Distillation to provide state-compatible privileged reference guidance for multi-turn agents by explicitly determining when

Key concepts

State–Reference Mismatch: This occurs when a successful past trajectory reference describes an action starting from a different state than the agent is currently in. For example, the reference might show reaching Product A, but the student is currently on Product B. This mismatch causes the teacher to penalize valid actions because they don't align with the reference's specific starting point.
State-Matched Routing: This mechanism determines if a successful trajectory reference is relevant for the agent's current situation. It checks if the current execution state satisfies the requirements of a specific position in the reference trajectory. If it matches, routing selects that position as compatible, ensuring guidance is only sought when applicable.
Contextualized Self-Distillation: This involves augmenting the privileged teacher's guidance with information about the current situation. When a state match occurs, SMRC-SD constructs a context bundle including the successful trajectory segment, a summary of the agent's reached progress, and a compatible continuation. This provides dense, relevant supervision tailored to the agent's actual state.

Terminology

Distillation — 6× in this paper · explained in 2 episodes
Self-Distillation — 2× in this paper · explained in 7 episodes
Ablation — 1× in this paper · explained in 1 episode
Grounding — 1× in this paper · explained in 3 episodes
On-Policy Distillation — 1× in this paper · explained in 3 episodes

Summary

The gist: SMRC-SD introduces State-Matched Routing and Contextualized Self-Distillation to provide state-compatible privileged reference guidance for multi-turn agents by explicitly determining when and how a successful trajectory should guide an on-policy student

State–Reference Mismatch in Privileged OPD

Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student’s response at every turn with access to training-only references, such as successful trajectories When the privileged reference is a successful trajectory, a direct approach is to provide the complete trajectory to the teacher at every turn In interactive environments, however, earlier actions continually change the execution state, including the agent’s location, inventory, object properties, and completed subgoals. This creates a local-validity problem for trajectory references because Each transition in a successful trajectory is tied to the pre-action state from which it was demonstrated. This failure mode is called state–reference mismatch, where the reference reaches Product A from Results A, whereas the student is on Product B and samples Back to Search.

How it works

SMRC-SD addresses this mismatch by treating a successful trajectory as a state-indexed training resource. At each turn, SMRC-SD matches the reached execution state against the states represented along the reference trajectory. If a match exists, it localizes a compatible continuation and constructs privileged teacher context from the successful trajectory, current-state summary, and grounded candidate. Otherwise, the turn remains optimized by the GRPO objective but receives no reference-conditioned distillation signal.

The process involves two coupled stages:

State-Matched Routing determines whether the reference supports the reached state and selects its latest compatible position. This is achieved by checking if the current signature satisfies position k’s task-relevant requirements and if the action must also ground to an action in the current admissible-action list.
Contextualized Self-Distillation then uses that position to ground teacher guidance in the current state. The teacher context is constructed as Render(pg, Summary(σt), a˜t,kt) which provides global structure, reached progress, and a localized continuation.

Policy Gains and Ablations

Across ALFWorld and WebShop, SMRC-SD consistently outperforms unconditional FullPath-SD. With Qwen3-1.7B, it improves task success from 0.746 to 0.865 on ALFWorld and from 0.574 to 0.693 on WebShop. Controlled routing and context ablations support both selecting state-compatible turns and constructing state-compatible teacher context as contributors to these gains. The analysis shows that Routing is the dominant contributor, while the complete contextualized bundle provides an additional gain under the matched-only route.

Key Findings and Interpretive Limits

The study demonstrates that SMRC-SD consistently improves the deployed policy over unconditional FullPath-SD across both model families and environments. The analysis of fixed-state teacher interventions shows that the A–C contrast is +0.070 [+0.012, +0.153], indicating that compatibility identifies where the successful path provides a reliable basis for SDL. Furthermore, the structured-state matcher recovers 98.8% of the turns found by the history matcher on ALFWorld and succeeds on all 781 audited structured-state matches. The study concludes that Executable references should be treated as conditional plans whose local validity is established before they supervise a multi-turn policy.

Implementation Details

The training loss is defined as LGRPO + λSDL where the routing mask wt determines whether SDL is applied, and the context cSMRC t is constructed for matched turns,. The process involves constructing compact signatures σ¯g,k for reference positions and σt for the current state to facilitate matching. The environment adapter constructs these signatures from ordinary observations and successful executed transitions to expose a shared interface between reference and student constructors. Evaluation starts from the ordinary initial state after removing all privileged components, ensuring that neither the policy nor the teacher receives a reference path, signature, matcher output, state summary, or candidate. The final response length for SMRC-SD is 78.8 tokens on ALFWorld compared to 142.6 tokens for FullPath-SD.

Experimental Setup

The evaluation uses two complementary multi-turn interactive benchmarks: ALFWorld and WebShop. Metrics include Average@4, Pass@4 for ALFWorld, and Acc/Score for WebShop. Baselines compared include Vanilla, GRPO, Skill-SD, SDAR, FullPath-SD (our primary trajectory-reference baseline), Dynamic Context (which changes the matched-turn context), and SMRC-SD (which combines matched-only routing with contextualized guidance) The experiments were conducted using Qwen2.5-3BInstruct and Qwen3-1.7B models. All privileged components are training-only, meaning All privileged components are training-only. The infrastructure utilized four NVIDIA H800 GPUs for the primary runs.

Model / environment Ckpt./turn Prompt/resp. Cap/util.

Qwen3 / ALFWorld 250 / 30 2,048 / 512 28,000 / 0.55

Qwen2.5 / ALFWorld 150 / 50 2,048 / 1,024 28,000 / 16

truth mismatch incidence.

Candidate-plus-suffix replay is positive continuation validation: it establishes successful reconnection for 792 of 800 audited history matches and all 781 audited structured-state matches. The fixed-anchor intervention measures context-induced teacher preference and complements rather than replaces end-to-end policy training.

J Interpretive Limits

The study establishes that Matched and unmatched are operational matcher labels, not action-correctness labels, and match coverage is not ground-. The results show that the candidate-plus-suffix replay is positive continuation validation: it establishes successful reconnection for 792 of 800 audited history matches and all 781 audited structured-state matches. The fixed-anchor intervention measures context-induced teacher preference and complements rather than replaces end-to-end policy training. The study concludes that Executable references should be treated as conditional plans whose local validity is established before they supervise a multi-turn policy.

Component Mean Median p95

Total 0.833 0.569 1.335 Context / FullPath-SD 112.9 112 148 175 Context / SMRC-SD 133.8 134 160

CPU runtime.

The index construction takes about 437.1 ms on average to construct once. The CPU time per matched turn for SMRC-SD is around 161 milliseconds. The total CPU time for the context rendering is approximately 859 milliseconds.

G Response Length and Style

SMRC-SD remains close to GRPO in response length, while FullPath-SD and especially Skill-SD are longer and more repetitive. The final checkpoint response statistics show SMRC-SD's repeated 4-gram rate is 8.8%. All privileged components are training-only, meaning All privileged components are training-only.

I Hyperparameters and Reproducibility

The reported experiments use GRPO for every rollout and add SDL only for the distillation methods. The SMRC-SD loss is defined as LGRPO + λSDL where the routing mask wt determines whether SDL is applied. The random seeds used are fixed to 0 across environment sampling, equal-count SDL turn selection, and WebShop goal-order sampling. The validation temperature is set to 0.4 for evaluation.

K3 Estimator

The chosen-token SDL estimator is defined by LSDL = P t wtlK3 t t,i m resp t,i where the denominator remains the number of tokens in the unmodified response mask.

Improvements for AI systems

State-Matched Routing (SMRC) can be used to explicitly determine when and how a privileged trajectory should guide an on-policy student by verifying whether the student’s current execution state matches a supported state along the reference trajectory. This ensures that distillation is only applied when the reference lacks locally compatible guidance, mitigating the previously identified state–reference mismatch.
Contextualized Self-Distillation (CSD) can be constructed to ground supervision by creating a state-conditioned teacher context from the successful trajectory, grounding supervision in the state actually reached, by combining the complete path, a one-line current-state summary, and the grounded candidate. This provides the teacher with highly relevant information for re-scoring.
The system can be improved by implementing an executable reference audit using a structured matcher to select candidates that preserve continuation semantics, as demonstrated by how the structured state matcher recognizes supported states reached through different histories while still checking the execution facts needed by the next action. This ensures that guidance is based on task progress, not just superficial observation similarity.
The model can be trained using a loss function structured as LGRPO + λSDL where SDL is applied conditionally based on matching, allowing for abstention removes only the privileged loss on an unmatched turn, not that turn or trajectory from on-policy learning. This allows the agent to maintain its exploration and policy updates even when the reference guidance is unreliable.
The system can be optimized by using state-aware agent context by having environment adapters reconstruct execution progress from ordinary observations and successful executed transitions, which allows the teacher to score based on what is actually relevant to the current state, as opposed to relying solely on static reference data.