Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Same Evidence, Different Answers".

Jane: Here's the gist of the paper. The gist The proposed Canonical-Context On-Policy Distillation (CCOPD) method addresses the failure of large language models to maintain consistency when information is revealed gradually across multiple turns by aligning a student model's behavior on contaminated, RAW-SHARDED histories with that of a frozen teacher conditioned on a clean, full prompt.

Tom: Problem Formulation and Self-Anchored Drift The core difficulty lies in the fragility of canonical-context consistency, where models fail to yield the same outcomes when task information is provided incrementally through a RAW-SHARDED conversation (Page 1).

Jane: This failure mode is termed self-anchored drift, which occurs because earlier assistant replies can introduce "unsupported assumptions" that later distort the final answer (Page 1).

Tom: The paper formalizes canonical-context consistency as the requirement that a reliable model should preserve its final-answer distribution under the change of presentation: π(y h(q)) ≈ π(y c(q)) (Page 3).

Title and authors: Tom: So we're looking at this paper called Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models today. It’s tackling that problem where an AI solves a task when you give it everything in one go, but then it messes up when the information comes in pieces over several turns.

Jane: Exactly. We're talking about how we can make sure that if the user gives the same complete set of facts in a long conversation, the AI gives you the exact same final answer regardless of how those facts were delivered incrementally.

Lu: The core idea here is that this failure mode is what they call self-anchored drift, where earlier assistant replies create assumptions that later distort the whole outcome.

Meng: So, it's like if you give an AI a set of clues one by one, and its first clue makes it guess something wrong that ruins the final result.

Tom: Right. And this paper proposes Canonical-Context On-Policy Distillation, or CCOPD, to fix that by aligning a student model with a teacher model.

Jane: So how does that actually work mechanically? How do we get the student to behave like the teacher when it’s being fed information in small chunks?

Lu: They use two roles for the same base model during training. There's a frozen teacher, which is conditioned on the clean, full prompt—the perfect context. And there's a trainable student that receives evidence incrementally through those raw-sharded interactions.

Tom: So the objective is to make sure that the student’s behavior along its own path matches what the teacher does when it has all that clean context available.

Jane: That sounds like a form of distillation, but applied to sequential interaction history rather than just static inputs. What's the specific math they use to measure this alignment?

Lu: They define canonical-context consistency as a requirement that the reliable model should preserve its final-answer distribution under the change of presentation, which they write as π(y h(q)) ≈ π(y c(q)) ><ref:2605.30251#pg3>.

Meng: That discrepancy is measured by something they call Ψπ, which identifies where the sharded path induces continuation preferences that are different from those induced by the clean presentation.

Tom: So they are quantifying this drift mathematically so they can train against it.

Jane: This takes us to the improvements section—what's actually new about CCOPD compared to just letting models learn from more data?

Lu: They frame it as a self-distillation objective that enforces canonical-context consistency under contaminated multi-turn histories, which is different because they are targeting this robustness during training, not by adding a repair loop at test time.

Title and authors: Tom: It’s not about retraining the model on more examples of sharded data; it’s about teaching it *how* to handle the contamination from its own partial responses.

Jane: That's a big conceptual shift, moving from just training accuracy to training consistency under difficult circumstances.

Lu: They implement this objective by minimizing the average deviation along the student rollout, which they write as Lccopd(q, yˆ) = Ψ¯θ(q, yˆ) ><ref:2605.30251#pg4>. This is calculated using what they call same-prefix on-policy supervision where the student and teacher score the same prefix under different contexts.

Meng: So they’re comparing how the student and teacher score a given prefix—say, a sequence of user inputs—when one sees it in a sharded history versus when it sees it in the full context.

Tom: That’s smart; they are forcing that alignment on the very early stages of response generation.

Jane: And what did they find when they tested this on math conversations, which is where these issues usually pop up?

Lu: They restricted training to math conversations from GSM8K and GSM8K-Aug. Strikingly, the math-only post-training signal transfers well beyond just math, improving RAW-SHARDED accuracy on math and yielding zeroshot gains across five non-math settings ><ref:2605.30251#pg5>.

Tom: Zeroshot gains across five non-math settings is significant because it shows the mechanism isn't just specialized for arithmetic; it generalizes its reasoning structure.

Jane: It also noted that CCOPD yields a three-twenty percent average relative improvement over the unmodified base model on RAWSHARDED performance, while largely preserving FULL and CONCAT performance ><ref:2605.30251#pg1>.

Meng: From an engineering standpoint, that transfer beyond math is what makes it interesting for us because we don't always have perfectly clean data for every domain.

Lu: The mechanism diagnostics show that CCOPD maintains stronger user-evidence focus across RAW-SHARDED response rounds after process replies enter the context, which is shown by increased SAAR and a flattened assistant-associated canonical deviation ><ref:2605.30251#pg4>.

Tom: SAAR, that Self-Anchor Attention Ratio, seems to be a key metric here for seeing if the model is looking at the right stuff in the transcript.

Jane: It sounds like CCOPD makes the model better at focusing on what the user has actually provided rather than getting distracted by its own prior commitments ><ref:2605.30251#pg4>.

Lu: They also showed that tests on matched polluted histories show that CCOPD reduces sensitivity to misleading anchors when complete user evidence is available, because the base model collapses under pollution while CCOPD remains near clean performance ><ref:2605.30251#pg8>.

Title and authors: Meng: That means if you have all the facts eventually, this method helps it ignore the noise introduced by those early mistakes.

Tom: It’s a nice stabilization effect during the conversation flow.

Jane: So we’ve covered how they fix self-anchored drift using on-policy distillation to enforce consistency between partial and full contexts. Where does this leave us regarding what these models can actually do in the real world?

Lu: They evaluated it across six task families, including math word problems, code generation, function calling, text-to-SQL, table-to-text generation, and long-context summarization ><ref:2605.30251#pg5>.

Tom: That’s a broad range of capabilities they tested on.

Meng: For deployment at scale like we do in production environments, what are the practical limitations of this CCOPD approach?

Lu: They did note that the method only optimizes the final answer state after all user evidence is available, which means it cannot correct errors made by the FULL teacher itself ><ref:2605.30251#pg9>.

Jane: That’s a fair point. So it helps anchor the final result but doesn't fix an error that happened much earlier in the sequence if that earlier turn was already contaminated.

Tom: It stops the drift from compounding, which is useful, even if it can't undo the initial mistake.

Lu: The paper concludes by reframing RAW-SHARDED multi-turn failure as self-anchored canonical-context drift ><ref:2605.30251#pg6>. They also noted that CCOPD adds no inferencetime component and improves on-policy sharded accuracy after math-only training, and transfers zeroshot to five non-math task types ><ref:2605.30251#pg6>.

Jane: It seems the real win here is the robustness it gives to models when they are interacting over a long transcript where the context gets messy ><ref:2605.30251#pg6>. So, what does this mean for how we design these conversational agents?

Tom: It suggests that for any model used in a multi-turn setting, we should consider training it with this kind of self-distillation to stabilize its internal representation against history contamination.

Lu: They are pushing the idea that canonical-context consistency should be internalized as a core model capability ><ref:2605.30251#pg2>.

Meng: If we were deploying this today, I’d want to know if the training cost is prohibitive for every single model in our pipeline.

Tom: The paper focuses on how to train one base model effectively, and the math shows that it's an efficient way to transfer knowledge across contexts ><ref:2605.30251#pg5>.

Jane: It’s about making sure that when we deploy these conversational agents, they don't suddenly start hallucinating or deviating wildly just because the user is talking back to them in a messy way ><ref:2605.30251#pg6>. We’ve got a lot more on the nature of attention sinks shaping decoding strategies coming up next.

The paper's summary: Tom: So we’re looking at this paper titled "Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models" and what it means for how we train these massive models to handle conversation history.

Jane: Basically, the paper is tackling that problem where an AI solves a task when you give it everything in one go, but then it messes up when the information comes in pieces over several turns. It’s about making sure that if the user gives the same complete set of facts in a long conversation, the AI gives you the exact same final answer regardless of how those facts were delivered incrementally.

Tom: Yeah, and they call this failure mode self-anchored drift. It happens because earlier assistant replies can introduce assumptions that later distort the whole outcome when the information is sharded.

Lu: The paper formalizes canonical-context consistency as a requirement that a reliable model should preserve its final-answer distribution under the change of presentation, which they write as pi(y h(q)) equals pi(y c(q)). They measure this gap with something called psi pi.

Jane: So how does this distillation objective actually work in practice? How do you get the student model to behave like the teacher model when it’s being fed information in small chunks?

Tom: They use a frozen teacher, which is conditioned on the clean, full prompt—the perfect context. And they have a trainable student that receives evidence incrementally through those raw-sharded interactions. The objective is to align the student’s behavior along its own path with the teacher’s canonical full-context behavior.

Lu: The loss function they use to enforce this alignment is Lccopd(q, yˆ) equals psi¯theta(q, yˆ). This loss is calculated using what they call same-prefix on-policy supervision.

Jane: Same prefix supervision means the student and teacher score the same sequence of user inputs under different contexts—one context being a sharded history, and the other being the full context.

Tom: Exactly. It’s forcing that alignment on the very early stages of response generation by comparing how they score those initial pieces of information in both scenarios.

Jane: They found that even when training just on math conversations from GSM8K, that signal transfers well beyond math, improving accuracy in five other non-math settings zero shot.

Lu: That’s significant because it suggests the underlying mechanism isn't just specialized for arithmetic; it generalizes its reasoning structure across different tasks.

Tom: And they noted a thirty two percent average relative improvement over the unmodified base model on raw sharded performance, while still keeping the full and concat performance pretty stable.

Jane: So this method seems to make models much more robust when they are interacting over a long transcript where the context gets messy, even if it can't fix an error that happened much earlier in the sequence.

Lu: That’s a fair point. The paper concludes by reframing RAW-SHARDED multi-turn failure as self-anchored canonical-context drift and suggesting this distillation is a way to internalize canonical-context consistency as a core model capability.

Tom: It does mean we should be considering this kind of self-distillation during training for any model used in a multi-turn setting to stabilize its internal representation against history contamination.

Jane: So, what’s next? We’re going to look at how they actually test this robustness using these mechanism diagnostics like the SAAR ratio.

The paper's improvements: Tom: So we're looking at how they suggest improving this CCOPD method to make it even better than just training on math data alone.

Jane: The paper suggests that the key improvement is actually making sure the model learns to reduce sensitivity to contamination from earlier assistant turns when complete user evidence is available.

Tom: So, instead of just stopping the drift, they’re trying to make the model actively ignore misleading anchors when it sees all the facts finally. They use probes like SAAR for that.

Lu: They showed that CCOPD increases layer SAAR and flattens a measure called assistant-associated canonical deviation, which shows the model is focusing better on what the user has actually provided over earlier assistant commitments.

Jane: That sounds like a direct fix for that self-anchored drift we talked about earlier, where those early mistakes keep pulling the final answer off track.

Tom: Exactly. It moves the mechanism from just enforcing consistency to actively grounding itself better in what’s currently present in the user evidence.

Meng: From an engineering standpoint, if this actually works at scale, it means we don't have to constantly re-verify conversation history after every single turn; the model gets more reliable as the conversation goes on.

Jane: It really changes how you think about deployment because instead of relying solely on perfect full context, you can build in a way for the model to self-correct based on what it’s seen so far.

Lu: They also found that CCOPD reduces sensitivity to misleading anchors when complete user evidence is available, meaning the base model collapses under pollution while CCOPD stays near clean performance.

Tom: That’s a strong result because it means we can trust the model more in messy, real-world conversations where context gets polluted by prior turns.

Jane: So this moves us toward systems that are more resilient to the kind of human interaction we see every day, where things get complicated and sometimes confusing over time.

Lu: The implication is that canonical-context consistency should be internalized as a core capability, not just something you patch on with some distillation loss during training.

Tom: Yeah, so instead of thinking about a perfect clean prompt at the start, you’re thinking about how the model handles the gradual messy inflow of information.

Jane: And this leads us perfectly into what they tested next: how robust is this across different types of tasks beyond just math word problems?

Conclusion: Tom: So to wrap up, we’ve seen how Canonical-Context On-Policy Distillation addresses self-anchored drift by forcing a student model to match a frozen teacher when information is fed incrementally across multiple turns.

Jane: Essentially, this paper shows that by training the model to preserve its final answer distribution under different presentation styles, you get much more reliable behavior in complex conversational flows.

Tom: The results are interesting because they show significant zero shot gains in non-math tasks and a measurable improvement over the unmodified base model on raw sharded performance, specifically a thirty two percent average relative improvement.

Lu: From my view, the major win is that this approach makes canonical-context consistency something you can actually enforce during training, rather than just hoping it emerges.

Meng: For deployment at scale I’m curious about how much computational overhead that distillation process adds to a model pipeline when we have thousands of versions to run.

Lalam: I’m confident this is the right direction because the mechanism diagnostics show stronger user evidence focus across response rounds, which translates directly into better grounding for users.

Jane: It really changes our mindset on how we build these agents; it suggests that robustness to context contamination needs to be baked into the training objective itself.

Tom: So, moving forward, we’re looking at how this distillation signal can be used in more complex architectures, perhaps integrating it with those test-time hinting ideas we touched on earlier.

Lu: The authors did mention a limitation though: the method only optimizes the final answer state after all user evidence is available; it cannot correct errors made by the full teacher itself during an intermediate step.

Meng: That’s a fair limitation; it doesn't fix mistakes that happen when the teacher is still generating its response, just when we look at the final output.

Jane: It’s about stopping the drift from compounding, which is a meaningful improvement for user experience in long conversations, even if it can't undo every single early mistake.

Tom: I think this paper on "Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models" gives us a concrete way to stabilize these models before they go into production environments.

Lu: It proves that we can internalize consistency as a core capability rather than just a post-hoc fix.

Meng: I’m still watching the scale of these experiments; it’s one thing to see thirty two percent improvement on GSM8K, but that doesn't tell me if it holds up when we push the complexity to things like code generation or function calling.

Lalam: I think the evidence across those six task families is solid; I'm sure this method will be useful in making our models more dependable for multi-turn interactions.

Jane: It’s a solid piece of research that shows how to tackle one of the trickiest problems in building conversational AI today.