Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Rethinking Relative Policy Differences Transfer of On-Policy Distillation".

Jane: The gist The log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper now called "Rethinking Relative Policy Differences Transfer of On-Policy Distillation."

Jane: It seems like the main idea is they are shifting away from needing a super strong teacher to train a model.

Lu: Exactly. They develop this unified view where the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization.

Meng: So they're not just matching the final distribution anymore; they're using the difference between how two policies change as a learning signal.

Tom: Right. The paper claims this formulation induces an on-policy update for the target model, and it recovers conventional on-policy distillation as a special case, but it opens up any-to-any transfer across models with different scales or training stages.

Jane: That sounds like a big deal because traditionally, you needed that teacher to be substantially more capable than the student just to get anything useful from distillation.

Lalam: The paper formalizes this by looking at a proxy pair where the expert is obtained from the reference by solving a one-step KL-regularized optimization problem with token-level task reward.

Tom: And the optimal solution to that problem yields this relative policy difference, which they call delta phi of s a equal one over tau r star of s a minus log z phi of s.

Jane: That difference is defined only by the proxy pair itself, not by some target reference model that we're trying to match. That sounds like it simplifies things a lot in terms of model compatibility.

Lu: It does simplify the setup because this difference doesn't depend on the target reference at all.

Meng: But then they have to figure out if this specific difference actually helps the target model, which is where the core question lies.

Tom: The systematic experiments yield three main findings regarding when these relative policy differences provide effective supervision. First, a stronger expert isn't necessarily better for supervision.

Jane: That’s counter-intuitive because we usually think the bigger teacher gives better instruction, but they found that improvements from task-directed training like reinforcement learning or supervised fine-tuning transfer consistently.

Lu: And improvements that come just from model scale alone transfer weakly or not at all, which points to the mechanism being more important than just raw power.

Tom: Second, the expert and reference serve complementary roles. The expert provides the capability to be transferred, but the reference filters out shared behavior and isolates the relevant residual capability for transfer.

Meng: So think of it like a filter that removes what both models already know, leaving only what's new or useful to pass along.

Paper summary: Jane: And third, even a useful difference doesn't transfer to every target model. Success requires the induced update to move the target toward the expert and generalize beyond just sampled rollouts.

Tom: That’s a crucial caveat. Raw policy distance doesn't predict this compatibility, but an appropriate reference can rescue otherwise unsuccessful transfers.

Lu: When we look at how much capability is available, moving the expert from early to regular and late reinforcement learning checkpoints raises the gp from twenty-two point nine to twenty-six point eighty-nine and then to thirty-two point twenty-eight and the gt tracks that trend.

Meng: That shows that even just changing when you stopped training the expert makes a difference in how much signal you can transfer.

Jane: So, what does this mean for someone just listening? It means that if you're trying to train a model using distillation, you can't just pick the strongest model available; you have to carefully select your proxy pair and your reference policy.

Tom: The paper unifies these various relative policy update transfer methods under this broader policy-optimization view, which they call A2-OPD.

Jane: So A2-OPD is this any-to-any formulation that decouples the proxy pair defining the relative difference from the target receiving it, with conventional on-policy distillation recovered as a special case.

Lu: It means we don't need to worry about matching an absolute distribution anymore; we just focus on this relative improvement signal.

Meng: But what about deployment? If I’m shipping models, how does this formulation change the cost or the complexity of setting up these proxy pairs for production?

Tom: Well, the paper shows that transfer patterns persist across different tasks. Configurations that succeed or fail on math show the same qualitative outcomes on code.

Jane: They found positive transfers between Qwen3 models in both math and code, showing gains of twenty-six point eight and twenty-five point nine for math, and plus two point three and plus one point nine for code <ref:2610.13205#pg3>.

Lu: But they explicitly show that transferring a difference from a Qwen3 model to a Qwen2 point five target fails in both domains, producing gains of negative one point seven and negative six point four.

Meng: So the success or failure of this transfer isn't just about the models; it’s about what kind of capability you are trying to move and what your target can actually handle.

Tom: The paper unifies various relative policy update transfer methods under a broader policy-optimization view, leading to A2-OPD, an any-to-any formulation that decouples the proxy pair defining the relative difference from the target receiving it.

Paper summary: Jane: It really helps clarify when this kind of relative policy difference transfer provides effective supervision and offers practical guidance for selecting proxy pairs and targets.

Lu: The key is understanding that a useful difference must induce an update that the target can follow and generalize from, and an appropriate reference can recover some otherwise unsuccessful transfers.

Meng: So if I'm trying to get a model to learn something new, I need to find a reference that helps filter out the noise while leaving the right signal for my target.

Tom: The paper itself states that the outcome is not a function of policy distance because the performance realization ratio and transferred math accuracy don't show a monotone relationship with the proxy base KL.

Jane: That’s important because it means you can’t just look at how far apart two policies are and expect it to tell you if they will work together in distillation.

Lu: Matching the reference generation restores both overlap with the expert and validation accuracy, while cross-generation references make both collapse.

Meng: That tells me that for practical purposes, I need to be careful about which reference model I use; a good one can actually fix a bad transfer situation.

Tom: So, this whole framework is about defining this implicit reward through the log-probability ratio between the proxy expert and proxy reference.

Jane: It sounds like they are providing a much more principled way to think about how to reuse costly post-training outcomes across different systems.

Lu: The implication for broader AI research is that we should be focusing less on absolute policy matching and more on these residual policy changes that arise from specific training methods.

Meng: For deployment, it suggests we need a system where the reference model acts as a dynamic filter during the transfer process to maximize utility.

Tom: That’s exactly what they propose; it’s about using the reference to isolate that residual capability for transfer.

Jane: So, to wrap up, this work provides A2-OPD, which decouples the proxy pair defining the relative difference from the target receiving it.

Lu: It unifies these methods under a broader policy-optimization view and clarifies when this any-to-any relative policy difference transfer provides effective supervision.

Meng: We should be looking at how to engineer these proxy pairs and references based on what we know about task-directed training effects versus just model scaling effects.

Tom: The paper title, "Rethinking Relative Policy Differences Transfer of On-Policy Distillation," suggests a fundamental rethinking of the supervision signal itself.

Jane: It’s a shift from matching an absolute final output to understanding the direction of improvement between policies at different stages.

Conclusion: Tom: So we’ve been looking at this paper called "Rethinking Relative Policy Differences Transfer of On-Policy Distillation." It’s about how we can transfer skills between different AI models without needing a super strong teacher for every single task.

Jane: Right, so instead of just matching the final output exactly, they're focusing on the difference between policies at different training stages and using that as a signal.

Lu: The authors are proposing this new way to define an implicit reward using the log-probability ratio between a proxy expert and a proxy reference policy.

Meng: It seems like they’ve unified a bunch of different transfer methods into one framework called A2-OPD, which decouples the difference definition from the target model itself.

Tom: Exactly, and it recovers conventional on-policy distillation as just one specific case within this bigger policy optimization view.

Jane: What does this mean for us? It suggests we can move away from needing that perfect teacher model to get useful results when transferring knowledge across models with different training setups or scales.

Lu: The main finding is that the relative policy difference itself isn't the whole story; it has to induce an update in the target model that it can actually follow and generalize from.

Tom: That’s a key caveat, because raw policy distance doesn't predict whether two models will work together well for distillation.

Jane: And they show that an appropriate reference model can actually help rescue transfers that would otherwise fail, which is interesting because you usually expect a good teacher to make everything work.

Lu: The paper also shows that the expert sets the maximum capability available, and the reference acts like a filter to isolate just the residual skill we want to pass along.

Meng: From an engineering standpoint, this suggests we need to be very careful about selecting our proxy pairs and references because a bad reference can actually make a good transfer setup fail.

Tom: It really shifts the focus from just matching distributions to understanding the direction of improvement between policies at different learning checkpoints.