Rethinking Relative Policy Differences Transfer of On-Policy Distillation

arXiv:2610.13205 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Rethinking Relative Policy Differences Transfer of On-Policy Distillation".

Jane: The gist The log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper now called "Rethinking Relative Policy Differences Transfer of On-Policy Distillation."

Jane: It seems like the main idea is they are shifting away from needing a super strong teacher to train a model.

Lu: Exactly. They develop this unified view where the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization.

Meng: So they're not just matching the final distribution anymore; they're using the difference between how two policies change as a learning signal.

Tom: Right. The paper claims this formulation induces an on-policy update for the target model, and it recovers conventional on-policy distillation as a special case, but it opens up any-to-any transfer across models with different scales or training stages.

Jane: That sounds like a big deal because traditionally, you needed that teacher to be substantially more capable than the student just to get anything useful from distillation.

Lalam: The paper formalizes this by looking at a proxy pair where the expert is obtained from the reference by solving a one-step KL-regularized optimization problem with token-level task reward.

Tom: And the optimal solution to that problem yields this relative policy difference, which they call delta phi of s a equal one over tau r star of s a minus log z phi of s.

Jane: That difference is defined only by the proxy pair itself, not by some target reference model that we're trying to match. That sounds like it simplifies things a lot in terms of model compatibility.

Lu: It does simplify the setup because this difference doesn't depend on the target reference at all.

Meng: But then they have to figure out if this specific difference actually helps the target model, which is where the core question lies.

Tom: The systematic experiments yield three main findings regarding when these relative policy differences provide effective supervision. First, a stronger expert isn't necessarily better for supervision.

Jane: That’s counter-intuitive because we usually think the bigger teacher gives better instruction, but they found that improvements from task-directed training like reinforcement learning or supervised fine-tuning transfer consistently.

Lu: And improvements that come just from model scale alone transfer weakly or not at all, which points to the mechanism being more important than just raw power.

Tom: Second, the expert and reference serve complementary roles. The expert provides the capability to be transferred, but the reference filters out shared behavior and isolates the relevant residual capability for transfer.

Meng: So think of it like a filter that removes what both models already know, leaving only what's new or useful to pass along.

Paper summary: Jane: And third, even a useful difference doesn't transfer to every target model. Success requires the induced update to move the target toward the expert and generalize beyond just sampled rollouts.

Tom: That’s a crucial caveat. Raw policy distance doesn't predict this compatibility, but an appropriate reference can rescue otherwise unsuccessful transfers.

Lu: When we look at how much capability is available, moving the expert from early to regular and late reinforcement learning checkpoints raises the gp from twenty-two point nine to twenty-six point eighty-nine and then to thirty-two point twenty-eight and the gt tracks that trend.

Meng: That shows that even just changing when you stopped training the expert makes a difference in how much signal you can transfer.

Jane: So, what does this mean for someone just listening? It means that if you're trying to train a model using distillation, you can't just pick the strongest model available; you have to carefully select your proxy pair and your reference policy.

Tom: The paper unifies these various relative policy update transfer methods under this broader policy-optimization view, which they call A2-OPD.

Jane: So A2-OPD is this any-to-any formulation that decouples the proxy pair defining the relative difference from the target receiving it, with conventional on-policy distillation recovered as a special case.

Lu: It means we don't need to worry about matching an absolute distribution anymore; we just focus on this relative improvement signal.

Meng: But what about deployment? If I’m shipping models, how does this formulation change the cost or the complexity of setting up these proxy pairs for production?

Tom: Well, the paper shows that transfer patterns persist across different tasks. Configurations that succeed or fail on math show the same qualitative outcomes on code.

Jane: They found positive transfers between Qwen3 models in both math and code, showing gains of twenty-six point eight and twenty-five point nine for math, and plus two point three and plus one point nine for code <ref:2610.13205#pg3>.

Lu: But they explicitly show that transferring a difference from a Qwen3 model to a Qwen2 point five target fails in both domains, producing gains of negative one point seven and negative six point four.

Meng: So the success or failure of this transfer isn't just about the models; it’s about what kind of capability you are trying to move and what your target can actually handle.

Tom: The paper unifies various relative policy update transfer methods under a broader policy-optimization view, leading to A2-OPD, an any-to-any formulation that decouples the proxy pair defining the relative difference from the target receiving it.

Paper summary: Jane: It really helps clarify when this kind of relative policy difference transfer provides effective supervision and offers practical guidance for selecting proxy pairs and targets.

Lu: The key is understanding that a useful difference must induce an update that the target can follow and generalize from, and an appropriate reference can recover some otherwise unsuccessful transfers.

Meng: So if I'm trying to get a model to learn something new, I need to find a reference that helps filter out the noise while leaving the right signal for my target.

Tom: The paper itself states that the outcome is not a function of policy distance because the performance realization ratio and transferred math accuracy don't show a monotone relationship with the proxy base KL.

Jane: That’s important because it means you can’t just look at how far apart two policies are and expect it to tell you if they will work together in distillation.

Lu: Matching the reference generation restores both overlap with the expert and validation accuracy, while cross-generation references make both collapse.

Meng: That tells me that for practical purposes, I need to be careful about which reference model I use; a good one can actually fix a bad transfer situation.

Tom: So, this whole framework is about defining this implicit reward through the log-probability ratio between the proxy expert and proxy reference.

Jane: It sounds like they are providing a much more principled way to think about how to reuse costly post-training outcomes across different systems.

Lu: The implication for broader AI research is that we should be focusing less on absolute policy matching and more on these residual policy changes that arise from specific training methods.

Meng: For deployment, it suggests we need a system where the reference model acts as a dynamic filter during the transfer process to maximize utility.

Tom: That’s exactly what they propose; it’s about using the reference to isolate that residual capability for transfer.

Jane: So, to wrap up, this work provides A2-OPD, which decouples the proxy pair defining the relative difference from the target receiving it.

Lu: It unifies these methods under a broader policy-optimization view and clarifies when this any-to-any relative policy difference transfer provides effective supervision.

Meng: We should be looking at how to engineer these proxy pairs and references based on what we know about task-directed training effects versus just model scaling effects.

Tom: The paper title, "Rethinking Relative Policy Differences Transfer of On-Policy Distillation," suggests a fundamental rethinking of the supervision signal itself.

Jane: It’s a shift from matching an absolute final output to understanding the direction of improvement between policies at different stages.

Conclusion: Tom: So we’ve been looking at this paper called "Rethinking Relative Policy Differences Transfer of On-Policy Distillation." It’s about how we can transfer skills between different AI models without needing a super strong teacher for every single task.

Jane: Right, so instead of just matching the final output exactly, they're focusing on the difference between policies at different training stages and using that as a signal.

Lu: The authors are proposing this new way to define an implicit reward using the log-probability ratio between a proxy expert and a proxy reference policy.

Meng: It seems like they’ve unified a bunch of different transfer methods into one framework called A2-OPD, which decouples the difference definition from the target model itself.

Tom: Exactly, and it recovers conventional on-policy distillation as just one specific case within this bigger policy optimization view.

Jane: What does this mean for us? It suggests we can move away from needing that perfect teacher model to get useful results when transferring knowledge across models with different training setups or scales.

Lu: The main finding is that the relative policy difference itself isn't the whole story; it has to induce an update in the target model that it can actually follow and generalize from.

Tom: That’s a key caveat, because raw policy distance doesn't predict whether two models will work together well for distillation.

Jane: And they show that an appropriate reference model can actually help rescue transfers that would otherwise fail, which is interesting because you usually expect a good teacher to make everything work.

Lu: The paper also shows that the expert sets the maximum capability available, and the reference acts like a filter to isolate just the residual skill we want to pass along.

Meng: From an engineering standpoint, this suggests we need to be very careful about selecting our proxy pairs and references because a bad reference can actually make a good transfer setup fail.

Tom: It really shifts the focus from just matching distributions to understanding the direction of improvement between policies at different learning checkpoints.

Rong Wu, Daocheng Fu, Yu Yang, Xuemeng Yang, Siqi Li, Shu Zou, Yao Zhu, Botian Shi, Bo Zhang

Zhejiang University · Shanghai Artificial Intelligence Laboratory · Fudan University · The Australian National University · Renmin University of China

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

Code: https://github.com/KnowledgeXLab/A2-OPD

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization, which enables any-to-any transfer across models that

Key concepts

Log-Probability Ratio: This ratio between a proxy expert and a proxy reference defines an implicit reward signal for policy optimization. It quantifies the relative difference in how likely two policies are to generate specific outputs, allowing the system to learn from this difference rather than just matching an absolute distribution.
On-Policy Distillation (OPD): Conventional OPD trains a target policy to exactly match a fixed teacher policy by minimizing the reverse KL divergence. The new formulation shows that this is equivalent to maximizing a JOPD objective, where the relative log-difference acts as an implicit reward driving the update.
Proxy Pair: A proxy pair consists of an expert and a reference policy derived from solving a specific KL-regularized optimization problem. This pair defines the relative policy difference ($\Delta\phi(s, a)$), which is crucial because this difference is defined solely by the proxy pair and not dependent on the final target model.
Any-to-Any Transfer: This capability means that a policy difference learned between any two models (the expert and reference) can be successfully transferred to any other target model, regardless of its scale or training stage. Success depends on the induced update moving the target toward the expert and generalizing beyond sampled rollouts.

Terminology

Distillation — 1× in this paper · explained in 2 episodes
On-Policy Distillation — 1× in this paper · explained in 3 episodes

Summary

The gist The log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization, which enables any-to-any transfer across models that differ in scale, training stage or generation.

How it works

On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert’s absolute distribution The paper develops a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation.

Conventional OPD trains a target policy to match a fixed teacher policy by minimizing the reverse KL objective This objective is equivalent to maximizing JOPD(θ) which gives a KL-regularized policy optimization view where the relative log-difference log πT − log πref acts as a reward-like signal that drives the update, while the KL term anchors the policy to πref.

The paper formalizes this by considering a proxy pair (π+ϕ, πϕ) where the proxy expert is obtained from the proxy reference by solving a one-step KL-regularized optimization problem with token-level task reward r∗(s, a). The optimal solution to this problem yields the relative policy difference ∆ϕ(s, a) = 1/τ r∗(s, a) − log Zϕ(s). This difference is defined entirely by the proxy pair and does not depend on the target reference.

Key Findings on Policy Differences

The systematic experiments yield three main findings regarding when relative policy differences provide effective supervision. First, a stronger expert is not necessarily a better source of supervision. Improvements acquired through task-directed training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts.

Characterizing Transfer Success

The paper characterizes when policy differences are useful and transferable by asking three specific questions: (i) Which policy differences provide useful supervision? (ii) Which capabilities of the expert are transferred, and what role does the reference play? (iii) Which target models can effectively realize a useful policy difference? The results reveal that transfer succeeds when the induced update moves the target toward the expert and generalizes beyond sampled rollouts. Raw policy distance does not predict this compatibility; critically, an appropriate reference can rescue otherwise unsuccessful transfer.

Role of Expert and Reference

The expert sets how much capability is available, as moving the expert from the early to the regular and late RL checkpoints raises GP from 22.09 to 26.89 and 32.28, and GT tracks this trend. The reference isolates the residual capability for transfer by acting as a filter that removes the capability already shared with the expert and leaves only the residual for transfer. Token-level overlap reveals this filtering effect, showing that a reference that shares more high-probability tokens with an expert removes more of the expert’s policy change, leaving a smaller residual for transfer.

Target Compatibility and Generalization

Transfer depends on target compatibility: a useful difference must induce an update that the target can follow and generalize from, and an appropriate reference can recover some otherwise unsuccessful transfers. The outcome is not a function of policy distance, as the performance realization ratio and transferred math accuracy do not show a monotone relationship with the proxy–target base KL. Matching the reference generation restores both overlap with the expert and validation accuracy, while cross-generation references make both collapse.

Generalizability Across Domains

Transfer patterns persist across tasks: configurations that succeed or fail on math show the same qualitative outcomes on code. Transfer between Qwen3 models is positive on both math (+26.8 and +25.9) and code (+2.3 and +1.9), while transferring a Qwen3-derived difference to a Qwen2.5 target fails in both domains, producing gains of −1.7 and −6.4, respectively. The success and failure patterns persist across domains.

Conclusion

The work unifies various relative policy update transfer methods under a broader policy-optimization view, leading to A2-OPD, an any-to-any formulation that decouples the proxy pair defining the relative difference from the target receiving it. The findings clarify when any-to-any relative policy difference transfer provides effective supervision and offer practical guidance for selecting proxy pairs and targets.

How it works

The formulation leads to A2-OPD, an any-to-any formulation that decouples the proxy pair defining the relative difference from the target receiving it, with conventional OPD recovered as a special case. The paper unifies these methods under a broader policy-optimization view.

Key Findings on Policy Differences

Policy differences induced by task-directed training, including RL and SFT, transfer consistently, whereas differences due to model scaling alone are often ineffective. The paper also finds that the proxy expert supplies the capability to be transferred, while the proxy reference determines the residual presented to the target.

Characterizing Transfer Success

Transfer depends on target compatibility: a useful difference must induce an update that the target can follow and generalize from, and an appropriate reference can recover some otherwise unsuccessful transfers. The results reveal that transfer succeeds when the induced update moves the target toward the expert and generalizes beyond sampled rollouts.

Role of Expert and Reference

Target Compatibility and Generalization

Generalizability Across Domains

Improvements for AI systems

Bold header: Task-Directed Training Transfer

The system can effectively transfer policy improvements acquired through task-specific training, such as reinforcement learning or supervised fine-tuning, which transfer consistently. This means that improvements from optimizing toward an explicit or implicit task reward are reliably transferred to the target model, regardless of the absolute capability difference between the expert and reference.

Bold header: Complementary Role of Expert and Reference

The AI system can leverage a proxy pair where the expert provides the capability, while the reference filters out behaviors already shared by the two models. This mechanism allows for isolating the relevant residual for transfer, ensuring that only task-relevant updates are transferred rather than relying on raw policy distance.

Bold header: Target Compatibility Assessment

The system can determine if a useful policy difference is transferable by checking whether the induced update moves the target toward the expert and generalize beyond sampled rollouts. This prevents applying differences where they fail, as it recognizes that Raw policy distance does not predict this compatibility.