EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

arXiv:2607.11012 · cs.CL · Submitted 2026-07-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models".

Title and authors: Tom: So we started by looking at the title and authors of this paper called "EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models". The authors are Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, and Gengsheng Li.

Jane: They are trying to make on-policy distillation methods much more usable by creating a framework that handles the differences between those methods cleanly. It’s focused on making it easy for people to actually implement these complex ideas without getting lost in setup details.

Lu: The real implication here is that they are moving away from having every method require custom modifications to a single system. Instead, they've created this layered architecture where the user only touches the configuration layer, and the specific distillation logic stays localized.

Meng: That localization sounds smart because when we move models to production, we hate hidden dependencies. If I can just swap out one YAML file for a different supervision technique without touching the entire training pipeline, that’s a big win for maintainability at scale.

Lalam: It seems like they are solving the problem where existing on-policy distillation methods have very different ways of handling supervision, tokenizer compatibility, and how you access the teacher model during training. EASYOPD aims to standardize those interfaces across all those different needs.

The paper's summary: Tom: In terms of what they’re actually doing, the paper says that conventional language model distillation often uses fixed teacher-generated data, which isn't always good because it doesn't match the specific states a student policy encounters during training.

Jane: So they are shifting the focus to on-policy distillation, which means instead of relying only on fixed sequences from a teacher, you collect supervision directly from outputs generated by the student policy itself. This happens during training when the student generates its own responses or trajectories.

Lu: The paper points out that existing on-policy distillation methods have significant differences in how they handle things like the form of supervision, whether they can work with different tokenizers, and who has access to the teacher model during training. That fragmentation makes it really hard to reproduce their results or extend them easily.

Meng: So what does this mean for us? It means we stop having to write custom code for every single distillation technique we want to try. We use one shared engine, verl, and then just register our method's specific rules through hooks.

Lalam: They introduce EASYOPD as a three-layer framework: the user layer for configuration, the EasyOPD layer that holds the method-specific logic, and then verl which is just the execution substrate. This separation is key to achieving that unified system interface they promised.

The paper's improvements: Tom: The main improvement they suggest is this three-layer structure designed to separate configuration, local logic, and execution. They show how method modules connect to this shared backend through specific extension points for things like loss construction and reward processing.

Jane: This design turns what used to be a framework-wide change into something that can happen locally within each method module. So instead of modifying the whole system for a new distillation objective, you just extend that one module.

Lu: They emphasize that this allows for different supervision interfaces, like token-level objectives or rubric-based feedback, to all coexist because the method modules define exactly how they get and use that feedback independently.

Meng: I like the idea of method-local diagnostics too. If a training run fails, you don't just see a generic error; you get supervision-specific diagnostics that tell you exactly where the failure happened in your distillation process. That helps debug much faster than just looking at the final task performance score alone.

Lalam: They also reuse verl as the execution substrate, which is a big deal because it means they inherit all of verl's capabilities for rollout generation, worker orchestration, and distributed training without having to reimplement all that distributed infrastructure themselves.

Conclusion: Tom: So wrapping up the discussion on "EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models", the core message is that they’ve successfully integrated on-policy distillation methods onto a shared verl backend while keeping their specific objectives and task performance profiles intact.

Jane: They show that even though these methods have different supervision interfaces, they can all share the same execution substrate, which makes them much more reproducible and easier to deploy across different research goals.

Lu: The framework demonstrates that you can keep your method logic local while letting verl remain the stable execution substrate for distributed training, which is a solid design choice for keeping things maintainable.

Meng: For production, this means we get flexibility without sacrificing stability. We can adapt our distillation strategy quickly by changing configuration files instead of rewriting core infrastructure code every time we want to test a new approach.

Lalam: It’s about having a method-oriented framework that separates user configuration, method-local supervision logic, and distributed execution through those thin dispatch hooks so different methods can share the backend while keeping their own objectives and diagnostics separate.

Tom: That's it for today on EASYOPD. It really shows how to manage complexity in these kinds of advanced distillation tasks by making the user configuration separate from the execution substrate.

Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Gengsheng Li, Zhepei Hong, Chang Wu, Pengfei Liu

University of Science and Technology of China Department of LLM Research Institute National University of Singapore

cs.CL

Submitted: 2026-07-13

Updated: 2026-11-18

Comments: 10 pages, 2 figures

Code: https://github.com/lds-ustc/EasyOPD

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: EasyOPD presents an on-policy distillation framework built on verl that separates user-side configuration, method-specific supervision logic…

Key concepts

Three-layer architecture
The framework is structured in three layers: a user layer for configuration, an EASYOPD layer that holds method-specific logic, and verl for distributed execution. This separation keeps the system organized, allowing users to define how their specific distillation method works without altering the core infrastructure.
Method-local supervision logic
This allows each distillation method to define its own unique way of getting, transforming, consuming, and diagnosing supervision signals. Instead of a single loss function for everyone, methods can use different interfaces like token-level objectives or rubric feedback.
verl execution substrate
EasyOPD reuses verl as the execution engine for distributed training. Method logic connects to this shared backend through thin hooks, meaning the core distributed training infrastructure remains stable and reused across different distillation techniques.

Terminology

Summary

EasyOPD presents an on-policy distillation framework built on verl that separates user-side configuration, method-specific supervision logic, and verl execution The authors introduce EASYOPD to address the fragmentation in existing on-policy distillation methods by providing a unified system interface for heterogeneous supervision interfaces while reusing a common distributed backend

Framework Architecture

EASYOPD adopts a three-layer architecture where the user layer provides configuration, the EASYOPD layer encapsulates method-local OPD logic, and verl provides distributed execution The method modules connect to the shared backend through explicit extension points for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation This design turns OPD integration from framework-wide modification into method-local extension

Unified Workflow and Invocation

The unified OPD workflow decomposes an OPD run into four reusable stages: method invocation, configuration resolution, supervision construction, and distributed execution EasyOPD makes this shared process explicit and standardizes invocation and dispatch without requiring all methods to share a single loss or teacher interface Declarative invocation allows users to select methods through method-specific YAML configurations without manually editing rollout workers or trainer-side components

Method-Local Supervision Logic

The core of EasyOPD is localizing OPD-specific supervision logic, allowing each method to define how supervision is obtained, transformed, consumed, and diagnosed This allows for different interfaces such as token-level distillation objectives or rubric-based feedback where methods localize feedback acquisition and training-signal construction Furthermore, implemented method modules expose supervision-specific diagnostics in addition to task performance, helping users debug failures that final task performance alone cannot explain

Execution Substrate and Hooks

EasyOPD reuses verl as the execution substrate instead of reimplementing distributed training, inheriting rollout generation, worker orchestration, optimization, and distributed execution from verl Method-local supervision logic connects to this substrate through a small set of thin hooks that serve as dispatch boundaries for loss hooks, rollout hooks for trajectory metadata, reward hooks for judge or rubric feedback, alignment hooks for token or span-level mapping, and teacher-sidecar hooks This design keeps method logic local while verl remains the shared backend

Experimental Results

Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectives and task-dependent performance profiles Cross-tokenizer OPD methods improved the unweighted average over the base student, with SimCT obtaining the highest average (53.5, +3.3 over the base) On-policy self-distillation showed that SDPO’s gains are largest on Chemistry and Biology (+19. p. 52 points at one hour, +6. p. 57 and +6.0 at five hours), while GRPO is stronger on GSM8K and on the five-hour Tool Use result Step-wise OPD showed that response-level OPD exceeds GRPO on all three benchmarks, and SOD obtained the highest observed score (39. p. 595 +5.1 over response-level OPD)

Conclusion

EASYOPD is a method-oriented framework that integrates on-policy distillation methods on a shared verl backend, separating user-side configuration, method-local supervision logic, and distributed execution through thin dispatch hooks Across cross-tokenizer OPD, on-policy selfdistillation, and step-wise OPD, methods with different supervision interfaces share this backend while keeping their own objectives and diagnostics Future work will broaden method coverage and larger-model evaluation

References

Microsoft: Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen et al. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-LoRAs. p. 6 arXiv preprint arXiv:2503. p. 601743 Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk et al. p. 6 2024. On-policy distillation of language models: Learning from self-generated mistakes. p. 6 In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Vienna, Austria Abdolmaged Alkhulaifi, Fahad Alsahli, and Irfan Ahmad. p. 6 2021. Knowledge distillation in deep learning and its applications. p. 6 PeerJ Computer Science, 7:e474 Jacob Austin, Augustus Odena et al. p. 6 2021. Program synthesis with large language models. p. 6 arXiv preprint arXiv:2108. p. 607732 Nicolas Boizard, Kevin El Haddad et al. p. 6 2025. Towards cross-tokenizer distillation: The universal logit distillation loss for LLMs. p. 6 Transactions on Machine Learning Research (TMLR) Karl Cobbe, Vineet Kosaraju et al. p. 6 2021. Training verifiers to solve math word problems. p. 6 arXiv preprint arXiv:2110. p. 614168 Kehua Feng, Keyan Ding et al. p. 6 2024. SciKnowEval: Evaluating multi-level scientific knowledge of large language models. p. 6 arXiv preprint arXiv:2406. p. 609098 Yukang Feng, Jianwen Sun et al. p. 6 2026a. Longclibench: A preliminary benchmark and study for longhorizon agentic programming in command-line interfaces. p. 6 arXiv preprint arXiv:2602. p. 614337 Dayuan Fu, Shenyu Wu et al. p. 6 2026b. Argo: Asynchronous rollout with human guidance for research agent optimization. p. 6 OpenReview preprint Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. p. 6 2024. MiniLLM: Knowledge distillation of large language models. p. 6 In The Twelfth International Conference on Learning Representations Geoffrey E. p. 6 Hinton, Oriol Vinyals, and Jeffrey Dean. p. 6 2015. Distilling the knowledge in a neural network. p. 6 arXiv preprint arXiv:1503. p. 602531 Jonas Hübotter et al. p. 6 2026. Reinforcement learning via self-distillation. p. 6 arXiv preprint arXiv:2601. p. 6 p. 6 p. 620802 Naman Jain, King Han et al. p. 6 2025. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. p. 6 In Proceedings of the 13th International Conference on Learning Representations (ICLR 2025) Yoon Kim and Alexander M. p. 6 Rush. 2016. Sequencelevel knowledge distillation. p. 6 In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas Jongwoo Ko, Sungnyun Kim et al. p. 6 2024. DistiLLM: Towards streamlined distillation for large language models. p. 6 In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235, Vienna, Austria Gengsheng Li, Mao Zheng et al. p. 6 2026a. On-policy distillation with curriculum turn-level guidance for multi-turn agents. p. 6 arXiv preprint arXiv:2606. p. 615912 Keyu Li, Junhao Shi et al. p. 6 2026b. Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token realworld contexts. p. 6 arXiv preprint arXiv:2601.11044 Hunter Lightman, Vineet Kosaraju et al. p. 6 2024. Let’s verify step by step. p. 6 In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024) Alexander Lin, Jeremy Wohlwend et al. p. 6 2020. Autoregressive knowledge distillation through imitation learning. p. 7 In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 6121–6133, Online Event Kevin Lu and Thinking Machines Lab. p. 7 2025. On-policy distillation. p. 0 Thinking Machines Lab: Connectionism Mathematical Association of America. p. 7 2026. MAA invitational competitions: American invitational mathematics examination (AIME). p. 7 https://maa. p. 7org/maa-invitational-competitions/. p. 7 Accessed: 2026-07-01 Benjamin Minixhofer, Ivan Vulic, and Edoardo Maria Ponti. p. 7 2025. Universal cross-tokenizer distillation via approximate likelihood matching. p. 7 In Advances in Neural Information Processing Systems 38 (NeurIPS 2025) Qwen Team. p. 7 2025. Qwen3 technical report. p. 7 arXiv preprint arXiv:2505. p. 709388 Zhihong Shao, Peiyi Wang et al. p. 7 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. p. 7 arXiv preprint arXiv:2402. p. 703300 Guangming Sheng, Chi Zhang et al. p. 7 2024. Hybridflow: A flexible and efficient rlhf framework. p. 7 arXiv preprint arXiv: 2409. p. 719256 Mingyang Song and Mao Zheng. p. 6 2026. A survey of onpolicy distillation for large language models. p. 7 arXiv preprint arXiv:2604. p. 700626 Jie Sun, Mao Zheng et al. p. 6 2026. Simct: Recovering lost supervision for cross-tokenizer on-policy distillation. p. 7 arXiv preprint arXiv:2605. p. 7 p. 707711 Qiaoyu Tang, Ziliang Deng et al. p. 7 2023. ToolAlpaca: Generalized tool learning for language models with 3000 simulated cases. p. 7 arXiv preprint arXiv:2306. p. 705301 Leandro von Werra, Younes Belkada et al. p. 7 2020. TRL: Transformers reinforcement learning. p. 7 https://github. p. 0 p. 0com/huggingface/trl Yunze Wu, Dayuan Fu et al. p. 7 2025. Innovatorbench: Evaluating agents’ ability to conduct innovative LLM research. p. 7 arXiv preprint arXiv:2510. p. 727598 Yang Xiao, Mohan Jiang et al. p. 6 2025. Limi: Less is more for agency. p. 7 arXiv preprint arXiv:2509. p. 717567 An Yang, Baosong Yang et al. p. 7 2024a. Qwen2.5 technical report. p. 6 arXiv preprint arXiv:2412. p. 715115 Chuanpeng Yang et al. p. 7 2024b. Survey on knowledge distillation for large language models: Methods, evaluation, and application. p. 7 arXiv preprint arXiv:2407. p. 701885 Ji Zeng, Dayuan Fu et al. p. 6 2026. davinci-dev: Agentnative mid-training for software engineering. p. 7 arXiv preprint arXiv:2601.18418 Songming Zhang, Xue Zhang et al. p. 7 2024. Dual-space knowledge distillation for large language models. p. 7 In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 18164–18181, Miami, Florida, USA Songming Zhang et al. p. 7 2026. KDFlow: A userfriendly and efficient knowledge distillation framework for large language models. p. 7 arXiv preprint arXiv:2603. p. 601875 Qiyong Zhong, Mao Zheng et al. p. 6 2026. Sod: Step-wise on-policy distillation for small language model agents. p. 7 arXiv preprint arXiv:2605.07725 Zilin Zhu, Chengxing Xie et al. p. 7 2025. slime: An llm post-training framework for rl scaling. p. 7 https://github.com/THUDM/slime GitHub repository Corresponding author: Xin Lv

A Framework Comparison Details

Table 5 provides the framework-specific evidence underlying the ratings in Table 1 The comparison reflects publicly documented functionality as of July 2026, using the versions or commits listed in Appendix A, where per-cell evidence is also provided EasyOPD builds on verl 0.5.0, and we compare against TRL 0. p. 9 p. 927.0, the slime repository (accessed 2025-06-19), and the KDFlow implementation (Zhang et al. p. 9 p. 9, 2026) Table 6 lists the released methods used in this work and the dispatch hooks each one registers Every method generates on-policy rollouts through verl’s shared rollout engine; the hooks in the table are the additional method-local extension points a method registers on top of this shared pipeline Cross-tokenizer alignment in SimCT reuses the shared simple teacher sidecar and is applied inside its loss path The rollout-metadata hook attaches per-step or trajectory annotations and is distinct from rollout generation itself

Experimental Setup Details

All experiments run on a node with 8× NVIDIA H20 96GB GPUs (driver 535.161.08, CUDA 12. p. 84) under Ubuntu 22. p. 804, using the released OpenAgentRL conda environment Cross-tokenizer OPD used GRPO with a cross-tokenizer KL reward and singlesample rollouts at temperature 0. p. 86 On-policy self-distillation used top-k logit distillation with k = 100 and a symmetric KL interpolation α = 0. p. 85 Step-wise OPD used a GRPO-optimized Qwen3-4B as the teacher with SOD applying step-wise adaptive weighting All methods are implemented in EASYOPD and run on the same verlbased infrastructure (Sheng et al. p. 8, 2024)

Overall Findings

Across the three case studies, EASYOPD runs the evaluated methods on the same verl backend while preserving their distinct supervision requirements Results should be read within each setting rather than as a cross-regime ranking The framework successfully demonstrates that different OPD settings can share the same execution substrate while retaining their original supervision designs Future work will broaden method coverage and larger-model evaluation

Table 5

Table 5 provides the framework-specific evidence supporting the ratings in Table 1 Ratings reflect publicly documented functionality of the following versions as of July 2026, using the versions or commits listed in Appendix A, where per-cell evidence is also provided EasyOPD builds on verl 0.5.0, and we compare against TRL 0.27.0, the slime repository (accessed 2025-06-19), and the KDFlow implementation (Zhang et al., 2026) Table 6 lists the released methods used in this work and the dispatch hooks each one registers Every method generates on-policy rollouts through verl’s shared rollout engine; the hooks in the table are the additional method-local extension points a method registers on top of this shared pipeline Cross-tokenizer alignment in SimCT reuses the shared simple teacher sidecar and is applied inside its loss path Method Loss GKD × × Logit SimCT × Cross-tok. p. 8 SDPO × Self-distill. p. 9 p. 9 SOD × Step-wise

Table 6

Table 6 lists the released methods used in this work and the dispatch hooks each one registers Every method generates on-policy rollouts through verl’s shared rollout engine; the hooks in the table are the additional method-local extension points a method registers on top of this shared pipeline Cross-tokenizer alignment in SimCT reuses the shared simple teacher sidecar and is applied inside its loss path Method Loss GKD × × Logit SimCT × Cross-tok. SDPO × Self-distill.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified OPD workflow allows for a single entry point to run three different OPD settings—cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD—by changing only the method name and its YAML configuration. This enables users to switch between distinct supervision interfaces without manually editing rollout workers or loss assembly.

  2. Method local logic allows each OPD method to define how supervision is obtained, transformed, consumed, and diagnosed independently of a single loss-only abstraction. For example, the cross-tokenizer implementation reports metrics like overlap-vocabulary size and valid-span ratio, helping users debug failures that final task performance alone cannot explain.

  3. Execution through thin verl hooks ensures that EasyOPD reuses verl for rollout generation, model execution, and optimization while connecting method local supervision logic via a small set of dispatch boundaries. This design balances expressiveness and maintainability, allowing method logic to remain local while keeping verl as the stable execution substrate.

Sources

Related papers