EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models
summary
The gist
EasyOPD presents an on-policy distillation framework built on verl that separates user-side configuration, method-specific supervision logic…
In short
EasyOPD is a framework that unifies different on-policy distillation methods for large language models. It achieves this by separating user configuration, method-specific supervision logic, and distributed execution. This allows various methods to share a common verl backend while maintaining their unique training objectives and diagnostics.
Key concepts
- Three-layer architecture
- The framework is structured in three layers: a user layer for configuration, an EASYOPD layer that holds method-specific logic, and verl for distributed execution. This separation keeps the system organized, allowing users to define how their specific distillation method works without altering the core infrastructure.
- Method-local supervision logic
- This allows each distillation method to define its own unique way of getting, transforming, consuming, and diagnosing supervision signals. Instead of a single loss function for everyone, methods can use different interfaces like token-level objectives or rubric feedback.
- verl execution substrate
- EasyOPD reuses verl as the execution engine for distributed training. Method logic connects to this shared backend through thin hooks, meaning the core distributed training infrastructure remains stable and reused across different distillation techniques.
Terminology used across episodes
This episode discusses
- EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models · Paper Radio
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Program Synthesis with Large Language Models
- Training Verifiers to Solve Math Word Problems
- SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
- LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
- daVinci-Env: Open SWE Environment Synthesis at Scale · Paper Radio
- Distilling the Knowledge in a Neural Network
- Reinforcement Learning via Self-Distillation
- On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents · Paper Radio
- AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
- Qwen3 Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- HybridFlow: A Flexible and Efficient RLHF Framework
- A Survey of On-Policy Distillation for Large Language Models · Paper Radio
- SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation · Paper Radio
- ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
- InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research
- LIMI: Less is More for Agency
- Qwen2.5 Technical Report
- Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
The paper
EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models · Read on arXiv
Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Gengsheng Li, Zhepei Hong, Chang Wu, Pengfei Liu
University of Science and Technology of China Department of LLM Research Institute National University of Singapore
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models".
Title and authors: Tom: So we started by looking at the title and authors of this paper called "EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models". The authors are Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, and Gengsheng Li.
Jane: They are trying to make on-policy distillation methods much more usable by creating a framework that handles the differences between those methods cleanly. It’s focused on making it easy for people to actually implement these complex ideas without getting lost in setup details.
Lu: The real implication here is that they are moving away from having every method require custom modifications to a single system. Instead, they've created this layered architecture where the user only touches the configuration layer, and the specific distillation logic stays localized.
Meng: That localization sounds smart because when we move models to production, we hate hidden dependencies. If I can just swap out one YAML file for a different supervision technique without touching the entire training pipeline, that’s a big win for maintainability at scale.
Lalam: It seems like they are solving the problem where existing on-policy distillation methods have very different ways of handling supervision, tokenizer compatibility, and how you access the teacher model during training. EASYOPD aims to standardize those interfaces across all those different needs.
The paper's summary: Tom: In terms of what they’re actually doing, the paper says that conventional language model distillation often uses fixed teacher-generated data, which isn't always good because it doesn't match the specific states a student policy encounters during training.
Jane: So they are shifting the focus to on-policy distillation, which means instead of relying only on fixed sequences from a teacher, you collect supervision directly from outputs generated by the student policy itself. This happens during training when the student generates its own responses or trajectories.
Lu: The paper points out that existing on-policy distillation methods have significant differences in how they handle things like the form of supervision, whether they can work with different tokenizers, and who has access to the teacher model during training. That fragmentation makes it really hard to reproduce their results or extend them easily.
Meng: So what does this mean for us? It means we stop having to write custom code for every single distillation technique we want to try. We use one shared engine, verl, and then just register our method's specific rules through hooks.
Lalam: They introduce EASYOPD as a three-layer framework: the user layer for configuration, the EasyOPD layer that holds the method-specific logic, and then verl which is just the execution substrate. This separation is key to achieving that unified system interface they promised.
The paper's improvements: Tom: The main improvement they suggest is this three-layer structure designed to separate configuration, local logic, and execution. They show how method modules connect to this shared backend through specific extension points for things like loss construction and reward processing.
Jane: This design turns what used to be a framework-wide change into something that can happen locally within each method module. So instead of modifying the whole system for a new distillation objective, you just extend that one module.
Lu: They emphasize that this allows for different supervision interfaces, like token-level objectives or rubric-based feedback, to all coexist because the method modules define exactly how they get and use that feedback independently.
Meng: I like the idea of method-local diagnostics too. If a training run fails, you don't just see a generic error; you get supervision-specific diagnostics that tell you exactly where the failure happened in your distillation process. That helps debug much faster than just looking at the final task performance score alone.
Lalam: They also reuse verl as the execution substrate, which is a big deal because it means they inherit all of verl's capabilities for rollout generation, worker orchestration, and distributed training without having to reimplement all that distributed infrastructure themselves.
Conclusion: Tom: So wrapping up the discussion on "EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models", the core message is that they’ve successfully integrated on-policy distillation methods onto a shared verl backend while keeping their specific objectives and task performance profiles intact.
Jane: They show that even though these methods have different supervision interfaces, they can all share the same execution substrate, which makes them much more reproducible and easier to deploy across different research goals.
Lu: The framework demonstrates that you can keep your method logic local while letting verl remain the stable execution substrate for distributed training, which is a solid design choice for keeping things maintainable.
Meng: For production, this means we get flexibility without sacrificing stability. We can adapt our distillation strategy quickly by changing configuration files instead of rewriting core infrastructure code every time we want to test a new approach.
Lalam: It’s about having a method-oriented framework that separates user configuration, method-local supervision logic, and distributed execution through those thin dispatch hooks so different methods can share the backend while keeping their own objectives and diagnostics separate.
Tom: That's it for today on EASYOPD. It really shows how to manage complexity in these kinds of advanced distillation tasks by making the user configuration separate from the execution substrate.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI