Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Rethinking Black-Box On-Policy Distillation".
Jane: Here's the gist of the paper. On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios It contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10× gain in sample efficiency.
Tom: The ROPD Framework The ROPD framework instantiates black-box OPD by distilling textual teacher responses into structured, prompt-specific rubrics for student reward computation The framework operates in two stages: (1) Rubric Induction, which extracts a common set of criteria from teacher and student responses, and (2) Rubric-based Verification, which evaluates student rollouts against these criteria to compute rewards for on-policy optimization During Rubric Induction, the framework collects a set of teacher responses Y T x = 1 and student rollouts Y S x = 1 sampled from πt and πθ, respectively.
Jane: This is converted into a set of prompt-specific rubrics Cx = Rubricator(x,Y T x, Y S x) = 1 and K=12 criteria.
Tom: How the training signal is built The reward for an i-th student rollout is computed as the weighted pass rate: si = PK k=1 wkvi,k / (PK k=1 wk + ϵ), where vi,k = 1 indicates that y S i satisfies criterion ρk.
Paper summary: Lu: Thinking about the title and what we’ve seen, ROPD is essentially shifting the supervisory signal from raw probabilities to high-level, structured rubrics derived from teacher responses.
Jane: So if you boil it down for someone who doesn't read papers, this paper suggests that instead of trying to perfectly copy every word the teacher says, we can teach a student model by giving it rules about what a good answer looks like, based on comparing teacher and student attempts.
Tom: It’s about using the teacher as a springboard rather than just a mirror for the student's logic. The key finding is that this semantic guidance leads to better performance and much more efficient training compared to methods that rely solely on logit matching >
Meng: From an engineering standpoint, if you can generate these rubrics from text responses, it’s a way to keep the distillation process accessible even when the teacher model isn't open-source or fully controllable >
Lalam: It gives us a way to distill capabilities without needing deep access into the teacher's internal state, which is a big win for proprietary models.
Jane: So what does this mean for how we think about distillation in general? It suggests that the future of on-policy distillation might lean more toward cultivating clearer semantic guidance rather than just chasing dense numerical signals >
Conclusion: Tom: So we've been looking at how we can distill knowledge from large language models when we don't have access to their internal settings, and this paper is titled "Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis."
Jane: That sounds a bit heavy for a title. What does that actually mean for someone listening right now?
Tom: Basically, they’re taking on the old idea of on-policy distillation, which relies on the teacher's internal scores called logits, and they’re rethinking it by using something else: rubrics.
Lu: The authors are looking at how you can create these rubrics from just text—from comparing what a teacher wrote to what a student model generated—and then using those rules to score the student's attempts for training.
Meng: So, instead of trying to match the teacher's exact probability output, you’re building a set of criteria that says "this kind of answer is good" and "that kind of answer is bad."
Jane: That makes sense. So, what's the main result they found? What did they prove about this rubric approach?
Tom: They showed that these text-only rubrics can actually beat the logit-based methods on some math problems, like competition math benchmarks.
Lu: Specifically, it found that their rubric rewards track correctness much better than just looking at the teacher's likelihood score. The difference is significant—we're talking a zero point nine zero versus something near zero for the logit method.
Jane: A huge gap there. So, what does that mean practically for someone who just wants to train a model?
Tom: It means you can get up to ten times more data efficiency when using this rubric approach compared to the older methods. The training gets repaired better, and it regresses less on the things that were already working well.
Meng: Ten times more data is good for production, but I still have concerns about those benchmarks. Are these rubrics robust enough for real-world mistakes?
Lu: They found that this approach is actually more robust to model divergence too. It keeps training stable even if you’re comparing two models with very different ways of reasoning.
Jane: So they're arguing that moving from matching numbers to matching high-level logic rules is the way forward for distillation, even when the teacher is a black box?
Tom: Yeah, it positions rubric-based distillation as a really flexible alternative because you don't need to open up the teacher’s inner workings. Next up, we look at what this actually means for building better AI systems on top of this idea.
Junfeng Fang, Zhepei Hong, Mao Zheng, Mingyang Song, Gengsheng Li, Houcheng Jiang, Dan Zhang, Haiyun Guo, Xiang Wang†,, Tat-Seng Chua
National University of Singapore · University of Science and Technology of China · Tencent
cs.LG, cs.AI
Submitted: 2026-05-08
Updated: 2026-11-10
Comments: Preprint. Code is available at https://github.com/Peregrine123/ROPD_official
Code: https://github.com/Peregrine123/ROPD_official
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 100/100
The gist: On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios It contend that structured semantic…
Key concepts
- On-Policy Distillation (OPD)
- A training method where a student model learns by optimizing its own responses against its current policy. Instead of matching the teacher's internal probabilities (logits), OPD uses feedback—like scores or rubrics—to guide the student to generate better outputs based on what it has already produced.
- Rubric Induction
- The process where the framework automatically creates a set of rules or criteria for judging responses. This is done by contrasting multiple teacher responses with the student's current attempts. The goal is to extract a common, structured set of quality standards that apply to the specific task.
- Group Relative Policy Optimization (GRPO)
- The specific optimization algorithm used in ROPD. It allows the student model to learn from rewards calculated based on how well its rollouts satisfy the induced rubrics compared to other rollouts. This enables optimization at a response level, using weighted pass rates as rewards.
- Semantic Guidance vs. Logit-Matching
- The core shift in distillation strategy. Instead of matching fine-grained token probabilities (logits), ROPD focuses on high-level reasoning principles by distilling behavioral gaps into structured rubrics. This semantic guidance filters out 'stochastic noise' and prioritizes logical fidelity over surface-form mimicry.
Terminology
Summary
On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios It contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10× gain in sample efficiency. p. 0
The ROPD Framework
The ROPD framework instantiates black-box OPD by distilling textual teacher responses into structured, prompt-specific rubrics for student reward computation The framework operates in two stages: (1) Rubric Induction, which extracts a common set of criteria from teacher and student responses, and (2) Rubric-based Verification, which evaluates student rollouts against these criteria to compute rewards for on-policy optimization During Rubric Induction, the framework collects a set of teacher responses Y T x = 1 and student rollouts Y S x = 1 sampled from πt and πθ, respectively. This is converted into a set of prompt-specific rubrics Cx = Rubricator(x,Y T x, Y S x) = 1 and K=12 criteria. p. 2
How the training signal is built
The reward for an i-th student rollout is computed as the weighted pass rate: si = PK k=1 wkvi,k / (PK k=1 wk + ϵ), where vi,k = 1 indicates that y S i satisfies criterion ρk. The framework utilizes Group Relative Policy Optimization (GRPO) to optimize the student from response-level rewards The GRPO reward for rollout y S j is computed as R(y S j) = PK k=1 wk · I[passk] / (PK k=1 wk z) weighted pass rate. p. 2
Where the gains come from
The advantages of the ROPD paradigm extend far beyond its inherent flexibility, such as supporting cross-architecture distillation without tokenizer alignment Conceptually, ROPD functions as a semantic filter: while token-level logits often reflect stochastic phrasing variations that offer negligible value for distillation, ROPD isolates task-level reasoning principles by distilling behavioral gaps into structured rubrics. This shift from logit-matching to semantic guidance yields a profound empirical gain, up to a 10× boost in sample efficiency. p. 1
Performance and Robustness
ROPD consistently ranks first across all 14 benchmark configurations in black-box settings In white-box settings, ROPD remains highly competitive with, and often surpasses, advancing logit-based OPD methods The results demonstrate that for complex reasoning tasks, rubric-based signals can serve as a flexible alternative to logit-based signals. Furthermore, ROPD exhibits superior robustness to model divergence, ensuring stable convergence even across models with markedly disparate reasoning trajectories. p. 1
Analysis of Mechanism
The analysis reveals that rubric rewards achieve 0.90 AUC versus the teacher’s near-random 0. p. 535 for correctness alignment ROPD uses the teacher as a springboard, not a mirror, as its accuracy and rubric rewards scale synchronously while its logit actively declines. p. 6 The transition in checkpoint dynamics shows that ROPD improves more and regresses less in every rubric category compared to LOPD. The analysis confirms that by filtering out the “stochastic noise” of token-level logit distributions, ROPD ensures the optimizer prioritizes logical fidelity over surface-form mimicry. p. 6
Conclusion
In this work, we introduce ROPD, a minimalist yet potent framework for rubric-based OPD By shifting the supervisory signal from probabilities to high-level rubrics, ROPD reconciles competitive performance with accessibility ROPD not only achieves a 10× boost in data utilization efficiency but also exhibits superior robustness across disparate model capabilities. These findings suggest that the future of OPD may lie in the cultivation of clearer semantic guidance rather than solely in the pursuit of denser numerical signals. p. 8 As a versatile and scalable baseline, ROPD paves the way for efficient and interpretable distillation in the era of increasingly opaque, high-performance LLMs. p. 8
References
[1] MAA. Aime 2024: American invitational mathematics examination, 2024
[2] MAA. Aime 2025: American invitational mathematics examination, 2025
[3] HMMT. Hmmt 2025: Harvard-mit mathematics tournament, 2025
[4] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu
[5] An Yang et al. Qwen3 technical report
[6] Rishabh Agarwal et al. On-policy distillation of language models: Learning from self-generated mistakes
[7] Kevin Lu and Thinking Machines Lab. On-policy distillation
[8] Yuxian Gu et al. Minillm: Knowledge distillation of large language models
[9] Bangjun Xiao et al. Mimo-v2-flash technical report
[10] DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence
[11] David Rein et al. Gpqa: A graduate-level google-proof q&a benchmark
[12] Rahul K. Arora et al. p. 9 Healthbench: Evaluating large language models towards improved human health
[13] Jeffrey Zhou et al. Instruction-following evaluation for large language models
[14] Gemma Team, Google DeepMind. Gemma 3 technical report
[15] OpenAI. Introducing gpt-5. p. 92
[16] Yuanda Xu et al. Tip: Token importance in on-policy distillation
[17] Yaxuan Li et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe
[18] Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models
[19] Zhihong Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models
[20] Jing Xiong et al. Ovd: On-policy verbal distillation
[21] Tianzhu Ye et al. Black-box on-policy distillation of large language models
[22] Wenkai Yang et al. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation
[23] Anisha Gunjal et al. Rubrics as rewards: Reinforcement learning beyond verifiable domains
[24] Woogyeol Jin et al. Entropy-aware on-policy distillation of language models
[25] Dongxu Zhang et al. Fast and effective on-policy distillation from reasoning prefixes
[26] Yecheng Wu et al. Lightning opd: Efficient post-training for large reasoning models with offline on-policy distillation
[27] Yuanda Xu et al. Paced: Distillation and onpolicy self-distillation at the frontier of student competence
[28] Binbin Zheng et al. Scope: Signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting
[29] Xue Zhang et al. A dual-space framework for general knowledge distillation of large language models
[30] Yuqian Fu et al. Revisiting on-policy distillation: Empirical failure modes and simple fixes
[31] Aasheesh Singh et al. Orpo-distill: Mixed-policy preference optimization for cross-architecture llm distillation
[32] Yiyang Shen et al. Reinforcement learning-based knowledge distillation with llm-as-a-judge
[33] Tianci Liu et al. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment
[34] Zenan Huang et al. Reinforcement learning with rubric anchors
[35] Rulin Shao et al. Dr tulu: Reinforcement learning with evolving rubrics for deep research
[36] Yifei Xu et al. Sibylsense: Adaptive rubric learning via memory tuning and adversarial probing
[37] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network
[38] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation
Appendix
§A Related Work (Complete Version)........ 14
§B Qualitative Analysis and Case Studies B Qualitative Analysis and Case Studies
§C Hyperparameters and Training Configuration C Hyperparameters and Training Configuration
§D Prompt Templates D Prompt Templates
§E Additional Figures and Analysis E Additional Figures and Analysis
§F Algorithm Pseudocode and Method Details F Algorithm Pseudocode and Method Details
A Related Work (Complete Version)
This section provides the complete Related Work discussion with full context and citations. Knowledge distillation and on-policy distillation. p. 2 Knowledge distillation (KD) transfers the behavior of a large teacher model into a smaller student, and is widely used to adapt or compress language models Classical KD matches teacher soft targets on a fixed data distribution [37], and Sequence-Level Knowledge Distillation (SeqKD) extends this to generation by substituting teacherdecoded sequences for label-level targets Both are offline and suffer from exposure bias: training follows teacher-forced trajectories, while inference exposes the student to its own prefixes and errors, creating a mismatch between the distributions seen at training and test time. On-policy distillation (OPD) addresses this by training on student-generated sequences: MiniLLM optimizes reverse-KL on sampled responses [8], Generalized Knowledge Distillation (GKD) learns from self-generated mistakes with teacher feedback [6], and recent work scales this recipe to reasoning post-training. p. 8 Despite this progress, these methods share a common assumption: they require token-level teacher information such as logits, which is unavailable for proprietary teachers and difficult to align across different architectures or vocabularies. p. 8 ROPD studies the complementary black-box regime where the teacher exposes only text responses, enabling on-policy distillation when token-level supervision is infeasible. p. 8 Black-box On-policy Distillation. p. 2 Recent black-box methods use various response-level signals: ORPO-Distill constructs preference pairs from mixed-policy traces [31]; GAD trains a discriminator for co-evolving rewards; OVD uses discrete verbal trajectory scores, avoiding token alignment and reducing memory cost; and RL-based KD trains from scalar evaluator rewards over unlabeled data. p. 8 ROPD instead makes the distillation interface explicit by deriving shared rubrics from multiple teacher answers and current student rollouts, verifying each rollout against these criteria, and using the resulting weighted pass rates as on-policy rewards. p. 8 Rubric-based Reinforcement Learning. p. 8 Reinforcement learning with verifiable rewards (RLVR) has driven strong gains in math and code [19], but its reliance on binary correctness limits it to domains with deterministic ground truth Rubrics address this by decomposing response quality into structured, multi-dimensional criteria, extending RL to open-ended tasks. p. 8 ROPD instead induces rubrics from the contrast between multi-teacher answers and on-policy student rollouts, converting them via a verifier into weighted passrate rewards for Group Relative Policy Optimization (GRPO). p. 8
Limitation and Future Work
While ROPD demonstrates the efficacy and flexibility of rubric-based rewards for OPD, we identify two primary limitations First, our evaluation mainly focuses on formal reasoning, such as Mathematics, Medicine, and Science Second, ROPD depends on the instruction-following of the Rubricator and Verifier. More broadly, these limitations point to a larger research opportunity regarding how to design, validate, and calibrate such principles for scalable distillation. p. 8
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI systems can achieve up to a 10 times gain in sample efficiency by using rubric-based OPD instead of logit-based OPD, as shown in Figure 1(a). This allows for training with significantly fewer samples while maintaining or exceeding performance on complex reasoning tasks.
-
The improved system can be made robust across different model architectures because ROPD's criterion referenced rubrics provide an absolute supervisory signal that remains informative even for low-quality responses, as demonstrated by the 50% relative improvement in AIME24 performance when substituting a less capable student model.
-
The system gains superior generalization stability because ROPD’s high-level semantic guidance ensures stable convergence even across models with markedly disparate reasoning trajectories, unlike logit-based OPD which typically requires similar reasoning patterns.
-
The improved system can better separate logical fidelity from stylistic mimicry because the rubric decomposes response quality into discrete, verifiable criteria, preventing fabricated derivations from hiding behind well-structured prose by yielding a 3.6 times wider margin of discrimination over scalar judges in case studies.
-
The AI system can prioritize substantive reasoning over superficial mimicry because ROPD’s signal is more effective for complex reasoning tasks;
By filtering out the 'stochastic noise' of token-level logit distributions, ROPD ensures the optimizer prioritizes logical fidelity over surface-form mimicry.
Sources
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Qwen3 Technical Report
- MiMo-V2-Flash Technical Report
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Instruction-Following Evaluation for Large Language Models
- Gemma 3 Technical Report
- TIP: Token Importance in On-Policy Distillation
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- A Survey of On-Policy Distillation for Large Language Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- OVD: On-policy Verbal Distillation
- Black-Box On-Policy Distillation of Large Language Models
- Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- Entropy-Aware On-Policy Distillation of Language Models
- Fast and Effective On-policy Distillation from Reasoning Prefixes
- Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
- PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence
- SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
Related papers
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- (Mis)generalization of Helpful-Only Fine-tuning
- Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine
- Flow-Direct: Feedback-Efficient and Reusable Guidance for Flow Models via Non-Parametric Guidance Field