SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillation
cs.AI
Submitted: 2026-09-29
Updated: 2026-09-29
Code: https://github.com/Miteto-sudo/SAKI
License: http://creativecommons.org/licenses/by/4.0/
The gist: On-policy distillation (OPD) reduces train-test state mismatch by training a student on its own generated trajectories, but weak students may visit teacher-misaligned prefixes where supervision is
Terminology
Abstract
On-policy distillation (OPD) reduces train-test state mismatch by training a student on its own generated trajectories, but weak students may visit teacher-misaligned prefixes where supervision is less representative. We introduce SAKI (Supervision Allocation with KL-constrained Interpolation), which combines a KL-constrained teacher-guided rollout with maximal coupling and reuses realized accept/correction events to route token-level supervision. Accepted positions retain sampled-token reverse-KL supervision, while correction positions receive direct supervision on the teacher's highest-probability token. Under maximal coupling, the correction probability is exactly TV(p t, q t), so the same trust-region radius controls rollout deviation and upper-bounds intervention and specialized-supervision frequency. We further implement an engine-resident speculative verifier that preserves the exact-q trajectory distribution and coupling semantics while improving matched-workload rollout throughput by 4.22x. Across seven mathematical reasoning benchmarks, SAKI improves the matched teacher-guided baseline in Mean@8 and Pass@8 for both 1.7B and 0.6B students. Placement controls and fixed-prefix analysis further support correction-triggered routing as a conflict-adaptive supervision signal.
Sources
- Accelerating Large Language Model Decoding with Speculative Sampling
- Weak-Driven Learning: How Weak Agents make Strong Agents Stronger
- Measuring Mathematical Problem Solving With the MATH Dataset
- Distilling the Knowledge in a Neural Network
- ZipRL: Adaptive Multi-Turn Context Compression with Hindsight Response Replay
- Draft-OPD: On-Policy Distillation for Speculative Draft Models
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Trust-Region Behavior Blending for On-Policy Distillation
- When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards
- AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
- Trust Region On-Policy Distillation
- Qwen3 Technical Report
- Your Group-Relative Advantage Is Biased
- Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems