Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
Ved Sriraman, Peihan Liu, Daniel Hsu, Adam Block
cs.LG, cs.AI, stat.ML
Submitted: 2026-06-29
Code: https://github.com/plau666/NAIL
License: http://creativecommons.org/licenses/by/4.0/
The gist: Imitation Learning is a natural framework for learning in sequential decision-making systems and has emerged as the dominant paradigm through which we understand language model training.
Terminology
Abstract
Imitation Learning is a natural framework for learning in sequential decision-making systems and has emerged as the dominant paradigm through which we understand language model training. A central puzzle is that, while in theory offline IL can be horizon-free and optimal, in practice online methods such as on-policy distillation often outperform offline methods such as supervised fine-tuning. We propose a noisy expert model to explain this gap, in which the learner only has access to a noisy version of the expert's policy, but wishes to compete against the reward achieved by a clean expert, motivated by the fact that in many applications, e.g. training language models to perform long chains of thought, the expert is often imperfect. In this setting, we show a sharp separation between offline and online IL. Offline learning from noisy trajectories is fundamentally hard: to compete with the clean expert, the sample complexity must grow exponentially, in contradistinction to the clean expert setting where no explicit horizon dependence exists. In contrast, we prove that online interaction with the noisy expert via a novel variant of OPD enables polynomial dependence on the horizon in general. We further show that, under a natural condition on the expert noise distribution, which we show to be necessary for any horizon-free sample complexity, one can obtain such a guarantee, although our proposed algorithm sacrifices statistical efficiency in its dependence on the size of the policy class. Our analysis leads to an alternative loss function that is commonly considered empirically for LM training. We further provide algorithms and lower bounds, and extend our results to the more realistic setting of unknown corruption when the clean expert is deterministic, thereby providing a theoretical foundation for why OPD can outperform SFT when training language models from imperfect teachers.
Sources
- Phi-4 Technical Report
- Qwen Technical Report
- Training Verifiers to Solve Math Word Problems
- The Statistical Complexity of Interactive Decision Making
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- Reinforcement Learning via Self-Distillation
- Gemma 3 Technical Report
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
- 2 OLMo 2 Furious
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- Policy Distillation
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- A Survey of On-Policy Distillation for Large Language Models
- Qwen3 Technical Report
- Self-Distilled RLVR
- Embarrassingly Simple Self-Distillation Improves Code Generation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks