MAS-OPD: On-Policy Distillation for Multi-agent Systems
cs.CL
Submitted: 2026-09-28
Updated: 2026-09-28
Code: https://github.com/huggingface/Math-Verify
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multi-agent systems (MAS) split a task across specialized roles and are promising on complex tasks, yet a prevailing approach relies on inference-time orchestration alone.
Terminology
Abstract
Multi-agent systems (MAS) split a task across specialized roles and are promising on complex tasks, yet a prevailing approach relies on inference-time orchestration alone. General-purpose APIs are costly and hard to customize, while small models with role prompts rarely develop stable role competence or reliable collaboration, so post-training a MAS jointly is central. Most attempts use reinforcement learning, whose team-level reward leaves undetermined which step of which agent brought about the outcome, while local rewards need redesigning per task. On-policy distillation (OPD) gives token-level teacher supervision on trajectories the student samples, a denser signal needing no local reward, yet is underexplored for the interdependent agents of a MAS. Two difficulties arise: building complementary specialization from a judgement of which role a behavior belongs to while preserving the knowledge all roles need, and turning cross-agent collaborative information into supervision OPD can exploit. We present MAS-OPD, where Role-Advantage Specialization defines the role advantage as the difference between the teacher signals under target and non-target role conditions, and Privileged Attribution for Coordination attributes an interaction conflict to its source and supplies it to the teacher alone as privileged information. Extensive experiments on code and mathematics benchmarks show that MAS-OPD attains the highest mean score at both student scales and leads the agents to develop clearer role specialization and more effective collaborative behavior.
Sources
- Small Language Models are the Future of Agentic AI
- SODA: Semi On-Policy Black-Box Distillation for Large Language Models
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- The False Promise of Imitating Proprietary LLMs
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
- Measuring Coding Challenge Competence With APPS
- Reinforcement Learning via Self-Distillation
- Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- MARFT: Multi-Agent Reinforcement Fine-Tuning
- Generative Agents: Interactive Simulacra of Human Behavior
- Privileged Information Distillation for Language Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Self-Distillation Enables Continual Learning
- A Survey of On-Policy Distillation for Large Language Models
- Qwen3 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering