When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs
cs.AI, cs.LG
Submitted: 2026-05-22
Updated: 2026-11-02
Code: https://github.com/XHMY/marl-llm-workflows
License: http://creativecommons.org/licenses/by/4.0/
The gist: Outcome-reward RL can train every role of a multi-agent LLM workflow.
Terminology
Abstract
Outcome-reward RL can train every role of a multi-agent LLM workflow. Its benefit is hard to predict because the reward assigns no credit to individual roles and a shared policy trades pooled data against specialization. We ask when this training improves a workflow over its base model and single-agent RL, and what each role learns. We train three workflows (Eval-Opt, Voting, Orch-Workers) with GRPO on math and code at three Qwen3 scales, with one policy for all roles (Shared-Policy, SP) or one per role type (Isolated-Policy, IP). Every workflow gains by adapting to its structure; this does not make a role better than training it alone. Single-agent RL in the same structure matches or beats trained Voting and Orch-Workers, and within the grid only Eval-Opt under IP exceeds it. Eval-Opt's evaluator learns when to retry and its generator learns to complete the retried attempts, a joint strategy that a single policy trained alone does not acquire. Under SP, per-output token averaging sets each role's gradient share, a hidden hyperparameter absent from single-agent RL, and the solving roles learn slowly; reweighting it gives SP the speed and peak of IP, and its decline. Under IP, a parallel role, whose instances share one adapter, takes one optimizer step per instance per training step and can drift after the peak as its entropy rises and accuracy declines. Merging those steps delays the decline. Our code is at: https://github.com/XHMY/marl-llm-workflows
Sources
- End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- MARFT: Multi-Agent Reinforcement Fine-Tuning
- LLM Collaboration With Multi-Agent Reinforcement Learning
- The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
- Qwen3 Technical Report
- Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning
- Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data
- Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents