ASTOR: Multi-Task Code Reinforcement Learning via Utility-Driven Coordination
cs.SE, cs.AI
Submitted: 2026-05-07
Updated: 2026-10-29
Comments: Accepted to NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) with verifiable rewards has proven effective at post-training LLMs for coding…
Terminology
Abstract
Reinforcement learning (RL) with verifiable rewards has proven effective at post-training LLMs for coding, yet deploying separate task-specific specialists incurs costs that scale with the number of tasks, motivating a unified multi-task RL (MTRL) approach. However, existing MTRL methods treat all coding tasks uniformly, relying on fixed data curricula under a shared optimization strategy, ultimately limiting the effectiveness of multi-task training. To address these limitations, we propose ASTOR, a multi-tASk code reinforcement learning framework via uTility-driven coORdination. Centered on task utility, a signal capturing each task learning potential and cross-task synergy, ASTOR comprises two coupled modules: 1) Hierarchical Utility-Routed Data Scheduling module hierarchically allocates training budget and prioritizes informative prompts, steering training toward the most valuable data and 2) Adaptive Utility-Calibrated Policy Optimization module dynamically scales per-task KL regularization, matching update constraints to each tasks current training state. Experiments on two widely-used LLMs across four representative coding tasks demonstrate that ASTOR consistently improves a single model across all tasks, outperforming the best task-specific specialist by 9.0%-9.5% and surpassing the strongest MTRL baseline by 7.5%-12.8%.
Sources
- Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents
- CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
- Kimi k1.5: Scaling Reinforcement Learning with LLMs
- Learning to Generate Unit Test via Adversarial Reinforcement Learning
- Imbalanced Gradients in RL Post-Training of Multi-Task LLMs
- Qwen2.5-Coder Technical Report
- Qwen3 Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
- Competition-Level Code Generation with AlphaCode
- Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging
- MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters