DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training
cs.AI, cs.LG, cs.SE
Submitted: 2026-09-03
Updated: 2026-09-03
Code: https://github.com/IBM/draco
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none.
Terminology
Abstract
Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.
Sources
- Reinforcement Learning for Long-Horizon Interactive LLM Agents
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Group-in-Group Policy Optimization for LLM Agent Training
- EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
- SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models
- RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
- Qwen2.5 Technical Report
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
- YaRN: Efficient Context Window Extension of Large Language Models
- Online Rubrics Elicitation from Pairwise Comparisons
- ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents
- Discretizing Reward Models
- The Verification Horizon: No Silver Bullet for Coding Agent Rewards
- SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
- Spurious Rewards: Rethinking Training Signals in RLVR
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- OpenClaw-RL: Train Any Agent Simply by Talking
- Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
- Learning CLI Agents with Structured Action Credit under Selective Observation
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems