SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
cs.AI, cs.LG
Submitted: 2026-09-24
Updated: 2026-09-24
Comments: 41 pages, 13 figures. Code: https://github.com/SLCA-GRPO/SLCA-GRPO ; Dataset: https://huggingface.co/datasets/YanZhanPKU/SLCA-GRPO-Datasets
Code: https://github.com/SLCA-GRPO/SLCA-GRPO
License: http://creativecommons.org/licenses/by/4.0/
The gist: Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries.
Terminology
Abstract
Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens. Consequently, gradient noise from summary generation leaks into tool-decision tokens, causing cross-segment credit misattribution and brittle optimization. In this work, we propose SLCA-GRPO, a framework incorporating Segment-Locked Credit Assignment (SLCA). To enable scalable exploration without costly real APIs and stable training, we first construct the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure. Building on this, SLCA decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), SLCA routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination (the dominant cross-segment credit misattribution channel) within each policy update. On a 7B backbone, SLCA-GRPO accelerates convergence and outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on τ squared-Bench under the same training budgets, achieving higher accuracy with reduced tool redundancy and costs.
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards
- Qwen2.5 Technical Report
- Qwen3 Technical Report
- Proximal Policy Optimization Algorithms
- Solving math word problems with process- and outcome-based feedback
- From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
- Self-Preference Bias in LLM-as-a-Judge
- Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design
- TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems