StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning
summary
The gist
Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents…
In short
StateAlign improves large language models' tool use by training them to make decisions based on task states. It addresses a problem where agents sometimes act prematurely or incorrectly when information is missing. The method constructs specific task states and uses reinforcement learning, including token-level contrastive rewards, to ensure agents ask for necessary information before executing tools, significantly reducing tool hallucination across different model sizes.
Key concepts
- Task States
- These are constructed representations of the current situation in a task. They vary based on available information, what tools can be used, and how far along the overall task is. The system creates five distinct states—like EXEC or NEED_USER—to give the model context-aware decision-making signals.
- StateContrastive Reinforcement Learning
- This is a specific training method where the model learns by comparing its reasoning and actions under different, paired task states. By rewarding outputs that are consistent with the current state while penalizing inconsistencies, it forces the agent to align its behavior precisely with the required context.
- State-Aware Tool Use
- This framework focuses on making tool use decisions that are aware of the task's current status. Instead of just executing tools blindly, StateAlign trains agents to recognize when they need to ask for more input or retrieve missing data before proceeding with an action.
- Token-Level State-Contrastive Reward
- This is a reward mechanism applied at the individual token level during training. It compares the model's generated reasoning and actions when presented with two different but related task states. This helps refine the output to be highly consistent with the specific state it is currently operating in, directly fighting tool hallucination.
Terminology used across episodes
This episode discusses
- StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning · Paper Radio
- The Llama 3 Herd of Models · Paper Radio
- From State to Action: OODA-Tool for Reliable Multi-Turn Tool Use · Paper Radio
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
- Qwen3 Technical Report
- ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset
- SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation · Paper Radio
- MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
- Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following · Paper Radio
- RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents
- Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
The paper
StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning · Read on arXiv
Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu
Rensselaer Polytechnic Institute · IBM Research
Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning".
Jane: Here's the gist of the paper. Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning The complete method further reduces tool hallucination across three Qwen3 sizes.
Tom: Pruning redundant reasoning tokens StateAlign is a reinforcement learning framework for state-aware, reliable tool use Starting from tool-use trajectories, we construct task states with different information availability, tool capabilities, and task progress.
Jane: The model is trained on these states with outcome rewards based on task completion We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination.
Tom: How the training signal is built We construct data from ToolMind (Yang et al., 2025b) and LoopTool-23k (Zhang et al., 2026a) We use Qwen3.8-27B to filter calls in ToolMind’s synthetic demonstrations that do not serve the request or contain unsupported arguments.
Paper summary: Tom: So looking at the title, "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning," it really sums up what they did—they internalized state awareness using reinforcement learning to make multi-turn tool use more reliable >
Jane: And what it means is that we're moving toward agents that aren't just good at one step, but can actually manage a whole process where they know exactly what information they need and when to stop trying >
Lu: The implication here is that if you want better tool use, you can’t just crank up the model size or the reward function; you have to explicitly teach it how to reason about the task state as it evolves >
Meng: From a deployment standpoint, this sounds like a way to stabilize performance when dealing with complex workflows where errors cascade easily across multiple tool calls >
Lalam: I'm confident in this direction because they are using concrete state definitions—like EXECUTE, NEED USER—to guide the agent's behavior instead of relying purely on statistical patterns learned from just a massive dataset >
Tom: So, for the listener who just wants to know what this paper does beyond the technical jargon, it’s about making AI agents think more like a human worker who checks their to-do list before picking up a tool >
Conclusion: Jane: So, to wrap up this discussion on StateAlign: it's basically about giving these large language models better instincts for using tools over a whole conversation, not just one single step.
Tom: Yeah, the title says internalizing state-aware decision-making through reinforcement learning. It sounds like they’re teaching the AI how to keep track of where it is in a task while it’s actually executing things.
Lu: What I find interesting is that they aren't just looking at the final answer; they are constructing these different task states—like needing user input or needing a specific tool result—and training the model on those specific conditions.
Meng: From my side, what really matters is that this helps fix those situations where the AI just keeps trying to use a tool when it should actually stop and ask you for more information, which is something that breaks things at scale.
Lalam: I'm confident in this because they have these five specific states: EXECUTE, NEED USER, NEED FETCH, INFEASIBLE, and STOP. That structure gives the model a clear map of what it needs to do next.
Jane: So for someone just listening on the go, this means when you ask an AI to build you a complex itinerary or solve a multi-step problem, it’s less likely to keep running tools blindly without checking if it has the right piece of information first.
Tom: Exactly. The numbers show that this method reduces tool hallucinations across three different sizes of Qwen3 models, which suggests it's pretty robust for different kinds of systems.
Lu: The paper shows behavioral improvements, like more consistent decisions across those task states and stronger recovery when a tool call fails. That's a big deal for reliable workflows.
Meng: But I gotta ask, how much does this training signal actually cost to generate? Is it just running these extra reward calculations on top of the existing model training loop?
Lalam: It adds that token-level state-contrastive reward, which compares different reasoning paths under the same task state. It’s an added layer of supervision that really tightens the consistency.
Jane: So what does this mean for the future of AI agents? Does this show us how to get them to handle longer, more complicated projects where they have to remember things across many turns?
Tom: It definitely points toward better long-term planning. The limitation they mentioned is that it was trained on these constructed states from shorter interactions, so extending it to very long workflows with constantly changing user goals is the next big challenge.
Lu: That’s the known boundary for now. We need to see if we can generalize those state definitions beyond what was used in those initial demonstrations.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation