StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning".
Jane: Here's the gist of the paper. Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning The complete method further reduces tool hallucination across three Qwen3 sizes.
Tom: Pruning redundant reasoning tokens StateAlign is a reinforcement learning framework for state-aware, reliable tool use Starting from tool-use trajectories, we construct task states with different information availability, tool capabilities, and task progress.
Jane: The model is trained on these states with outcome rewards based on task completion We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination.
Tom: How the training signal is built We construct data from ToolMind (Yang et al., 2025b) and LoopTool-23k (Zhang et al., 2026a) We use Qwen3.8-27B to filter calls in ToolMind’s synthetic demonstrations that do not serve the request or contain unsupported arguments.
Paper summary: Tom: So looking at the title, "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning," it really sums up what they did—they internalized state awareness using reinforcement learning to make multi-turn tool use more reliable >
Jane: And what it means is that we're moving toward agents that aren't just good at one step, but can actually manage a whole process where they know exactly what information they need and when to stop trying >
Lu: The implication here is that if you want better tool use, you can’t just crank up the model size or the reward function; you have to explicitly teach it how to reason about the task state as it evolves >
Meng: From a deployment standpoint, this sounds like a way to stabilize performance when dealing with complex workflows where errors cascade easily across multiple tool calls >
Lalam: I'm confident in this direction because they are using concrete state definitions—like EXECUTE, NEED USER—to guide the agent's behavior instead of relying purely on statistical patterns learned from just a massive dataset >
Tom: So, for the listener who just wants to know what this paper does beyond the technical jargon, it’s about making AI agents think more like a human worker who checks their to-do list before picking up a tool >
Conclusion: Jane: So, to wrap up this discussion on StateAlign: it's basically about giving these large language models better instincts for using tools over a whole conversation, not just one single step.
Tom: Yeah, the title says internalizing state-aware decision-making through reinforcement learning. It sounds like they’re teaching the AI how to keep track of where it is in a task while it’s actually executing things.
Lu: What I find interesting is that they aren't just looking at the final answer; they are constructing these different task states—like needing user input or needing a specific tool result—and training the model on those specific conditions.
Meng: From my side, what really matters is that this helps fix those situations where the AI just keeps trying to use a tool when it should actually stop and ask you for more information, which is something that breaks things at scale.
Lalam: I'm confident in this because they have these five specific states: EXECUTE, NEED USER, NEED FETCH, INFEASIBLE, and STOP. That structure gives the model a clear map of what it needs to do next.
Jane: So for someone just listening on the go, this means when you ask an AI to build you a complex itinerary or solve a multi-step problem, it’s less likely to keep running tools blindly without checking if it has the right piece of information first.
Tom: Exactly. The numbers show that this method reduces tool hallucinations across three different sizes of Qwen3 models, which suggests it's pretty robust for different kinds of systems.
Lu: The paper shows behavioral improvements, like more consistent decisions across those task states and stronger recovery when a tool call fails. That's a big deal for reliable workflows.
Meng: But I gotta ask, how much does this training signal actually cost to generate? Is it just running these extra reward calculations on top of the existing model training loop?
Lalam: It adds that token-level state-contrastive reward, which compares different reasoning paths under the same task state. It’s an added layer of supervision that really tightens the consistency.
Jane: So what does this mean for the future of AI agents? Does this show us how to get them to handle longer, more complicated projects where they have to remember things across many turns?
Tom: It definitely points toward better long-term planning. The limitation they mentioned is that it was trained on these constructed states from shorter interactions, so extending it to very long workflows with constantly changing user goals is the next big challenge.
Lu: That’s the known boundary for now. We need to see if we can generalize those state definitions beyond what was used in those initial demonstrations.
Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu
Rensselaer Polytechnic Institute · IBM Research
cs.LG
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/ZhexiLu/StateAlign
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents…
Key concepts
- Task States
- These are constructed representations of the current situation in a task. They vary based on available information, what tools can be used, and how far along the overall task is. The system creates five distinct states—like EXEC or NEED_USER—to give the model context-aware decision-making signals.
- StateContrastive Reinforcement Learning
- This is a specific training method where the model learns by comparing its reasoning and actions under different, paired task states. By rewarding outputs that are consistent with the current state while penalizing inconsistencies, it forces the agent to align its behavior precisely with the required context.
- State-Aware Tool Use
- This framework focuses on making tool use decisions that are aware of the task's current status. Instead of just executing tools blindly, StateAlign trains agents to recognize when they need to ask for more input or retrieve missing data before proceeding with an action.
- Token-Level State-Contrastive Reward
- This is a reward mechanism applied at the individual token level during training. It compares the model's generated reasoning and actions when presented with two different but related task states. This helps refine the output to be highly consistent with the specific state it is currently operating in, directly fighting tool hallucination.
Terminology
Summary
Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning The complete method further reduces tool hallucination across three Qwen3 sizes. p. 0 Our analysis shows that
agents sometimes execute when they should
ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies
missing information. We introduce StateAlign,
combining task-state construction with statecontrastive reinforcement learning.Where the paper says it
Pruning redundant reasoning tokens
StateAlign is a reinforcement learning framework for state-aware, reliable tool use Starting from tool-use trajectories, we construct task states with different information availability, tool capabilities, and task progress. The model is trained on these states with outcome rewards based on task completion We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination. p. 1 The model
is trained on these states with outcome rewards
based on task completion. We further design a
token-level state-contrastive reward that compares
the same generated reasoning and actions under
paired states to encourage outputs consistent with
the current task state and reduce tool hallucination.Where the paper says it
How the training signal is built
We construct data from ToolMind (Yang et al., 2025b) and LoopTool-23k (Zhang et al. p. 2 1 Constructing Paired Task States
We construct data from ToolMind (Yang et al. ,
2025b) and LoopTool-23k (Zhang et al. , 2026a), and the base models. ToolRL
uses Qwen3-8B trained on RLLA-4k; LoopTool
and AWM use released models. We use Qwen3. 8-27B to filter
calls in ToolMind’s synthetic demonstrations that
do not serve the request or contain unsupported arguments. NEED_USER removes a required user-provided value or retains a clarification
turn, requiring the agent to ask before executing. NEED_FETCH removes a relevant tool result or
introduces a lookup for a required identifier, so a
tool can obtain the missing information.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
We maximize J (θ) = Jout(θ) + λJpair(θ) Outcome supervision evaluates actions over an interaction, where a successful first action must lead to an appropriate follow-up action. State contrasts additionally train how generated reasoning and actions fit the current conditions by scoring tokens under both contexts using the same prefix pi,t. p. 3 State contrasts additionally train how
generated reasoning and actions fit the current conditions. For paired contexts (hi
, h˜
i), reference
paths zi
, z˜i describe appropriate actions and continuations. The pair objective provides supervision beyond
these tied outcomes. We prioritize groups with
varying rewards and retain a portion of paired successful groups, as detailed in Appendix B.Where the paper says it
Where the paper says it
Behavioral improvements
Experiments across five base models show improved performance on BFCL v4 multi-turn, ToolSandbox, and When2Call Behavioral analyses show more consistent decisions across task states, more complete multicall execution, and stronger recovery after tool errors. StateAlign leads turn success on two-call and threeor more-call BFCL turns StateAlign also achieves the highest ToolSandbox similarity on error-containing scenarios. p. 5 3 Multi-Step Execution and Recovery
Multi-step tasks require agents to update what remains after each tool result and continue until
the request is complete. Figure 5a shows that
StateAlign leads turn success on two-call and threeor-more-call BFCL turns. containing turns than on turns without errors. In
Table 4, StateAlign makes no call immediately after the first error and ultimately fails in 2. In
Table 4, StateAlign makes no call immediately after the first error and ultimately fails in 2. 7% of
error-containing turns, compared with 5. get_order(4521)
cancel_order(4521)
order 4521 found
cancelled
Order cancelled. GRPO objective
3 Learn from state contrasts
current state h
no id; cannot fetch
z: ask, then cancel
paired state ̃ h
id given: #4521
z:̃ cancel order 4521
base model
2 3
rollout A: asks
The order id is missing. These tools enable models to retrieve information, perform calculations, and execute operations requested by users. By incorporating tool results into subsequent reasoning, models
can gather information and complete tasks over
45 50 55
BFCL v4 multi-turn (%)
55
60
65
Hallucination rate (%)
Base
ToolRL
LoopTool
AWM
StateAlign
better ↘
(a) Unavailable tool
45 50 55
BFCL v4 multi-turn (%)
24
26
28
30
32
Base
ToolRL
AWM LoopTool
StateAlign
better ↘
(b) Distractor tool
Figure 1: Comparison of tool-use performance and reliability on Qwen3-8B.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations
StateAlign is trained on five types of constructed task states through short interactions with recorded feedback Training feedback is limited to the interactions covered by the source trajectories, which suggests extending this setup to longer workflows with changing user goals or tool behavior is a direction for future work. We evaluate five base models ranging from 4B to 14B parameters on text-based tool-use benchmarks. p. 8 Extending this setup to longer workflows with changing
user goals or tool behavior is a direction for future work. We evaluate five base models ranging
from 4B to 14B parameters on text-based tool-use
benchmarks. We evaluate five base models ranging
from 4B to 14B parameters on text-based tool-use
benchmarks. The effectiveness of the approach at
larger scales and in multimodal tool environments
remains to be studied.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
StateAlign improves tool-use reliability by combining task-state construction with statecontrastive reinforcement learning to train outputs that fit the current task state. This enables agents to
decide whether and how to act based on the current task state,
leading tomore consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.
-
StateAlign reduces tool hallucination by comparing generated reasoning and actions under paired states, which is achieved by training on these contrasts. This method has been shown to
reduce tool hallucination across three Qwen3 model sizes
compared with training on outcome rewards alone. -
The system can perform better in multi-turn scenarios because it learns from interaction outcomes that train agents to follow a sequence where
Asking for information should enable execution once it arrives; receiving a result should lead to further work or termination according to the remaining task.
This results inmore complete multi-call execution.
-
Agents exhibit superior error recovery by learning from state contrasts, which helps them address conditions before continuing. For example, StateAlign
makes no call immediately after the first error and ultimately fails in 2.7% of error-containing turns, compared with 5.7% for GRPO and 6.3% for LoopTool.
Abstract
Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.
Sources
- The Llama 3 Herd of Models
- From State to Action: OODA-Tool for Reliable Multi-Turn Tool Use
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
- Qwen3 Technical Report
- ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset
- SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
- MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
- Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
- RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents
- Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis