StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning

arXiv:2611.16304 · cs.LG · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning".

Jane: Here's the gist of the paper. Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning The complete method further reduces tool hallucination across three Qwen3 sizes.

Tom: Pruning redundant reasoning tokens StateAlign is a reinforcement learning framework for state-aware, reliable tool use Starting from tool-use trajectories, we construct task states with different information availability, tool capabilities, and task progress.

Jane: The model is trained on these states with outcome rewards based on task completion We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination.

Tom: How the training signal is built We construct data from ToolMind (Yang et al., 2025b) and LoopTool-23k (Zhang et al., 2026a) We use Qwen3.8-27B to filter calls in ToolMind’s synthetic demonstrations that do not serve the request or contain unsupported arguments.

Paper summary: Tom: So looking at the title, "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning," it really sums up what they did—they internalized state awareness using reinforcement learning to make multi-turn tool use more reliable >

Jane: And what it means is that we're moving toward agents that aren't just good at one step, but can actually manage a whole process where they know exactly what information they need and when to stop trying >

Lu: The implication here is that if you want better tool use, you can’t just crank up the model size or the reward function; you have to explicitly teach it how to reason about the task state as it evolves >

Meng: From a deployment standpoint, this sounds like a way to stabilize performance when dealing with complex workflows where errors cascade easily across multiple tool calls >

Lalam: I'm confident in this direction because they are using concrete state definitions—like EXECUTE, NEED USER—to guide the agent's behavior instead of relying purely on statistical patterns learned from just a massive dataset >

Tom: So, for the listener who just wants to know what this paper does beyond the technical jargon, it’s about making AI agents think more like a human worker who checks their to-do list before picking up a tool >

Conclusion: Jane: So, to wrap up this discussion on StateAlign: it's basically about giving these large language models better instincts for using tools over a whole conversation, not just one single step.

Tom: Yeah, the title says internalizing state-aware decision-making through reinforcement learning. It sounds like they’re teaching the AI how to keep track of where it is in a task while it’s actually executing things.

Lu: What I find interesting is that they aren't just looking at the final answer; they are constructing these different task states—like needing user input or needing a specific tool result—and training the model on those specific conditions.

Meng: From my side, what really matters is that this helps fix those situations where the AI just keeps trying to use a tool when it should actually stop and ask you for more information, which is something that breaks things at scale.

Lalam: I'm confident in this because they have these five specific states: EXECUTE, NEED USER, NEED FETCH, INFEASIBLE, and STOP. That structure gives the model a clear map of what it needs to do next.

Jane: So for someone just listening on the go, this means when you ask an AI to build you a complex itinerary or solve a multi-step problem, it’s less likely to keep running tools blindly without checking if it has the right piece of information first.

Tom: Exactly. The numbers show that this method reduces tool hallucinations across three different sizes of Qwen3 models, which suggests it's pretty robust for different kinds of systems.

Lu: The paper shows behavioral improvements, like more consistent decisions across those task states and stronger recovery when a tool call fails. That's a big deal for reliable workflows.

Meng: But I gotta ask, how much does this training signal actually cost to generate? Is it just running these extra reward calculations on top of the existing model training loop?

Lalam: It adds that token-level state-contrastive reward, which compares different reasoning paths under the same task state. It’s an added layer of supervision that really tightens the consistency.

Jane: So what does this mean for the future of AI agents? Does this show us how to get them to handle longer, more complicated projects where they have to remember things across many turns?

Tom: It definitely points toward better long-term planning. The limitation they mentioned is that it was trained on these constructed states from shorter interactions, so extending it to very long workflows with constantly changing user goals is the next big challenge.

Lu: That’s the known boundary for now. We need to see if we can generalize those state definitions beyond what was used in those initial demonstrations.

Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

Rensselaer Polytechnic Institute · IBM Research

cs.LG

Submitted: 2026-11-12

Updated: 2026-11-12

Code: https://github.com/ZhexiLu/StateAlign

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents…

Key concepts

Task States
These are constructed representations of the current situation in a task. They vary based on available information, what tools can be used, and how far along the overall task is. The system creates five distinct states—like EXEC or NEED_USER—to give the model context-aware decision-making signals.
StateContrastive Reinforcement Learning
This is a specific training method where the model learns by comparing its reasoning and actions under different, paired task states. By rewarding outputs that are consistent with the current state while penalizing inconsistencies, it forces the agent to align its behavior precisely with the required context.
State-Aware Tool Use
This framework focuses on making tool use decisions that are aware of the task's current status. Instead of just executing tools blindly, StateAlign trains agents to recognize when they need to ask for more input or retrieve missing data before proceeding with an action.
Token-Level State-Contrastive Reward
This is a reward mechanism applied at the individual token level during training. It compares the model's generated reasoning and actions when presented with two different but related task states. This helps refine the output to be highly consistent with the specific state it is currently operating in, directly fighting tool hallucination.

Terminology

Summary

Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning The complete method further reduces tool hallucination across three Qwen3 sizes. p. 0

Where the paper says it

Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning.

Pruning redundant reasoning tokens

StateAlign is a reinforcement learning framework for state-aware, reliable tool use Starting from tool-use trajectories, we construct task states with different information availability, tool capabilities, and task progress. The model is trained on these states with outcome rewards based on task completion We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination. p. 1

Where the paper says it

The model is trained on these states with outcome rewards based on task completion. We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination.

How the training signal is built

We construct data from ToolMind (Yang et al., 2025b) and LoopTool-23k (Zhang et al. p. 2

Where the paper says it

1 Constructing Paired Task States We construct data from ToolMind (Yang et al. , 2025b) and LoopTool-23k (Zhang et al.

, 2026a) We use Qwen3. p. 4
Where the paper says it

, 2026a), and the base models. ToolRL uses Qwen3-8B trained on RLLA-4k; LoopTool and AWM use released models.

8-27B to filter calls in ToolMind’s synthetic demonstrations that do not serve the request or contain unsupported arguments. p. 3
Where the paper says it

We use Qwen3. 8-27B to filter calls in ToolMind’s synthetic demonstrations that do not serve the request or contain unsupported arguments.

For each selected call, we identify its required tools and information, then modify the tool definitions or history to construct five types of states: EXEC supplies the information and tools for execution, NEED USER removes a required user-provided value or retains a clarification turn, requiring the agent to ask before executing, NEED FETCH removes a relevant tool result or introduces a lookup for a required identifier, so a tool can obtain the missing information, INFEASIBLE removes the required tool and its identified alternatives, and STOP includes a result completing the request and requires a reply without further calls. p. 3
Where the paper says it

NEED_USER removes a required user-provided value or retains a clarification turn, requiring the agent to ask before executing. NEED_FETCH removes a relevant tool result or introduces a lookup for a required identifier, so a tool can obtain the missing information.

Where the gains come from

We maximize J (θ) = Jout(θ) + λJpair(θ) Outcome supervision evaluates actions over an interaction, where a successful first action must lead to an appropriate follow-up action. State contrasts additionally train how generated reasoning and actions fit the current conditions by scoring tokens under both contexts using the same prefix pi,t. p. 3

Where the paper says it

State contrasts additionally train how generated reasoning and actions fit the current conditions. For paired contexts (hi , h˜ i), reference paths zi , z˜i describe appropriate actions and continuations.

This pair objective provides supervision beyond these tied outcomes, prioritizing groups with varying rewards and retaining a portion of paired successful groups. p. 4
Where the paper says it

The pair objective provides supervision beyond these tied outcomes. We prioritize groups with varying rewards and retain a portion of paired successful groups, as detailed in Appendix B.

Behavioral improvements

Experiments across five base models show improved performance on BFCL v4 multi-turn, ToolSandbox, and When2Call Behavioral analyses show more consistent decisions across task states, more complete multicall execution, and stronger recovery after tool errors. StateAlign leads turn success on two-call and threeor more-call BFCL turns StateAlign also achieves the highest ToolSandbox similarity on error-containing scenarios. p. 5

Where the paper says it

3 Multi-Step Execution and Recovery Multi-step tasks require agents to update what remains after each tool result and continue until the request is complete. Figure 5a shows that StateAlign leads turn success on two-call and threeor-more-call BFCL turns.

The paper shows that StateAlign makes no call immediately after the first error and ultimately fails in 2. p. 6
Where the paper says it

containing turns than on turns without errors. In Table 4, StateAlign makes no call immediately after the first error and ultimately fails in 2.

7% of error-containing turns, compared with 5. p. 6
Where the paper says it

In Table 4, StateAlign makes no call immediately after the first error and ultimately fails in 2. 7% of error-containing turns, compared with 5.

7% for GRPO and 6. p. 2
Where the paper says it

get_order(4521) cancel_order(4521) order 4521 found cancelled Order cancelled. GRPO objective 3 Learn from state contrasts current state h no id; cannot fetch z: ask, then cancel paired state ̃ h id given: #4521 z:̃ cancel order 4521 base model 2 3 rollout A: asks The order id is missing.

3% for LoopTool. p. 0
Where the paper says it

These tools enable models to retrieve information, perform calculations, and execute operations requested by users. By incorporating tool results into subsequent reasoning, models can gather information and complete tasks over 45 50 55 BFCL v4 multi-turn (%) 55 60 65 Hallucination rate (%) Base ToolRL LoopTool AWM StateAlign better ↘ (a) Unavailable tool 45 50 55 BFCL v4 multi-turn (%) 24 26 28 30 32 Base ToolRL AWM LoopTool StateAlign better ↘ (b) Distractor tool Figure 1: Comparison of tool-use performance and reliability on Qwen3-8B.

Limitations

StateAlign is trained on five types of constructed task states through short interactions with recorded feedback Training feedback is limited to the interactions covered by the source trajectories, which suggests extending this setup to longer workflows with changing user goals or tool behavior is a direction for future work. We evaluate five base models ranging from 4B to 14B parameters on text-based tool-use benchmarks. p. 8

Where the paper says it

Extending this setup to longer workflows with changing user goals or tool behavior is a direction for future work. We evaluate five base models ranging from 4B to 14B parameters on text-based tool-use benchmarks.

The effectiveness of the approach at larger scales and in multimodal tool environments remains to be studied. p. 8
Where the paper says it

We evaluate five base models ranging from 4B to 14B parameters on text-based tool-use benchmarks. The effectiveness of the approach at larger scales and in multimodal tool environments remains to be studied.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. StateAlign improves tool-use reliability by combining task-state construction with statecontrastive reinforcement learning to train outputs that fit the current task state. This enables agents to decide whether and how to act based on the current task state, leading to more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.

  2. StateAlign reduces tool hallucination by comparing generated reasoning and actions under paired states, which is achieved by training on these contrasts. This method has been shown to reduce tool hallucination across three Qwen3 model sizes compared with training on outcome rewards alone.

  3. The system can perform better in multi-turn scenarios because it learns from interaction outcomes that train agents to follow a sequence where Asking for information should enable execution once it arrives; receiving a result should lead to further work or termination according to the remaining task. This results in more complete multi-call execution.

  4. Agents exhibit superior error recovery by learning from state contrasts, which helps them address conditions before continuing. For example, StateAlign makes no call immediately after the first error and ultimately fails in 2.7% of error-containing turns, compared with 5.7% for GRPO and 6.3% for LoopTool.

Abstract

Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.

Sources

Related papers