StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning

summary

The gist

Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents…

In short

StateAlign improves large language models' tool use by training them to make decisions based on task states. It addresses a problem where agents sometimes act prematurely or incorrectly when information is missing. The method constructs specific task states and uses reinforcement learning, including token-level contrastive rewards, to ensure agents ask for necessary information before executing tools, significantly reducing tool hallucination across different model sizes.

Key concepts

Task States
These are constructed representations of the current situation in a task. They vary based on available information, what tools can be used, and how far along the overall task is. The system creates five distinct states—like EXEC or NEED_USER—to give the model context-aware decision-making signals.
StateContrastive Reinforcement Learning
This is a specific training method where the model learns by comparing its reasoning and actions under different, paired task states. By rewarding outputs that are consistent with the current state while penalizing inconsistencies, it forces the agent to align its behavior precisely with the required context.
State-Aware Tool Use
This framework focuses on making tool use decisions that are aware of the task's current status. Instead of just executing tools blindly, StateAlign trains agents to recognize when they need to ask for more input or retrieve missing data before proceeding with an action.
Token-Level State-Contrastive Reward
This is a reward mechanism applied at the individual token level during training. It compares the model's generated reasoning and actions when presented with two different but related task states. This helps refine the output to be highly consistent with the specific state it is currently operating in, directly fighting tool hallucination.

Terminology used across episodes

This episode discusses

The paper

StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning · Read on arXiv

Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

Rensselaer Polytechnic Institute · IBM Research

Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning".

Jane: Here's the gist of the paper. Reinforcement learning has improved the tooluse capabilities of large language models, but better task performance can coincide with increased tool hallucination Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information We introduce StateAlign, combining task-state construction with statecontrastive reinforcement learning The complete method further reduces tool hallucination across three Qwen3 sizes.

Tom: Pruning redundant reasoning tokens StateAlign is a reinforcement learning framework for state-aware, reliable tool use Starting from tool-use trajectories, we construct task states with different information availability, tool capabilities, and task progress.

Jane: The model is trained on these states with outcome rewards based on task completion We further design a token-level state-contrastive reward that compares the same generated reasoning and actions under paired states to encourage outputs consistent with the current task state and reduce tool hallucination.

Tom: How the training signal is built We construct data from ToolMind (Yang et al., 2025b) and LoopTool-23k (Zhang et al., 2026a) We use Qwen3.8-27B to filter calls in ToolMind’s synthetic demonstrations that do not serve the request or contain unsupported arguments.

Paper summary: Tom: So looking at the title, "StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning," it really sums up what they did—they internalized state awareness using reinforcement learning to make multi-turn tool use more reliable >

Jane: And what it means is that we're moving toward agents that aren't just good at one step, but can actually manage a whole process where they know exactly what information they need and when to stop trying >

Lu: The implication here is that if you want better tool use, you can’t just crank up the model size or the reward function; you have to explicitly teach it how to reason about the task state as it evolves >

Meng: From a deployment standpoint, this sounds like a way to stabilize performance when dealing with complex workflows where errors cascade easily across multiple tool calls >

Lalam: I'm confident in this direction because they are using concrete state definitions—like EXECUTE, NEED USER—to guide the agent's behavior instead of relying purely on statistical patterns learned from just a massive dataset >

Tom: So, for the listener who just wants to know what this paper does beyond the technical jargon, it’s about making AI agents think more like a human worker who checks their to-do list before picking up a tool >

Conclusion: Jane: So, to wrap up this discussion on StateAlign: it's basically about giving these large language models better instincts for using tools over a whole conversation, not just one single step.

Tom: Yeah, the title says internalizing state-aware decision-making through reinforcement learning. It sounds like they’re teaching the AI how to keep track of where it is in a task while it’s actually executing things.

Lu: What I find interesting is that they aren't just looking at the final answer; they are constructing these different task states—like needing user input or needing a specific tool result—and training the model on those specific conditions.

Meng: From my side, what really matters is that this helps fix those situations where the AI just keeps trying to use a tool when it should actually stop and ask you for more information, which is something that breaks things at scale.

Lalam: I'm confident in this because they have these five specific states: EXECUTE, NEED USER, NEED FETCH, INFEASIBLE, and STOP. That structure gives the model a clear map of what it needs to do next.

Jane: So for someone just listening on the go, this means when you ask an AI to build you a complex itinerary or solve a multi-step problem, it’s less likely to keep running tools blindly without checking if it has the right piece of information first.

Tom: Exactly. The numbers show that this method reduces tool hallucinations across three different sizes of Qwen3 models, which suggests it's pretty robust for different kinds of systems.

Lu: The paper shows behavioral improvements, like more consistent decisions across those task states and stronger recovery when a tool call fails. That's a big deal for reliable workflows.

Meng: But I gotta ask, how much does this training signal actually cost to generate? Is it just running these extra reward calculations on top of the existing model training loop?

Lalam: It adds that token-level state-contrastive reward, which compares different reasoning paths under the same task state. It’s an added layer of supervision that really tightens the consistency.

Jane: So what does this mean for the future of AI agents? Does this show us how to get them to handle longer, more complicated projects where they have to remember things across many turns?

Tom: It definitely points toward better long-term planning. The limitation they mentioned is that it was trained on these constructed states from shorter interactions, so extending it to very long workflows with constantly changing user goals is the next big challenge.

Lu: That’s the known boundary for now. We need to see if we can generalize those state definitions beyond what was used in those initial demonstrations.

More episodes

← Home