EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

The paper addresses the challenge of training large language model (LLM) agents for long-horizon tool use. The authors note that "Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground."

The core limitation of existing paradigms is identified as follows: existing approaches remain fundamentally dependent on access to real environments, either for generating executable interactions or for grounding and validating simulated ones. In conventional setups, the policy learns to act on externally supplied responses, leaving environment modeling outside the acting policy. The authors argue that an effective agent policy should not only act to accomplish the task but also model how the environment responds to its actions.

The paper introduces EnvACE, an agentic RL method that trains a single policy to both act and rehearse its environment. The key mechanism works as follows: At each turn, the policy first issues a tool call, then rehearses the environment response that the tool call would induce, and conditions its next action on this rehearsal.

The central concept is world rehearsal: "the world modeling capability that prior work assigns to a separate simulator is absorbed into the policy and executed inline, enabling the policy to internalize how its actions shape environment responses while acting and world modeling unfold as a unified process."

Formally, EnvACE assigns two roles to a shared policy πθ:

• Acting role: generates environment-facing actions: at ∼ πθ(· ht, ACT)

• Rehearsal role: generates the corresponding environment response: ôt ∼ πθ(· ht, at, REHEARSE)

The history is updated as ht+1 = ht ⊕ (at, ôt), and through repeated rehearsal, the relationship between actions and their environment responses is absorbed into θ, enabling the policy to internalize environment dynamics as an agent world model.

EnvACE optimizes the shared policy using a modified Group Relative Policy Optimization (GRPO) objective. For each role r (ACT or REHEARSE), a separate reward baseline is computed: µx,r = (1/Gx,r) Σ Rj, with advantages defined as Ai,m = Ri − µx,ri,m. Although the reward baselines are computed separately for the two roles, outputs from both roles jointly update the shared policy parameters θ, using the standard clipped GRPO objective.

At test time, the policy performs private rehearsals in parallel or sequentially, summarizes them into a rehearsal memory, and uses this memory to guide a committed execution in the external environment. The two modes are:

• Parallel: all attempts generated independently from the same context

• Sequential: each new attempt observes previous rehearsal trajectories with assessments and revision suggestions

The rehearsals remain private and do not alter the external environment.

Benchmarks: BFCL-v4 (function calling and tool use), τ2-Bench (stateful service environments covering Retail, Telecom, Airline), VitaBench (food delivery, in-store consumption, online travel, cross-domain), and FinMCP-Bench (financial agents via Model Context Protocol).

Baselines: Qwen3 models (1.7B, 4B, 8B), Qwen3 with standard GRPO, Simulator-8B (reasoning-model simulated feedback), TOUCAN-7B (synthesized trajectories from MCP environments), EnvScaler-8B (programmatic environment synthesis), AWM-8B/14B (code-driven environments with database states), and ScaleEnv-8B (interactive environments built from scratch).

Implementation: Main experiments use Qwen3-8B, trained on the CM2 dataset for 470 steps with learning rate 1e-6, batch size 16, four rollouts per prompt, KL coefficient 1e-4, up to 30 interaction turns, with Qwen3-30B-A3B as LLM judge during training.

EnvACE achieves an Overall score of 32.91%, outperforming all environment-scaling baselines with complete results on the three benchmarks. Specifically:

• On BFCL V4, EnvACE reaches 46.04%, outperforming Qwen3-8B and AWM-8B by 2.00% and 1.75%

• On τ2-Bench, it obtains the second-highest average of 36.7%, exceeding EnvScaler-8B, AWM-8B, and AWM-14B by 3.8%, 5.5%, and 6.0%

• On VitaBench, EnvACE achieves 16.0%, the best average among all 7B–8B methods

EnvACE achieves the best TF1 score of 46.78%, outperforming EnvScaler-8B and AWM-8B by 3.10% and 4.28%, respectively. It also obtains the highest tool precision of 54.04%.

On τ2-Bench, EnvACE improves the average score from 31.2% to 36.7%, a gain of 5.5% compared to standard GRPO, particularly benefiting stateful, multi-turn interactions.

Comparing EnvACE with a Per-role Policy variant using separate policies: On τ2-Bench, parameter sharing improves the average score from 35.5% to 36.7%, a gain of 1.2%. This supports the hypothesis that shared parameters allow rehearsal knowledge to be directly incorporated into acting.

Scaling from 1.7B to 8B improves the BFCL V4 average from 31.81% to 46.04%, a gain of 14.23%, and the τ2-Bench average from 15.3% to 36.7%, a gain of 21.4%. EnvACE outperforms standard GRPO at both scales.

The offline evaluation score increases from 30.0% at step 50 to 36.7% at step 470, showing that world rehearsal provides a sustained learning signal during RL training.

With rehearsal budget N=2: Parallel rehearsal with EnvACE achieves the best Overall score of 40.9%, improving the Non-TTS result of 36.7% by 4.2%. Notably, the gains do not arise from additional inference compute alone, but depend on the environment-response knowledge internalized through world rehearsal, since using the base model for rehearsal yields only marginal gains under parallel mode and underperforms Non-TTS under sequential mode.

Regarding rehearsal budget, increasing N from 1 to 2 further improves performance for all evaluated configurations, while at N=3 performance decreases likely because additional rehearsal trajectories lengthen the input and approach or exceed the effective context range.

The paper includes two case studies demonstrating EnvACE's behavior. In Case A, EnvACE anticipates a potential tool-call failure and repairs the invalid parameters before execution, whereas baseline agents issue an invalid tool call and require additional interaction steps to recover. In Case B, EnvACE recognizes that the required information for executing update reservation flights is incomplete and that issuing the write operation would lead to an invalid action, proactively replacing it with a read-only query, while baselines eventually execute the invalid write operation, resulting in failure.

The paper concludes that world rehearsal enables the policy to internalize environment dynamics as an agent world model and establishes it as an effective and scalable alternative to agent training based on real-environment interaction or separate external simulators.

Limitations: "Due to computational constraints, we evaluate EnvACE only up to the 8B scale, leaving its effectiveness on larger models for future investigation. In addition, our current evaluation focuses primarily on tool-interactive tasks, and extending world rehearsal to a broader range of agentic settings remains an interesting direction for future work."

The code is publicly available at https://github.com/Within-yao/EnvACE.

Based on this paper, an improved AI agent system can be built with the following specific capabilities:

• Internalize environment dynamics without needing real environments or external simulators. Instead of relying on costly executable environments or separate simulators, the policy learns to rehearse the environment’s responses to its own tool calls during training. This lets the system acquire an implicit world model from private, self-generated experience.

• Act and model the world with a single shared policy. By using one policy for both generating tool calls and generating predicted environment responses, the system directly transfers rehearsal knowledge into action selection. This improves decision-making in stateful, multi-turn tasks, e.g., boosting τ2-Bench from 31.2% (standard GRPO) to 36.7%.

• Avoid invalid tool calls before they happen. The system can anticipate potential failures during rehearsal and repair invalid parameters before executing the actual action. For example, it can detect that a tool call would fail and replace it with a corrected call in the same turn, reducing recovery steps.

• Defer or replace unsafe write operations when information is incomplete. The system can recognize that a write operation (e.g., updating a reservation) would be invalid because required information is missing, and instead issue a read-only query first, preventing execution failures.

• Perform safe, private test-time rehearsal. At inference, the system can generate multiple private rehearsals in parallel or sequentially, summarize them into a rehearsal memory, and use that memory to guide a committed execution. These rehearsals do not alter the external environment, enabling risk-free exploration and planning before acting.

• Scale its capabilities with model size. The approach works across model scales: scaling from 1.7B to 8B improves BFCL V4 by 14.23% and τ2-Bench by 21.4%, and it outperforms standard GRPO at every tested scale.

• Achieve stronger tool-use and function-calling performance. The improved system reaches 46.04% on BFCL V4, 36.7% average on τ2-Bench, 16.0% on VitaBench, and 46.78% TF1 on FinMCP-Bench, beating leading environment-scaling baselines that use real or synthesized environments.

• Sustain learning signal throughout RL training. Because the policy continuously rehearses environment responses, the system keeps improving over training steps (from 30.0% at step 50 to 36.7% at step 470), rather than saturating early.

• Leverage rehearsal budget effectively. Increasing rehearsal attempts from 1 to 2 improves performance for both parallel and sequential rehearsal modes, allowing the system to trade inference compute for higher task success when context length permits.