Reinforcement World Model Learning for LLM-based Agents
cs.CL
Submitted: 2026-02-05
Updated: 2026-11-01
Comments: neurips 2026 camera ready
Code: https://github.com/modelscope/evalscope
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) have achieved strong performance in language-centric tasks.
Terminology
Abstract
Large language models (LLMs) have achieved strong performance in language-centric tasks. However, in agentic settings, LLMs often struggle to anticipate action consequences and adapt to environment dynamics, highlighting the need for world-modeling capabilities in LLM-based agents. We propose Reinforcement World Model Learning (RWML), a self-supervised method that learns action-conditioned world models for LLM-based agents on textual states using sim-to-real gap rewards. Our method aligns simulated next states produced by the model with realized next states observed from the environment, encouraging consistency between internal world simulations and actual environment dynamics in a pre-trained embedding space. Unlike next-state token prediction, which prioritizes token-level fidelity (i.e., reproducing exact wording) over semantic equivalence and can lead to model collapse, our method provides a more robust training signal and is empirically less susceptible to reward hacking than LLM-as-a-judge. We evaluate our method on ALFWorld and τ squared Bench and observe significant gains over the base model, despite using no expert data, task-success rewards, or strong-LLM annotations - only an off-the-shelf embedding model as a distance metric. When combined with task-success rewards, our method outperforms direct task-success reward RL by 6.9 and 5.7 points on ALFWorld and τ squared Bench respectively; it further outperforms expert-data training on ALFWorld and is comparable to it on τ squared Bench.
Sources
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
- GUICourse: From General Vision Language Models to Versatile GUI Agents
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- CWM: An Open-Weights LLM for Research on Code Generation with World Models
- ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
- Group-in-Group Policy Optimization for LLM Agent Training
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- Dense Passage Retrieval for Open-Domain Question Answering
- Let's Verify Step by Step
- GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
- AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
- OpenAI o1 System Card
- Training Software Engineering Agents and Verifiers with SWE-Gym
- Current Agents Fail to Leverage World Model as Tool for Foresight
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data