Towards a Belief-Based World Model for LLM Agents
cs.AI
Submitted: 2026-08-31
Updated: 2026-09-30
Comments: pre-print
Code: https://github.com/skumar-ml/belief-world-models
Project page: https://worldmodels.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains.
Terminology
Abstract
Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains. Despite their strong reasoning capabilities, LLMs struggle with long-horizon tasks, especially under partial observability. World models are a promising way to enhance policy performance, both during training and inference. During inference, agents currently use world models to simulate the consequences of candidate actions before choosing an action, which can improve decision-making. However, we argue that simulation alone is an incomplete interface for decision-making under partial observability: simulation does not adequately capture uncertainty about the current state, which agents may need for accurate decision-making. We address this limitation with Belief-Based World Models (BB-WMs), which maintain a belief that LLMs can query to access information on what is known and uncertain about the current state. Before developing methods to learn accurate BB-WMs, this paper focuses on a more fundamental question: does exposing a world model's belief directly to an LLM policy improve decision-making? Our results show that giving LLM agents access to beliefs improves task performance under partial observability, while remaining complementary to existing simulation-based world models. Code: https://github.com/skumar-ml/belief-world-models.
Sources
- The Llama 3 Herd of Models
- Learning Latent Dynamics for Planning from Pixels
- GAIA-1: A Generative World Model for Autonomous Driving
- Think Locally, Explain Globally: Graph-Guided LLM Investigations via Local Reasoning and Belief Propagation
- Belief Memory: Agent Memory Under Partial Observability
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Debugging code world models
- StateAct: Enhancing LLM Base Agents via Self-prompting and State-tracking
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty
- CWM: An Open-Weights LLM for Research on Code Generation with World Models
- Qwen3 Technical Report
- Reinforcement World Model Learning for LLM-based Agents
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
- WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems