Generating Critic-Guided Synthetic Experience in Latent World Models
cs.LG, cs.AI
Submitted: 2026-10-29
Updated: 2026-11-11
License: http://creativecommons.org/licenses/by/4.0/
The gist: Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world.
Terminology
Abstract
Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the Q-function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of 7.1% on the average success rate.
Sources
- Cosmos World Foundation Model Platform for Physical AI
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
- Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
- Training Agents Inside of Scalable World Models
- Temporal Difference Learning for Model Predictive Control
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
- AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization
- When to Trust Your Model: Model-Based Policy Optimization
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Reinforcement Learning with Action Chunking
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Scalable Offline Model-Based RL with Action Chunks
- OGBench: Benchmarking Offline Goal-Conditioned RL
- Imitating Human Behaviour with Diffusion Models
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
- Behavior Regularized Offline Reinforcement Learning
- Hierarchical Planning with Latent World Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis