Learning in Dreams, Winning in Reality: A Continuous Dyna Loop for a Ten-Hero MOBA
cs.AI
Submitted: 2026-10-06
Updated: 2026-10-06
Comments: 15 pages, 11 figures. Code, policies, evaluation and videos: https://github.com/JordyKieto/puffermoba-dyna
Code: https://github.com/JordyKieto/puffermoba-dyna
Project page: https://oasis-model.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: World models are usually judged from the inside: by prediction loss, by the return a policy earns in imagination, or by how convincing their frames look.
Abstract
World models are usually judged from the inside: by prediction loss, by the return a policy earns in imagination, or by how convincing their frames look. We judge one from the outside. We learn a structured, multi-agent world model of a complete ten-hero MOBA (206 units, every hero acting every tick, games of up to 6,000 ticks), train a policy only inside it with 1,400-tick free-running imagined episodes, and measure that policy in the real game against the opponent the game ships with. The real game never provides a gradient; it provides the policy's own games as training data for the world model, and an online evaluation that selects and anchors the policy. Run as a continuous asynchronous Dyna loop, the policy wins 70.2% of real games as radiant (421 of 600; 95% CI 66.4-73.7) on seeds never used for any decision, up from 0% for dream training alone and 33.7% before the loop. It wins none as dire, and neither does the shipped opponent when it plays itself. Four findings explain the result. Model exploitation is invisible from inside the dream: every unanchored run collapsed within a few updates while no in-dream metric tracked the collapse. A world model that is accurate on its training corpus is badly wrong on the policy's own games, and Dyna repairs it there, which is worth +9.2 points of real win rate with the policy recipe held fixed. Finally, the policy inherits its world model's fidelity profile mechanic by mechanic: the model represents the macro game but not crowd control, cast timing or lethality, and the policy wins by map-wide pressure with almost no coordinated fighting. We release the world model, the dream-PPO harness, a world-model debugger, the evaluation protocol, and every policy and log.
Sources
- Diffusion Models Are Real-Time Game Engines
- Genie: Generative Interactive Environments
- Diffusion for World Modeling: Visual Details Matter in Atari
- Mastering Diverse Domains through World Models
- Transformers are Sample-Efficient World Models
- PufferLib: Making Reinforcement Learning Libraries and Environments Play Nice
- Dream to Control: Learning Behaviors by Latent Imagination
- When to Trust Your Model: Model-Based Policy Optimization
- Self-Correcting Models for Model-Based Reinforcement Learning
- World Models
- Recurrent World Models Facilitate Policy Evolution
- Objective Mismatch in Model-based Reinforcement Learning
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
- MOPO: Model-based Offline Policy Optimization
- MOReL : Model-Based Offline Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- Training language models to follow instructions with human feedback
- Dota 2 with Large Scale Deep Reinforcement Learning
- Deep Reinforcement Learning that Matters
- Deep Reinforcement Learning at the Edge of the Statistical Precipice
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems