AgentGarten: Code Worlds for Evolving Agents

arXiv:2610.12374 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "AgentGarten: Code Worlds for Evolving Agents".

Tom: The gist The framework introduces AgentGarten,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at AgentGarten today. This paper tackles that big problem of making interactive virtual worlds that are both faithful to the rules and realistic visually for agents. It claims they can achieve this by coupling simulators or game engines with a shared neural renderer one <ref:2610.12374#pg1,game engines with a shared neural renderer>.

Jane: Exactly. The core idea is that agents learn best through exploration and interaction, but what they learn is totally limited by the environments they practice in, which have to be consistent in state and rules, and realistic in how observations look compared to the real world one <ref:2610.12374#pg1,by the environments they practice in, which>.

Lu: AgentGarten addresses that bottleneck by coupling simulators or game engines with a shared neural renderer one <ref:2610.12374#pg1,game engines with a shared neural renderer>. The simulation backends keep track of a persistent world state and run program-defined interaction rules, while the renderer creates visual observations from structured conditions exported through a common interface.

Meng: So it separates the control—the state and the rules—from the visuals. That way, you get rigorous control over what's happening in the simulation without having to build a custom visual pipeline for every single scene one <ref:2610.12374#pg1>.

Lalam: And then they build this neural renderer by adapting a pretrained video model to geometry conditions, distilling it using something called Adversarial Forcing, and optimizing it for real-time interaction one <ref:2610.12374#pg1,a pretrained video model to geometry conditions>.

Tom: Adversarial Forcing is the trick there. It makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and it adds real-data adversarial supervision to improve its visual quality one <ref:2610.12374#pg1,makes history prefilling differentiable through exact replay, so that losses on later>.

Jane: It sounds complicated, but what it means is that as the agent interacts and sees more stuff, the visual representation gets better because it’s being trained against real data during that process.

Lu: The paper defines a code world as combining a scene program p, an engine that runs it, and this neural renderer R theta one <ref:2610.12374#pg1>. The engine updates the state and captures a structured condition ct from the camera, which is something like surface normals or depth or reference text one <ref:2610.12374#pg1>.

Tom: So you have this scene program defining the rules, and then the engine executes those rules to get a new state plus that structured condition. That's how they capture what's happening in the world at any given moment one <ref:2610.12374#pg1>.

Meng: And then the neural renderer R theta takes that structured condition along with some cached visual history to generate new observations, which are fed back to the agent one <ref:2610.12374#pg1>.

Jane: It’s a loop: actions lead to state updates, which lead to new observations, and those observations inform the next action. This whole setup is designed for real-time interaction where agents get immediate feedback one <ref:2610.12374#pg1>.

Tom: The empirical study shows that pretrained foundation model agents observing only through this neural rendering can rapidly learn semantic tool knowledge and improve outcomes across rounds by using written playbooks in things like hide-and-seek one <ref:2610.12374#pg1>.

Lu: In those examples, the agents perceive the world strictly through rendered observations, diagnose failures, and distill those insights into written playbooks that subsequent agents inherit and refine one <ref:2610.12374#pg1>.

Jane: It shows that these evolving agents don't just learn from trial and error; they learn to write down their strategies based on what they see visually one <ref:2610.12374#pg1>.

Tom: The paper evaluates code worlds across different appearances, like Halloween or Medieval Minecraft, and tasks like navigation or tool use one <ref:2610.12374#pg1>. The agents develop increasingly sophisticated strategies over rounds of interaction, such as building shelters or crossing ramps in those games one <ref:2610.12374#pg1>.

Meng: From an engineering standpoint, the ability to write new worlds as code and render them through the same interface means you can scale up both the number of environments and the difficulty alongside your agents one <ref:2610.12374#pg1>.

Jane: It suggests a future where agents keep evolving continuously through this loop of interaction and refinement rather than just learning in isolation one <ref:2610.12374#pg1>.

Tom: The title, "AgentGarten: Code Worlds for Evolving Agents," really captures what's going on here. It points to the idea that environments are no longer static; they are dynamic things that agents can interact with and change over time one <ref:2610.12374#pg1>.

Lu: And the authors point out that what an agent can learn is strictly bounded by the environment it practices in, which has to be faithful, with consistent state, rules, and dynamics one <ref:2610.12374#pg1,be faithful, with consistent state, rules, and dynamics>.

Jane: So AgentGarten gives us a way to build environments that are both faithful and realistic by dividing the work between a simulator or game engine and that shared neural renderer one <ref:2610.12374#pg1>.

Tom: It's about moving toward systems where new worlds can be written as code and rendered through the same interface, which allows those environments to grow in number and difficulty alongside the agents practicing in them one <ref:2610.12374#pg1>.

Conclusion: Tom: So we've been looking at AgentGarten today, and now we’re hitting the conclusion to see what all this actually means for agents moving forward.

Jane: It boils down to a framework that couples simulators or game engines with a shared neural renderer.

Lu: They’re trying to solve that problem of making interactive virtual worlds both faithful to the rules and realistic visually.

Meng: The paper shows they did this by separating the simulation backends—which keep track of state and rules—from the renderer, which generates visual observations from a common interface.

Lalam: And they built this neural renderer by adapting a video model, using something called Adversarial Forcing to distill it into a real-time system.

Tom: So the main point is that agents can now evolve through experience because you can write new worlds as code and render them through the same interface.

Jane: That lets environments grow in number and difficulty right along with the agents that practice in them.

Lu: It really suggests a future where we don't just have static environments, but these things that keep evolving through interaction.

Meng: From an engineering standpoint, it’s about moving past building custom visual assets for every single scene.

Lalam: The core contribution is this framework for code worlds that links persistent state to a shared real-time neural renderer through structured conditions.

Tom: So what does this mean if we think about the agents themselves? What they can learn is bounded by the environments they practice in, which must be consistent and realistic.

Jane: AgentGarten builds those environments so they are both faithful and realistic by dividing the work between a simulator or game engine and that shared neural renderer.

Lu: It's about moving toward systems where new worlds can be written as code and rendered through the same interface, which allows those environments to grow in number and difficulty alongside the agents practicing in them.

Meng: We’re seeing agents improve by distilling each round of experience into playbooks that subsequent agents inherit and refine.

Lalam: That loop—where they perceive, interact, diagnose failures, and distill insights—is what drives the evolution we're talking about here.

Tom: It’s a step toward agents that keep evolving through interactive experience instead of just learning in isolation.

Jane: And the authors are pointing toward this idea that environments can scale in both number and difficulty alongside their agents.

Lu: That’s a really interesting direction, connecting programmable dynamics with a shared real-time neural renderer.

Meng: It makes the whole interaction loop much more robust when you can code the world itself.

Lalam: We're expecting agents that keep evolving through interaction to come from that loop, and it's not just about playing a game anymore.

MirroS Technical Report

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-10

Comments: Project page: https://mirros-lab.github.io/agent-garten

Code: https://github.com/MirroS-Lab/AgentGarten

Project page: https://mirros-lab.github.io/agent-garten

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The framework introduces AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments It demonstrates a

Key concepts

AgentGarten
A framework that connects simulators or game engines with a shared neural renderer. It allows persistent world state and program-defined rules from the simulation side, while the renderer generates visual observations from structured conditions, enabling real-time interaction across different engines.
Neural Renderer Adaptation
Adapting a pretrained video model to render geometry conditions for interactive use. This involves converting it to block-causal generation and distilling it using Adversarial Forcing. This technique makes history prefilling differentiable, allowing losses on later predictions to improve how the renderer encodes prior observations, boosting visual quality.
Code World Formulation
A system combining a scene program (p), an execution engine that updates state and captures structured conditions (ct), and a neural renderer (R theta). The engine handles physics/logic, while the renderer uses these conditions to generate subsequent visual observations based on past history.
Adversarial Forcing
A distillation method used to train a few-step, block-causal neural renderer. It combines distribution matching and exact replay with a real-data adversarial objective. This process distills the teacher-forced model into a fast renderer trained on its own rollouts, improving visual fidelity.

Terminology

Summary

The gist The framework introduces AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments It demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared to millions for a conventional reinforcement learning counterpart

Framework Overview

AgentGarten is designed to address the bottleneck of achieving both faithful and realistic environments for agents by coupling simulators or game engines with a shared neural renderer The simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface This separation preserves rigorous control over state and rules while allowing compatible engines to share a single renderer New environments can thus be authored by coding agents from text or image inputs, then edited and extended entirely in code without creating bespoke visual assets for every scene

Neural Renderer Adaptation

Operating a video model as an interactive neural renderer requires it to strictly follow updated engine conditions, maintain visual consistency across extended rollouts, and synthesize observations in finegrained, short blocks so that agents receive immediate feedback after brief actions The team adapted a pretrained bidirectional video model to structured conditions, converted it to block-causal generation, and distilled it into a real-time renderer on a single GPU with Adversarial Forcing Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality

Code World Formulation

A code world combines a scene program p, an engine that executes it, and a neural renderer R theta The engine updates the state and captures a structured condition ct from the camera: (st+1, pi t+1) = fp(st, pi t, at), ct = hp(st, pi t), (1) where fp implements the state transition and hp renders the visible geometry The neural renderer generates subsequent observations from visual history x<t: xt = R theta(x<t, ct≤t, x0, y), t ≥ 1 (2)

Training and Distillation

The training of the neural renderer involves three stages:

  1. Geometry conditioning: This uses condition tokens derived from colorized depth or surface normals to condition the model

  2. Teacher-forcing adaptation: This partitions latents into fixed-length blocks and uses parallel teacher forcing to train the block-causal model

  3. Adversarial Forcing: This stage distills the teacher-forced model into a few-step renderer that is trained on its own rollouts, combining distribution matching, exact replay, and a real-data adversarial objective

Empirical Study of Evolving Agents

The empirical study demonstrates that pretrained foundation-model agents observing exclusively through neural rendering rapidly ground semantic tool knowledge and improve outcomes across rounds through written playbooks in hide-and-seek and several further distinct worlds The agents perceive the world strictly through rendered observations, diagnose failures, and distill key insights into written playbooks inherited by subsequent rounds In hide-and-seek, agents rapidly grounded abstract spatial knowledge into closed-loop physical execution, with ramp use and shelter construction emerging within a handful of games

Code World Examples

The paper evaluates code worlds across diverse appearances, tasks, and agent evolution Examples include diverse geometries like Halloween, Medieval Minecraft, and diverse tasks such as navigation, manipulation tool use, multi-agent racing from each player's camera The agents develop increasingly sophisticated strategies such as shelter-building and ramp-crossing over rounds of interaction

Contributions

The main contributions include:

**/A framework for code worlds that couples persistent, editable environment state to a shared real-time neural renderer through structured conditions, enabling programmable interaction and diverse visual observations across compatible engines without per-scene visual asset authoring 2. Adversarial Forcing, a distillation method for a few-step, block-causal neural renderer 3. An empirical study of evolving agents The authors conclude that agents that keep evolving through interaction to come from that loop 8 Conclusion, what an agent can learn is bounded by the environment it practices in, and AgentGarten builds environments that are both faithful and realistic by dividing the work between a simulator or game engine and a shared neural renderer 17 Conclusion, because a new world is written as code and rendered through the same interface, environments can grow in number and difficulty alongside the agents that practice in them 8 Conclusion, we expect agents that keep evolving through interaction to come from that loop >.">The gist The framework introduces AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments It demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared to millions for a conventional reinforcement learning counterpart The framework is significant because it enables agents to evolve through interactive experience by combining programmable dynamics with a shared real-time neural renderer 8 Conclusion, what an agent can learn is bounded by the environment it practices in 8 Conclusion, because a new world is written as code and rendered through the same interface, environments can grow in number and difficulty alongside the agents that practice in them 8 Conclusion, we expect agents that keep evolving through interaction to come from that loop >.">The gist The framework introduces AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments It demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared to millions for a conventional reinforcement learning counterpart The framework is significant because it enables agents to evolve through interactive experience by combining programmable dynamics with a shared real-time neural renderer 8 Conclusion, what an agent can learn is bounded by the environment it practices in 8 Conclusion, because a new world is written as code and rendered through the same interface, environments can grow in number and difficulty alongside the agents that practice in them 8 Conclusion, we expect agents that keep evolving through interaction to come from that loop >.">The gist The framework introduces AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments It demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared to millions for a conventional reinforcement learning counterpart The framework is significant because it enables agents to evolve through interactive experience by combining programmable dynamics with a shared real-time neural renderer 8 Conclusion, what an agent can learn is bounded by the environment it practices in 8 Conclusion, because a new world is written as code and rendered through the same interface, environments can grow in number and difficulty alongside the agents that practice in them 8 Conclusion, we expect agents that keep evolving through interaction to come from that loop >.

Improvements for AI systems

  1. Development of code worlds that couple persistent, editable environment state to a shared real-time neural renderer through structured conditions. This allows environments to be authored by coding agents from text or image inputs, then edited and extended entirely in code without creating bespoke visual assets for every scene.

  2. Implementation of Adversarial Forcing, which is described as a distillation method that combines distribution matching on self-rollouts with exact bitwise replay for history gradients, alongside a real-data adversarial objective. This improves the neural renderer by allowing it to synthesize observations in finegrained, short blocks so that agents receive immediate feedback after brief actions.

  3. Creation of agents that improve from their own experience: they play, review each round, and write playbooks that later agents inherit and build upon. This enables agents with general priors to ground abstract knowledge into closed-loop physical execution in environments like hide-and-seek, where strategies like shelter construction and ramp crossing emerging within a handful of games.

  4. A framework for scaling learning by allowing new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents. This moves beyond fixed environments to systems where a world is a program that allows for continuous evolution driven by agent-written playbooks.

  5. Evaluation of agent performance across diverse tasks, demonstrating that pretrained agents rapidly ground semantic tool knowledge and improve outcomes across rounds through written playbooks in hide-and-seek and several further distinct worlds. This confirms the efficacy of the framework for transferring abstract knowledge into physical execution without needing tabula rasa exploration.

  6. Enhancement of rendering fidelity via exact R1/R2 regularization, which avoids double-backward through fused attention kernels by computing gradients using a JVP and a surrogate, ensuring that the parameter gradients update the head using the full discriminator objective. This maintains visual quality over long rollouts by preventing degradation seen in methods relying solely on score distillation.

  7. Implementation of streaming inference techniques, utilizing custom Triton kernels to fuse operations around attention and a tiny decoder to reconstruct pixel frames in under 10 ms, achieving a throughput of over 35 frames per second. This enables the system to operate in real-time for interactive deployments with low-latency execution.

Sources

Related papers