WorldCast: Distributed Multiplayer World Models

arXiv:2610.12412 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "WorldCast: Distributed Multiplayer World Models".

Jane: The gist The WorldCast system is a distributed multiplayer world model that allows each player to run an independent client generating their own view while sharing only a world state,…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’ve looked at what "WorldCast: Distributed Multiplayer World Models" is all about—basically, moving away from systems where everyone has to talk to the same central brain and instead letting everyone handle their own view based on shared updates.

Jane: The thesis they are pushing is that you can generate independently controlled views while still making sure players look consistent across those different viewpoints.

Lu: They claim this works by having each player run a local client with its own video generator and a state model, and these clients share only the world state.

Meng: What’s the core mechanism they use for sharing that state? It seems like it's not just sending raw video frames around, but structured information.

Tom: Right. They are using player state fields, which project other players’ states onto each client’s camera-aligned token grid so the client knows exactly where and how to render others.

Jane: And they also use a scene state memory bank where clients can reuse generated observations of the environment, which helps maintain visual consistency across views over time.

Lu: The state model estimates player positions and scene depth from the video and controls by using a motion head for displacement prediction, a place head for absolute position on the map cell, and a depth head for coarse depth maps.

Meng: That’s quite a bit of estimation happening locally on each client during inference to build that shared state information. How do you manage the complexity there?

Tom: The training process conditions the video generation on this shared world state by minimizing a weighted flow-matching loss, increasing the weight when players are visible in those specific positions.

Jane: The results show that this distributed setup improves player rendering rates by over an order of magnitude compared to methods that generate everything together.

Lu: And they also showed it improves whole-round visual consistency on Counter-Strike two keeping image quality stable even over hour-long rollouts <ref:2610.12412#pg1>.

Meng: So, the paper is making a case for scaling up multiplayer experiences by distributing the computational load across multiple players' machines rather than bottlenecking it in one place.

Tom: Exactly. It’s about making generation scalable and consistent by keeping the heavy lifting local to each player’s machine while only exchanging necessary state information.

Jane: And that means they are achieving real-time generation without a centralized computational bottleneck, which is a major practical win for multiplayer systems.

Conclusion: Tom: So, looking at "WorldCast: Distributed Multiplayer World Models," the authors are really focused on showing how you can achieve consistent multiplayer views through this distributed generation framework.

Jane: They tackle the problem of coordinating multiple players in a shared environment by making each player responsible for its own view generation based on local and shared state information.

Lu: The key contribution is using camera-aligned player state fields to guide rendering, alongside a scene state memory bank to maintain visual coherence across different views.

Meng: For practical impact, this suggests that we can build multiplayer experiences that scale better because the computational cost doesn't just grow with the number of players in a centralized manner.

Tom: It’s about having independent clients exchange only player and scene states once per block, which means real-time generation is possible even with twenty or sixteen players.

Jane: And they’ve demonstrated that this setup keeps image quality stable over hour-long rollouts during tests on Counter-Strike two <ref:2610.12412#pg1>.

Lu: The paper suggests that distributed generation methods are a viable path forward for large multiplayer simulations where computational resources might be spread across many machines.

Meng: It moves the focus from centralized coordination to a system where local computation and state sharing drive performance and scalability.

Tom: That’s the essence of WorldCast—it’s about achieving consistent, scalable generation through localized decision-making and careful state management.

Ziyang Ye, Junchao Huang, Evelyn Zhang, Zhihao Xie, Ruicheng Zhang, Boyao Han, Litao Ban, Ziye Wang

CUHK-Shenzhen (The Chinese University of Hong Kong, Shenzhen) · SLAI (School of AI and Information Sciences) · Tsinghua SIGS (School of Information Science and Technology) · Voyager Research · Didi Chuxing · USTC

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 31 pages, 19 figures, 20 tables. Project page: https://ziyang-ye.github.io/WorldCast-Page

Code: https://github.com/ultralytics/ultralytics

Project page: https://ziyang-ye.github.io/WorldCast-Page

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The WorldCast system is a distributed multiplayer world model that allows each player to run an independent client generating their own view while sharing only a world state, achieving

Key concepts

Player State
This record tracks each player's position and attributes like facing direction, controls, and weapon at specific points in time. The system estimates the player's exact location from the generated video and control inputs using a specialized state model. This state is then projected into a camera-aligned field to tell other players exactly where they should be rendered.
Camera-Aligned Player State Field
This is a spatial guide constructed by projecting other players' states onto the local client's camera grid for every frame. It explicitly encodes information about coverage, relative depth, and facing direction. This field provides precise instructions to the renderer on where and how to draw other players in the scene.
Scene State
This component stores generated observations of the environment in a shared memory bank. Clients can reuse these stored latent frames from other clients' work to maintain a consistent visual appearance across different views. This allows for better scene consistency over longer periods, even when intermediate features are kept local.

Terminology

Summary

The gist The WorldCast system is a distributed multiplayer world model that allows each player to run an independent client generating their own view while sharing only a world state, achieving real-time generation and cross-view consistency through camera-aligned player state fields and shared scene memory banks

How it works

WorldCast separates shared world state from local rendering by assigning each of the P players an independent client that reads shared world state, generates its own view, and publishes updated state once per block This distributed generation framework involves each client running on its own GPU and combining a video generator Gθ with a state model that estimates position and scene depth from generated video and controls Consistent views are maintained through two complementary components of the shared world state: player state, which records players’ positions and attributes projected into a camera-aligned player state field that guides the rendering of other players Scene state stores generated observations in a shared memory bank, supplying content missing from each client’s recent context Clients exchange these states once per video block while keeping intermediate features local

Player State and Scene State

The player state contains one record s i n for each player i at block n, describing its position and other attributes, such as facing direction, controls, and weapon The position is estimated from generated video and controls by the state model (Sec. 3.4), while other attributes are obtained from the player’s inputs Camera-aligned player state field F i n,f is constructed by projecting other players' states onto client i’s camera-aligned token grid for each latent frame f This field encodes coverage, camera-relative depth, and facing direction, providing explicit spatial guidance about where other players should appear and which attributes to render Scene state composition allows clients to reuse generated observations of the environment, helping maintain consistent scene appearance across views through a memory bank where each entry stores a block of latent frames generated by one client

State Model and Memory Retrieval

The state model estimates player position and scene depth from generated latents and controls, enabling clients to update the shared world state without a game engine supplying these quantities at inference The state model architecture includes a motion head that predicts the player’s displacement between consecutive latent frames, a place head that provides an absolute position estimate on the current map by classifying the frame into a map cell and predicting its offset within the cell, and a depth head that predicts a coarse depth map from generated latents Before generating block n, client i retrieves the memory entry M i n that best covers scene content missing from its recent context by projecting points of all memory entries into the same cameras and defining reference surface depths The retrieval process bounds the generator’s input at twenty latent frames (four memory frames, twelve recent context frames, four target frames) for any number of clients

Training and Evaluation

WorldCast is trained in four stages, with Stage 2 conditioning Gθ on the shared world state, adding the player state field and scene state The loss function L minimizes weighted flow-matching loss where the foreground weight α k increases the loss at positions occupied by visible players In inference, each client executes the generation cycle of Sec. 3.1 with a four-step generator, which is distilled to a four-step generator by distribution matching trained on its own rollouts to reduce accumulated error Experiments on Counter-Strike 2 demonstrate WorldCast’s consistency, real-time per rendering rates over an order of magnitude higher than joint-generation methods

Performance and Consistency

WorldCast achieves a player rendering rate over ten times that of the joint-generation baseline, reaching 0.817 compared with 0.056 for Solaris and 0.074 for frame concatenation Shared scene state further improves visual consistency over whole rounds In closed-loop deployment, the agreement between clients improves after 30 s, with WorldCast showing a gain in Mat. Pix., Inliers, and RotErr compared to scene state off at inference The state model’s position error on generated video grows with time (Appendix G) Scene state significantly raises the agreement between clients after 30 s, with WorldCast showing a gain in Mat. Pix. by +20.1K and Inliers by +5.

Improvements for AI systems

  1. The improved system can generate independently controlled views for multiple players in real-time, achieving a player rendering rate over an order of magnitude higher than joint generation methods. This is accomplished by having each player runs a local client comprising a video generator and a state model that exchanges only shared state, enabling scalable multiplayer generation without a centralized computational bottleneck.

  2. The system can maintain cross-view consistency across hours of rollouts, as the shared scene state improves visual consistency over whole rounds, which is achieved by allowing clients to reuse one another’s generated observations to maintain consistent scene appearance across views.

  3. The system can provide accurate player placement in generated video, as the camera-aligned player state field improves player rendering rates by over an order of magnitude, and the state model estimates its own position and scene depth from generated video and controls to correct control-based extrapolation.

  4. The system can be deployed in a closed-loop manner where clients advance in lockstep, as each client runs on its own GPU and clients exchange only player state, allowing for concurrent execution of multiple players without requiring a centralized engine supplying positions and geometry at inference.

  5. The system can utilize memory retrieval to maintain scene appearance over long durations, as the scene state stores generated observations in a shared memory bank, supplying content missing from each client’s recent context, and clients retrieve the entry that best covers scene content missing from its recent context.

Sources

Related papers