Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction

The gist The Multi-agent Egocentric World Model (ME-World) jointly generates synchronized ego streams for multiple agents interacting through fine-grained actions in a shared world, improving shared-world consistency, action control, identity preservation, and video quality over existing methods

How it works

The core of ME-World is formulating multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared environment consistency, and consistent propagation of interaction-induced state updates. The model jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents’ target-view poses, and grounds generation with shared environment memory.

Joint Multi-Agent Generation

Since agents interact within the same environment, their ego streams are not independent. The model generates all streams jointly within a single pretrained video diffusion transformer (DiT) (Ali et al., 2025) as shown in Fig. 3 (b). This involves jointly denoising their latent sequences under the conditioning C, which allows cross-stream information exchange. The model applies identical rotary positional embeddings (RoPE) to all streams and distinguishes them by adding a learnable agent embedding e(n) to the tokens of n-th agent.

Shared Action Conditioning

To ensure consistency in cross-view actions, ME-World uses shared action conditioning, guiding each ego stream with all agent’s motion rather than by its own action alone. For each n-th agent, the action at frame t is decomposed into body and head motion as a(n)t = a(n)t,body, a(n)t,head. The body motion a(n)t,body is encoded as a self-action skeleton map capturing egovisible hand and arm motion. The head motion is represented with Plucker ray maps as X(n)ray = a(n)t,head. This unified pose condition represents both n-th agent’s own motion and the visible motions of other agents in n-th agent’s ego view.

Shared Environment Memory

To provide a common reference for scene layout, background appearance, and object state, ME-World introduces shared environment memory from the shared observation-history pool E. This is achieved through two paths:

1. Stream-Aligned Geometric Memory: This path transforms relevant observations in E into target-view conditions for each ego stream by warping history images according to the target ray condition X(n)ray. It uses SAM3 (Carion et al., 2026) to mask dynamic agent regions to avoid transferring outdated poses, yielding a warped RGB condition X(n)warp and a visibility mask M(n)vis.

2. Cross-Stream Anchor Memory: This path preserves selected history images from E in their original, unwarped form as shared scene references. It greedily selects K history observations to maximize the target-view regions not covered by previously selected anchors, allowing all ego streams to attend to the same anchors during generation.

Evaluation Protocol

The model is evaluated using shared-world consistency metrics, including Senv, Supdate, and Sid.

- Senv measures environment consistency only in regions not directly provided by geometric memory by computing DINOv3 feature similarity between these regions and corresponding regions in X(2)gen defined by co-visible correspondence. It is calculated as Senv = 1/2 Avg(p,q)∈C(n→m) env s(n→m)(p, q). If M(n)vis = 0, the region is not covered by warped history guidance.

- Supdate measures whether interaction-induced state changes are reflected consistently across agent views by computing DINOv3 feature similarity between these updated regions and the corresponding regions in X(2)gen defined by co-visible correspondences. It is calculated as Supdate = 1/2 Avg(p,q)∈C(n→m) update s(n→m)(p, q). If M(n)vis = 1 and M(n)update = 1, the region is updated from history.

- Sid evaluates identity consistency by comparing the DINOv3 feature of the detected interaction partner against multi-view reference images of the corresponding ground-truth character. It measures whether different ego streams depict the same counterpart in the shared world rather than drifting to inconsistent character identities. The final score is averaged over valid frames and both ego streams.

Performance Summary

ME-World consistently improves shared-world consistency, action control, identity preservation, and video quality on real and synthetic benchmarks. On the real benchmark, ME-World achieves 0.468 for Senv ↑ 0.466 for Supdate ↑ 0.881 for Other-Agent Action Control ↑ 0.656 for Video Quality ↑. On the synthetic benchmark, ME-World achieves a Senv of 0.433 and Supdate of 0.561. The ablation study validates that joint generation couples the predictions, shared action conditioning aligns embodied behavior, and shared environment memory anchors both streams to a common evolving world.

Future Directions

The framework offers directions for extension where jointly modeling actions and observations could enable agents to plan behaviors and anticipate their effects on the shared world. Efficient cross-agent communication and memory could support larger groups. Combined with efficient video generation, these extensions could enable real-time, interactive multi-agent world simulation. The model extends naturally to a larger number of agents without modifying the underlying architecture by simply appending an additional ego stream and its corresponding agent token. Autoregressive generation can be extended along the temporal dimension by autoregressively chaining multiple generation chunks to produce a 221-frame two-agent sequence while maintaining the interacting agents and shared environment over the extended horizon. The model is trained using rectified flow with an objective LFM = EZ0,ϵ,τ,C h fθ Z joint τ, τ, C − (ϵ gen − Z gen 0) 2/2 i.

Conclusion

ME-World introduces multi-agent egocentric world modeling where multiple agents interact in a shared world through their own egocentric viewpoints. The framework provides a method to generate multiple ego streams with shared action conditioning and environment memory, which improves shared-world consistency and fine-grained action control while maintaining strong video quality. This work advances world models toward simulating multiple embodied agents in a persistent and evolving world.

References

Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao et al. World simulation with video foundation models for physical ai. Yutong Bai, Danny Tran et al. Whole-body conditioned egocentric video prediction. Philip J. Ball et al. Egocontrol: Controllable egocentric video generation via 3d full-body poses. Hongjin Chen et al. Rhythm: Learning interactive whole-body control for dual humanoids. Kristen Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video. Dima Damen et al. Scaling egocentric vision: the epic-kitchens dataset. Qiao Gu et al. E3C: Video generation with 3d environmental memory and ego-exo human pose control. Alexey Gavryushin et al. Comind: Understanding collaborative human activity from multiple minds and views. Zexuan Yan et al. Egogenesis: Egocentric world-action modeling with online anchored projective memory and action-3d rope. Oriane Simeoni et al. Dinov3. DreamX Team et al. Dreamx-world 1.0: A general-purpose interactive world model. Enigma team. Introducing multiverse: The first ai multiplayer world model. Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all. NVIDIA et al. Cosmos world foundation model platform for physical ai. Poly Haven. Scene files. Habitat 3.0: A co-habitat for humans, avatars and robots. Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all. Lingni Ma et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. Sicheng Mo et al.

1. The ME-World model can generate synchronized ego streams that remain consistent in character identity, scene appearance, and interaction outcomes across agents, which allows for more realistic multi-agent simulations compared to existing methods.

2. ME-World achieves consistency by coupling generation, action conditioning, and environment evidence across agents, specifically through joint multi-agent generation (Sec. 3.2), shared action conditioning (Sec. 3.3), and grounded with shared environment memory (Sec. 3.4).

3. The system can maintain fine-grained embodied interaction consistency because it introduces shared action conditioning, guiding each ego stream with all agent’s motion rather than by its own action alone, which is important for finegrained embodied interaction, where hands, arms, body pose, and gaze-induced viewpoint changes must remain consistent across ego streams.

4. ME-World can accurately model shared environment dynamics by using a stream-aligned geometric memory path that transforms relevant observations into target-view conditions for each ego stream, providing geometry-aligned evidence for the frames being generated.

5. The system can assess its performance rigorously using novel metrics: Senv to measure environment consistency only in regions not directly provided by geometric memory and Supdate to measure whether interaction-induced state changes are reflected consistently across views.

6. The ME-World framework can preserve identity integrity through the Sid metric, which compares the generated partner's DINOv3 features against multi-view reference images of the corresponding groundtruth character to ensure different ego streams depict the same counterpart in the shared world.

7. The model's training can be made more robust by perturbing conditional latents during training by randomly conditioning on one or two frames, applying Gaussian noise with standard deviation 0.1 with probability 0.5 when two frames are used, to improve robustness to imperfect generated context.

8. The framework can be extended to larger groups of agents without architectural modification; for instance, adding a third agent simply involves appending an additional ego stream and projecting its poses into each ego view in the same manner as in the two-agent setting.

Egocentric world models predict first-person observations conditioned on an agent's actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose Multi-agent Egocentric World Model (ME-World), which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents' target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show ME-World improves shared-world consistency, action control, identity preservation, and video quality over existing methods.