WorldComposer: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation

arXiv:2604.15805 · cs.RO, cs.AI · Submitted 2026-04-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "WorldComposer: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation".

Title and authors: Rosa: Let's talk about the paper "WorldComposer: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation" and who came up with it. The authors are Jasper Lu, Zhenhao Shen, Yuanfei Wang, Shugao Liu, Shengqiang Xu, Shawn Xie, Jingkai Xu, Feng Jiang and Jade Yang.

Dev: They're from Peking University. It sounds like a team of researchers from different sides—robotics and simulation modeling—bringing together different skills to solve this data scarcity issue.

Taro: I'm curious about the title itself: "WorldComposer: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation." What does that really mean in plain language?

Rosa: It means they are generating high-fidelity simulations directly from real panoramas, and then using prompt-based editing to create diverse cousin scenes, all for robot learning and evaluation.

Dev: So the focus isn't just on making one perfect simulation, but on creating a whole family of slightly different simulations that help the robot learn more robustly.

Taro: That implies they are trying to make the simulation space much richer than what we typically get from just doing simple domain randomization in existing simulators.

Rosa: Right. They are moving beyond just making one static digital replica and creating an entire distribution of realistic, editable environments that cover a lot of ground for policy testing.

Dev: It sounds like they are trying to create a scalable data engine, which is something we've talked about before when discussing how hard it is to get enough real-world data for large foundation models.

Taro: So the implication here is that if you can generate diverse, editable simulation data this easily from existing real-world captures, then scaling up your policy training data becomes much less expensive and more effective.

Rosa: Precisely. They are showing that extensive scaling of data generation leads to significantly better generalization for the robotic policies being trained on these generated scenes.

The paper's summary: Dev: So, let's look at what the paper summarizes in detail about WorldComposer. It outlines a coarse-to-fine generation strategy, starting from a panorama to a static "Digital Twin," and then generating diverse "Digital Cousins" through prompt editing.

Rosa: The process starts with Marble, which is used to reconstruct the scene geometry and texture from the panorama, producing both three dee Gaussian Splats for rendering and a collision mesh for physics interaction <ref:2604.15805#pg3>.

Taro: So that's the first stage—getting the base structure down into a format usable by simulators like Isaac Sim through USD conversion.

Dev: After getting that twin, they use Marble again to generate cousins by using natural language prompts to modify the appearance and semantic layout of the scene while keeping the structural logic intact.

Rosa: That's how they get those diverse cousin scenes, environments that look different but still make sense structurally, which is a huge part of their contribution.

Taro: And they tackle a real-world challenge by stitching multiple room-level scenes together into a house-scale environment using panoramic feature matching for coarse alignment and then fine refinement with geometric ICP.

Dev: So the whole flow is from raw panorama to twin, then to diverse cousins via editing, and finally stitching those rooms together for navigation tests.

Rosa: And they don't stop there; they add a high-fidelity asset library including rigid objects, articulated objects like garments, and deformable objects like soft foods or fluids.

Taro: So it’s not just about the room layout; it’s about adding complex physical interactions so the robot can perform tasks like folding cloth or pouring water in these simulated environments.

Dev: And they use a language model to provide common-sense placement priors when populating those cousins with assets, which adds another layer of realism.

The paper's improvements: Rosa: Now let's look at what the authors suggest as improvements or key findings that make this work stand out. They focus heavily on how these diverse data augmentations actually help the robot policies generalize better.

Dev: The main improvement they highlight is that adding augmented data, specifically the Digital Cousin data, significantly boosts performance across all categories for generalizable robotic manipulation policies.

Taro: That's a strong claim because it suggests that just having more simulation data isn't always enough; the *type* of variation matters a lot.

Rosa: They show that a policy trained only on fifty twin scenes achieves generalization results similar to one trained on fifty real-world scenes, which is a major comparison point <ref:2604.15805#pg2>.

Dev: That implies that extensive scaling up data generation through these Digital Cousins provides the critical environmental fidelity and diversity needed to bridge the data scarcity gap for policies.

Taro: I also noticed they show a strong positive linear relationship between simulation and real-world success rates across three representative tasks, with a Pearson correlation coefficient of zero point nine one.

Rosa: That zero point nine one correlation is what's really impressive; it shows that their simulation evaluation aligns very well with the real world.

Dev: So, the improvement isn't just about creating pretty simulations; it’s about making those simulations a reliable proxy for real-world interaction, which helps us test policies before they ever hit the physical robot.

Conclusion: Rosa: To wrap up on WorldComposer, it gives us a generative framework that creates high-fidelity interactive simulations with diverse digital cousins from real-world panoramas. The key finding is that this approach improves policy robustness and establishes a strong correlation between simulation and the real world.

Dev: It validates the platform for generalizable robot learning and evaluation because it shows that scaling up data generation through these variations effectively boosts generalization of robotic manipulation policies, as shown by the scaling curves in figure six.

Taro: For me, what's important is that this setup provides a comprehensive evaluation platform for long-horizon tasks because they successfully validated zero-shot object goal navigation in those stitched multi-room environments with a success rate of sixty eight percent.

Rosa: That sixty eight percent success rate in the VLFM policy demonstrates that these stitched environments are physically robust and semantically consistent testbeds, supporting the validation of semantic reasoning and long-horizon navigation.

Dev: Overall, WorldComposer is positioning these generated Digital Cousins as a critical way to bridge the data scarcity gap for training policies by providing rich, diverse data augmentation from real-world captures.

Taro: It really shows that when you generate this kind of high-fidelity diversity—scene and object variations—the policy learns much better than if you just used standard domain randomization or simple texture changes.

Rosa: So we've looked at the WorldComposer framework, which uses generative modeling to create diverse simulation environments from real panoramas, and it provides a strong way to get more generalizable robot policies by scaling up data generation with high-fidelity Digital Cousins.

Peking University

cs.RO, cs.AI

Submitted: 2026-04-17

Updated: 2026-11-18

Code: https://github.com/isaac-sim/IsaacSim

Project page: https://stubborn111.github.io/WorldComposer

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 99/100

The gist: Learning robust robot policies in real-world environments requires diverse data augmentation, yet scaling real-world data collection is costly due to the need to acquiring physical assets and…

Key concepts

Digital Twins
These are initial 3D reconstructions of real-world panoramas created using a multimodal World Model. They capture the original scene's precise geometry and texture, providing the structural foundation for subsequent scene generation.
Digital Cousins
These are diverse variations of the Digital Twin generated by conditioning it on natural language prompts. They retain the original structure but introduce significant changes in visual appearance and semantic layout, creating a broad distribution of training data.
Multi-Room Stitching
This pipeline combines multiple room-level 3D scenes into a single, navigable house-scale environment. It uses feature matching for coarse alignment and geometric ICP for fine refinement to ensure the stitched rooms form a cohesive floor.
High-Fidelity Assets
The system populates generated scenes with realistic objects across three categories: rigid, articulated, and deformable (like cloth or soft foods). This ensures the simulation supports complex physical interactions necessary for robust manipulation tasks.

Terminology

Summary

Learning robust robot policies in real-world environments requires diverse data augmentation, yet scaling real-world data collection is costly due to the need to acquiring physical assets and reconfiguring environments. p. 0 We present a generative framework that establishes a generative real-to-sim mapping from real-world panoramas to high-fidelity simulation scenes, and further synthesizes diverse cousin scenes via semantic and geometric editing. p. 0 Combined with high-quality physics engines and realistic assets, the generated scenes support interactive manipulation tasks. p. 0 Experiments demonstrate that policies trained on our platform generalize better to unseen scenes and objects, showing that extensively scaling up data generation leads to significantly better generalization to unseen scene and object variations. p. 0

Generative Scene Construction

The framework WorldComposer constructs high-fidelity simulation scenes directly from real-world panoramas. This process involves a coarse-to-fine generation strategy, progressing from panoramas to static “Digital Twins”, and then to diverse “Digital Cousins”. p. 2 From Panorama to Digital Twin, the process leverages Marble [26], a multimodal World Model, to reconstruct the scene geometry and texture. p. 2 This reconstruction produces two key components: (i) Visual Rendering, a set of 3D Gaussian Splats that enables photorealistic rendering while capturing complex lighting and material effects; and (ii) Collision Mesh, a corresponding mesh that supports robot-environment interaction by providing rigid bodies for physics simulation. p. 3 To integrate these components into the Isaac Sim simulator, we utilize 3DGRUT to convert the Gaussian PLY data and collision meshes into the Universal Scene Description (USD) format. p. 3

Diverse Data Augmentation

A key limitation of simple real-to-sim Digital Twin scene generation is the lack of diversity, which hinders generalizable robot learning and evaluation. To address this, we leverage the editing capability of Marble by conditioning the model on natural language prompts (e. p. 3g., “a kitchen with wooden textures” or “modern style layout”) to modify the visual appearance and semantic layout of the reconstructed scene. p. 3 This process generates Digital Cousins, environments that retain the structural logic of the original Digital Twin while introducing significant visual and layout variations. p. 3

Multi-Room Composition

Since a 360-degree panorama typically covers only a single room, we need to stitch multiple room-level scenes into a house-scale, navigable environment. To this end, we introduce a robust multi-room stitching pipeline that spatially aligns discrete 3D Gaussian Splatting rooms generated by Marble into a cohesive floor. p. 3 This involves two steps:

  1. Coarse Alignment via Panoramic Feature Matching, which uses SuperPoint [12] and LightGlue [31] to extract local features and decompose the resulting essential matrix into a relative rotation Rab and unit translation tˆab.

  2. Fine Refinement via Geometric ICP, which applies Point-to-Plane Iterative Closest Point (ICP) [7] to minimize the geometric distance between sampled point clouds Pa,Pb.

High-Fidelity Interaction and Evaluation

To enable interactive simulation, we integrate the reconstructed scenes with high-quality assets supported by high-fidelity physics solvers. We curate a comprehensive asset library spanning three categories: Rigid Objects, Articulated Objects, and Deformable Objects. p. 3 For deformable objects, we employ PositionBased Dynamics for diverse garments; the Finite Element Method for elastic volumetric objects like soft foods; and Dynamic Grid Method to simulate fluids. p. 3 The final step involves populating the generated Digital Cousins with these assets using an LLM to provide common-sense, semantically grounded placement priors. p. 3

Generalization and Correlation

The framework serves as both a data engine for generalizable robot learning and a platform for generalization evaluation. We assess the efficacy of our framework by adding diverse generated Digital Cousin data for generalizable robotic manipulation policies. p. 5 Analysis shows that adding augmented data significantly boosts performance across all categories, and the policy trained solely on 50 Sim (Twin) data achieves generalization similar to the policy trained on 50 Real data. p. 5 Furthermore, a strong positive linear relationship is observed between simulation and real-world success rates across three representative tasks, with a Pearson correlation coefficient yielding a high value of r = 0. p. 691.

Navigation Validation

To validate that the stitched environments are suitable for scene-level robotic tasks, we conduct Zero-Shot ObjectGoal Navigation (ZSON) experiments using the movable XLeRobot. We employ VLFM [53], a zeroshot policy noted for its deployment efficiency, which achieves a 68% average success rate in stitched multi-room environments. p. 7 This demonstrates that our stitched multi-room environments provide a physically robust and semantically consistent testbed, effectively supporting the validation of semantic reasoning and long-horizon navigation. p. 7

Conclusion

We presented a generative framework that generates highfidelity, interactive simulations with diverse digital cousins from real-world panoramas. Improved policy robustness and strong real-sim correlation validate the platform for generalizable robot learning and evaluation. p. 7

Future Work

While Marble generates the whole mesh of the scene, we will further explore instance-level decomposition, leveraging 3D semantic segmentation to replace static elements. Additionally, to address texture discontinuities at room junctions, we plan to investigate cross-scene radiance field fusion and end-to-end 3DGS optimization. p. 8

References

[1] Jad Abou-Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Niko Suenderhauf, Maria Vittoria Minniti, and Laura Herlant. Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin. p. 8

[2] Johan Bjorck, Fernando Castaneda, Nikita Cherniadev, ˜Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. p. 8 Gr00t n1: An open foundation model for generalist humanoid robots. p. 8

[3] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. p. 8 pi 0: A vision-language-action flow model for general robot control. p. 8

[4] Chilam Cheang, Sijin Chen, Zhongren Cui, Yingdong Hu, Liqun Huang, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Xiao Ma, Hao Niu. p. 8

[5] Boyuan Chen, Tianyuan Zhang, Haoran Geng, Kiwhan Song. p. 8

[6] Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai. p. 8

[7] Yang Chen and Gerard Medioni. p. 8 Object modelling by registration of multiple range images. p. 8

[8] Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo. p. 8

[9] Cheng Chi, Zhenjia Xu, Siyuan Feng. p. 8

[10] Prithwish Dan, Kushal Kedia, Angela Chao. p. 8

[11] Matt Deitke, Rose Hendrix, Luca Weihs. p. 8

[12] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. p. 8 Superpoint: Self-supervised interest point detection and description. p. 8

[13] Haoran Geng, Feishi Wang, Songlin Wei. p. 8

[14] Xiangyu Guo, Zhanqian Wu, Kaixin Xiong. p. 8

[15] Xiaoshen Han, Minghuan Liu, Yilun Chen. p. 8

[16] Eric Heiden, Miles Macklin, Yashraj Narang. p. 8

[17] Zhiao Huang, Yuanming Hu, Tao Du. p. 8

[18] Physical Intelligence, Kevin Black, Noah Brown. p. 8

[19] Arhan Jain, Mingtong Zhang, Kanav Arora. p. 9

[20] Stephen James, Zicong Ma, David Rovick Arrojo. p. 9

[21] Yufei Jia, Guangyu Wang, Yuhang Dong. p. 9

[22] Guangqi Jiang, Haoran Chang, Ri-Zhao Qiu. p. 9

[23] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler. p. 9

[24] Geonung Kim, Janghyeok Han, Sunghyun Cho. p. 9

[25] Weijie Kong, Qi Tian, Zijian Zhang. p. 9

[26] World Labs. p. 9 Marble, 2025. p. 9

[27] Chengshu Li, Fei Xia, Roberto Mart´ın-Mart´ın. p. 9

[28] Chengshu Li, Ruohan Zhang, Josiah Wong. p. 9

[29] Xinhai Li, Jialin Li, Ziheng Zhang. p. 9

[30] Zeyi Li, Jade Yang, Jingkai Xu. p. 9

[31] Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. p. 9 LightGlue: Local Feature Matching at Light Speed. p. 9

[32] Bo Liu, Yifeng Zhu, Chongkai Gao. p. 9

[33] Haoran Lu, Ruihai Wu, Yitong Li. p. 9

[34] Abhiram Maddukuri, Zhenyu Jiang, Lawrence Yunliang Chen. p. 9

[35] Yao Mu, Tianxing Chen, Zanxin Chen. p. 9

[36] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang. p. 9

[37] NVIDIA. p. 4 Isaac Sim.

[38] NVIDIA,:, Arslan Ali, Junjie Bai. p. 9

[39] OpenAI. Sora, 2024. p. 8

[40] OpenAI. Sora 2, 2025. p. 8

[41] Nicholas Pfaff, Evelyn Fu, Jeremy Binagia.

[42] Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun. p. 8

[43] Xuanchi Ren, Tianchang Shen, Jiahui Huang. p. 8

[44] Johannes Lutz Schonberger and Jan-Michael Frahm. p. 8 Structure-from-motion revisited. p. 3

[45] Daniel Seita, Pete Florence, Jonathan Tompson. p. 8

[46] Mustafa Shukor, Dana Aubakirova, Francesco Capuano. p. 9

[47] Andrew Szot, Alexander Clegg, Eric Undersander. p. 8

[48] GigaWorld Team, Angen Ye, Boyuan Wang. p. 8

[49] Team Wan, Ang Wang, Baole Ai. p. 8

[50] Yian Wang, Juntian Zheng, Zhehuan Chen. p. 9

[51] Yufei Wang, Zhou Xian, Feng Chen. p. 9

[52] Zhou Xian, Bo Zhu, Zhenjia Xu. p. 9

[53] Naoki Yokoyama, Sehoon Ha, Dhruv Batra. p. 9

[54] Tony Z Zhao, Vikash Kumar, Sergey Levine. p. 8

[55] Haoyu Zhen, Qiao Sun, Hongxin Zhang. p. 9

Additional Experiments

Effectiveness over Domain Randomization is shown in Table XIII, where the results indicate that while standard Domain Randomization (Lighting, Texture) yields moderate gains, it cannot alter scene structures. Ablating variation types shows both Cousin (Scene & Object) independently outperform classical DR. p. 1 Integrating them via WorldComposer achieves peak performance, clarifying that our gains stem from scene and object diversity, not sheer data volume or simple texture randomization. Scaling curves on more tasks demonstrate strong, consistent data efficiency, with performance reliably scaling up as simulated cousins increase. p. 0

Detailed Quantitative Evaluation

The high r-value of r = 0.91 demonstrates that our simulation evaluation aligns with the real world. p. 7 This result establishes our system as a high-fidelity, reliable testbed for evaluating generalist policies at scale prior to realworld deployment. p. 7 The scaling trend in Fig. p. 6 6 demonstrates that largescale data with variation generated by WorldComposer can effectively boost generalization of robotic manipulation policy. p. 6 This confirms that “Digital Cousins” provide the critical environmental fidelity and diversity necessary to bridge the data scarcity gap. p. 6

The navigation results in Table IV report a success rate of 68% for VLFM in stitched multi-room environments. This shows that our stitched multi-room environments provide a physically robust and semantically consistent testbed, effectively supporting the validation of semantic reasoning and long-horizon navigation. p. 7

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Make robot policies more robust to unseen scene variations by training on extensively scaled Digital Cousin data, as extensively scaling up data generation leads to significantly better generalization to unseen scene and object variations. This allows the AI system to achieve high success rates in generalization settings by sampling from a broad and effectively unbounded training distribution.

  2. Enable zero-shot navigation in complex environments by constructing large-scale, navigable spaces through multi-room stitching, which provides a comprehensive evaluation platform for long-horizon tasks and supports the validation of semantic reasoning and long-horizon navigation. The AI system can traverse house-scale floorplans to locate interactive assets without prior training.

  3. Improve the reliability of simulation evaluation by leveraging the strong correlation between simulation and real-world performance, as The high r-value demonstrates that our simulation evaluation aligns with the real world, meaning A policy failure in our Digital Cousins, caused by collision or dynamic mismanagement, serves as a reliable predictor of failure in the real world. This provides a high-fidelity proxy for real-world interaction for policy testing.

  4. Enhance manipulation capabilities across diverse physical interactions by populating generated environments with a high-quality asset library covering rigid, articulated, and deformable objects, allowing the system to handle complex dynamics such as Fold Cloth, Pour Water, and Assemble Burger. This enables the robot to perform tasks requiring precise grasping of varied object types.

  5. Improve policy performance on large foundation models by augmenting them with Digital Cousin data, as this augmentation yields positive gains, particularly in the Set Tableware task, proving that our high-fidelity visual generation is beneficial to largescale pre-trained robotic foundation model. This allows foundation models like pi 0 to exhibit improved generalization when trained with diverse generated data.

Sources

Related papers