WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes

arXiv:2605.15843 · cs.CV · Submitted 2026-05-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes".

Jane: Here's the gist of the paper. WorldAct presents a framework that converts monolithic 3D Gaussian Splatting scenes into editable and interaction-ready scenes.

Tom: The resulting scenes support object-level editing, collision-aware manipulation, and embodied task execution while preserving global scene coherence.

Jane: Pruning redundant reasoning tokens The core challenge addressed is that recent 3D world modeling systems typically produce static, monolithic assets with limited editability and physical interaction.

Tom: WorldAct tackles this by using a multimodal agent to guide scene decomposition, identify actionable objects, reconstruct geometrically aligned object-level meshes for interaction, and restore the residual background via 3D inpainting.

Paper summary: Tom: So we’ve gone through the mechanics of how WorldAct takes monolithic three deeGS scenes and turns them into object-centric environments that you can actually interact with <ref:2605.15843#pg1>. The authors are Jichen Hu, Jiawei Guo, Jiazhong Cen, Chen Yang, Sikuang Li, and Wei Shen >

Jane: And they’re arguing that this framework is useful because it solves the problem of static three dee models being too rigid for embodied simulation and content creation > <ref:2605.15843#pg1>

Lu: The core idea is using an agent to guide scene decomposition and then reconstructing those objects as meshes for manipulation while keeping the background consistent through inpainting >

Meng: From my side, I see the practical implication as a tool that lets us move beyond just viewing three dee scenes to actively manipulating them in a simulated environment > <ref:2605.15843#pg1>

Lalam: It’s about moving from static assets to structured scenes where individual objects can be moved and interacted with by an AI system >

Tom: That’s the high level view. The paper shows how they achieved this conversion into editable and interaction-ready scenes while keeping the original visual appearance of the scene intact >

Jane: It depends on a few things, like the quality of your starting three dee world model and whether you're dealing with dynamic scenes or physical properties like mass and friction, which they admit isn't fully handled yet > <ref:2605.15843#pg1>

Conclusion: Tom: So we’re wrapping up this look at WorldAct, which is titled "Activating Monolithic three dee Worlds into Interactive-Ready Object-Centric Scenes." Basically, they took those big, static three dee scenes and made them something you can actually touch and move things in.

Jane: That’s the core idea: taking these massive world representations and turning them into scenes where you can edit individual objects, like picking up a chair or moving a table around.

Lu: The authors are Hu, Guo, Cen, Yang, Li, and Shen. Their main goal is fixing that problem where three dee models are just big blobs that don't let you interact with anything specific.

Meng: So what does this actually *do*? Does it make the world instantly playable for a robot? Because from where I sit shipping production models, I worry about how much manual cleanup goes into making a monolithic scene usable.

Tom: Well, WorldAct uses an agent to figure out which parts of that big scene are actual objects. It looks at the camera views and finds things you can actually grab or move.

Jane: And then it reconstructs those identified objects as proper meshes so a robot can actually grip them, while filling in the empty space around them smartly.

Lalam: The process involves decomposing the scene into object subsets and then using inpainting to fix the background where those objects used to be, keeping everything visually consistent.

Tom: That reconstruction part is key. It’s not just guessing; they use an optimization score guided by a loss function that checks for foreground consistency against the original image data.

Lu: They also have this two-stage alignment process at the end where they use iterative closest point matching and then differentiable rendering to make sure those new objects fit perfectly into the background structure.

Meng: But I gotta ask, how much does this actually change things for someone building a simulation? Is it just a neat visual trick, or can I actually use this to train a robot arm?

Jane: The paper suggests that this lets robotic manipulators do things like grasping and placing objects in ways that were really hard before. It gives them something physical to work with.

Tom: Right, and for content creators, it means you could edit scenes—add or remove objects, change their scale—and keep the whole scene looking right.

Lu: The results show a significant boost in how well the AI can actually recall and identify those objects compared to what they got before using this agent guidance.

Jane: So, for someone just listening to the show, WorldAct means taking a huge, uneditable three dee environment and making it structured enough for an AI to actually *act* on it.

Shanghai Jiao Tong University · Huawei Inc.

cs.CV

Submitted: 2026-05-15

Updated: 2026-11-05

Comments: Project page: https://sjtu-deepvisionlab.github.io/WorldAct

Project page: https://sjtudeepvisionlab.github.io/WorldAct

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: WorldAct presents a framework that converts monolithic 3D Gaussian Splatting scenes into editable and interaction-ready scenes.

Key concepts

Scene Decomposition
This process breaks down a large 3D scene into smaller, manageable parts representing distinct objects. The system uses an agent to identify which parts are objects and reconstruct them into detailed meshes suitable for physical interaction, separating them from the background.
Interactable Object Discovery
A vision-language agent analyzes video frames to find objects of interest. It generates text prompts, and a segmentation model then creates masks showing exactly where each identified object is located in the scene, guiding the decomposition process.
Collision-Aware Manipulation
This capability allows robots to physically interact with objects in the reconstructed scene. By creating collision proxy meshes, the system ensures that movements like grasping and placing objects are physically plausible and avoid unwanted collisions during task execution.

Terminology

Summary

WorldAct presents a framework that converts monolithic 3D Gaussian Splatting scenes into editable and interaction-ready scenes. p. 0 The resulting scenes support object-level editing, collision-aware manipulation, and embodied task execution while preserving global scene coherence. p. 0

Pruning redundant reasoning tokens

The core challenge addressed is that recent 3D world modeling systems typically produce static, monolithic assets with limited editability and physical interaction. WorldAct tackles this by using a multimodal agent to guide scene decomposition, identify actionable objects, reconstruct geometrically aligned object-level meshes for interaction, and restore the residual background via 3D inpainting. p. 0 This process enables downstream simulation tasks such as robotic manipulation and scene rearrangement. p. 1

How the training signal is built

The pipeline begins by rendering a camera trajectory to obtain multi-view observations for object discovery and segmentation. An agent-driven interactable object discovery stage uses a vision-language agent to analyze keyframes and generate text prompts for operable objects. p. 3 For each prompt, video segmentation using SAM3 is performed to produce binary masks indicating the pixel region occupied by the object corresponding to the prompt. p. 3

Where the gains come from

The scene decomposition stage involves estimating a Gaussian subset Gm for each object om by optimizing a learnable soft assignment score si,m through mask inverse rendering. This optimization is guided by the projection loss Lm seg, which encourages foreground consistency and suppresses false positives in background regions. p. 4 Following decomposition, the background is completed using DiffuEraser to produce inpainted frames, and a lightweight collision proxy mesh Mbg is constructed from Gcomp bg using Poisson reconstruction. p. 4

Application to Embodied Simulation

The framework supports object-level physical interaction, such as grasping and placement, which is difficult for conventional monolithic 3D world representations. This capability allows a robotic manipulator to actively interact with objects in reconstructed scenes, providing a practical basis for embodied AI tasks like rearrangement and task planning. p. 7 Furthermore, WorldAct supports high-quality object-level scene editing and reconstruction where users can add, remove, move objects to new locations, and modify their scale or texture while maintaining visual coherence. p. 7

User Study and Auxiliary Agent Evaluation

A Mean Opinion Score (MOS) user study with 20 participants was conducted to evaluate the perceptual quality of the converted scenes across four dimensions: Overall Quality, Surface Completeness, Boundary Cleanliness, and Naturalness. The results showed that while scene-level quality remained similar to the original static scenes, object-level fidelity improved significantly. p. 8 An auxiliary agent evaluation using GPT-5. p. 85 served as a reproducible qualitative audit for checking whether the proposed criteria captured meaningful visual failure modes. p. 6 The quantitative results showed that the Interactable Object Recall Rate increased substantially by more than a factor of three on the standard MWM-easy dataset when using agent guidance. p. 6

Scene Assembly and Alignment

The final stage involves aligning and assembling generated objects into the completed background Gcomp bg using a two-stage alignment procedure. First, an initial pose is estimated using Iterative Closest Point (ICP) between the generated object mesh Mgen m and the point set derived from Gm. p. 5 Second, the object pose is refined through differentiable rendering by minimizing a loss function Lalign that enforces consistency with projected object masks, encourages plausible support relationships, and penalizes collisions. p. 5 This process yields a decomposed representation that supports object-level editing and manipulation. p. 5

Conclusion

WorldAct successfully converts monolithic 3DGS scenes into object-decomposed environments for editing and interaction. The framework preserves the visual appearance of the original scene while enabling basic object-level editing, placement, and embodied interaction. p. 8 The current framework depends on the quality of the input 3D world model and does not yet handle dynamic scenes or physical properties such as mass and friction. p. 8 Future work will focus on addressing these limitations. p. 8

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Agent-Driven Automation: The system can automatically identify operable objects, decompose scenes, restore backgrounds, and reconstruct object assets without manual annotation by employing a agent-looped pipeline that leverages a vision-language agent to generate text prompts and SAM3 for segmentation.

  2. Scene Decomposition: The framework can decompose monolithic 3D Gaussian Splatting scenes into object-level Gaussian subsets and residual background by optimizing a learnable soft assignment score using mask inverse rendering, ensuring that the separation process is guided by consistent foreground and background suppression terms in the loss function.

  3. Object-Level Editing and Manipulation: The system supports object-level editing, collision-aware manipulation, and embodied task execution because it builds simplified collision geometry from the restored background and refines extracted instances into clean object assets before assembling them back into a scene where objects are independently represented.

  4. High-Quality Asset Generation: The system can generate high-quality 3DGS representations and textured meshes for isolated objects using SAM3, guided by an agent that automatically selects the optimal viewpoint for each object by evaluating visibility, occlusion levels, and semantic confidence across all frames in the trajectory.

  5. Scene Restoration Fidelity: The framework ensures visual quality is preserved during reconstruction through a two-stage alignment procedure involving Iterative Closest Point (ICP) followed by differentiable rendering optimization to minimize Lmask, Lcontact, and Lpenetration losses when placing generated objects back into the scene.

Sources

Related papers