SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image
cs.CV, cs.RO
Submitted: 2026-06-02
Updated: 2026-11-03
Comments: Project Page: https://snuvclab.github.io/SimuScene/
Project page: https://snuvclab.github.io/SimuScene/Teaser
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation.
Terminology
Abstract
Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware compositional reconstruction methods mainly address this as a post-hoc layout correction, leaving the underlying geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals that drive amodal shape resampling, with gravity-axis stretching as a lightweight correction for minor errors. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in humanoid control and robot-arm manipulation tasks.
Sources
- GPT-4 Technical Report
- Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents
- Training Diffusion Models with Reinforcement Learning
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- SAM 3: Segment Anything with Concepts
- SAM 3D: 3Dfy Anything in Images
- KTO: Model Alignment as Prospect Theoretic Optimization
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models
- Flash Sculptor: Modular 3D Worlds from Objects
- DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation
- PhiP-G: Physics-Guided Text-to-3D Compositional Scene Generation
- PAT3D: Physics-Augmented Text-to-3D Scene Generation
- Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
- Decoupled Weight Decay Regularization
- 3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework
- DINOv3
- Gemini: A Family of Highly Capable Multimodal Models
- MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details
- PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Image
- Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs