InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models
summary
The gist
InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D…
In short
InHabit automatically generates a massive, photorealistic dataset of interacting humans in 3D scenes using image foundation models. It employs a render-generate-lift process to transfer commonsense knowledge from 2D images to create physically plausible human poses and interactions. This scalable data augments existing training sets for embodied AI tasks like contact estimation.
Key concepts
- Render–Generate–Lift Principle
- This is the core method where a 3D scene is rendered, a vision-language model suggests an action, an image editor places a human into the scene, and finally, optimization lifts that result into a physically accurate 3D body. This ensures the generated humans interact plausibly with the environment.
- VLM for Affordance-Driven Prompting
- A vision-language model analyzes the rendered scene to propose contextually relevant actions, like 'reaching' or 'leaning,' based on what is visible. This allows the system to generate semantically appropriate interactions tailored specifically to the environment shown in the 2D input images.
- Physical Grounding and Refinement
- This step uses an optimization-based method inspired by PhySIC to reconstruct inserted humans into 3D. It minimizes several physical constraints, such as projection error and penetration depth, ensuring the resulting human bodies are physically sound and correctly aligned with the scene geometry.
Terminology used across episodes
This episode discusses
- InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models · Paper Radio
- GPT-4 Technical Report
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
- GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction
The paper
InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models · Read on arXiv
NIKITA KISTER, PRADYUMNA YM, ISTVÁN SÁRÁNDI, JIAYI WANG, ANNA KHOREVA, GERARD PONS-MOLL
University of Tübingen · Bosch Center of Artificial Intelligence Center of Artificial Intelligence Germany · Zalando SE
Embodied AI and human-centric 3D scene understanding require data of people meaningfully interacting with diverse environments, yet such data remains scarce. Real-world capture has steadily pushed toward more scenes, but is bound to studios and at most around a hundred scenes, while synthetic datasets scale easily but place people without regard to what the scene affords. We argue that useful human-scene interaction data requires diversity in both scenes and interactions, and we show this with our proposed data engine InHabit. InHabit is a fully automatic engine that renders an existing 3D scene, uses 2D foundation models to propose and depict context-appropriate interactions, and lifts them into SMPL-X bodies, where the known scene geometry makes lifting well-constrained. Applied to Habitat-Matterport3D, InHabit produces InHabitants, which pairs photorealistic images with SMPL-X bodies and complete scene geometry across 6.6k rooms, about 60x more scenes than the largest captured dataset. We illustrate the quality and value of such large-scale human-scene interaction data through a variety of experiments. In a perceptual study, our interactions are preferred over two prior placement methods in 78% of cases. Across three tasks - 1) contact estimation, 2) human-scene reconstruction, and 3) generating humans in scenes - training with InHabitants outperforms training on synthetic data with exact ground truth (BEDLAM), motion-capture data (TRUMANS), and real RGB-D captures (PROX); e.g., GRAFT trained on InHabitants reaches a contact F1 of 0.594 on PROX, versus 0.492 when trained on PROX itself. Project page: https://virtualhumans.mpi-inf.mpg.de/inhabit/
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models".
Jane: Here's the gist of the paper. InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset.
Tom: The Core Methodology InHabit follows a render–generate–lift principle: given a rendered 3D scene, a vision–language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry <ref:2604.19673#pg6>.
Jane: This process is applied to HabitatMatterport3D to produce InHabitants, which contains 78K samples across approximately 800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images <ref:2604.19673#pg3>.
Tom: Data Generation Pipeline The generation of scene-aware humans in the 2D space involves two foundation models: a vision–language model (VLM) and an image-editing model <ref:2604.19673#pg6>.
Title and authors: Tom: So we're looking at this paper today called "InHabit: Scaling three dee Human-Scene Interaction Data With Image Foundation Models <ref:2604.19673#pg1>." Jane, what's the main idea behind that title?
Jane: It basically says they're scaling up data generation for three dee human and scene interactions using image foundation models to get photorealistic datasets <ref:2604.19673#pg1>. It points out that getting real data of people interacting with diverse three dee environments is hard because MoCap capture is expensive and limited, while existing synthetic methods use simple geometric rules <ref:2604.19673#pg1>.
Lu: They're suggesting that instead of relying only on those simple geometric heuristics, we can leverage the commonsense knowledge foundation models have learned from massive amounts of 2D internet data to propose contextually meaningful actions for humans in a scene <ref:2604.19673#pg1>.
Meng: So they're taking something from a 2D model and trying to force it into three dee scenes without the human actually being there for capture <ref:2604.19673#pg1>. That sounds ambitious, but what's the actual process they’re describing?
Tom: The core idea is this render-generate-lift principle where a rendered scene gets contextually meaningful actions from a vision language model, an image editing model inserts a human based on that proposal, and then an optimization procedure lifts it into physically plausible SMPL-X bodies aligned with the scene geometry.
Jane: That's the key mechanism—it’s not just throwing people into scenes randomly; they are using the 2D foundation models to guide *what* the person should be doing in that specific environment <ref:2604.19673#pg1>.
Lu: And they call this process InHabitants, which is a large dataset with over seventy-eight thousand samples across about eight hundred building-scale scenes, and it includes complete SMPL-X bodies and images.
Meng: seventy-eight thousand samples sounds like a lot of data to generate for. What kind of scale are we talking about in terms of the environments they used?
Tom: They used HabitatMatterportthree dee for this, which means these are building-scale scenes, giving them complete three dee geometry and SMPL-X bodies <ref:2604.19673#pg1>.
The paper's summary: Jane: So the summary highlights that while older work like PiGraphs or PROX only get data from a handful of scenes, and MoCap methods are limited to maybe thirty scenes, InHabit tackles that by using 2D foundation models which have implicitly acquired rich commonsense knowledge <ref:2604.19673#pg1>.
Lu: They argue that existing synthetic datasets are scalable but they just place prerecorded human motions into scenes, making the humans merely physically valid but not truly engaging with the environment in a semantic way.
Tom: InHabit moves past that by using a vision language model for affordance-driven prompting to suggest semantically appropriate interactions tailored to the visible environment, supplemented by an action-driven strategy with generic categories like "reaching" or "leaning."
Meng: So the VLM proposes *what* to do, and then an image editing model realizes it visually in 2D before they try to make it three dee <ref:2604.19673#pg1>. That sounds like a pipeline where errors can accumulate quickly.
Lalam: I'm confident that this transfer of knowledge from 2D foundation models addresses the scarcity problem by leveraging those implicit interaction priors, and my literature scan confirms the direction is strong for large-scale data generation > <ref:2604.19673#pg1>
Jane: And then they refine that in three dee using an optimization-based method inspired by PhySIC, which minimizes terms like Lproj, Ldepth, Lcontact, and others to ensure the resulting bodies are physically plausible and aligned with the scene geometry <ref:2604.19673#pg1>.
Tom: So we have a proposal for interaction from a VLM, a visual realization from an image editing model, and then a physical grounding through optimization to get those three dee SMPL-X bodies into place <ref:2604.19673#pg1>.
The paper's improvements: Tom: The paper points out that the resulting InHabitants dataset is characterized by high diversity, showing the most diverse interactions, target objects, and poses compared to prior work like TRUMANS and PROX-D.
Jane: That diversity is key because it means participants selected InHabit as strongly preferred over GenZI and POSA++ in seventy-eight point one percent of cases when they considered realism, physical feasibility, and pose naturalness.
Lu: Quantitatively, they show that InHabit yields renderings most CLIP-aligned with the prompt text, and its contact estimations best match prompt-based expectations.
Meng: That’s interesting because it means we can use this data to improve downstream tasks like contact estimation methods such as DECO on the DAMON benchmark by augmenting standard training data.
Tom: And they also show that feed-forward HSI reconstruction models consistently improve when trained on this dataset, achieving stronger performance on the PROX benchmark.
Lu: What's really compelling is their claim that this synthetic data has shown to improve SOTA contact estimation methods like DECO by providing high-quality contact supervision that is generated fully automatically, which outperforms labor-intensive capture datasets like TRUMANS.
Conclusion: Jane: So to wrap up, InHabit demonstrates that internet-scale 2D foundation models encode enough knowledge of human–environment interactions to serve as a practical and scalable alternative to expensive three dee motion capture for training embodied AI methods <ref:2604.19673#pg1>.
Tom: They show that this is extensible, suggesting the principle can be used for video generation and dynamic scenes in the future. The core idea is leveraging those 2D models to get interaction data without needing expensive real-world capture infrastructure <ref:2604.19673#pg1>.
Lu: From a representation learning standpoint, this moves us toward training embodied agents using data that captures complex social scenarios like "a couple dancing or moving furniture together," which was not supported by prior work.
Meng: From an engineering standpoint, the challenge for deployment will be managing the scale of generating seventy-eight thousand samples across hundreds of scenes efficiently and ensuring the optimization procedure doesn't just produce physically valid but useless data.
Lalam: I'm sure that InHabit is a significant step because it shows that we can generate large-scale, diverse interaction data synthetically by grounding it in existing powerful 2D models > <ref:2604.19673#pg1>
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI