InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models".
Jane: Here's the gist of the paper. InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset.
Tom: The Core Methodology InHabit follows a render–generate–lift principle: given a rendered 3D scene, a vision–language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry <ref:2604.19673#pg6>.
Jane: This process is applied to HabitatMatterport3D to produce InHabitants, which contains 78K samples across approximately 800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images <ref:2604.19673#pg3>.
Tom: Data Generation Pipeline The generation of scene-aware humans in the 2D space involves two foundation models: a vision–language model (VLM) and an image-editing model <ref:2604.19673#pg6>.
Title and authors: Tom: So we're looking at this paper today called "InHabit: Scaling three dee Human-Scene Interaction Data With Image Foundation Models <ref:2604.19673#pg1>." Jane, what's the main idea behind that title?
Jane: It basically says they're scaling up data generation for three dee human and scene interactions using image foundation models to get photorealistic datasets <ref:2604.19673#pg1>. It points out that getting real data of people interacting with diverse three dee environments is hard because MoCap capture is expensive and limited, while existing synthetic methods use simple geometric rules <ref:2604.19673#pg1>.
Lu: They're suggesting that instead of relying only on those simple geometric heuristics, we can leverage the commonsense knowledge foundation models have learned from massive amounts of 2D internet data to propose contextually meaningful actions for humans in a scene <ref:2604.19673#pg1>.
Meng: So they're taking something from a 2D model and trying to force it into three dee scenes without the human actually being there for capture <ref:2604.19673#pg1>. That sounds ambitious, but what's the actual process they’re describing?
Tom: The core idea is this render-generate-lift principle where a rendered scene gets contextually meaningful actions from a vision language model, an image editing model inserts a human based on that proposal, and then an optimization procedure lifts it into physically plausible SMPL-X bodies aligned with the scene geometry.
Jane: That's the key mechanism—it’s not just throwing people into scenes randomly; they are using the 2D foundation models to guide *what* the person should be doing in that specific environment <ref:2604.19673#pg1>.
Lu: And they call this process InHabitants, which is a large dataset with over seventy-eight thousand samples across about eight hundred building-scale scenes, and it includes complete SMPL-X bodies and images.
Meng: seventy-eight thousand samples sounds like a lot of data to generate for. What kind of scale are we talking about in terms of the environments they used?
Tom: They used HabitatMatterportthree dee for this, which means these are building-scale scenes, giving them complete three dee geometry and SMPL-X bodies <ref:2604.19673#pg1>.
The paper's summary: Jane: So the summary highlights that while older work like PiGraphs or PROX only get data from a handful of scenes, and MoCap methods are limited to maybe thirty scenes, InHabit tackles that by using 2D foundation models which have implicitly acquired rich commonsense knowledge <ref:2604.19673#pg1>.
Lu: They argue that existing synthetic datasets are scalable but they just place prerecorded human motions into scenes, making the humans merely physically valid but not truly engaging with the environment in a semantic way.
Tom: InHabit moves past that by using a vision language model for affordance-driven prompting to suggest semantically appropriate interactions tailored to the visible environment, supplemented by an action-driven strategy with generic categories like "reaching" or "leaning."
Meng: So the VLM proposes *what* to do, and then an image editing model realizes it visually in 2D before they try to make it three dee <ref:2604.19673#pg1>. That sounds like a pipeline where errors can accumulate quickly.
Lalam: I'm confident that this transfer of knowledge from 2D foundation models addresses the scarcity problem by leveraging those implicit interaction priors, and my literature scan confirms the direction is strong for large-scale data generation > <ref:2604.19673#pg1>
Jane: And then they refine that in three dee using an optimization-based method inspired by PhySIC, which minimizes terms like Lproj, Ldepth, Lcontact, and others to ensure the resulting bodies are physically plausible and aligned with the scene geometry <ref:2604.19673#pg1>.
Tom: So we have a proposal for interaction from a VLM, a visual realization from an image editing model, and then a physical grounding through optimization to get those three dee SMPL-X bodies into place <ref:2604.19673#pg1>.
The paper's improvements: Tom: The paper points out that the resulting InHabitants dataset is characterized by high diversity, showing the most diverse interactions, target objects, and poses compared to prior work like TRUMANS and PROX-D.
Jane: That diversity is key because it means participants selected InHabit as strongly preferred over GenZI and POSA++ in seventy-eight point one percent of cases when they considered realism, physical feasibility, and pose naturalness.
Lu: Quantitatively, they show that InHabit yields renderings most CLIP-aligned with the prompt text, and its contact estimations best match prompt-based expectations.
Meng: That’s interesting because it means we can use this data to improve downstream tasks like contact estimation methods such as DECO on the DAMON benchmark by augmenting standard training data.
Tom: And they also show that feed-forward HSI reconstruction models consistently improve when trained on this dataset, achieving stronger performance on the PROX benchmark.
Lu: What's really compelling is their claim that this synthetic data has shown to improve SOTA contact estimation methods like DECO by providing high-quality contact supervision that is generated fully automatically, which outperforms labor-intensive capture datasets like TRUMANS.
Conclusion: Jane: So to wrap up, InHabit demonstrates that internet-scale 2D foundation models encode enough knowledge of human–environment interactions to serve as a practical and scalable alternative to expensive three dee motion capture for training embodied AI methods <ref:2604.19673#pg1>.
Tom: They show that this is extensible, suggesting the principle can be used for video generation and dynamic scenes in the future. The core idea is leveraging those 2D models to get interaction data without needing expensive real-world capture infrastructure <ref:2604.19673#pg1>.
Lu: From a representation learning standpoint, this moves us toward training embodied agents using data that captures complex social scenarios like "a couple dancing or moving furniture together," which was not supported by prior work.
Meng: From an engineering standpoint, the challenge for deployment will be managing the scale of generating seventy-eight thousand samples across hundreds of scenes efficiently and ensuring the optimization procedure doesn't just produce physically valid but useless data.
Lalam: I'm sure that InHabit is a significant step because it shows that we can generate large-scale, diverse interaction data synthetically by grounding it in existing powerful 2D models > <ref:2604.19673#pg1>
NIKITA KISTER, PRADYUMNA YM, ISTVÁN SÁRÁNDI, JIAYI WANG, ANNA KHOREVA, GERARD PONS-MOLL
University of Tübingen · Bosch Center of Artificial Intelligence Center of Artificial Intelligence Germany · Zalando SE
cs.CV
Submitted: 2026-04-21
Updated: 2026-10-28
Project page: https://virtualhumans.mpi-inf.mpg.de/inhabit
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D…
Key concepts
- Render–Generate–Lift Principle
- This is the core method where a 3D scene is rendered, a vision-language model suggests an action, an image editor places a human into the scene, and finally, optimization lifts that result into a physically accurate 3D body. This ensures the generated humans interact plausibly with the environment.
- VLM for Affordance-Driven Prompting
- A vision-language model analyzes the rendered scene to propose contextually relevant actions, like 'reaching' or 'leaning,' based on what is visible. This allows the system to generate semantically appropriate interactions tailored specifically to the environment shown in the 2D input images.
- Physical Grounding and Refinement
- This step uses an optimization-based method inspired by PhySIC to reconstruct inserted humans into 3D. It minimizes several physical constraints, such as projection error and penetration depth, ensuring the resulting human bodies are physically sound and correctly aligned with the scene geometry.
Terminology
Summary
InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset. p. 0 In contrast, 2D foundation
models trained at internet scale have acquired commonsense knowledge
of human–environment interactions. To transfer this knowledge to 3D, we
introduce InHabit, an automatic and scalable data generator for populating
3D scenes with interacting humans.Where the paper says it
The Core Methodology
InHabit follows a render–generate–lift principle: given a rendered 3D scene, a vision–language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry <ref:2604.19673#pg6>. This process is applied to HabitatMatterport3D to produce InHabitants, which contains 78K samples across approximately 800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images <ref:2604. p. 0 and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry. Applied to HabitatMatterport3D, InHabit produces InHabitants, the first large-scale photorealistic 3D human—scene interaction dataset, with 78K samples across
∼800 building-scale scenes with complete 3D geometry, SMPL-X bodies,
and images.Where the paper says it
Data Generation Pipeline
The generation of scene-aware humans in the 2D space involves two foundation models: a vision–language model (VLM) and an image-editing model <ref:2604.19673#pg6>. The VLM is used for , a couple dancing or moving furniture together (fig. Given the VLM’s textual interaction proposals, the image-editing
model serves as the visual realization engine, generating body poses,
proportions, and spatial placements adapted to the local scene context.affordance-driven prompting
to propose semantically appropriate interactions tailored to the visible environment, supplemented by an action-driven strategy using generic action categories like reaching
or leaning
<ref:2604.19673#pg6>. The image-editing model then serves as the visual realization engine, generating body poses and spatial placements adapted to the local scene context based on these proposals <ref:2604. p. 2Where the paper says it
Physical Grounding and Refinement
The lifting step reconstructs inserted humans into 3D using an optimization-based method inspired by PhySIC <ref:2604.19673#pg6>. The objective function minimizes terms such as Lproj, Ldepth, Lcontact, Lpenetration, and regLreg to ensure the resulting bodies are physically plausible and aligned with the known 3D scene geometry <ref:2604. p. 3 Right: 3D scene-aware lifting—the inserted
humans are recovered as SMPL-X meshes and optimized with scene geometry and proximity-based contact constraints for physically plausible 3D HSI. where Lproj enforces 2D reprojection consistency, Ldepth aligns
the visible human surface with the estimated depth, Lcontact encourages plausible contact with scene surfaces, and Lpenetration penalizes
intersections with the scene geometry. To remove implausible or incomplete
human insertions and ensure high-quality data, we apply three
complementary, automated post-reconstruction filters. We discard
examples with strongly truncated humans, generations where the
edited image contains unindended scene changes besides inserting
the human, and finally we remove human mesh reconstructions
with implausibly small volumes.Where the paper says it
Where the paper says it
Dataset Utility and Impact
The resulting InHabitants dataset is characterized by high diversity, showing the most diverse interactions, target objects, and poses compared to prior work like TRUMANS and PROX-D <ref:2604.19673#pg6>. This synthetic data has been shown to improve downstream tasks; specifically, augmenting standard training data with InHabitants improves the SOTA contact estimation method DECO on the DAMON benchmark <ref:2604. p. 1 We demonstrate the value of the InHabitants data on two downstream tasks. First, augmenting standard training data with our
samples improves the SOTA contact estimation method DECO [Tripathi et al. 2026] consistently improve when trained on our data, achieving
stronger performance on the PROX benchmark. We will release the
dataset and code publicly.Where the paper says it
Where the paper says it
Evaluation Against Baselines
In a perceptual user study, InHabit was strongly preferred over GenZI and POSA++ in 78.1% of cases when participants selected the single best result considering realism, physical feasibility, and pose naturalness <ref:2604. p. 5 We conducted a
perceptual user study, where 41 participants were shown triplets of generations made with InHabit, GenZI and
POSA++ at corresponding scene locations, in random order, as interactive
3D visualizations. They were asked to
select the single best result, considering the realism of human–scene interaction, physical feasibility and pose
naturalness. 2, humans generated by InHabit yield renderings
most CLIP-aligned with the prompt text, and have body contacts
that best match prompt-based expectations. 2 Data for Contact Estimation
Body contact estimation predicts, from a single RGB image, which
body-surface vertices touch the scene.Where the paper says it
Where the paper says it
Conclusion
InHabit demonstrates that internet-scale 2D foundation models encode enough knowledge of human–environment interactions to serve as a practical, scalable alternative to expensive 3D motion capture for training embodied AI methods <ref:2604.19673#pg6>. The principle is extensible to video generation and dynamic scenes in the future <ref:2604. p. 6 We showed that InHabit outperforms
existing placement methods both quantitatively and in a perceptual user study, and that augmenting training data with InHabitants
improves state-of-the-art contact estimation and feed-forward HSI
reconstruction on established benchmarks. Currently, InHabit generates static poses within static 3D scenes, but the principle can be
extended to video generation and dynamic scenes in the future as
the underlying generative models continue to evolve.Where the paper says it
Acknowledgments
Special thanks RVH members for the help and discussion Nikita Kister thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support <ref:2604.19673#pg6>. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pradyumna <ref:2604. p. 7 Acknowledgments
Acknowledgments: Special thanks RVH members for the help and
discussion Nikita Kister thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. The
authors thank the International Max Planck Research School for
Intelligent Systems (IMPRS-IS) for supporting Pradyumna. The
authors thank the International Max Planck Research School for
Intelligent Systems (IMPRS-IS) for supporting Pradyumna. István
Sárándi and Gerard Pons-Moll were supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) – 409792180 (Emmy Noether
Programme, project: Real Virtual Humans). István
Sárándi and Gerard Pons-Moll were supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) – 409792180 (Emmy Noether
Programme, project: Real Virtual Humans). GPM is a member of
the Machine Learning Cluster of Excellence, EXC number 2064/1
– Project number 390727645 and is supported by the Carl Zeiss
Foundation.Where the paper says it
Where the paper says it
Where the paper says it
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, et al. 2023. GPT-4 Technical Report <ref:2604. p. 7 References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, et al. GPT-4 Technical Report.Where the paper says it
Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun Wu, Alexander Clegg, and C. p. 7 08774 (2023). Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun
Wu, Alexander Clegg, and C. Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun
Wu, Alexander Clegg, and C. CIRCLE: Capture In Rich Contextual
Environments.Where the paper says it
Where the paper says it
Michael J. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. p. 7 In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. BEDLAM:
A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion.Where the paper says it
Where the paper says it
Tim Brooks, Aleksander Holynski, and Alexei A. p. 7 In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR). Tim Brooks, Aleksander Holynski, and Alexei A. Tim Brooks, Aleksander Holynski, and Alexei A. InstructPix2Pix: Learning
to Follow Image Editing Instructions.Where the paper says it
Where the paper says it
Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. p. 7 In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR). Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-Term Human Motion Prediction with Scene Context.Where the paper says it
Where the paper says it
Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. p. 7 In Proceedings of
the European Conference on Computer Vision (ECCV). Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. Human3R: Everyone Everywhere All at Once.Where the paper says it
Where the paper says it
Angela Dai, Angel X. p. 7 In International Conference on
Learning Representations (ICLR). Angela Dai, Angel X. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and
Matthias Nießner. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and
Matthias Nießner. ScanNet: Richly-annotated 3D Reconstructions of Indoor
Scenes.Where the paper says it
Where the paper says it
Where the paper says it
Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. p. 7 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and
Grégory Rogez. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and
Grégory Rogez. Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and
Grégory Rogez. PoseScript: 3D Human Poses from Natural Language.Where the paper says it
Where the paper says it
Where the paper says it
Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. 2024. PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation <ref:2604. p. 7 Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez. PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation.Where the paper says it
Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, and Chunhua Shen. p. 7 PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation. Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, and
Chunhua Shen. Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, and
Chunhua Shen. Diffusion Models are Efficient Data Generators for Human
Mesh Recovery.Where the paper says it
Where the paper says it
Google DeepMind. p. 1 Image-editing
models [Brooks et al. 2023; Google DeepMind 2025a] can synthesize
what it looks like to cook at a stove or read on a couch, as they have
seen millions of such images, and vision–language models [Achiam
et al. Image-editing
models [Brooks et al. 2023; Google DeepMind 2025a] can synthesize
what it looks like to cook at a stove or read on a couch, as they have
seen millions of such images, and vision–language models [Achiam
et al. Google DeepMind. 5: Our Most Intelligent AI Model.Where the paper says it
Where the paper says it
Where the paper says it
Google DeepMind. 2025b. Gemini 3 <ref:2604. p. 7 https://deepmind. google/models/gemini/.Where the paper says it
Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll. p. 7 google/models/gemini/. Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten
Sattler, and Gerard Pons-Moll. Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten
Sattler, and Gerard Pons-Moll. Interaction Replica: Tracking Human-Object
Interaction and Scene Changes From Human Motion.Where the paper says it
Where the paper says it
Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. p. 7 In International Conference on
3D Vision (3DV). Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. Human
POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in
Large Scenes from Body-Mounted Sensors.Where the paper says it
Where the paper says it
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. p. 7 In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR). Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Resolving 3D Human Pose Ambiguities with 3D Scene Constraints.Where the paper says it
Where the paper says it
Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J. p. 7 In Proceedings
of the IEEE/CVF International Conference on Computer Vision (ICCV). Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J. Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J. Populating 3D Scenes by Learning Human-Scene Interaction.Where the paper says it
Where the paper says it
Chun-Hao P. p. 7 In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Chun-Hao P. Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina
Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina
Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Capturing
and Inferring Dense Full-Body Human-Scene Contact.Where the paper says it
Where the paper says it
Where the paper says it
Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. p. 7 In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR). Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu
Liu, Yixin Zhu, and Siyuan Huang. Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu
Liu, Yixin Zhu, and Siyuan Huang. Scaling Up Dynamic Human-Scene Interaction Modeling.Where the paper says it
Where the paper says it
Hyeonwoo Kim, Sangwon Baik, and Hanbyul Joo. p. 7 In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). Hyeonwoo Kim, Sangwon Baik, and Hanbyul Joo. Hyeonwoo Kim, Sangwon Baik, and Hanbyul Joo. DAViD: Modeling Dynamic
Affordance of 3D Objects Using Pre-trained Video Diffusion Models.Where the paper says it
Where the paper says it
Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo. p. 7 In Proceedings
of the IEEE/CVF International Conference on Computer Vision (ICCV). Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo. Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo. Beyond the
Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained
2D Diffusion Models.Where the paper says it
Where the paper says it
Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. p. 7 In Proceedings of the European Conference on Computer Vision
(ECCV). Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. Efros, and Krishna Kumar Singh. Efros, and Krishna Kumar Singh. Putting People in Their Place: AffordanceAware Human Insertion into Scenes.Where the paper says it
Where the paper says it
Where the paper says it
Lei Li and Angela Dai. p. 7 In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR). Lei Li and Angela Dai. Lei Li and Angela Dai. GenZI: Zero-Shot 3D Human-Scene Interaction Generation.Where the paper says it
Where the paper says it
Naureen Mahmood, Nima Ghorbani, Nikolaus F. p. 7 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR). Naureen Mahmood, Nima Ghorbani, Nikolaus F. Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Troje, Gerard Pons-Moll, and Michael J. AMASS: Archive of Motion Capture as Surface Shapes.Where the paper says it
Where the paper says it
Where the paper says it
Priyanka Patel, Chun-Hao P. p. 8 InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement • 9
of the IEEE/CVF International Conference on Computer Vision (ICCV). Priyanka Patel, Chun-Hao P. Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Huang, Joachim Tesch, David T. Hoffmann, Shashank
Tripathi, and Michael J. Hoffmann, Shashank
Tripathi, and Michael J. AGORA: Avatars in Geography Optimized for
Regression Analysis.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. p. 8 on Computer Vision and Pattern
Recognition (CVPR). Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Osman,
Dimitrios Tzionas, and Michael J. Osman,
Dimitrios Tzionas, and Michael J. Expressive Body Capture: 3D Hands,
Face, and Body From a Single Image.Where the paper says it
Where the paper says it
Where the paper says it
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Dhruv Batra, and Roozbeh Mottaghi. p. 8 In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR). Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Dhruv Batra, and
Roozbeh Mottaghi. Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Dhruv Batra, and
Roozbeh Mottaghi. 0: A Co-Habitat for Humans, Avatars and Robots.Where the paper says it
Where the paper says it
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. p. 8 In International Conference on Learning Representations (ICLR). Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets,
Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew
Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets,
Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew
Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. HabitatMatterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied
AI.Where the paper says it
Where the paper says it
Manolis Savva, Angel X. p. 7 Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and
Matthias Nießner. 08238
Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. PiGraphs: Learning Interaction Snapshots from Observations.Where the paper says it
Where the paper says it
Where the paper says it
Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael J. p. 8 ACM Transactions on Graphics (TOG) 35, 4 (2016). Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas,
and Michael J. Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas,
and Michael J. DECO: Dense Estimation of 3D Human-Scene Contact
In The Wild.Where the paper says it
Where the paper says it
Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. p. 8 8001–8013. Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen,
Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen,
Wenwei Zhang, Kai Chen, Tianfan Xue, et al. EmbodiedScan: A Holistic MultiModal 3D Perception Suite Towards Embodied AI.Where the paper says it
Where the paper says it
Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. p. 8 In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR). Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes.Where the paper says it
Where the paper says it
Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, and Gerard Pons-Moll. p. 8 In
Advances in Neural Information Processing Systems (NeurIPS). Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, and
Gerard Pons-Moll. Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, and
Gerard Pons-Moll. PhySIC: Physically Plausible 3D Human-Scene Interaction
and Contact from a Single Image.Where the paper says it
Where the paper says it
Pradyumna YM, Yuxuan Xue, Yue Chen, Nikita Kister, István Sárándi, and Gerard PonsMoll. p. 8 In ACM SIGGRAPH Asia. Pradyumna YM, Yuxuan Xue, Yue Chen, Nikita Kister, István Sárándi, and Gerard PonsMoll. GRAFT: Geometric Refinement and Fitting Transformer for Human
Scene Reconstruction. arXiv:2604.Where the paper says it
Where the paper says it
Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. p. 8 org/abs/2604. 19624
Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. 19624
Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. PLACE:
Proximity learning of articulation and contact in 3D environments.Where the paper says it
Where the paper says it
Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, and Gerard Pons-Moll. p. 8 IEEE, 642–651. Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, and Gerard
Pons-Moll. Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, and Gerard
Pons-Moll. COUCH: Towards Controllable Human-Chair Interactions.Where the paper says it
Where the paper says it
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. p. 8 In
Proceedings of the European Conference on Computer Vision (ECCV). Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining
Hong, and Chuang Gan. Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining
Hong, and Chuang Gan. 3D-VLA: A 3D Vision-Language-Action Generative
World Model.Where the paper says it
Where the paper says it
--- Page 1 ---
InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated generation of large-scale 3D human–scene interaction data using a render–generate–lift principle allows for populating 3D scenes with interacting humans across varied environments and actions. This enables large-scale generation of interaction data for embodied 3D scene understanding, moving beyond simple geometric heuristics.
-
Transferring commonsense knowledge from 2D foundation models to the 3D domain via a Vision–Language Model (VLM) and image-editing model proposes contextually meaningful actions tailored to the visible environment. This addresses the scarcity of data by leveraging
implicit interaction priors in 2D foundation models.
-
The system can generate a first large-scale photorealistic 3D human—scene interaction dataset, InHabitants, containing over 78k samples across ∼800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images. This dataset augments standard training data for RGB-based 3D human—scene reconstruction and contact estimation.
-
The resulting data improves SOTA contact estimation methods like DECO by providing high-quality contact supervision that is
generated fully automatically,
which outperforms labor-intensive capture datasets like TRUMANS. -
InHabitants can be used to consistently improve feed-forward HSI reconstruction models, achieving stronger performance on benchmarks by training them on the synthetic data, as shown in Table 4 and Table 5 comparisons for GRAFT and Human3R methods.
-
The system can generate diverse interactions involving multiple people, which was
not supported by prior work,
allowing for the modeling of complex social scenarios likea couple dancing or moving furniture together.
Abstract
Embodied AI and human-centric 3D scene understanding require data of people meaningfully interacting with diverse environments, yet such data remains scarce. Real-world capture has steadily pushed toward more scenes, but is bound to studios and at most around a hundred scenes, while synthetic datasets scale easily but place people without regard to what the scene affords. We argue that useful human-scene interaction data requires diversity in both scenes and interactions, and we show this with our proposed data engine InHabit. InHabit is a fully automatic engine that renders an existing 3D scene, uses 2D foundation models to propose and depict context-appropriate interactions, and lifts them into SMPL-X bodies, where the known scene geometry makes lifting well-constrained. Applied to Habitat-Matterport3D, InHabit produces InHabitants, which pairs photorealistic images with SMPL-X bodies and complete scene geometry across 6.6k rooms, about 60x more scenes than the largest captured dataset. We illustrate the quality and value of such large-scale human-scene interaction data through a variety of experiments. In a perceptual study, our interactions are preferred over two prior placement methods in 78% of cases. Across three tasks - 1) contact estimation, 2) human-scene reconstruction, and 3) generating humans in scenes - training with InHabitants outperforms training on synthetic data with exact ground truth (BEDLAM), motion-capture data (TRUMANS), and real RGB-D captures (PROX); e.g., GRAFT trained on InHabitants reaches a contact F1 of 0.594 on PROX, versus 0.492 when trained on PROX itself. Project page: https://virtualhumans.mpi-inf.mpg.de/inhabit/
Sources
- GPT-4 Technical Report
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
- GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs