InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models

arXiv:2604.19673 · cs.CV · Submitted 2026-04-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "InHabit: Scaling 3D Human-Scene Interaction Data With Image Foundation Models".

Jane: Here's the gist of the paper. InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset.

Tom: The Core Methodology InHabit follows a render–generate–lift principle: given a rendered 3D scene, a vision–language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry <ref:2604.19673#pg6>.

Jane: This process is applied to HabitatMatterport3D to produce InHabitants, which contains 78K samples across approximately 800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images <ref:2604.19673#pg3>.

Tom: Data Generation Pipeline The generation of scene-aware humans in the 2D space involves two foundation models: a vision–language model (VLM) and an image-editing model <ref:2604.19673#pg6>.

Title and authors: Tom: So we're looking at this paper today called "InHabit: Scaling three dee Human-Scene Interaction Data With Image Foundation Models <ref:2604.19673#pg1>." Jane, what's the main idea behind that title?

Jane: It basically says they're scaling up data generation for three dee human and scene interactions using image foundation models to get photorealistic datasets <ref:2604.19673#pg1>. It points out that getting real data of people interacting with diverse three dee environments is hard because MoCap capture is expensive and limited, while existing synthetic methods use simple geometric rules <ref:2604.19673#pg1>.

Lu: They're suggesting that instead of relying only on those simple geometric heuristics, we can leverage the commonsense knowledge foundation models have learned from massive amounts of 2D internet data to propose contextually meaningful actions for humans in a scene <ref:2604.19673#pg1>.

Meng: So they're taking something from a 2D model and trying to force it into three dee scenes without the human actually being there for capture <ref:2604.19673#pg1>. That sounds ambitious, but what's the actual process they’re describing?

Tom: The core idea is this render-generate-lift principle where a rendered scene gets contextually meaningful actions from a vision language model, an image editing model inserts a human based on that proposal, and then an optimization procedure lifts it into physically plausible SMPL-X bodies aligned with the scene geometry.

Jane: That's the key mechanism—it’s not just throwing people into scenes randomly; they are using the 2D foundation models to guide *what* the person should be doing in that specific environment <ref:2604.19673#pg1>.

Lu: And they call this process InHabitants, which is a large dataset with over seventy-eight thousand samples across about eight hundred building-scale scenes, and it includes complete SMPL-X bodies and images.

Meng: seventy-eight thousand samples sounds like a lot of data to generate for. What kind of scale are we talking about in terms of the environments they used?

Tom: They used HabitatMatterportthree dee for this, which means these are building-scale scenes, giving them complete three dee geometry and SMPL-X bodies <ref:2604.19673#pg1>.

The paper's summary: Jane: So the summary highlights that while older work like PiGraphs or PROX only get data from a handful of scenes, and MoCap methods are limited to maybe thirty scenes, InHabit tackles that by using 2D foundation models which have implicitly acquired rich commonsense knowledge <ref:2604.19673#pg1>.

Lu: They argue that existing synthetic datasets are scalable but they just place prerecorded human motions into scenes, making the humans merely physically valid but not truly engaging with the environment in a semantic way.

Tom: InHabit moves past that by using a vision language model for affordance-driven prompting to suggest semantically appropriate interactions tailored to the visible environment, supplemented by an action-driven strategy with generic categories like "reaching" or "leaning."

Meng: So the VLM proposes *what* to do, and then an image editing model realizes it visually in 2D before they try to make it three dee <ref:2604.19673#pg1>. That sounds like a pipeline where errors can accumulate quickly.

Lalam: I'm confident that this transfer of knowledge from 2D foundation models addresses the scarcity problem by leveraging those implicit interaction priors, and my literature scan confirms the direction is strong for large-scale data generation > <ref:2604.19673#pg1>

Jane: And then they refine that in three dee using an optimization-based method inspired by PhySIC, which minimizes terms like Lproj, Ldepth, Lcontact, and others to ensure the resulting bodies are physically plausible and aligned with the scene geometry <ref:2604.19673#pg1>.

Tom: So we have a proposal for interaction from a VLM, a visual realization from an image editing model, and then a physical grounding through optimization to get those three dee SMPL-X bodies into place <ref:2604.19673#pg1>.

The paper's improvements: Tom: The paper points out that the resulting InHabitants dataset is characterized by high diversity, showing the most diverse interactions, target objects, and poses compared to prior work like TRUMANS and PROX-D.

Jane: That diversity is key because it means participants selected InHabit as strongly preferred over GenZI and POSA++ in seventy-eight point one percent of cases when they considered realism, physical feasibility, and pose naturalness.

Lu: Quantitatively, they show that InHabit yields renderings most CLIP-aligned with the prompt text, and its contact estimations best match prompt-based expectations.

Meng: That’s interesting because it means we can use this data to improve downstream tasks like contact estimation methods such as DECO on the DAMON benchmark by augmenting standard training data.

Tom: And they also show that feed-forward HSI reconstruction models consistently improve when trained on this dataset, achieving stronger performance on the PROX benchmark.

Lu: What's really compelling is their claim that this synthetic data has shown to improve SOTA contact estimation methods like DECO by providing high-quality contact supervision that is generated fully automatically, which outperforms labor-intensive capture datasets like TRUMANS.

Conclusion: Jane: So to wrap up, InHabit demonstrates that internet-scale 2D foundation models encode enough knowledge of human–environment interactions to serve as a practical and scalable alternative to expensive three dee motion capture for training embodied AI methods <ref:2604.19673#pg1>.

Tom: They show that this is extensible, suggesting the principle can be used for video generation and dynamic scenes in the future. The core idea is leveraging those 2D models to get interaction data without needing expensive real-world capture infrastructure <ref:2604.19673#pg1>.

Lu: From a representation learning standpoint, this moves us toward training embodied agents using data that captures complex social scenarios like "a couple dancing or moving furniture together," which was not supported by prior work.

Meng: From an engineering standpoint, the challenge for deployment will be managing the scale of generating seventy-eight thousand samples across hundreds of scenes efficiently and ensuring the optimization procedure doesn't just produce physically valid but useless data.

Lalam: I'm sure that InHabit is a significant step because it shows that we can generate large-scale, diverse interaction data synthetically by grounding it in existing powerful 2D models > <ref:2604.19673#pg1>

NIKITA KISTER, PRADYUMNA YM, ISTVÁN SÁRÁNDI, JIAYI WANG, ANNA KHOREVA, GERARD PONS-MOLL

University of Tübingen · Bosch Center of Artificial Intelligence Center of Artificial Intelligence Germany · Zalando SE

cs.CV

Submitted: 2026-04-21

Updated: 2026-10-28

Project page: https://virtualhumans.mpi-inf.mpg.de/inhabit

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D…

Key concepts

Render–Generate–Lift Principle
This is the core method where a 3D scene is rendered, a vision-language model suggests an action, an image editor places a human into the scene, and finally, optimization lifts that result into a physically accurate 3D body. This ensures the generated humans interact plausibly with the environment.
VLM for Affordance-Driven Prompting
A vision-language model analyzes the rendered scene to propose contextually relevant actions, like 'reaching' or 'leaning,' based on what is visible. This allows the system to generate semantically appropriate interactions tailored specifically to the environment shown in the 2D input images.
Physical Grounding and Refinement
This step uses an optimization-based method inspired by PhySIC to reconstruct inserted humans into 3D. It minimizes several physical constraints, such as projection error and penetration depth, ensuring the resulting human bodies are physically sound and correctly aligned with the scene geometry.

Terminology

Summary

InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset. p. 0

Where the paper says it

In contrast, 2D foundation models trained at internet scale have acquired commonsense knowledge of human–environment interactions. To transfer this knowledge to 3D, we introduce InHabit, an automatic and scalable data generator for populating 3D scenes with interacting humans.

The Core Methodology

InHabit follows a render–generate–lift principle: given a rendered 3D scene, a vision–language model proposes contextually meaningful actions, an image-editing model inserts a human, and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry <ref:2604.19673#pg6>. This process is applied to HabitatMatterport3D to produce InHabitants, which contains 78K samples across approximately 800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images <ref:2604. p. 0

Where the paper says it

and an optimization procedure lifts the edited result into physically plausible SMPL-X bodies aligned with the scene geometry. Applied to HabitatMatterport3D, InHabit produces InHabitants, the first large-scale photorealistic 3D human—scene interaction dataset, with 78K samples across ∼800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images.

19673#pg3>.

Data Generation Pipeline

The generation of scene-aware humans in the 2D space involves two foundation models: a vision–language model (VLM) and an image-editing model <ref:2604.19673#pg6>. The VLM is used for affordance-driven prompting to propose semantically appropriate interactions tailored to the visible environment, supplemented by an action-driven strategy using generic action categories like reaching or leaning <ref:2604.19673#pg6>. The image-editing model then serves as the visual realization engine, generating body poses and spatial placements adapted to the local scene context based on these proposals <ref:2604. p. 2

Where the paper says it

, a couple dancing or moving furniture together (fig. Given the VLM’s textual interaction proposals, the image-editing model serves as the visual realization engine, generating body poses, proportions, and spatial placements adapted to the local scene context.

19673#pg6>.

Physical Grounding and Refinement

The lifting step reconstructs inserted humans into 3D using an optimization-based method inspired by PhySIC <ref:2604.19673#pg6>. The objective function minimizes terms such as Lproj, Ldepth, Lcontact, Lpenetration, and regLreg to ensure the resulting bodies are physically plausible and aligned with the known 3D scene geometry <ref:2604. p. 3

Where the paper says it

Right: 3D scene-aware lifting—the inserted humans are recovered as SMPL-X meshes and optimized with scene geometry and proximity-based contact constraints for physically plausible 3D HSI. where Lproj enforces 2D reprojection consistency, Ldepth aligns the visible human surface with the estimated depth, Lcontact encourages plausible contact with scene surfaces, and Lpenetration penalizes intersections with the scene geometry.

19673#pg6>. Post-reconstruction quality control involves discarding examples with strongly truncated humans or implausibly small human mesh volumes <ref:2604. p. 3
Where the paper says it

To remove implausible or incomplete human insertions and ensure high-quality data, we apply three complementary, automated post-reconstruction filters. We discard examples with strongly truncated humans, generations where the edited image contains unindended scene changes besides inserting the human, and finally we remove human mesh reconstructions with implausibly small volumes.

19673#pg6>.

Dataset Utility and Impact

The resulting InHabitants dataset is characterized by high diversity, showing the most diverse interactions, target objects, and poses compared to prior work like TRUMANS and PROX-D <ref:2604.19673#pg6>. This synthetic data has been shown to improve downstream tasks; specifically, augmenting standard training data with InHabitants improves the SOTA contact estimation method DECO on the DAMON benchmark <ref:2604. p. 1

Where the paper says it

We demonstrate the value of the InHabitants data on two downstream tasks. First, augmenting standard training data with our samples improves the SOTA contact estimation method DECO [Tripathi et al.

19673#pg6>. Furthermore, feed-forward HSI reconstruction models consistently improve when trained on this dataset, achieving stronger performance on the PROX benchmark <ref:2604. p. 1
Where the paper says it

2026] consistently improve when trained on our data, achieving stronger performance on the PROX benchmark. We will release the dataset and code publicly.

19673#pg6>.

Evaluation Against Baselines

In a perceptual user study, InHabit was strongly preferred over GenZI and POSA++ in 78.1% of cases when participants selected the single best result considering realism, physical feasibility, and pose naturalness <ref:2604. p. 5

Where the paper says it

We conducted a perceptual user study, where 41 participants were shown triplets of generations made with InHabit, GenZI and POSA++ at corresponding scene locations, in random order, as interactive 3D visualizations. They were asked to select the single best result, considering the realism of human–scene interaction, physical feasibility and pose naturalness.

19673#pg6>. Quantitative metrics also show that InHabit yields renderings most CLIP-aligned with the prompt text, and its contact estimations best match prompt-based expectations <ref:2604. p. 5
Where the paper says it

2, humans generated by InHabit yield renderings most CLIP-aligned with the prompt text, and have body contacts that best match prompt-based expectations. 2 Data for Contact Estimation Body contact estimation predicts, from a single RGB image, which body-surface vertices touch the scene.

19673#pg6>.

Conclusion

InHabit demonstrates that internet-scale 2D foundation models encode enough knowledge of human–environment interactions to serve as a practical, scalable alternative to expensive 3D motion capture for training embodied AI methods <ref:2604.19673#pg6>. The principle is extensible to video generation and dynamic scenes in the future <ref:2604. p. 6

Where the paper says it

We showed that InHabit outperforms existing placement methods both quantitatively and in a perceptual user study, and that augmenting training data with InHabitants improves state-of-the-art contact estimation and feed-forward HSI reconstruction on established benchmarks. Currently, InHabit generates static poses within static 3D scenes, but the principle can be extended to video generation and dynamic scenes in the future as the underlying generative models continue to evolve.

19673#pg6>.

Acknowledgments

Special thanks RVH members for the help and discussion Nikita Kister thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support <ref:2604.19673#pg6>. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pradyumna <ref:2604. p. 7

Where the paper says it

Acknowledgments Acknowledgments: Special thanks RVH members for the help and discussion Nikita Kister thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pradyumna.

19673#pg6>. István Sárándi and Gerard Pons-Moll were supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 409792180 (Emmy Noether Programme, project: Real Virtual Humans) <ref:2604. p. 7
Where the paper says it

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pradyumna. István Sárándi and Gerard Pons-Moll were supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 409792180 (Emmy Noether Programme, project: Real Virtual Humans).

19673#pg6>. GPM is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1–Project number 390727645 and is supported by the Carl Zeiss Foundation <ref:2604. p. 7
Where the paper says it

István Sárándi and Gerard Pons-Moll were supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 409792180 (Emmy Noether Programme, project: Real Virtual Humans). GPM is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645 and is supported by the Carl Zeiss Foundation.

19673#pg6>.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, et al. 2023. GPT-4 Technical Report <ref:2604. p. 7

Where the paper says it

References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, et al. GPT-4 Technical Report.

19673#pg9>.

Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun Wu, Alexander Clegg, and C. p. 7

Where the paper says it

08774 (2023). Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun Wu, Alexander Clegg, and C.

Karen Liu. 2023. CIRCLE: Capture In Rich Contextual Environments <ref:2604. p. 7
Where the paper says it

Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun Wu, Alexander Clegg, and C. CIRCLE: Capture In Rich Contextual Environments.

19673#pg10>.

Michael J. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang.

2023. BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion <ref:2604. p. 7
Where the paper says it

Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion.

19673#pg9>.

Tim Brooks, Aleksander Holynski, and Alexei A. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Tim Brooks, Aleksander Holynski, and Alexei A.

Efros. 2023. InstructPix2Pix: Learning to Follow Image Editing Instructions <ref:2604. p. 7
Where the paper says it

Tim Brooks, Aleksander Holynski, and Alexei A. InstructPix2Pix: Learning to Follow Image Editing Instructions.

19673#pg9>.

Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik.

2020. Long-Term Human Motion Prediction with Scene Context <ref:2604. p. 7
Where the paper says it

Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-Term Human Motion Prediction with Scene Context.

19673#pg9>.

Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. p. 7

Where the paper says it

In Proceedings of the European Conference on Computer Vision (ECCV). Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll.

2026. Human3R: Everyone Everywhere All at Once <ref:2604. p. 7
Where the paper says it

Yue Chen, Xingyu Chen, Yuxuan Xue, Anpei Chen, Yuliang Xiu, and Gerard Pons-Moll. Human3R: Everyone Everywhere All at Once.

19673#pg10>.

Angela Dai, Angel X. p. 7

Where the paper says it

In International Conference on Learning Representations (ICLR). Angela Dai, Angel X.

Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. p. 7
Where the paper says it

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.

2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes <ref:2604. p. 7
Where the paper says it

Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes.

19673#pg10>.

Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez.

p. 7
Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez.

2022. PoseScript: 3D Human Poses from Natural Language <ref:2604. p. 7
Where the paper says it

Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. PoseScript: 3D Human Poses from Natural Language.

19673#pg10>.

Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. 2024. PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation <ref:2604. p. 7

Where the paper says it

Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez. PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation.

19673#pg10>.

Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, and Chunhua Shen. p. 7

Where the paper says it

PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation. Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, and Chunhua Shen.

2025. Diffusion Models are Efficient Data Generators for Human Mesh Recovery <ref:2604. p. 7
Where the paper says it

Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, and Chunhua Shen. Diffusion Models are Efficient Data Generators for Human Mesh Recovery.

19673#pg10>.

Google DeepMind. p. 1

Where the paper says it

Image-editing models [Brooks et al. 2023; Google DeepMind 2025a] can synthesize what it looks like to cook at a stove or read on a couch, as they have seen millions of such images, and vision–language models [Achiam et al.

p. 1
Where the paper says it

Image-editing models [Brooks et al. 2023; Google DeepMind 2025a] can synthesize what it looks like to cook at a stove or read on a couch, as they have seen millions of such images, and vision–language models [Achiam et al.

2025a. Gemini 2.5: Our Most Intelligent AI Model <ref:2604. p. 7
Where the paper says it

Google DeepMind. 5: Our Most Intelligent AI Model.

19673#pg9>.

Google DeepMind. 2025b. Gemini 3 <ref:2604. p. 7

Where the paper says it

https://deepmind. google/models/gemini/.

19673#pg9>.

Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll. p. 7

Where the paper says it

google/models/gemini/. Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll.

2024. Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion <ref:2604. p. 7
Where the paper says it

Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll. Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion.

19673#pg9>.

Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. p. 7

Where the paper says it

In International Conference on 3D Vision (3DV). Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll.

2021. Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors <ref:2604. p. 7
Where the paper says it

Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll. Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors.

19673#pg9>.

Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J.

Black. 2019. Resolving 3D Human Pose Ambiguities with 3D Scene Constraints <ref:2604. p. 7
Where the paper says it

Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Resolving 3D Human Pose Ambiguities with 3D Scene Constraints.

19673#pg9>.

Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J.

Black. 2021. Populating 3D Scenes by Learning Human-Scene Interaction <ref:2604. p. 7
Where the paper says it

Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J. Populating 3D Scenes by Learning Human-Scene Interaction.

19673#pg9>.

Chun-Hao P. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Chun-Hao P.

Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. p. 7
Where the paper says it

Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J.

Black. 2022. Capturing and Inferring Dense Full-Body Human-Scene Contact <ref:2604. p. 7
Where the paper says it

Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Capturing and Inferring Dense Full-Body Human-Scene Contact.

19673#pg9>.

Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang.

2024. Scaling Up Dynamic Human-Scene Interaction Modeling <ref:2604. p. 7
Where the paper says it

Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling Up Dynamic Human-Scene Interaction Modeling.

19673#pg9>.

Hyeonwoo Kim, Sangwon Baik, and Hanbyul Joo. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Hyeonwoo Kim, Sangwon Baik, and Hanbyul Joo.

2025. DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Models <ref:2604. p. 7
Where the paper says it

Hyeonwoo Kim, Sangwon Baik, and Hanbyul Joo. DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Models.

19673#pg9>.

Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo.

2024. Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models <ref:2604. p. 7
Where the paper says it

Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo. Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models.

19673#pg9>.

Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. p. 7

Where the paper says it

In Proceedings of the European Conference on Computer Vision (ECCV). Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A.

Efros, and Krishna Kumar Singh. p. 7
Where the paper says it

Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. Efros, and Krishna Kumar Singh.

2023. Putting People in Their Place: AffordanceAware Human Insertion into Scenes <ref:2604. p. 7
Where the paper says it

Efros, and Krishna Kumar Singh. Putting People in Their Place: AffordanceAware Human Insertion into Scenes.

19673#pg9>.

Lei Li and Angela Dai. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Lei Li and Angela Dai.

2024. GenZI: Zero-Shot 3D Human-Scene Interaction Generation <ref:2604. p. 7
Where the paper says it

Lei Li and Angela Dai. GenZI: Zero-Shot 3D Human-Scene Interaction Generation.

19673#pg9>.

Naureen Mahmood, Nima Ghorbani, Nikolaus F. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Naureen Mahmood, Nima Ghorbani, Nikolaus F.

Troje, Gerard Pons-Moll, and Michael J. p. 7
Where the paper says it

Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J.

Black. 2019. AMASS: Archive of Motion Capture as Surface Shapes <ref:2604. p. 7
Where the paper says it

Troje, Gerard Pons-Moll, and Michael J. AMASS: Archive of Motion Capture as Surface Shapes.

19673#pg9>.

Priyanka Patel, Chun-Hao P. p. 8

Where the paper says it

InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement • 9 of the IEEE/CVF International Conference on Computer Vision (ICCV). Priyanka Patel, Chun-Hao P.

Huang, Joachim Tesch, David T. p. 8
Where the paper says it

Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T.

Hoffmann, Shashank Tripathi, and Michael J. p. 8
Where the paper says it

Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael J.

Black. 2021. AGORA: Avatars in Geography Optimized for Regression Analysis <ref:2604. p. 8
Where the paper says it

Hoffmann, Shashank Tripathi, and Michael J. AGORA: Avatars in Geography Optimized for Regression Analysis.

19673#pg9>.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. p. 8

Where the paper says it

on Computer Vision and Pattern Recognition (CVPR). Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A.

Osman, Dimitrios Tzionas, and Michael J. p. 8
Where the paper says it

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Osman, Dimitrios Tzionas, and Michael J.

Black. 2019. Expressive Body Capture: 3D Hands, Face, and Body From a Single Image <ref:2604. p. 8
Where the paper says it

Osman, Dimitrios Tzionas, and Michael J. Expressive Body Capture: 3D Hands, Face, and Body From a Single Image.

19673#pg9>.

Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Dhruv Batra, and Roozbeh Mottaghi. p. 8

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Dhruv Batra, and Roozbeh Mottaghi.

2024. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots <ref:2604. p. 8
Where the paper says it

Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Dhruv Batra, and Roozbeh Mottaghi. 0: A Co-Habitat for Humans, Avatars and Robots.

19673#pg9>.

Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. p. 8

Where the paper says it

In International Conference on Learning Representations (ICLR). Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra.

2021. HabitatMatterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI <ref:2604. p. 8
Where the paper says it

Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. HabitatMatterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI.

19673#pg9>.

Manolis Savva, Angel X. p. 7

Where the paper says it

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.

Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. p. 8
Where the paper says it

08238 Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner.

2016. PiGraphs: Learning Interaction Snapshots from Observations <ref:2604. p. 8
Where the paper says it

Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. PiGraphs: Learning Interaction Snapshots from Observations.

19673#pg9>.

Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael J. p. 8

Where the paper says it

ACM Transactions on Graphics (TOG) 35, 4 (2016). Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael J.

Black. 2023. DECO: Dense Estimation of 3D Human-Scene Contact In The Wild <ref:2604. p. 8
Where the paper says it

Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael J. DECO: Dense Estimation of 3D Human-Scene Contact In The Wild.

19673#pg9>.

Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. p. 8

Where the paper says it

8001–8013. Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al.

2024. EmbodiedScan: A Holistic MultiModal 3D Perception Suite Towards Embodied AI <ref:2604. p. 8
Where the paper says it

Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. EmbodiedScan: A Holistic MultiModal 3D Perception Suite Towards Embodied AI.

19673#pg9>.

Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. p. 8

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang.

2022. HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes <ref:2604. p. 8
Where the paper says it

Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes.

19673#pg9>.

Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, and Gerard Pons-Moll. p. 8

Where the paper says it

In Advances in Neural Information Processing Systems (NeurIPS). Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, and Gerard Pons-Moll.

2025. PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image <ref:2604. p. 8
Where the paper says it

Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, and Gerard Pons-Moll. PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image.

19673#pg9>.

Pradyumna YM, Yuxuan Xue, Yue Chen, Nikita Kister, István Sárándi, and Gerard PonsMoll. p. 8

Where the paper says it

In ACM SIGGRAPH Asia. Pradyumna YM, Yuxuan Xue, Yue Chen, Nikita Kister, István Sárándi, and Gerard PonsMoll.

2026. GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction <ref:2604. p. 8
Where the paper says it

GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction. arXiv:2604.

19673#pg9>.

Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. p. 8

Where the paper says it

org/abs/2604. 19624 Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang.

2020. PLACE: Proximity learning of articulation and contact in 3D environments <ref:2604. p. 8
Where the paper says it

19624 Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. PLACE: Proximity learning of articulation and contact in 3D environments.

19673#pg9>.

Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, and Gerard Pons-Moll. p. 8

Where the paper says it

IEEE, 642–651. Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, and Gerard Pons-Moll.

2022. COUCH: Towards Controllable Human-Chair Interactions <ref:2604. p. 8
Where the paper says it

Xiaohan Zhang, Bharat Lal Bhatnagar, Vladimir Guzov, Sebastian Starke, and Gerard Pons-Moll. COUCH: Towards Controllable Human-Chair Interactions.

19673#pg9>.

Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. p. 8

Where the paper says it

In Proceedings of the European Conference on Computer Vision (ECCV). Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan.

2024. 3D-VLA: A 3D Vision-Language-Action Generative World Model <ref:2604. p. 8
Where the paper says it

Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3D-VLA: A 3D Vision-Language-Action Generative World Model.

19673#pg9>.

--- Page 1 ---

InHabit introduces an automatic and scalable data generator for populating 3D scenes with interacting humans by leveraging image foundation models to transfer commonsense knowledge from 2D to 3D, enabling the creation of a large-scale photorealistic human—scene interaction dataset.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated generation of large-scale 3D human–scene interaction data using a render–generate–lift principle allows for populating 3D scenes with interacting humans across varied environments and actions. This enables large-scale generation of interaction data for embodied 3D scene understanding, moving beyond simple geometric heuristics.

  2. Transferring commonsense knowledge from 2D foundation models to the 3D domain via a Vision–Language Model (VLM) and image-editing model proposes contextually meaningful actions tailored to the visible environment. This addresses the scarcity of data by leveraging implicit interaction priors in 2D foundation models.

  3. The system can generate a first large-scale photorealistic 3D human—scene interaction dataset, InHabitants, containing over 78k samples across ∼800 building-scale scenes with complete 3D geometry, SMPL-X bodies, and images. This dataset augments standard training data for RGB-based 3D human—scene reconstruction and contact estimation.

  4. The resulting data improves SOTA contact estimation methods like DECO by providing high-quality contact supervision that is generated fully automatically, which outperforms labor-intensive capture datasets like TRUMANS.

  5. InHabitants can be used to consistently improve feed-forward HSI reconstruction models, achieving stronger performance on benchmarks by training them on the synthetic data, as shown in Table 4 and Table 5 comparisons for GRAFT and Human3R methods.

  6. The system can generate diverse interactions involving multiple people, which was not supported by prior work, allowing for the modeling of complex social scenarios like a couple dancing or moving furniture together.

Abstract

Embodied AI and human-centric 3D scene understanding require data of people meaningfully interacting with diverse environments, yet such data remains scarce. Real-world capture has steadily pushed toward more scenes, but is bound to studios and at most around a hundred scenes, while synthetic datasets scale easily but place people without regard to what the scene affords. We argue that useful human-scene interaction data requires diversity in both scenes and interactions, and we show this with our proposed data engine InHabit. InHabit is a fully automatic engine that renders an existing 3D scene, uses 2D foundation models to propose and depict context-appropriate interactions, and lifts them into SMPL-X bodies, where the known scene geometry makes lifting well-constrained. Applied to Habitat-Matterport3D, InHabit produces InHabitants, which pairs photorealistic images with SMPL-X bodies and complete scene geometry across 6.6k rooms, about 60x more scenes than the largest captured dataset. We illustrate the quality and value of such large-scale human-scene interaction data through a variety of experiments. In a perceptual study, our interactions are preferred over two prior placement methods in 78% of cases. Across three tasks - 1) contact estimation, 2) human-scene reconstruction, and 3) generating humans in scenes - training with InHabitants outperforms training on synthetic data with exact ground truth (BEDLAM), motion-capture data (TRUMANS), and real RGB-D captures (PROX); e.g., GRAFT trained on InHabitants reaches a contact F1 of 0.594 on PROX, versus 0.492 when trained on PROX itself. Project page: https://virtualhumans.mpi-inf.mpg.de/inhabit/

Sources

Related papers