LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes

arXiv:2610.12069 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes".

Jane: The gist: LIVIN introduces a benchmark for spatial and embodied intelligence built on digital twins of 30 diverse lived-in homes to evaluate 3D detection, reconstruction, navigation,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Welcome back to the show. We’re talking about a new benchmark called LIVIN, which is really focusing on spatial and embodied intelligence in digital twins of homes. It’s been getting a lot of attention in the AI community lately, so we wanted to break down what it actually means for us.

Jane: Exactly, Tom. So basically, this paper introduces LIVIN as a way to test how well robots can handle the messy reality of living spaces—not just empty rooms, but actual homes with furniture and stuff everywhere. It sets a new standard for testing spatial intelligence in realistic domestic environments.

Lu: What's really impressive is the scale of what they built. They used digital twins from thirty diverse lived-in homes, which is a massive dataset for training or evaluating models to get good at recognizing and interacting with cluttered spaces like real houses do <ref:2610.12069#pg1,30 diverse lived-in homes>.

Meng: Three zero homes sounds like a lot of data, but what’s the actual output? Are we talking about just pretty pictures, or something functional that tests real robotic capabilities?

Tom: It’s functional, Meng. They aren't just making scenes; they are creating digital twins that preserve observed layouts and everyday objects. This means you can test things like three dee detection, reconstruction, navigation, and even physical manipulation in a setting that feels genuinely lived-in <ref:2610.12069#pg2,3D detection, reconstruction, navigation, and>.

Jane: And the construction process itself is interesting because it uses a human-in-the-loop workflow. It involves instance recognition, architectural reconstruction, and object generation with stage-wise human verification to make sure everything matches what was actually seen in the real homes.

Lu: They are using different strategies for different objects too. Articulated objects, like chairs or drawers, are modeled using GPT-six Astra through Blender code to keep their kinematic structure intact. Non-articulated items are generated by Rodin from enhanced reference images and then placed to match the captured arrangements.

Meng: That sounds technically heavy but necessary for realism. So, what’s the actual performance being measured on this LIVIN benchmark? What are they testing specifically?

Tom: They are testing four main tasks: three dee detection, which is about object localization and geometry recovery under clutter and occlusion; three dee reconstruction, which assesses geometry recovery in those same dense arrangements; navigation, which targets long-horizon humanoid navigation requiring robots to reason about routes through constrained spaces; and loco-manipulation <ref:2610.12069#pg2>.

Title and authors: Jane: Navigation is a big one because it requires the robot to execute physically feasible wholebody motion through spaces where everything is crowded and you have to plan a long route. And they found GPT-six Astra performed best overall in that task, achieving a success rate of forty-one point four eight percent, which they called the highest SPL.

Lu: They also looked at loco-manipulation, which evaluates physical interaction like pick-and-place or pushing, and they noted that GPT-six Astra achieved the highest success rates across all four task families in this benchmark. It really shows its strength in coupling locomotion with dependent manipulation subgoals.

Meng: So, if we look at the results for detection and reconstruction, what’s the main challenge they found when comparing different methods? Is it that the data is too cluttered?

Tom: Yes, they found that current methods remain challenged by the dense object arrangements, occlusions in those homes, limited free space around objects, and constrained interaction regions. In three dee detection specifically, accurate localization of detected objects doesn't guarantee complete scene coverage <ref:2610.12069#pg2>.

Jane: It sounds like the environment itself is the main hurdle for existing AI models when they try to operate in these realistic settings. So how does this benchmark help us understand where we are now?

Lu: LIVIN provides three key contributions here. First, LIVIN-Scenes, which are those thirty diverse lived-in homes that give us observation-grounded digital twins preserving realistic layouts and object arrangements under natural spatial constraints <ref:2610.12069#pg1,30 diverse lived-in homes>.

Meng: The second contribution is that human-in-the-loop reconstruction workflow, which combines agentic reasoning with code-based modeling and image-conditioned three dee generation to rebuild structure, articulated objects, and belongings with stage-wise verification <ref:2610.12069#pg2>. That sounds like a very robust way to build high fidelity data.

Tom: And the third thing is LIVIN-Bench itself. It’s a unified evaluation suite covering three dee detection, three dee reconstruction, navigation, and loco-manipulation to evaluate scene understanding and robot mobility in these realistic household environments <ref:2610.12069#pg2,reconstruction, navigation, and loco-manipulation>.

Jane: Overall, the results suggest that LIVIN better captures the object richness and interaction-relevant structure of lived-in household environments compared to what we’ve seen before. It sets a testbed for driving progress toward more robust and generalizable embodied systems.

Title and authors: Lu: It really highlights how much richer the structure of real household objects is than many existing simulation resources capture, especially concerning articulation and dense packing.

Meng: So, for practical application, what does this mean for building robots that can actually live in these places? Are we closer to something deployable?

Tom: It moves us toward systems that are more robust when encountering the real world's spatial constraints. The paper concludes by emphasizing that existing methods struggle with dense object arrangements, frequent occlusion, and physically constrained interactions. This gives us a clear direction for future research on embodied AI in home environments.

Jane: So we’ve seen how LIVIN is constructed, what tasks it tests, and the performance of models like GPT-six Astra in those areas. It really shows the gap between simulated environments and actual lived-in realities.

Lu: The work provides a concrete framework for moving beyond approximate scene replicas toward observation-grounded digital twins that preserve real household layouts, rich object instances, and interactable articulation.

Meng: It sounds like the next step is figuring out how to deploy models that can handle that kind of dense interaction reliably without needing constant manual correction during training.

Tom: Right. And as we wrap up this deep dive into LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes, it’s clear that the challenge isn't just rendering pretty scenes anymore; it’s capturing the complex, constrained reality of everyday life for spatial intelligence.

Jane: It’s a really important tool for anyone working on embodied AI who wants to ensure their systems can handle those messy, real-world domestic scenarios.

Lu: This benchmark really pushes the field to focus on building models that understand not just geometry, but also the physical relationships and constraints within these complex settings.

Meng: And I think seeing how GPT-six Astra handled loco-manipulation across all those tasks gives us a good idea of what high performance looks like in terms of integrated capability.

Tom: That’s what we were talking about. We’ll be looking at how these findings influence the next wave of embodied system development, and we'll be right back after the break.

The paper's summary: Tom: So, we’ve talked about how LIVIN is built—using digital twins from thirty different homes to test robots in real messes. Now we need to talk about what the paper actually says about this benchmark itself and why it matters for embodied AI in homes.

Jane: Right, so the core idea of the LIVIN benchmark is that current testing methods aren't good enough for messy, lived-in spaces because they lack realism regarding clutter and how things are arranged daily.

Lu: The authors show that this benchmark isn't just about collecting pictures; it’s a structured way to build observation-grounded digital twins. They created a human-in-the-loop workflow where people look at the results and correct them before moving on.

Meng: That human oversight part is key, because it means the resulting data—the digital twins—is supposed to be much more faithful to how a real house actually looks, including those tricky object arrangements.

Tom: Exactly. The paper contributes three main things: first, these LIVIN-Scenes themselves; they are thirty diverse homes that capture real layouts and objects under natural spatial constraints.

Jane: Second, this human-in-the-loop reconstruction workflow; it’s how they build those twins by combining agentic reasoning with code modeling for articulated objects and image conditioning for everyday belongings.

Lu: And the third contribution is LIVIN-Bench; which is this unified evaluation suite testing three D detection, three D reconstruction, navigation, and loco-manipulation all at once.

Tom: What this means for us in the broader AI space is that we finally have a proper testbed to push spatial and embodied intelligence. It’s not just about navigating an empty hallway anymore; it’s about interacting with a kitchen or a cluttered living room.

Jane: The results they present show that existing methods still struggle because of those dense object arrangements, the occlusions, and the limited free space you find in actual homes.

Meng: So while GPT-six Astra showed top performance in navigation and manipulation, they also flagged some issues with transitioning from just approaching a target to actually grabbing it. That’s a practical limitation for building reliable physical systems.

Lu: The paper concludes that LIVIN captures the object richness and interaction structure of these environments better than previous resources because it models those physical constraints deeply.

Tom: It sets a new bar for what we expect from spatial understanding in robotics, moving us toward systems that can handle the unpredictable nature of real-world domestic settings.

Jane: And this benchmark is now a tool to drive progress toward more robust and generalizable embodied systems that can actually work reliably in everyday environments.

Lu: Now that we see what’s missing—the dense packing, the occlusion—we have a much clearer picture of where the next generation of embodied AI needs to focus its training and modeling efforts.

The paper's improvements: Tom: So, we’ve seen how LIVIN is built—using digital twins from thirty different homes to test robots in real messes, and now we’re talking about what the paper suggests are the specific improvements they propose for this benchmark itself.

Jane: Right, so it sounds like they aren't just presenting a static set of tests; they’re proposing ways to make the testing process even more robust and useful for developing better AI systems.

Lu: The first improvement is around scene construction—they call it the Automated Human-in-the-Loop Scene Construction. It lets the system do instance recognition, architectural reconstruction, and object placement while humans review and correct intermediate results at every stage.

Meng: That’s significant for engineering because it tackles the problem of fidelity head-on; it ensures that when we build these digital twins, they are actually accurate representations of what’s in a real home.

Tom: And then there's the second improvement, which focuses on three dee detection accuracy under clutter. They suggest leveraging methods like Boxer to get better F1 scores for object localization even when things are dense and occluded.

Jane: But they also flag that even with high detection accuracy, you still have to worry about complete scene coverage; it’s not just about finding the object, it’s about knowing the whole environment.

Lu: The third improvement is in navigation, where they suggest a mixed strategy for robots—the robot selects between GR00T and CAT based on the current context, switching to CAT when more obstacle avoidance capability is needed.

Tom: That sounds like a smarter decision-making process for the robot itself, rather than just following one fixed path through a crowded room.

Jane: And for loco-manipulation, they highlight GPT-six Astra again because it achieved a sixty-one point nine percent success rate on articulated object manipulation, which is really high for that kind of physical interaction.

Meng: So what this means practically is that the next iteration of robotic training won't just use standard navigation; it will use these smarter decision-making layers to handle tight spaces better.

Lu: The implication is that by focusing on these specific improvements—the human-in-the-loop building and the adaptive navigation strategies—we can push embodied AI into those complex, real-world domestic scenarios much more effectively.

Tom: Essentially, they are showing us how to move from just testing robots in controlled sims to testing them in environments that actually mimic the complexity of real living.

Jane: And this benchmark is now positioned as the go-to resource for anyone wanting to evaluate spatial understanding and physical interaction in realistic home settings.

Lu: Now that we’ve covered the construction and the improvements, we have to ask—how do these results compare to other recent work on embodied AI?

Conclusion: Tom: So we’ve gone through how LIVIN was constructed, what tasks they tested—detection, reconstruction, navigation, and manipulation—and what improvements they suggested for the benchmark itself.

Jane: It boils down to this: LIVIN provides a much more rigorous and realistic testbed for spatial and embodied intelligence in homes than anything we’ve seen before.

Lu: The authors are pushing us toward systems that understand not just geometry, but the actual physical constraints and object relationships found in lived-in spaces.

Meng: For practical engineering, this means that when we train robots for real-world deployment, we have a much clearer picture of the failure modes we need to fix—specifically those dense packing and interaction struggles.

Lalam: From an AI perspective, this work shows that culture around AI development needs to shift toward creating these high-fidelity training grounds so the resulting models are genuinely useful in complex human environments.

Tom: It changes how we think about testing embodied systems because it forces us to address the messiness of real life instead of just clean simulation.

Jane: The paper essentially argues that current methods aren't capturing enough detail about object richness and interaction structure in these digital twins.

Lu: So, LIVIN-Scenes and LIVIN-Bench give us a concrete framework for evaluating how well spatial intelligence translates into physical action within constrained areas.

Tom: It’s a really solid contribution because it moves the needle on creating more robust embodied systems that actually work in messy, real homes.

Jane: And moving forward, we need to see if these high-fidelity digital twins can eventually lead to more generalizable AI agents that can operate reliably anywhere.

Meng: I’m curious if this level of fidelity is something we can realistically achieve with current computational resources or if it requires a massive jump in training data volume.

Lalam: The vision here is a world where AI systems are not just smart in theory, but actually capable of navigating and interacting safely within the intricate reality of human dwellings.

Tom: That’s the big picture we’re talking about. We’ve seen how LIVIN was built, what it tests, and its suggested improvements for testing itself.

Jane: It really solidifies that the future of embodied AI in home environments depends on capturing that deep spatial understanding accurately.

Lu: So next time we look at this research, we should keep an eye on how these new construction workflows integrate with agentic reasoning to see what else they discover.

Peijun Xu, Chuansen Nie, Yiyang He, Yinuo Bai, Jingyang Liu, Kuixiang Shao, Yuyang Jiao, Kuanhao Xia, Jiayi Zhu, Zitian Yang, Yanqi Zhang, Tianye Tan

ShanghaiTech University

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-09

Code: https://github.com/NVlabs/GR00T-WholeBodyControl

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: LIVIN introduces a benchmark for spatial and embodied intelligence built on digital twins of 30 diverse lived-in homes to evaluate 3D detection, reconstruction, navigation, and

Key concepts

Digital Twins of Lived-in Homes
These are highly realistic 3D replicas of actual homes. They capture not just the room layout but also the specific furniture arrangements and everyday objects found in real living spaces. This realism is crucial because robots need to understand these dense, cluttered environments to navigate and interact with them effectively.
Human-in-the-Loop Workflow
This is a process where humans actively guide the creation of the digital twins. It involves steps like recognizing objects, reconstructing the architecture, and generating/placing furniture. Humans review intermediate results to correct errors before moving to the next step, ensuring high fidelity.
Loco-manipulation
This task evaluates a robot's ability to physically interact with its environment. It includes complex actions like picking up objects, pushing items, and performing long-horizon tasks that require combining movement (locomotion) with precise manipulation of objects within the constrained space of a home.

Terminology

Summary

The gist: LIVIN introduces a benchmark for spatial and embodied intelligence built on digital twins of 30 diverse lived-in homes to evaluate 3D detection, reconstruction, navigation, and loco-manipulation in realistic domestic environments.

LIVIN Benchmark Construction

LIVIN is constructed from digital twins of 30 diverse lived-in homes to capture realistic household layouts, furnishings, and daily-use objects The construction process involves a human-in-the-loop workflow comprising instance recognition, architectural reconstruction, and object generation and placement This workflow treats articulated and non-articulated objects with different reconstruction strategies, where articulated objects are modeled by GPT-6 Astra through Blender code to preserve their kinematic structure Non-articulated objects are generated by Rodin from enhanced reference images and then placed to match the captured arrangements Each construction stage is followed by human review, allowing intermediate results to be inspected and corrected against the source observations before proceeding

Evaluation Tasks

LIVIN supports four benchmarks for joint evaluation:

  1. 3D detection, which evaluates object localization and geometry recovery under clutter, occlusion, and dense household arrangements Methods are compared using metrics such as F-Score-S and F1

  2. 3D reconstruction, which assesses geometry recovery under clutter, occlusion, and dense household arrangements Methods like Fire3D and ShapeR show varying performance in this task

  3. Navigation, which targets long-horizon humanoid navigation, requiring robots to reason about routes and execute physically feasible wholebody motion through constrained spaces GPT-6 Astra performed best overall in this task, achieving a 41.48% success rate, the highest SPL

  4. Loco-manipulation, which evaluates physical interaction, including pick-and-place, pushing, articulated-object manipulation, and longhorizon tasks that couple locomotion with dependent manipulation subgoals GPT-6 Astra achieved the highest success rates across all four task families in this benchmark

Performance Findings

Evaluations show that current methods remain challenged by the dense object arrangements, occlusions, limited free space, and constrained interaction regions found in realistic lived-in homes In 3D detection, results indicate that accurate localization of detected objects does not ensure complete scene coverage For navigation, GPT-6 Astra showed superior performance compared to OmniNav and DualVLN, although it can still be limited by imperfect scene understanding in complex household environments In loco-manipulation, while GPT-6 Astra achieved the highest success rates, difficulties were noted in transitioning from target approach to manipulation for grasping tasks

Key Contributions

The work presents three main contributions:

  1. LIVIN-Scenes: 30 diverse lived-in homes, providing observation-grounded digital twins that preserve realistic household layouts, everyday object arrangements, and rich object categories under natural spatial constraints

  2. A human-in-the-loop reconstruction workflow: This workflow combines agentic reasoning, code-based modeling, and image-conditioned 3D generation to reconstruct architectural structure, articulated objects, and everyday belongings with stage-wise human verification

  3. LIVIN-Bench: This is a "unified evaluation suite for 3D and embodied tasks, spanning 3D detection, 3D reconstruction, navigation, and loco-manipulation to evaluate scene understanding, robot mobility, and physical interaction in realistic household environments"

The results demonstrate that LIVIN better captures the object richness and interaction-relevant structure of lived-in household environments This benchmark provides a testbed for driving progress toward more robust and generalizable embodied systems. The paper concludes by highlighting that existing methods struggle with dense object arrangements, frequent occlusion, and physically constrained interactions.

--- Page 1 ---

LIVIN: BENCHMARKING SPATIAL AND EMBODIED INTELLIGENCE IN DIGITAL TWINS OF LIVED-IN HOMES Peijun Xu1, Chuansen Nie1, Yiyang He1, Yinuo Bai1,2 Jingyang Liu1 Kuixiang Shao1 Yuyang Jiao1 Kuanhao Xia1 Jiayi Zhu1 Zitian Yang1 Yanqi Zhang Tianye TanTianye TanShuwei DiJunyi XuJingyi YuJiayuan Gugujy

ABSTRACTRealistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction. Existing resources often trade off scale, real-world correspondence, and interaction readiness, leaving a gap in faithful, interactive replicas of how real homes are actually arranged. To this end, we introduce LIVIN, a benchmark for spatial and embodied intelligence built on digital twins of 30 diverse lived-in homes. These replicas preserve observed room layouts, furniture configurations, and everyday belongings. To construct them, we design a human-in-the-loop workflow comprising instance recognition, architectural reconstruction, and object generation and placement We evaluate four tasks in LIVIN: 3D detection, 3D reconstruction, navigation, and loco-manipulation Our evaluations show that current methods remain challenged by the dense object arrangements, occlusions, limited free space, and constrained interaction regions found in realistic lived-in homes. We hope LIVIN will help advance embodied AI in real-world homes, from spatial understanding to robotic interaction, and ultimately bring embodied intelligence into everyday home environments.

1 INTRODUCTIONEveryday homes impose dense and highly variable spatial constraints on robot behavior: furniture limits traversable space, belongings create clutter and occlusion, and surrounding objects affect how a target can be reached or manipulated. Household robots therefore require both spatial intelligence to understand objects, geometry, and spatial relationships, as well as embodied intelligence to act within these constraints. Environmental diversity is crucial for generalization; for example, π0.5 reports improved performance as mobile-manipulation training scales from 3 to 104 locations (Intelligence et al., 2025). While simulation can scale such experience, constructing realistic, interactive replicas of diverse lived-in homes remains challenging. Existing resources address different aspects of the problem. Artist-designed datasets and procedural generators provide diverse layouts and object configurations (Fu et al., 2021; Deitke et al., 2022; Raistrick et al., 2024), but are synthesized rather than grounded in observed homes. Other efforts such as MetaScenes and InternScenes (Yu et al., 2025; Zhong et al., 2026) convert real-world scans into simulatable scenes with detailed object arrangements. These pipelines, however, rely heavily on asset retrieval and provide limited coverage of articulated household objects. More recently, agentic systems such as SceneSmith (Pfaff et al., 2026) automate increasing portions of scene and asset construction. Together, these advances make it possible to move beyond approximate scene replicas toward observation-grounded digital twins that preserve real household layouts, rich object instances, and interactable articulation, enabling spatial understanding and embodied behavior to be evaluated in the challenging realistic environments.

2 RELATED WORKReal-to-Sim Generation. Recent 3D generative models have substantially improved the conversion of real-world observations into reusable virtual assets and structured scenes. At the object level, methods such as TRELLIS, Clay, and SAR3D recover detailed geometry and appearance from sparse visual inputs (Xiang et al., 2025; 2026; Zhang et al., 2024), while some works further extend 3D generation to articulated and interaction-ready assets with explicit part structures and motion parameters (Liu et al., 2024; Zhou et al., 2026; Gao et al., 2025; Wang et al., 2026). At the scene level, recent methods such as MIDI and SceneGen generate compositional 3D environments by jointly modeling object geometry and scene layout from visual or structured conditions (Huang et al., 2025; Meng et al., 2026), while CAST, SAM3D and RecGen reconstruct 3D scenes by recovering object geometry and estimating object poses from visual observations (Yao et al., 2025; Chen et al., 2026b; Zadaianchuk et al., 2026). LIVIN extends this direction to complete lived-in homes through a human-in-the-loop agentic workflow that reconstructs architectural structure, scene layout, articulated objects, and high-fidelity everyday objects, producing more complete and faithful replicas of real household environments.

Indoor Scene Datasets. Existing indoor scene resources are built through several distinct approaches. Artist-designed datasets such as 3D-FRONT and HSSD offer high-quality structured interiors but require substantial manual effort (Fu et al., 2021; Khanna et al., 2024).

Improvements for AI systems

  1. Bold header: Automated Human-in-the-Loop Scene Construction

The improved system can perform instance recognition, architectural reconstruction, and object generation and placement using a workflow where intermediate results are reviewed and corrected by humans against the source observations at each stage, ensuring high fidelity to observed layouts and object arrangements in real homes.

  1. Bold header: Enhanced 3D Detection Accuracy

The system can improve localization under clutter by leveraging methods that show high F1 scores, such as Boxer, while addressing the challenge that accurate localization of detected objects does not ensure complete scene coverage, suggesting an integrated approach to object recovery and scene understanding.

  1. Bold header: Robust Long-Horizon Navigation

The improved navigation system can achieve higher success rates by utilizing a mixed strategy where the robot selects between GR00T and CAT based on the current context, allowing it to switch to CAT when additional obstacle-avoidance capability is needed, improving its ability to navigate narrow or cluttered spaces.

  1. Bold header: High-Fidelity Articulated Object Modeling

The system can generate articulated objects using a method that jointly model[s] its geometry, materials, articulation, and placement through Blender code, preserving their kinematic structure while fitting the surrounding environment, including modeling internal compartments, drawer boxes, handles.

  1. Bold header: Improved Loco-Manipulation Success

The system can achieve higher success rates in complex tasks by utilizing GPT-6 Astra which achieves a 61.9% SR on articulated-object manipulation and demonstrating superior performance over other models in evaluating physical interaction within constrained household tasks.

Sources

Related papers