Spatial Foundry: A Multimodal Dataset for Spatial Intelligence
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Spatial Foundry: A Multimodal Dataset for Spatial Intelligence".
Jane: Here's the gist of the paper. Spatial Foundry introduces a collection of 5,280 furnished residential 3D scene configurations with aligned visual, geometric, and temporal data to enable spatial intelligence applications.
Tom: Data Modalities and Scene Representation The dataset combines floor plans, perspective and panoramic RGB, metric depth, point clouds, meshes, instance-level 3D bounding boxes, and camera trajectories paired with video.
Jane: The representation of observation t is conceptually expressed as os t = (I s t, D s t, Π s t, T wc, τ s t).
Tom: Perspective observations use calibrated pinhole views and panoramic observations use an equirectangular camera model with a declared orientation.
Paper summary: Tom: Welcome back to the show. We're talking about a new dataset called Spatial Foundry, which is a collection of five thousand two hundred eighty furnished residential three dimensional scene configurations with aligned visual, geometric, and temporal data.
Jane: It sounds like it's trying to solve that problem where just having pictures doesn't tell you the real relationship between what you see and the space itself.
Lu: Exactly. The whole thesis of Spatial Foundry is about connecting visual observations to the geometry, objects, and persistent state of an environment because images alone leave those relationships implicit, while disconnected annotations make it hard to study appearance or camera motion under a common reference point >
Meng: So what does this dataset actually include beyond just pictures? I need to know the concrete data points here >
Tom: It includes floor plans, perspective and panoramic RGB images, metric depth maps, point clouds, meshes, instance level three dimensional bounding boxes for objects, and camera trajectories paired with video >
Jane: That’s a lot of different ways to look at the same scene all at once. The paper claims this collection is important because it links these modalities together using shared scene coordinates across views and time >
Lu: The construction workflow they describe is key here. It connects residential design rules to editable Blender scenes and aligned synthetic observations through a process where GPT-six generates structured layouts, and then human reviewers check the spatial usability >
Meng: So it takes design priors, structures them, puts them in Blender with assets like Infinigen, and then adds the temporal data from camera movements and object state changes >
Tom: Right. And they focus on separating changes in observation from changes in the environment itself to keep things clean >
Jane: Then we have these controlled data generation workflows, which let you fix the camera path to see before and after an edit, or hold the path fixed to compare different scene edits >
Lu: Multiple observer cameras can share the same object state sequence, which extends that supervision to cross view consistency and shared world modeling across different viewpoints >
Meng: That sounds useful for training models where you want them to understand how different agents perceive a common environment, but what's the actual cost of running this setup at scale?
Paper summary: Tom: Well, the paper mentions that these applications span floor plan conditioned three dimensional generation, geometric reconstruction, instance level perception, camera conditioned video generation, shared world modeling and scene editing >
Jane: It’s connecting layout constraints to the three dimensional structure and appearance of a place you can actually walk through virtually >
Lu: Geometric reconstruction uses perspective and panoramic RGB observations to estimate depth or construct a representation that supports novel views because they have metric geometry attached >
Meng: And instance level spatial perception lets models localize an object, estimate its extent, and associate it with observations from other viewpoints, which is important for robotics maybe >
Tom: Camera conditioned video generation uses trajectory video pairs to learn how to generate corresponding sequences of views under specified motion because you have the trajectory and the video paired up >
Jane: So what does this mean for someone who just wants to understand a room? It means they can study how objects move or change when you look at them from different angles, given the original floor plan >
Lu: The collection spans five thousand two hundred eighty furnished configurations with aligned observations and editable source scenes which lets you render additional observations under specified cameras and object configurations >
Meng: That sounds like a massive storage requirement for handling all that geometry and state data, but they do have these editable source scenes, which is good because it means you can change the environment itself >
Tom: The construction process preserves the correspondence between plan elements and their three dimensional realization producing those paired floor plans and furnished environments with explicit geometry and object transforms >
Jane: It’s linking those layout constraints directly to the physical structure and how everything looks on top of it >
Lu: The pipeline for aligned multimodal data links rendered observations to camera calibration, trajectories, geometry, object annotations, and scene state records which expresses the shared spatial and temporal associations among all those modalities >
Meng: And they detail how this works by describing the exported camera record which specifies the depth convention and coordinate transform connecting instance annotations to the same RGB depth observation >
Tom: So when you look at a camera trajectory, you see positions p of time and orientations R of time that connect rooms and floors through those movements >
Paper summary: Jane: That means if you have an object annotation, its location in the world is tied to that specific camera's view and its recorded depth measurement >
Lu: Object translation, rotation, insertion, removal, and appearance changes update explicit scene state so you can study how those visual consequences look when the object moves or changes >
Meng: Fixed camera paths give you observations before and after an edit to isolate scene changes specifically to the environment itself rather than just camera movement >
Tom: And holding the path fixed gives you before and after edit pairs, which is a good way to check if an object change is what you think it is >
Jane: It’s all about having these paired floor plans and furnished environments that connect layout constraints to three dee structure and appearance so you can test things systematically > <ref:2611.17318#pg1>
Lu: The diversity of paired environments connects these tasks to a broad range of residential configurations, spanning US style layouts with varied room compositions and furnishing arrangements >
Meng: So it’s not just one type of house; it covers the variety in how people actually furnish their living spaces >
Tom: The data organization supports floor plan conditioned three dimensional scene generation, geometric reconstruction, instance level perception, camera conditioned video generation, multi agent shared world modeling and state conditioned scene editing >
Jane: It’s covering everything from understanding a static room to predicting what the view will look like during exploration and interaction >
Lu: The applications span static scene understanding and the prediction of observations during exploration and interaction because you can use this data for both things >
Meng: So, in short, it’s a big dataset that lets you test models on a really rich variety of real-world spatial scenarios but it relies heavily on having this structured input data to work properly >
Jane: It’s a really comprehensive resource for anyone doing research in spatial intelligence that needs to tie together the layout, the three dee geometry, and the visual experience > <ref:2611.17318#pg1>
Lu: The construction workflow connects residential design priors to editable Blender scenes and aligned synthetic observations through rule conditioned layout generation and human verification >
Paper summary: Meng: So, they’re not just throwing a bunch of scans together; they are building them following some kind of design logic first >
Tom: And the post processing extracts evaluated mesh geometry in a common world coordinate system connecting object annotations to paired 2D layouts and reference surface cloud from the scene retaining the instance and semantic associations of its source geometry > <ref:2611.17318#pg3>
Jane: That post processing step is crucial because it connects everything back into that single world coordinate system where you can actually use it for things like geometric reconstruction >
Lu: The mobile scan simulation combines reference surface samples, virtual scanner locations, rendered appearance, and geometric perturbations to approximate scanner occlusion by retaining the nearest sampled surface point in each bin >
Meng: And they also export a camera record that specifies the depth convention and coordinate transform connecting instance annotations to the same RGB depth observation >
Lu: Object translation rotation insertion removal and appearance changes update explicit scene state so you can study how those visual consequences look when the object moves or changes >
Meng: Fixed camera paths yield corresponding observations before and after an edit isolating scene changes to the environment itself rather than just camera movement >
Tom: And multiple observer cameras share the same object state sequence providing cross view observations of each change, which is really interesting for modeling how different agents perceive a common change >
Jane: It’s all about having paired floor plans and furnished environments that connect layout constraints to three dee structure and appearance so you can test things systematically across different views > <ref:2611.17318#pg1>
Meng: So it’s not just one type of house; it covers the variety in how people actually furnish their living spaces, which is important for real world deployment >
Meng: So, in short, it’s a big dataset that lets you test models on a really rich variety of real world spatial scenarios but it relies heavily on having this structured input data to work properly >
Lu: The mobile scan simulation combines reference surface samples, virtual scanner locations, rendered appearance and geometric perturbations to approximate scanner occlusion by retaining the nearest sampled surface point in each bin >
Meng: So it’s not just one type of house; it covers the variety in how people actually furnish their living spaces, which is important for
Conclusion: Tom: So, we're wrapping up our look at Spatial Foundry, which is this dataset of five thousand two hundred eighty furnished residential scenes with all that visual and spatial data attached to them >
Jane: It’s a huge collection, Tom. The point of the paper is really about providing these aligned observations so you can test models that understand how space works in three dimensions, not just looking at flat pictures >
Lu: They built a whole pipeline connecting the design rules, like floor plans, to actual three dee geometry and then pairing that with camera movements and object changes so the data is totally consistent across time and different views >
Meng: So what's the actual big picture here? What does this mean when you're trying to build something that needs to understand a physical space reliably?
Tom: Well, it gives AI models a way to learn about spatial relationships by showing them how the layout connects directly to the three dee structure and how those structures change when things move inside >
Jane: It’s connecting the constraints of a floor plan—like where walls are—to what that space actually looks like in three dimensions, which is super useful for generating realistic environments or even for robotics that needs to navigate unseen spaces >
Lu: The authors focus on making sure the data generation process is controlled, meaning they can separate changes in the camera movement from actual changes happening to the house itself so you get clean comparisons >
Meng: That control is important because when you're shipping models, you need reproducible results. If a model makes a mistake, you want to know if it was because the model got confused by a weird camera angle or if it misunderstood how the furniture was actually placed >
Tom: It means they are providing not just raw photos but structured data that links every single observation—from RGB images to depth maps—to the exact position and state of every object in the scene >
Jane: And that alignment is what lets these different applications work, like reconstructing a room from just a few pictures or even generating new video sequences based on how a camera moves >
Lu: It’s moving beyond just seeing what's there to understanding how to interact with it, whether that's editing the scene or having multiple agents learn how to perceive the same environment >
Meng: So this collection is basically a massive training ground for making AI systems genuinely aware of physical space rather than just pattern matching on isolated images >
Tom: Yeah, and the authors put all this together with a construction workflow that starts with design rules and ends up with these perfectly aligned data sets ready for serious spatial intelligence applications >
Jane: It’s a big step toward making AI models capable of understanding the 'why' behind what they see in a physical space, not just the 'what' >
Lu: And next, we'll look at how they actually use this to reconstruct three dee geometry from those images and what kind of results they are seeing there >
Ethan Lin, Yang Song
Ranolabs
cs.CV
Submitted: 2026-11-12
Updated: 2026-11-12
Comments: Project Page: https://ranolabs.com/Blog/spatial-foundry
Project page: https://facebookresearch.github.io/projectaria_tools/docs/open_d
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: Spatial Foundry introduces a collection of 5,280 furnished residential 3D scene configurations with aligned visual, geometric, and temporal data to enable spatial intelligence applications.
Key concepts
- Data Modalities
- This refers to the different types of data used in the dataset. It includes floor plans, perspective/panoramic RGB images, metric depth maps, point clouds, meshes, 3D bounding boxes for objects, and camera trajectories paired with video. These modalities are all aligned to provide a comprehensive view of the scene.
- Scene Construction Workflow
- This is the process used to create the final furnished scenes. It starts by using GPT-6 to generate structured floor plans based on residential rules, which are then reviewed by humans for usability. This layout is then assembled using Infinigen and Blender assets to produce the final source scenes, ensuring a link between design rules and 3D reality.
- Controlled Data Generation
- This technique allows researchers to separate changes in the scene from changes in the camera's view. By fixing camera paths, researchers can compare observations before and after an edit, isolating what actually changed in the environment. This method supports reproducible comparisons for spatial perception tasks.
- Spatial Intelligence Applications
- The dataset supports six key areas: floor-plan-conditioned 3D generation, geometric reconstruction from visual data, instance-level perception to locate objects, camera-conditioned video generation based on motion, shared world modeling among multiple cameras, and state-conditioned scene editing. These applications connect layout constraints to 3D structure and appearance.
Terminology
Summary
Spatial Foundry introduces a collection of 5,280 furnished residential 3D scene configurations with aligned visual, geometric, and temporal data to enable spatial intelligence applications. p. 0
Data Modalities and Scene Representation
The dataset combines floor plans, perspective and panoramic RGB, metric depth, point clouds, meshes, instance-level 3D bounding boxes, and camera trajectories paired with video. The representation of observation t is conceptually expressed as os t = (I s t, D s t, Π s t, T wc, τ s t). p. 3 Perspective observations use calibrated pinhole views and panoramic observations use an equirectangular camera model with a declared orientation. p. 3 Object records include instance identifiers, names or semantic descriptions where available, metric bounds, and transforms. p. 3 Table 1 shows the correspondence between modalities such as Floor plans Layout, room boundaries, metric geometry. p. 4
Scene Construction Workflow
The construction workflow connects residential design priors to editable Blender scenes and aligned synthetic observations. p. 4 The process involves several steps:
** GPT-6 generates structured layouts from residential rules and human reviewers check their spatial usability. The assembly uses Infinigen and Blender assembly with assets to produce the furnished source scenes. p. 1 Reference geometry supports virtual scanning and calibrated rendering, while camera trajectories and object-state changes produce aligned temporal observations. p. 5 The central organizing principle is to separate changes in observation from changes in the environment. p. 1**
Controlled Data Generation Workflows
The collection supports controlled comparisons by separating camera variation from scene variation. Specifically, fixed camera paths yield corresponding observations before and after an edit. p. 7 Holding the camera path fixed provides before-and-after edit pairs. p. 3 Multiple observer cameras can share the same object-state sequence, extending this supervision to cross-view consistency and shared world modeling. p. 3
Applications in Spatial Intelligence
The aligned data supports six application areas derived from the dataset. These applications include:
** Floor-plan-conditioned 3D generation, geometric reconstruction, instance-level perception, camera-conditioned video generation, shared world modeling, and scene editing. Paired floor plans and furnished environments connect layout constraints to 3D structure and appearance. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1**
** Geometric reconstruction involves using perspective and panoramic RGB observations to estimate depth or construct a representation that supports novel views. p. 8 Instance-level spatial perception allows models to localize an object, estimate its extent, and associate it with observations from other viewpoints. p. 8 Camera-conditioned video generation uses trajectory–video pairs to learn to generate corresponding sequences of views under specified motion. p. 8**
** Multi-agent collaboration and shared world modeling allow multiple observer cameras to be associated with a common sequence of scene states for learning how different agents perceive and update a common environment. p. 9 State-conditioned scene editing supports controlled object translation, rotation, insertion, removal, and appearance changes. p. 9**
Key Contributions
The report makes three main contributions to the field. These include:
** A collection of furnished residential environments with metric geometry, editable assets, and diverse layout and furnishing configurations. A pipeline for aligned multimodal and sequential data linking rendered observations to camera calibration, trajectories, geometry, object annotations, and scene-state records. p. 1 Controlled data-generation workflows that separate camera variation from scene variation supporting reproducible comparisons for spatial perception and generation. p. 2**
** The collection spans 5,280 furnished configurations with aligned observations and editable source scenes. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 The source scenes allow additional observations to be rendered under specified cameras and object configurations. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1**
** Paired floor plans and furnished environments connect layout constraints to 3D structure and appearance. Aligned observations and state records connect camera motion and object changes to their visual consequences. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1**
** The diversity of paired environments connects these tasks to a broad range of residential configurations. p. 8 The collection spans US-style residential layouts with varied room compositions and furnishing arrangements. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 The source assets retain metric geometry and object structure, supporting perspective and panoramic RGB observations, depth maps, meshes, point clouds, and instance-level 3D bounding boxes. p. 1**
** The construction workflow connects residential design priors to editable Blender scenes and aligned synthetic observations. This workflow involves rule-conditioned layout generation, human verification, and Blender assembly with Infinigen assets. p. 5**
** Scene construction preserves the correspondence between plan elements and their 3D realization producing paired floor plans and furnished environments. p. 6 The source assets retain explicit geometry and object transforms. p. 3**
** The pipeline for aligned multimodal data links rendered observations to camera calibration, trajectories, geometry, object annotations, and scene-state records. p. 1 This representation expresses the shared spatial and temporal associations among the modalities. p. 3**
** The construction process involves GPT-6 generating structured layouts from residential rules and human reviewers checking connectivity and clearance. p. 5 The assembly combines the accepted layout with Infinigen assets, materials, and furniture. p. 5**
** Post-processing extracts evaluated mesh geometry in a common world coordinate system connecting object annotations to paired 2D layouts. p. 6 Reference surface cloud is sampled from the scene retaining the instance and semantic associations of its source geometry. p. 6**
** The mobile-scan simulation combines reference surface samples, virtual scanner locations, rendered appearance, and geometric perturbations to approximate scanner occlusion. p. 6 Visibility is approximated by retaining the nearest sampled surface point in each bin. p. 6**
** The exported camera record specifies the depth convention and coordinate transform connecting instance annotations to the same RGB-D observation. p. 7 Camera trajectories connect rooms and floors through positions p(t) and orientations R(t). p. 7**
** Object translation, rotation, insertion, removal, and appearance changes update explicit scene state. p. 7 Fixed camera paths yield corresponding observations before and after an edit isolating scene changes. p. 7 Multiple observer cameras share the same object-state sequence providing cross-view observations of each change. p. 7**
** Editable source scenes support camera and scene controls recorded separately for task-specific data augmentation. p. 9 The collection spans US-style residential layouts with varied room compositions and furnishing arrangements. The source assets retain metric geometry and object structure supporting perspective and panoramic RGB observations. p. 1**
** The data organization supports floor-plan-conditioned 3D scene generation, geometric reconstruction, instance-level spatial perception, camera-conditioned video generation, multi-agent shared world modeling, and state-conditioned scene editing. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 Paired floor plans and furnished environments connect layout constraints to 3D structure and appearance. Aligned observations and state records connect camera motion and object changes to their visual consequences.**
** The applications span static scene understanding and the prediction of observations during exploration and interaction. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 Paired floor plans and furnished environments connect layout constraints to 3D structure and appearance. Aligned observations and state records connect camera motion and object changes to their visual consequences.**
** The collection spans US-style residential layouts with varied room compositions and furnishing arrangements. The source assets retain metric geometry and object structure supporting perspective and panoramic RGB observations, depth maps, meshes, point clouds, and instance-level 3D bounding boxes. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1**
** The collection spans 5,280 furnished configurations with aligned observations and editable source scenes. The source scenes allow additional observations to be rendered under specified cameras and object configurations.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Camera-conditioned video generation can be improved by using
Trajectory–video pairs associate each native frame with a camera pose and timestamp
to generate sequences conditioned ona target camera path.
This allows models to learn how an environment is observed under specified motion, extending capabilities beyond local viewpoint changes. -
Shared world modeling can be enhanced by leveraging the organization where
Multiple observer cameras share the same object-state sequence, providing cross-view observations of each change.
This supportsmulti-agent observation prediction
andcross-view state association
to learn how different agents perceive and update a common environment. -
State-conditioned scene editing can be refined by using the explicit records where
Holding the camera path fixed yields paired sequences that isolate scene changes; varying the viewpoints reveals the same edited state from different locations.
This enablesaction-conditioned observation generation, edit localization, view-consistent scene editing, and prediction of persistent object changes.
-
Floor-plan conditioned 3D scene generation can be improved by using
Paired floor plans and furnished environments connect layout constraints to 3D structure and appearance,
which supportsfloor-plan-to-3D generation, furnished scene completion, layoutconditioned interior synthesis, and retrieval of 3D environments from 2D plans.
This links architectural constraints directly to complete 3D realization. -
Instance-level spatial perception can be enhanced by utilizing
Persistent instance identifiers support cross-view association
linkingScene geometry with identifiers, transforms, bounding boxes, dimensions, and available semantic descriptions.
This allows models to localize objects and estimate their extent across different views accurately.
Sources
- Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation
- The Replica Dataset: A Digital Replica of Indoor Spaces
- ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes
- Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs