Spatial Foundry: A Multimodal Dataset for Spatial Intelligence

summary

The gist

Spatial Foundry introduces a collection of 5,280 furnished residential 3D scene configurations with aligned visual, geometric, and temporal data to enable spatial intelligence applications.

In short

Spatial Foundry provides a dataset of 5,280 furnished residential 3D scenes with aligned visual and geometric data for spatial intelligence applications. It combines floor plans, RGB images, depth maps, and camera trajectories to create structured observations. The system enables tasks like 3D generation and scene editing by linking design priors to editable Blender scenes.

Key concepts

Data Modalities
This refers to the different types of data used in the dataset. It includes floor plans, perspective/panoramic RGB images, metric depth maps, point clouds, meshes, 3D bounding boxes for objects, and camera trajectories paired with video. These modalities are all aligned to provide a comprehensive view of the scene.
Scene Construction Workflow
This is the process used to create the final furnished scenes. It starts by using GPT-6 to generate structured floor plans based on residential rules, which are then reviewed by humans for usability. This layout is then assembled using Infinigen and Blender assets to produce the final source scenes, ensuring a link between design rules and 3D reality.
Controlled Data Generation
This technique allows researchers to separate changes in the scene from changes in the camera's view. By fixing camera paths, researchers can compare observations before and after an edit, isolating what actually changed in the environment. This method supports reproducible comparisons for spatial perception tasks.
Spatial Intelligence Applications
The dataset supports six key areas: floor-plan-conditioned 3D generation, geometric reconstruction from visual data, instance-level perception to locate objects, camera-conditioned video generation based on motion, shared world modeling among multiple cameras, and state-conditioned scene editing. These applications connect layout constraints to 3D structure and appearance.

Terminology used across episodes

This episode discusses

The paper

Spatial Foundry: A Multimodal Dataset for Spatial Intelligence · Read on arXiv

Ethan Lin, Yang Song

Ranolabs

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Spatial Foundry: A Multimodal Dataset for Spatial Intelligence".

Jane: Here's the gist of the paper. Spatial Foundry introduces a collection of 5,280 furnished residential 3D scene configurations with aligned visual, geometric, and temporal data to enable spatial intelligence applications.

Tom: Data Modalities and Scene Representation The dataset combines floor plans, perspective and panoramic RGB, metric depth, point clouds, meshes, instance-level 3D bounding boxes, and camera trajectories paired with video.

Jane: The representation of observation t is conceptually expressed as os t = (I s t, D s t, Π s t, T wc, τ s t).

Tom: Perspective observations use calibrated pinhole views and panoramic observations use an equirectangular camera model with a declared orientation.

Paper summary: Tom: Welcome back to the show. We're talking about a new dataset called Spatial Foundry, which is a collection of five thousand two hundred eighty furnished residential three dimensional scene configurations with aligned visual, geometric, and temporal data.

Jane: It sounds like it's trying to solve that problem where just having pictures doesn't tell you the real relationship between what you see and the space itself.

Lu: Exactly. The whole thesis of Spatial Foundry is about connecting visual observations to the geometry, objects, and persistent state of an environment because images alone leave those relationships implicit, while disconnected annotations make it hard to study appearance or camera motion under a common reference point >

Meng: So what does this dataset actually include beyond just pictures? I need to know the concrete data points here >

Tom: It includes floor plans, perspective and panoramic RGB images, metric depth maps, point clouds, meshes, instance level three dimensional bounding boxes for objects, and camera trajectories paired with video >

Jane: That’s a lot of different ways to look at the same scene all at once. The paper claims this collection is important because it links these modalities together using shared scene coordinates across views and time >

Lu: The construction workflow they describe is key here. It connects residential design rules to editable Blender scenes and aligned synthetic observations through a process where GPT-six generates structured layouts, and then human reviewers check the spatial usability >

Meng: So it takes design priors, structures them, puts them in Blender with assets like Infinigen, and then adds the temporal data from camera movements and object state changes >

Tom: Right. And they focus on separating changes in observation from changes in the environment itself to keep things clean >

Jane: Then we have these controlled data generation workflows, which let you fix the camera path to see before and after an edit, or hold the path fixed to compare different scene edits >

Lu: Multiple observer cameras can share the same object state sequence, which extends that supervision to cross view consistency and shared world modeling across different viewpoints >

Meng: That sounds useful for training models where you want them to understand how different agents perceive a common environment, but what's the actual cost of running this setup at scale?

Paper summary: Tom: Well, the paper mentions that these applications span floor plan conditioned three dimensional generation, geometric reconstruction, instance level perception, camera conditioned video generation, shared world modeling and scene editing >

Jane: It’s connecting layout constraints to the three dimensional structure and appearance of a place you can actually walk through virtually >

Lu: Geometric reconstruction uses perspective and panoramic RGB observations to estimate depth or construct a representation that supports novel views because they have metric geometry attached >

Meng: And instance level spatial perception lets models localize an object, estimate its extent, and associate it with observations from other viewpoints, which is important for robotics maybe >

Tom: Camera conditioned video generation uses trajectory video pairs to learn how to generate corresponding sequences of views under specified motion because you have the trajectory and the video paired up >

Jane: So what does this mean for someone who just wants to understand a room? It means they can study how objects move or change when you look at them from different angles, given the original floor plan >

Lu: The collection spans five thousand two hundred eighty furnished configurations with aligned observations and editable source scenes which lets you render additional observations under specified cameras and object configurations >

Meng: That sounds like a massive storage requirement for handling all that geometry and state data, but they do have these editable source scenes, which is good because it means you can change the environment itself >

Tom: The construction process preserves the correspondence between plan elements and their three dimensional realization producing those paired floor plans and furnished environments with explicit geometry and object transforms >

Jane: It’s linking those layout constraints directly to the physical structure and how everything looks on top of it >

Lu: The pipeline for aligned multimodal data links rendered observations to camera calibration, trajectories, geometry, object annotations, and scene state records which expresses the shared spatial and temporal associations among all those modalities >

Meng: And they detail how this works by describing the exported camera record which specifies the depth convention and coordinate transform connecting instance annotations to the same RGB depth observation >

Tom: So when you look at a camera trajectory, you see positions p of time and orientations R of time that connect rooms and floors through those movements >

Paper summary: Jane: That means if you have an object annotation, its location in the world is tied to that specific camera's view and its recorded depth measurement >

Lu: Object translation, rotation, insertion, removal, and appearance changes update explicit scene state so you can study how those visual consequences look when the object moves or changes >

Meng: Fixed camera paths give you observations before and after an edit to isolate scene changes specifically to the environment itself rather than just camera movement >

Tom: And holding the path fixed gives you before and after edit pairs, which is a good way to check if an object change is what you think it is >

Jane: It’s all about having these paired floor plans and furnished environments that connect layout constraints to three dee structure and appearance so you can test things systematically > <ref:2611.17318#pg1>

Lu: The diversity of paired environments connects these tasks to a broad range of residential configurations, spanning US style layouts with varied room compositions and furnishing arrangements >

Meng: So it’s not just one type of house; it covers the variety in how people actually furnish their living spaces >

Tom: The data organization supports floor plan conditioned three dimensional scene generation, geometric reconstruction, instance level perception, camera conditioned video generation, multi agent shared world modeling and state conditioned scene editing >

Jane: It’s covering everything from understanding a static room to predicting what the view will look like during exploration and interaction >

Lu: The applications span static scene understanding and the prediction of observations during exploration and interaction because you can use this data for both things >

Meng: So, in short, it’s a big dataset that lets you test models on a really rich variety of real-world spatial scenarios but it relies heavily on having this structured input data to work properly >

Jane: It’s a really comprehensive resource for anyone doing research in spatial intelligence that needs to tie together the layout, the three dee geometry, and the visual experience > <ref:2611.17318#pg1>

Lu: The construction workflow connects residential design priors to editable Blender scenes and aligned synthetic observations through rule conditioned layout generation and human verification >

Paper summary: Meng: So, they’re not just throwing a bunch of scans together; they are building them following some kind of design logic first >

Tom: And the post processing extracts evaluated mesh geometry in a common world coordinate system connecting object annotations to paired 2D layouts and reference surface cloud from the scene retaining the instance and semantic associations of its source geometry > <ref:2611.17318#pg3>

Jane: That post processing step is crucial because it connects everything back into that single world coordinate system where you can actually use it for things like geometric reconstruction >

Lu: The mobile scan simulation combines reference surface samples, virtual scanner locations, rendered appearance, and geometric perturbations to approximate scanner occlusion by retaining the nearest sampled surface point in each bin >

Meng: And they also export a camera record that specifies the depth convention and coordinate transform connecting instance annotations to the same RGB depth observation >

Lu: Object translation rotation insertion removal and appearance changes update explicit scene state so you can study how those visual consequences look when the object moves or changes >

Meng: Fixed camera paths yield corresponding observations before and after an edit isolating scene changes to the environment itself rather than just camera movement >

Tom: And multiple observer cameras share the same object state sequence providing cross view observations of each change, which is really interesting for modeling how different agents perceive a common change >

Jane: It’s all about having paired floor plans and furnished environments that connect layout constraints to three dee structure and appearance so you can test things systematically across different views > <ref:2611.17318#pg1>

Meng: So it’s not just one type of house; it covers the variety in how people actually furnish their living spaces, which is important for real world deployment >

Meng: So, in short, it’s a big dataset that lets you test models on a really rich variety of real world spatial scenarios but it relies heavily on having this structured input data to work properly >

Lu: The mobile scan simulation combines reference surface samples, virtual scanner locations, rendered appearance and geometric perturbations to approximate scanner occlusion by retaining the nearest sampled surface point in each bin >

Meng: So it’s not just one type of house; it covers the variety in how people actually furnish their living spaces, which is important for

Conclusion: Tom: So, we're wrapping up our look at Spatial Foundry, which is this dataset of five thousand two hundred eighty furnished residential scenes with all that visual and spatial data attached to them >

Jane: It’s a huge collection, Tom. The point of the paper is really about providing these aligned observations so you can test models that understand how space works in three dimensions, not just looking at flat pictures >

Lu: They built a whole pipeline connecting the design rules, like floor plans, to actual three dee geometry and then pairing that with camera movements and object changes so the data is totally consistent across time and different views >

Meng: So what's the actual big picture here? What does this mean when you're trying to build something that needs to understand a physical space reliably?

Tom: Well, it gives AI models a way to learn about spatial relationships by showing them how the layout connects directly to the three dee structure and how those structures change when things move inside >

Jane: It’s connecting the constraints of a floor plan—like where walls are—to what that space actually looks like in three dimensions, which is super useful for generating realistic environments or even for robotics that needs to navigate unseen spaces >

Lu: The authors focus on making sure the data generation process is controlled, meaning they can separate changes in the camera movement from actual changes happening to the house itself so you get clean comparisons >

Meng: That control is important because when you're shipping models, you need reproducible results. If a model makes a mistake, you want to know if it was because the model got confused by a weird camera angle or if it misunderstood how the furniture was actually placed >

Tom: It means they are providing not just raw photos but structured data that links every single observation—from RGB images to depth maps—to the exact position and state of every object in the scene >

Jane: And that alignment is what lets these different applications work, like reconstructing a room from just a few pictures or even generating new video sequences based on how a camera moves >

Lu: It’s moving beyond just seeing what's there to understanding how to interact with it, whether that's editing the scene or having multiple agents learn how to perceive the same environment >

Meng: So this collection is basically a massive training ground for making AI systems genuinely aware of physical space rather than just pattern matching on isolated images >

Tom: Yeah, and the authors put all this together with a construction workflow that starts with design rules and ends up with these perfectly aligned data sets ready for serious spatial intelligence applications >

Jane: It’s a big step toward making AI models capable of understanding the 'why' behind what they see in a physical space, not just the 'what' >

Lu: And next, we'll look at how they actually use this to reconstruct three dee geometry from those images and what kind of results they are seeing there >

More episodes

← Home