VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching
summary
The gist
Reconstructing the complete geometry of a scene from a single RGB image remains challenging—especially when inferring hidden structures where visual evidence is incomplete <ref:2605.31466#pg3>…
In short
VolFill reconstructs complete 3D scene geometry from a single RGB image by predicting a volumetric distance function instead of just visible surfaces. It uses a hybrid VAE and flow matching to sample from the distribution of possible geometries, allowing it to infer hidden structures with high fidelity, outperforming traditional methods.
Key concepts
- Truncated Unsigned Distance Function (TUDF)
- This is a way to represent 3D scene geometry as a grid where each point stores the distance to the nearest surface. It avoids ambiguity by defining the value based on the closest physical boundary, regardless of how complex or connected that geometry is.
- Generative Framework
- Instead of predicting one single geometry (like traditional methods), this framework learns to sample from all possible valid geometries consistent with the input image. This allows it to produce sharp, physically plausible completions in areas where visual evidence is missing.
- Hybrid 3D VAE
- This model uses a two-part structure: a sparse encoder efficiently processes the initial distance function data, and a dense bottleneck compresses this information into a manageable latent space. This allows the generative part of the model to work effectively with high-resolution scene details.
- Geometry-Conditioned Flow Matching
- This technique trains the model by learning how to move noise toward a target geometry distribution. It uses two inputs: global context from a foundation model and local anchors from visible geometry, ensuring the generated 3D structure is both globally sensible and locally accurate.
Terminology used across episodes
This episode discusses
- VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching · Paper Radio
- ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
- ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation
- Dens3R: A Foundation Model for 3D Geometry Prediction
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models
- Submanifold Sparse Convolutional Networks
- CameraCtrl: Enabling Camera Control for Text-to-Video Generation
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction
- LRM: Large Reconstruction Model for Single Image to 3D
- SCRREAM: SCan, Register, REnder And Map:A Framework for Annotating Accurate and Dense 3D Indoor Scenes with a Benchmark
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction
- From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
- Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model
- LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning
- Depth Anything 3: Recovering the Visual Space from Any Views
- Flow Matching for Generative Modeling
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- SyncDreamer: Generating Multiview-consistent Images from a Single-view Image
- Point-E: A System for Generating 3D Point Clouds from Complex Prompts
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
The paper
VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching · Read on arXiv
N/A
Reconstructing the complete geometry of a scene from a single RGB image remains challenging, especially when inferring hidden structures where visual evidence is incomplete. We introduce VolFill, a generative framework that predicts the 3D structure of the complete scene rather than relying on traditional pixel-aligned regression. Our method utilizes a hybrid 3D VAE to compress sparse truncated unsigned distance function grids into a compact latent space, paired with a latent Diffusion Transformer that denoises this representation to recover the complete scene. We condition the generation on a geometry foundation model, leveraging rich spatial priors for robust reasoning. Unlike existing methods limited by per-ray constraints or unstructured point-cloud queries, VolFill provides a structured representation that supports direct surface extraction and occupancy queries at scale. Extensive experiments on the SCRREAM and NRGB-D datasets demonstrate that our approach significantly outperforms current amodal baselines, providing a robust foundation for holistic spatial understanding.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching".
Jane: Here's the gist of the paper. Reconstructing the complete geometry of a scene from a single RGB image remains challenging—especially when inferring hidden structures where visual evidence is incomplete <ref:2605.31466#pg3> VolFill, a generative framework that predicts the 3D structure of the complete scene rather than relying on traditional pixel-aligned regression <ref:2605.31466#pg2>.
Tom: Problem Formulation The paper addresses the task of amodal scene reconstruction from a single RGB image <ref:2605.31466#pg4> Unlike traditional pixel-aligned methods that recover only visible surfaces, our goal is to predict the complete 3D geometry within the observed view frustum <ref:2605.31466#pg8>.
Jane: We represent the scene geometry using a Truncated Unsigned Distance Function <ref:2605.31466#pg9> Let S denote the set of all physical surfaces in the scene contained within the view frustum For each voxel center p within our grid, the value V (p) is defined as the distance to the nearest surface in S, truncated to a maximum value τ: V (p) = min(dist(p, S), τ).
Tom: The unsigned formulation avoids this ambiguity because distance-to-nearest-surface is well-defined for any geometry regardless of topology.
Title and authors: Tom: So we’re talking about VolFill: Single-View Amodal three dee Scene Reconstruction with Volumetric Flow Matching <ref:2605.31466#pg1,VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching>. The title itself points to the two big ideas: single-view input and amodal reconstruction>
Jane: That amodal part is key, because it means they are aiming for structures that exist even where there’s no direct visual evidence, which is what makes it useful for things like navigation>
Lu: They tackle the constraint mentioned earlier—that pixel-aligned methods are strictly limited to visible surfaces—by shifting the focus to inferring hidden structures where geometric evidence is partially absent>
Meng: So they’re not trying to get a perfect, deterministic mesh from the image; they're trying to model the space of possibilities for what that scene could look like>
Tom: Right. And they use this generative framework to produce something sharp and physically plausible, following the learned manifold of real-world structures>
Lalam: I see how it relates to their methodology, which involves a hybrid three dee VAE to bridge the gap between high-resolution TUDF grids and the latent space needed for generation> <ref:2605.31466#pg1,a hybrid 3D VAE to>
Jane: That hybrid setup sounds like they’re trying to keep the fine detail while making the model manageable in terms of computation>
Tom: And that leads us into how they handle that representation, which is where their specific architectural choices come into play>
The paper's summary: Tom: The summary explains that standard L1 or L2 losses just predict the average geometry, which often leads to over-smoothed results in occluded areas>
Jane: That makes sense. If you only have visible data, averaging everything tends to smooth out the gaps where nothing is visible>
Lu: VolFill moves away from that by sampling from P(VI), which produces sharper completions because it follows the learned distribution of real-world scene structures>
Meng: But this sounds computationally expensive. Generating a whole distribution instead of just one point estimate usually means a lot more complexity in training and running inference>
Tom: That’s where their hybrid three dee VAE comes in to manage that complexity, converting sparse TUDF inputs into a dense latent space at some highly compressed scale> <ref:2605.31466#pg1>
Lalam: They use a sequence of sparse three dee convolutions with strided downsampling to reduce the spatial resolution by sixteen times before densifying at the bottleneck> <ref:2605.31466#pg1>
Jane: So they’re essentially compressing the high-detail geometry into a manageable, dense representation that their generative model can actually work with>
Tom: And then they use Geometry-Conditioned Flow Matching to learn a velocity field that transports noise toward those desired scene TUDF latents>
The paper's improvements: Jane: The authors suggest two specific improvements to make this work better. First, they introduce a dual conditioning strategy>
Tom: That dual conditioning uses global semantic-geometric context and local structural anchors to guide the flow matching process>
Lu: They use Geometry Foundation Models, or GFMs, to condition the DiT on frozen features extracted from those models, which gives it a strong prior on three dee structure> <ref:2605.31466#pg1>
Meng: Using a foundation model as a prior sounds powerful because it provides that high-level context you need when the local visual evidence is weak>
Lalam: They also have Visible-Latent Inpainting, which explicitly anchors the generative process to what we can actually see by integrating known visible geometry directly into the latent space>
Tom: So, if I understand this right, they are using both a big scene understanding model and explicit visible data to guide the reconstruction>
Jane: That’s a good way to put it. The paper shows that relying only on the visible latent fails to recover those unobserved structures, which is what this dual conditioning strategy fixes>
Conclusion: Tom: So, wrapping up, VolFill shifts reconstruction from deterministic pixel-aligned regression to modeling a structured generative manifold>
Jane: It produces cleaner point clouds and smoother meshes compared to some previous methods like NOVA3R and LaRI, which are what the paper compares it against>
Lu: The evolution of the model shows that it rapidly converges on the coarse scene layout by time three, and then recovers sharp, detailed structures around time eight>
Meng: From a deployment standpoint, they show that this hybrid design achieves good reconstruction quality while keeping memory cost linear with surface area instead of cubically with volume>
Lalam: The final point is that for amodal three dee scene reconstruction using VolFill, the dual conditioning strategy combines global visual context and explicit geometric grounding to achieve the highest fidelity> <ref:2605.31466#pg1>
Tom: So, it’s a generative framework for amodal three dee scene reconstruction, moving us toward physically plausible completions in occluded regions by modeling the distribution of structures> <ref:2605.31466#pg1>
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound