Unified Panoramic Geometry Estimation via Multi-View Foundation Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Unified Panoramic Geometry Estimation via Multi-View Foundation Models".
Jane: Here's the gist of the paper. Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images in a single forward pass By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360◦ scenes from single panoramas Pruning redundant reasoning tokens The framework is designed to process an input RGB panorama by using a shared geometry transformer backbone to predict a sky mask, scale-invariant (SI) depth, surface normals, and coarse metric depth The metric branch obtains the final absolute depth by aligning the SI depth with coarse metric predictions and masking the sky The normal branch produces the final surface normal map by masking predicted orientations with sky segmentation The model is built on top of deterministic, feed-forward foundation models like DA3, which are designed for (perspective) multi-view input and offer a natural synergy with the cubemap format How the training signal is built The adaptation involves formatting the panoramic input as a six-face cubemap and supplying fixed camera matrices K alongside axis-aligned extrinsics Ei, i = 1,..., 6 Implicit encoder synchronization is achieved by fine-tuning the ViT encoder on panoramic data without structural modifications, allowing the network to naturally learn to route context and synchronize features across adjacent cubemap faces guided by fixed camera tokens Spherically aware decoder padding integrates cross-face valid padding into all convolutional and interpolation operations within the decoder architecture to ensure continuous spherical sampling A mixed training regime alternates between two data streams: processing full six-face configurations for panoramic batches and isolating a single real-world image for perspective batches, dynamically disabling cross-face padding for perspective samples Where the gains come from The unified multi-task architecture equips the backbone with multiple, coupled task heads to jointly reason about related geometric properties and efficiently extract a comprehensive 3D representation in a single forward pass The scale-invariant depth branch uses a composite loss function Ldepth that balances per-pixel precision, local smoothness, and surface alignment The metric scale is inferred by computing an optimal log-space shift β∗ = arg minβ (ˆzSI + β) − z∗2 and recovering the absolute metric depth as Zˆm = exp(βˆ)ZˆSI Sky segmentation is handled by a lightweight branch that reads out geometric cues from intermediate decoder features and fuses them with semantic tokens to mask sky regions Pruning redundant reasoning tokens The authors introduce ZüriPano, a novel dataset of real-world outdoor panoramas with associated high-accuracy LiDAR scans to address the lack of benchmark data for rigorous evaluation in long-range, outdoor scenarios The method achieves state-of-the-art performance and excellent zero-shot performance across a wide range of scenes in both indoor and outdoor environments PaGeR establishes a new state of the art for several benchmarks by reparameterizing panoramic geometry as a structured multi-view problem Where the gains come from PaGeR consistently outperforms existing methods across all datasets, reducing the Absolute Relative Error (AbsRel) on the challenging outdoor ZüriPano dataset from 18.27 to 9.36 The framework maintains high accuracy indoors while establishing a commanding lead on the ZüriPano metric benchmark with an RMSE of 530.85 Furthermore, PaGeR sets a new state of the art for surface normal estimation on the Structured3D dataset, achieving a Mean Angular Error of 5.49◦ Pruning redundant reasoning tokens The ablation studies show that while the unified model provides significant advantage, independent full metric depth models outperform their counterparts in terms of absolute metric depth accuracy when the ViT backbone is fully unlocked In the unified model, allowing gradients from the metric scale head to flow back into the shared ViT fundamentally conflicts with other tasks, leading to a severe degradation in SI depth and surface normal predictions This suggests that while efficient, the shared representation is capacity-constrained when balancing competing gradients Where the gains come from The framework's computational efficiency is also noted, with inference time being 0.5 seconds for a full 2K panorama and peak memory consumption of 12.8 GB on a single consumer-grade GPU The loss weights are detailed in Table 7, showing that the Gradient Loss λgrad is set to 40.0, and the Normals Consistency Loss λnorm is set to 0.6 The training resolution used for cubemap faces is maintained at 504 × 504 pixels per cubemap face, which assembles into a 2K equirectangular panorama The mixed data regime balances 80k synthetic panoramas from Structured3D with 10k real perspective images to mitigate the sim-to-real domain gap Pruning redundant reasoning tokens The limitations of the model include susceptibility to depth ambiguities caused by complex material variations and potential geometric or photometric misalignments at face boundaries in rare, structurally complex scenes These boundary artifacts are minimal but indicate that unconstrained data-driven attention cannot always guarantee absolute boundary consistency The model can also produce unreliable predictions when encountering specular, reflective, or transparent surfaces Where the gains come from The PanoInfinigen dataset provides complete, high-quality ground truth for both depth and surface normals across a variety of indoor and outdoor scenes at native 4K resolution The ZüriPano dataset uses the Leica RTC360 LiDAR scanner to capture 100 panoramic scans across 11 distinct urban locations in Zürich, Switzerland The framework's compatibility with alternative geometric transformers is maintained by adopting the architectural principles of perspective VGFMs The authors thank Hothifa Smair and the Parametra team for granting authorization to use the iCity Blender add-on for data generation Pruning redundant reasoning tokens The paper concludes that PaGeR successfully lifts robust representations of perspective depth models into the spherical domain by combining a synchronized cubemap representation with separate decoding heads The framework transfers established perspective priors to multiple 360° geometric tasks without requiring extensive retraining The architectural principles of the framework maintain compatibility with alternative geometric transformers, offering a generalized methodology for bridging the domain gap between perspective and panoramic 3D scene understanding Where the gains come from The results demonstrate that PaGeR establishes a new state of the art in zero-shot panoramic reconstruction across diverse indoor and outdoor environments The framework's balanced capability extends directly to absolute scale recovery, as detailed in Table 2, where it maintains high accuracy indoors and outperforms recent baselines on the ZüriPano metric benchmark The authors provide qualitative comparisons showing that PaGeR recovers sharper boundaries alongside a globally continuous scene layout The framework's ability to recover complex outdoor topographies while preserving intricate indoor structural arrangements is highlighted in the point cloud reconstructions The best and second-best results are indicated in bold and underlined in Table 1 The authors provide qualitative comparisons of surface normals estimation shown alongside the RGB input and ground-truth depth on panoramas from the Structured3D dataset The best result for surface normals is achieved by PaGeR, achieving a Mean Angular Error of 5.
Title and authors: Tom: Now let’s talk about where they claim the results actually come from, because that’s usually where you find out if this thing is just a clever trick or something solid.
Jane: Well, the authors introduce ZüriPano, which is a novel dataset of real world outdoor panoramas with high accuracy LiDAR scans to address the lack of benchmark data for rigorous evaluation in long range outdoor scenarios.
Lu: That dataset is key because it allows them to test their method on challenging long-range outdoor scenes where existing benchmarks might be lacking.
Meng: So they didn't just train on synthetic stuff; they curated this real world data with LiDAR, which gives them a high quality ground truth for testing.
Lalam: PaGeR establishes a new state of the art for several benchmarks by reparameterizing panoramic geometry as a structured multi view problem, and that is what allows them to achieve this performance.
Tom: Let’s look at the numbers they brought up. They mention consistently outperforming existing methods across all datasets and reducing the Absolute Relative Error on the challenging outdoor ZüriPano dataset from eighteen point two seven down to nine point three six.
Jane: That is a significant drop in error, especially when we think about how difficult it is to get accurate depth in complex outdoor environments where things like terrain vary a lot.
Lu: They also maintain high accuracy indoors while establishing a commanding lead on the ZüriPano metric benchmark with an Root Mean Square Error of five hundred thirty point eight five.
Meng: That RMSE number sounds like it’s hitting a really low error rate for metric depth in the outdoor setting. How does that translate to actual usability for, say, an autonomous vehicle?
Lalam: Furthermore, PaGeR sets a new state of the art for surface normal estimation on the Structuredthree dee dataset by achieving a Mean Angular Error of five point four nine degrees.
Tom: five point four nine degrees is what we see when surface orientation needs to be very precise, which is important for things like how light reflects off a wall or object.
The paper's summary: Jane: So wrapping this up, the paper concludes that PaGeR successfully lifts robust representations of perspective depth models into the spherical domain by combining a synchronized cubemap representation with separate decoding heads.
Lu: The main point is that they transfer established perspective priors to multiple three hundred sixty degree geometric tasks without needing extensive retraining, which is a big deal for generalization <ref:2605.26368#pg1>.
Tom: It seems like the core idea is that by using this unified model, you get these multi task heads working together efficiently in one forward pass.
Meng: I have to ask about the trade-offs though; they mention that while the unified model is efficient, independent full metric depth models actually outperform their counterparts when you fully unlock the ViT backbone.
Lalam: That suggests that while the shared representation is efficient, it can be capacity constrained when trying to balance competing gradients from those different tasks.
Tom: So even though PaGeR is faster and handles multiple things at once, if you want pure absolute metric depth accuracy on a single task, a dedicated model might still win out.
Jane: It sounds like the authors are saying this framework is really strong for zero shot reconstruction across diverse indoor and outdoor settings.
Lu: Exactly; they demonstrate that PaGeR establishes a new state of the art in zero shot panoramic reconstruction across diverse indoor and outdoor environments.
Meng: And for deployment, inference time is noted at zero point five seconds for a full two K panorama, which is pretty reasonable for real world applications.
Lalam: The paper titled "Unified Panoramic Geometry Estimation via Multi-View Foundation Models" successfully lifts robust representations of perspective depth models into the spherical domain by combining a synchronized cubemap representation with separate decoding heads.
Tom: That’s the main thing to remember for this work, adapting foundation models for panoramas is becoming a mature technique.
Jane: It certainly seems like they are setting a new benchmark for how we approach three dee understanding from single images in this domain <ref:2605.26368#pg1>.
The paper's improvements: Tom: So, to recap, PaGeR takes those powerful three dee foundation models that were trained on regular perspective images and turns them into something that works for panoramas too, by using a shared geometry transformer backbone.
Jane: Right, but what they’re really showing us here is how this unified architecture handles the different tasks—the depth, the surface normals, and even the sky mask—all in one single forward pass.
Lu: The real trick is how they train it. They take a panoramic image and treat it like a six-face cubemap, and they mix training data from both perspective images and these panoramic views together in a very specific way.
Meng: I’m curious about that mixed training regime. How much of the training is actually from the real world versus just synthetic stuff?
Jane: They alternate between processing full six-face configurations for panoramic batches and isolating a single real world image for perspective batches, dynamically turning off that cross-face padding when they are looking at a real image.
Tom: That makes sense. It lets the model learn the structure from the panoramic format without losing the three dee prior it got from those standard perspective models.
Lu: And that’s what allows PaGeR to generalize so well; it learns how to route context and synchronize features across adjacent cubemap faces guided by fixed camera tokens, even without major architectural changes.
Jane: So, the main improvement here is taking something trained for one type of view—perspective—and making it work robustly on three hundred sixty degree spherical scenes just by changing how you feed it the training data.
Tom: And they tackle that scale problem too, which is huge. They have a separate branch for scale-invariant depth and then an extra step to calculate the absolute metric depth by finding that optimal log space shift.
Meng: That’s important because if you don't get absolute metric depth, you can't really do anything with it in applications like robotics where you need to know the exact distance.
Jane: Exactly. They are decoupling the scale estimation from the main depth prediction, which means they recover that true physical size of things better than before.
Lu: But there’s a catch, and this is where I see some real engineering talk coming up in their limitations section. The authors admit that when you fully unlock the ViT backbone for metric depth, it actually conflicts with the other tasks, causing a degradation in the scale invariant depth predictions.
Tom: So, the efficiency of having those multiple heads working together comes at a cost if you try to force them all to do everything perfectly at once.
Jane: It suggests that while PaGeR is fantastic for getting a good general understanding quickly, if you need absolute metric depth on something very specific, you might still have better off with a dedicated model.
Lu: That’s the trade-off they laid out clearly—efficiency versus peak performance on one specific task.
Tom: And on the data side, they’ve introduced ZüriPano to test it rigorously in real outdoor scenarios, which is a good move for showing it works outside of the synthetic training environment.
Jane: It really shows that this approach isn't just neat for indoor scenes; it’s actually holding up well when dealing with the complexity of outdoor environments and those tricky material variations.
Lu: The conclusion is that they successfully transfer those established perspective priors to multiple three hundred sixty degree geometric tasks without needing massive retraining, which is a strong point for making these foundation models practical.
Tom: So, PaGeR isn't just another tool; it’s a way to bridge the gap between how we usually see things from one spot and how we can reconstruct the whole world around us from just one view.
Conclusion: Tom: So, to wrap up, PaGeR takes those powerful three dee foundation models that were trained on regular perspective images and turns them into something that works for panoramas too, by using a shared geometry transformer backbone.
Jane: Right, but what they’re really showing us here is how this unified architecture handles the different tasks—the depth, the surface normals, and even the sky mask—all in one single forward pass.
Lu: The real trick is how they train it. They take a panoramic image and treat it like a six-face cubemap, and they mix training data from both perspective images and these panoramic views together in a very specific way.
Meng: I’m curious about that mixed training regime. How much of the training is actually from the real world versus just synthetic stuff?
Jane: They alternate between processing full six-face configurations for panoramic batches and isolating a single real world image for perspective batches, dynamically turning off that cross-face padding when they are looking at a real image.
Tom: That makes sense. It lets the model learn the structure from the panoramic format without losing the three dee prior it got from those standard perspective models.
Lu: And that’s what allows PaGeR to generalize so well; it learns how to route context and synchronize features across adjacent cubemap faces guided by fixed camera tokens, even without major architectural changes.
Jane: So, the main improvement here is taking something trained for one type of view—perspective—and making it work robustly on three hundred sixty degree spherical scenes just by changing how you feed it the training data.
Tom: And they tackle that scale problem too, which is huge. They have a separate branch for scale-invariant depth and then an extra step to calculate the absolute metric depth by finding that optimal log space shift.
Meng: That’s important because if you don't get absolute metric depth, you can't really do anything with it in applications like robotics where you need to know the exact distance.
Jane: Exactly. They are decoupling the scale estimation from the main depth prediction, which means they recover that true physical size of things better than before.
Lu: But there’s a catch, and this is where I see some real engineering talk coming up in their limitations section. The authors admit that when you fully unlock the ViT backbone for metric depth, it actually conflicts with the other tasks, causing a degradation in the scale invariant depth predictions.
Tom: So, the efficiency of having those multiple heads working together comes at a cost if you try to force them all to do everything perfectly at once.
Jane: It suggests that while PaGeR is fantastic for getting a good general understanding quickly, if you need absolute metric depth on something very specific, you might still have better off with a dedicated model.
Lu: That’s the trade-off they laid out clearly—efficiency versus peak performance on one specific task.
Tom: And on the data side, they’ve introduced ZüriPano to test it rigorously in real outdoor scenarios, which is a good move for showing it works outside of the synthetic training environment.
Jane: It really shows that this approach isn't just neat for indoor scenes; it’s actually holding up well when dealing with the complexity of outdoor environments and those tricky material variations.
Lu: The conclusion is that they successfully transfer those established perspective priors to multiple three hundred sixty degree geometric tasks without needing massive retraining, which is a strong point for making these foundation models practical.
Tom: So, PaGeR isn't just another tool; it’s a way to bridge the gap between how we usually see things from one spot and how we can reconstruct the whole world around us from just one view.
Jane: It is a solid piece of work, really showing how to adapt massive models for completely new geometry domains.
Lu: Next week, we’re looking at something totally different—how these models handle video consistency over long sequences.
Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger, Denis Rozumny, Konrad Schindler, Nikolai Kalischek
ETH Zurich · Google
cs.CV, cs.AI
Submitted: 2026-05-25
Updated: 2026-11-10
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 95/100
The gist: Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view…
Key concepts
- PaGeR (Panoramic Geometry Reconstruction)
- A framework designed to adapt 3D foundation models trained on single-view perspective imagery for use with panoramic data. It transforms the panoramic input into a six-face cubemap format to leverage existing perspective priors while learning to reconstruct consistent 360° scenes from a single panorama.
- Unified Multi-task Architecture
- The model uses a shared geometry transformer backbone with multiple coupled task heads. This allows the network to jointly reason about different geometric properties—like depth, surface normals, and sky masks—in one forward pass, leading to more efficient and comprehensive 3D scene representation.
- Metric Depth Recovery
- The system estimates an initial scale-invariant depth first. It then recovers the absolute metric depth by computing an optimal log-space shift ($eta^*$) between the predicted SI depth and ground truth, effectively inferring a global scale factor necessary for accurate spatial reasoning.
Terminology
Summary
Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images in a single forward pass By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360◦ scenes from single panoramas
Pruning redundant reasoning tokens
The framework is designed to process an input RGB panorama by using a shared geometry transformer backbone to predict a sky mask, scale-invariant (SI) depth, surface normals, and coarse metric depth The metric branch obtains the final absolute depth by aligning the SI depth with coarse metric predictions and masking the sky The normal branch produces the final surface normal map by masking predicted orientations with sky segmentation The model is built on top of deterministic, feed-forward foundation models like DA3, which are designed for (perspective) multi-view input and offer a natural synergy with the cubemap format
How the training signal is built
The adaptation involves formatting the panoramic input as a six-face cubemap and supplying fixed camera matrices K alongside axis-aligned extrinsics Ei, i = 1,..., 6 Implicit encoder synchronization is achieved by fine-tuning the ViT encoder on panoramic data without structural modifications, allowing the network to naturally learn to route context and synchronize features across adjacent cubemap faces guided by fixed camera tokens Spherically aware decoder padding integrates cross-face valid padding into all convolutional and interpolation operations within the decoder architecture to ensure continuous spherical sampling A mixed training regime alternates between two data streams: processing full six-face configurations for panoramic batches and isolating a single real-world image for perspective batches, dynamically disabling cross-face padding for perspective samples
Where the gains come from
The unified multi-task architecture equips the backbone with multiple, coupled task heads to jointly reason about related geometric properties and efficiently extract a comprehensive 3D representation in a single forward pass The scale-invariant depth branch uses a composite loss function Ldepth that balances per-pixel precision, local smoothness, and surface alignment The metric scale is inferred by computing an optimal log-space shift β∗ = arg minβ (ˆzSI + β) − z∗2 and recovering the absolute metric depth as Zˆm = exp(βˆ)ZˆSI Sky segmentation is handled by a lightweight branch that reads out geometric cues from intermediate decoder features and fuses them with semantic tokens to mask sky regions
The authors introduce ZüriPano, a novel dataset of real-world outdoor panoramas with associated high-accuracy LiDAR scans to address the lack of benchmark data for rigorous evaluation in long-range, outdoor scenarios The method achieves state-of-the-art performance and excellent zero-shot performance across a wide range of scenes in both indoor and outdoor environments PaGeR establishes a new state of the art for several benchmarks by reparameterizing panoramic geometry as a structured multi-view problem
PaGeR consistently outperforms existing methods across all datasets, reducing the Absolute Relative Error (AbsRel) on the challenging outdoor ZüriPano dataset from 18.27 to 9.36 The framework maintains high accuracy indoors while establishing a commanding lead on the ZüriPano metric benchmark with an RMSE of 530. p. 6 By
decoupling metric scale estimation from the structural backbone, our independent domain heads
successfully specialize to indoor or outdoor structures while sharing the same underlying transformer
features. On the ZüriPano metric benchmark, PaGeR establishes a commanding lead with an RMSE
of 530. To evaluate higher-order geometric consistency, we report surface normal estimation on the Structured3D dataset in Table 3. PaGeR sets a new state of the art, outperforming specialized architectures
including PanoNormal and HyperSphere.Where the paper says it
Where the paper says it
The ablation studies show that while the unified model provides significant advantage, independent full metric depth models outperform their counterparts in terms of absolute metric depth accuracy when the ViT backbone is fully unlocked In the unified model, allowing gradients from the metric scale head to flow back into the shared ViT fundamentally conflicts with other tasks, leading to a severe degradation in SI depth and surface normal predictions This suggests that while efficient, the shared representation is capacity-constrained when balancing competing gradients
The framework's computational efficiency is also noted, with inference time being 0.5 seconds for a full 2K panorama and peak memory consumption of 12. p. 5 At inference time, our unified framework processes a full 2K panorama in 0. 5 seconds
6 while consuming 12. 8 GB of memory, allowing for deployment on a single consumer-grade GPU. The primary
loss L1 measures the absolute discrepancy scaled by the predicted aleatoric confidence cp. We
complement this with an edge-aware gradient penalty Lgrad to preserve discontinuities at object
boundaries, and a normal consistency loss Lnorm that enforces geometric alignment via the cosine
similarity between ground-truth orientations n
∗
and surface normals nˆ, derived analytically from the
predicted depth maps.Where the paper says it
Where the paper says it
Where the paper says it
The limitations of the model include susceptibility to depth ambiguities caused by complex material variations and potential geometric or photometric misalignments at face boundaries in rare, structurally complex scenes These boundary artifacts are minimal but indicate that unconstrained data-driven attention cannot always guarantee absolute boundary consistency The model can also produce unreliable predictions when encountering specular, reflective, or transparent surfaces
The PanoInfinigen dataset provides complete, high-quality ground truth for both depth and surface normals across a variety of indoor and outdoor scenes at native 4K resolution The ZüriPano dataset uses the Leica RTC360 LiDAR scanner to capture 100 panoramic scans across 11 distinct urban locations in Zürich, Switzerland The framework's compatibility with alternative geometric transformers is maintained by adopting the architectural principles of perspective VGFMs The authors thank Hothifa Smair and the Parametra team for granting authorization to use the iCity Blender add-on for data generation
The paper concludes that PaGeR successfully lifts robust representations of perspective depth models into the spherical domain by combining a synchronized cubemap representation with separate decoding heads The framework transfers established perspective priors to multiple 360° geometric tasks without requiring extensive retraining The architectural principles of the framework maintain compatibility with alternative geometric transformers, offering a generalized methodology for bridging the domain gap between perspective and panoramic 3D scene understanding
The results demonstrate that PaGeR establishes a new state of the art in zero-shot panoramic reconstruction across diverse indoor and outdoor environments The framework's balanced capability extends directly to absolute scale recovery, as detailed in Table 2, where it maintains high accuracy indoors and outperforms recent baselines on the ZüriPano metric benchmark The authors provide qualitative comparisons showing that PaGeR recovers sharper boundaries alongside a globally continuous scene layout The framework's ability to recover complex outdoor topographies while preserving intricate indoor structural arrangements is highlighted in the point cloud reconstructions The best and second-best results are indicated in bold and underlined in Table 1 The authors provide qualitative comparisons of surface normals estimation shown alongside the RGB input and ground-truth depth on panoramas from the Structured3D dataset The best result for surface normals is achieved by PaGeR, achieving a Mean Angular Error of 5.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified geometry estimation from single panoramic images: PaGeR can simultaneously reconstruct
highly detailed scale-invariant depth, absolute metric depth, surface normals, and sky segmentation masks across both indoor and outdoor environments
in a single forward pass. -
Cross-domain generalization for 3D reconstruction: The framework achieves
Zero-shot generalization to unseen indoor and outdoor scenes,
allowing it to perform high-quality 3D reconstruction without needing specific training for every new environment type. -
Robust metric scale recovery: PaGeR decouples metric estimation from the scale-invariant branch, recovering an
absolute metric depth
by inferring a global scale factor, which is crucial for accurate spatial reasoning in applications like robotics and autonomous driving. -
Seamless panoramic representation adaptation: By using cubemaps instead of equirectangular projection and employing a
mixed training regime with both synthetic panoramas and real perspective imagery,
the model overcomes geometric distortions while retaining therich 3D prior of the underlying foundation model.
-
High-fidelity surface geometry prediction: The multi-task architecture allows for the joint decoding of depth, surface orientation, and sky masks, which is validated by achieving a new state of the art in surface normal estimation on Structured3D with a Mean Angular Error of 5.49 degrees.
Abstract
Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.
Sources
- ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
- PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery
- PanoNormal: Monocular Indoor 360{\deg} Surface Normal Estimation
- Multi-task Geometric Estimation of Depth and Surface Normal from Monocular 360{\deg} Images
- DreamCube: 3D Panorama Generation via Multi-plane Synchronization
- $360^o$ Surface Regression with a Hyper-Sphere Loss
- Decoupled Weight Decay Regularization
- UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler
- VGGT-360: Geometry-Consistent Zero-Shot Panoramic Depth Estimation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs