PVDepth: Panoramic Video Depth Estimation via Geometry-Aware Spatiotemporal Adaptation
cs.CV
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: ICML 2026, Project Page: https://chuanxinsong.github.io/pvdepth-project-page
Code: https://github.com/ChuanxinSong/PVDepth
Project page: https://chuanxinsong.github.io/pvdepth-project-page
License: http://creativecommons.org/licenses/by/4.0/
The gist: Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models.
Terminology
Abstract
Panoramic video depth estimation is pivotal for applications such as Virtual Reality and World Models. However, advancements in this field are impeded by two primary obstacles: the scarcity of large-scale training data and the unique spatiotemporal challenges of Equirectangular Projection (ERP), which hinder the direct transfer of perspective models. In this paper, we first present PanoCARLA, a large-scale synthetic RGB-D panoramic video dataset, featuring natural motion trajectories and drone-like roaming perspectives. Building on this foundation, we propose PVDepth, an end-to-end framework adapted from perspective video depth models. To tackle ERP-specific geometric distortions and consequent non-linear temporal dynamics, we introduce two core mechanisms: (1) A Progressive Sphere-aware Noise Initialization strategy that anneals the noise distribution from planar to spherical, guiding the model to adapt to non-uniform information density; and (2) A Cube-rectified Temporal Modeling module that incorporates an auxiliary cubemap temporal branch to rectify non-linear temporal dynamics in the ERP domain. Extensive experiments demonstrate that PVDepth achieves superior performance, generating geometrically accurate and temporally consistent depth sequences. Code and data will be released at https:// github.com/ChuanxinSong/PVDepth.
Sources
- Joint 2D-3D-Semantic Data for Indoor Scene Understanding
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- AirSim360: A Panoramic Simulation Platform within Drone View
- RPG360: Robust 360 Depth Estimation with Perspective Foundation Models and Graph Optimization
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction
- DA$^{2}$: Depth Anything in Any Direction
- GenEx: Generating an Explorable World
- DINOv2: Learning Robust Visual Features without Supervision
- HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels
- Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation
- Matrix-3D: Omnidirectional Explorable 3D World Generation
- PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs