OneCanvas: 3D Scene Understanding via Panoramic Reprojection
cs.CV, cs.AI, cs.LG, cs.RO
Submitted: 2026-06-17
Updated: 2026-09-27
Comments: NeurIPS 2026 (Spotlight). Project page: https://baranowskibrt.github.io/onecanvas/ Code and model: https://github.com/baranowskibrt/onecanvas
Project page: https://baranowskibrt.github.io/onecanvas
License: http://creativecommons.org/licenses/by/4.0/
The gist: Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning.
Terminology
Abstract
Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Namely, each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization or aggregation across overlapping views. A 3D position embedding of the patch's metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated reasoning from a specific viewpoint, a common requirement in robotics and embodied AI. Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial reasoning tasks, with answer distributions controlled to reduce spatial reasoning shortcuts. OneCanvas achieves state-of-the-art results on SQA3D and VSI-Bench and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than its competitors closest in benchmark performance.
Sources
- Qwen3-VL Technical Report
- SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
- Thinking with Spatial Code for Physical-World Video Reasoning
- 3D Aware Region Prompted Vision Language Model
- Video-R1: Reinforcing Video Reasoning in MLLMs
- Visuospatial Cognitive Assistant
- LiftProj: Space Lifting and Projection-Based Panorama Stitching
- PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding
- PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning
- When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
- Do 3D Large Language Models Really Understand 3D Spatial Relationships?
- SpaceR: Reinforcing MLLMs in Video Spatial Reasoning
- Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
- Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
- Visual Spatial Tuning
- SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
- PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models