OpenView: Empowering MLLMs with Out-of-view VQA
cs.CV
Submitted: 2025-12-21
Updated: 2026-11-06
Comments: Accepted at NeurIPS 2026. Code: https://github.com/q1xiangchen/OpenView
Code: https://github.com/q1xiangchen/OpenView
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent multimodal large language models (MLLMs) show great potential in natural image understanding.
Terminology
Abstract
Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason about objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline that leverages panoramic imagery for large-scale and diverse multi-choice VQA synthesis, where full-scene coverage enables flexible view framing while preserving global scene awareness. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, multiple MLLMs, upon empowered by OpenView, can consistently boost their performance, uplifted from 48.3% to 62.3% on average. Code, benchmark, and data are available at https://github.com/q1xiangchen/OpenView.
Sources
- GPT-4 Technical Report
- Qwen2.5-VL Technical Report
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Evaluating Object Hallucination in Large Vision-Language Models
- Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation
- Towards Understanding Camera Motions in Any Video
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Ovis2.5 Technical Report
- SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
- VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs