OpenView: Empowering MLLMs with Out-of-view VQA

arXiv:2512.18563 · cs.CV · Submitted 2025-12-21 · Read on arXiv

cs.CV

Submitted: 2025-12-21

Updated: 2026-11-06

Comments: Accepted at NeurIPS 2026. Code: https://github.com/q1xiangchen/OpenView

Code: https://github.com/q1xiangchen/OpenView

License: http://creativecommons.org/licenses/by/4.0/

The gist: Recent multimodal large language models (MLLMs) show great potential in natural image understanding.

Terminology

Abstract

Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason about objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline that leverages panoramic imagery for large-scale and diverse multi-choice VQA synthesis, where full-scene coverage enables flexible view framing while preserving global scene awareness. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, multiple MLLMs, upon empowered by OpenView, can consistently boost their performance, uplifted from 48.3% to 62.3% on average. Code, benchmark, and data are available at https://github.com/q1xiangchen/OpenView.

Sources

Related papers