Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning
cs.CV, cs.AI, cs.LG, cs.LO
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 29 pages, 6 figures, 14 tables
Code: https://github.com/Yuzhou-Wang99/Hob-VL
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions.
Terminology
Abstract
Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves uneven gains while retaining substantial errors and inconsistencies. Hob-VL exposes these failures through executable reference answers and matched evaluations.
Sources
- Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models
- CLOSURE: Assessing Systematic Generalization of CLEVR Models
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- Controlled Reformulation Testing for Logical Consistency in Large Language Models
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
- Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
- Analysis of Boolean Functions
- Cycle-Consistency for Robust Visual Question Answering
- MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning
- DriveLM: Driving with Graph Visual Question Answering
- LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
- Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models