When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoning
cs.CV
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: 33 pages, 16 figures. Code: https://github.com/yunbeizhang/text-for-vision
Code: https://github.com/yunbeizhang/text-for-vision
License: http://creativecommons.org/licenses/by/4.0/
The gist: Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information the question needs, or because the solver fails to
Terminology
Abstract
Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information the question needs, or because the solver fails to use information that is present. We introduce a diagnostic protocol to distinguish these explanations. Using the same solver model and generation settings, we compare three input conditions: the original image, question-blind structure extracted by a vision-language model, or gold structure derived from the diagram source. Validity-triggered recovery tests truncation and schema failure, question-relevant fidelity measures preservation of answer-critical structure, and matched edge interventions test the effect of error location. On a reserved holdout of 240 public FlowGen diagrams, evaluated under a frozen protocol, gold structure reaches 87% accuracy while direct vision and learned text both remain below 30%. The aggregate comparison includes source-derived relation labels that may not be printed in the image and uses different learned and gold graph encodings, so it does not isolate extraction error alone. Retrying only invalid extractions makes nearly every public representation schema-valid yet leaves accuracy essentially unchanged. The public learned-text deficit relative to gold more than doubles with structural difficulty. Question-relevant topology predicts correctness better than whole-graph topology. In an exposed intervention study, a single answer-relevant edge edit reduces the primary solver's original-answer accuracy to near zero, while matched irrelevant edits largely preserve it. Supplied structure requires fewer solving tokens than vision, but learned acquisition removes this advantage at single use. These comparisons motivate evaluating acquired text by the answer-relevant evidence it preserves and by the solver's ability to use that representation.
Sources
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
- ICDAR 2019 Competition on Scene Text Visual Question Answering
- OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents
- The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs
- Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
- RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion with Matrix-Valued Trust for Multimodal Prediction under Modality Uncertainty
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
- MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
- Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planning
- Stop Comparing LLM Agents Without Disclosing the Harness
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models