From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
cs.CV, cs.AI, cs.CL
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: Published as a conference paper at COLM 2026
Code: https://github.com/expertailab/scigram
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images.
Terminology
Abstract
Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images. However, they continue to struggle with scientific diagrams, which are designed to convey functional or relational meaning rather than literal scenes. We therefore introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula. Our approach systematically extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision in the form of diagram captions and multiple-choice questions. Using this pipeline, we construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Despite relying on noisy web data and synthetic annotations, models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks, including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Furthermore, augmenting existing models such as LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram question answering. Our results highlight the effectiveness of terminology-grounded instruction generation as a general strategy for improving vision-language reasoning in scientific domains. To support future research in scientific diagram understanding, we release both the SciGram dataset and models.
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Pixtral 12B
- PaLM: Scaling Language Modeling with Pathways
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- The Llama 3 Herd of Models
- SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems
- LoRA: Low-Rank Adaptation of Large Language Models
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
- KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
- GPT-4 Technical Report
- Bidirectional Attention Flow for Machine Comprehension
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Gemini: A Family of Highly Capable Multimodal Models
- Memory Networks
- Qwen2 Technical Report
- Deep Modular Co-Attention Networks for Visual Question Answering
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models