From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists
cs.CV
Submitted: 2026-09-18
Updated: 2026-09-18
License: http://creativecommons.org/licenses/by/4.0/
The gist: Does a general vision--language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence
Terminology
Abstract
Does a general vision--language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677-0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17-20 of the top 20 heads per task with all-head rank correlations of 0.874-0.942, while redistributing their functional and causal strengths.
Sources
- Qwen3-VL Technical Report
- Nougat: Neural Optical Understanding for Academic Documents
- GLM-OCR Technical Report
- Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
- Optimal ablation for interpretability
- In-context Learning and Induction Heads
- olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
- Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model
- DeepSeek-OCR: Contexts Optical Compression
- Retrieval Head Mechanistically Explains Long-Context Factuality
- PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training
- Image-based table recognition: data, model, and evaluation
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models