From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

arXiv:2610.11818 · cs.CV, cs.AI · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "From Pixels to Structure".

Jane: The gist The study presents a comparative analysis of eight lightweight Vision-Language Models (VLMs) to perform Optical Character Recognition (OCR) and structured JSON extraction across three university heritage collections,

Tom: First, who's behind it and why it matters.

Paper summary: Lu: So putting it all together, this paper demonstrates that even relatively lightweight Vision-Language Models can be highly effective tools for structured extraction in specialized domains like heritage digitization.

Meng: The implication is that institutions with unique archival needs—like those dealing with jewelry inventories or historical architectural forms—can build their own private OCR pipelines using these open-source models instead of relying on external, costly services.

Tom: It’s about moving away from the black-box approach where you send sensitive documents to a commercial API just to get some text back, and instead having control over the entire process locally.

Jane: The authors highlight that their work provides guidance for heritage institutions looking to implement institution-controlled OCR-to-JSON extraction systems.

Lu: They found that reliability comes from instruction following, making sure the model binds extracted data to the correct fields in a JSON schema, and ensuring it can decode that JSON correctly without errors.

Meng: The paper suggests that for maximum accuracy across different collections, having separate dataset-specific checkpoints might be better than trying to use one universal checkpoint.

Tom: So the overall message is that you need to adapt these VLMs carefully—through fine-tuning and thoughtful preprocessing—to get a high-performing, private solution for your specific documents.

Conclusion: Tom: So we’re wrapping up this look at "From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction." This paper is about taking these big, heavy vision models and making them small enough to run privately on a regular computer.

Jane: Right. It focuses on using lightweight AI models, up to seven billion parameters, for things like reading old documents and turning that text into organized JSON data.

Lu: What’s really interesting here is how they test these models across three totally different university collections—prehistory texts, jewelry inventories, and architectural forms—to see if one model can handle it all.

Meng: From an engineering standpoint, the big question is how much effort you need to put in to make this work reliably when the input data is really messy and specialized.

Lalam: I see a lot of potential here for making archival work much more accessible by automating that extraction step so people don't have to transcribe everything manually.

Tom: So, what does this mean for us listening right now? It means we’re moving away from those huge, expensive commercial systems and toward something you can control on your own hardware.

Jane: Exactly. The authors show that with careful fine-tuning, you can get a private AI system that performs better than just reading the image and guessing the text.

Lu: They found that instruction following is super important here because it’s not just about getting words out of pixels; it’s about making sure the AI understands *which* piece of data belongs in *which* part of your structure.

Meng: And they even looked at how you can improve performance by tweaking the training process—things like optimizing the settings or using a few small image corrections before the model sees it.

Tom: So, for someone who just listens to this show, what’s the real deal? It’s that sophisticated document understanding isn't just for massive tech companies anymore; it can be made practical and private for specialized institutions.

Jane: And the study points toward a decision point: do you want one general model that works okay across all your collections, or do you want separate, super-tuned models for each specific archive? (Sound of gentle radio static fading back in)

Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen

Friedrich-Alexander–Universität Erlangen–Nürnberg

cs.CV, cs.AI

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 17 pages. Published in Document Analysis and Recognition - ICDAR 2026, LNCS vol. 16974, Springer. Code: https://github.com/uddipan77/Analysis-of-Lightweight-Vision-Language-Models-for-Document-OCR-and-Structured-Output-Generation

Journal ref: Document Analysis and Recognition - ICDAR 2026, Lecture Notes in Computer Science, vol. 16974, pp. 502-519, Springer, 2027

DOI: 10.1007/978-3-032-36039-7_30

Code: https://github.com/uddipan77/Analysis-of-Lightweight-Vision-Language-Models-for-Document-OCR-and-Str

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist The study presents a comparative analysis of eight lightweight Vision-Language Models (VLMs) to perform Optical Character Recognition (OCR) and structured JSON extraction across three

Key concepts

Constraint-Aware Prompting: This protocol ensures the models follow specific rules required by different interfaces. Some models need simple prompts for text recognition, while others allow complex instructions to generate structured JSON. This adaptation is crucial for making open-source VLMs work reliably in diverse archival settings.
Structured Output Quality Metrics: The paper uses three metrics—CER (Character Error Rate), ANLS* (Normalized Levenshtein Similarity), and mAP-F1—to judge the accuracy of the extracted text and its JSON format. These measures quantify how well the model captures both the correct words and the required structural schema for downstream use.
Cross-Dataset Transfer: This examines whether a single model trained on one set of documents performs well on others. Results show that some models, like Gemma-3, handle different document layouts better across collections. However, other models might perform worse when transferring knowledge between distinct datasets without specific fine-tuning.
Multi-Stage Training: This strategy involves training the model in several sequential steps rather than one long process. The study found that multi-stage training consistently improved the final performance and convergence of certain models, particularly Gemma-3, across all three tested university collections.

Terminology

Fine-Tuning — 40× in this paper · explained in 5 episodes
Adaptation — 9× in this paper · explained in 1 episode
Benchmark — 8× in this paper · explained in 2 episodes
Accuracy — 5× in this paper · explained in 1 episode
Instruction Tuning — 3× in this paper · explained in 2 episodes
Fidelity — 2× in this paper · explained in 2 episodes
LoRA Adapters — 2× in this paper · explained in 3 episodes
Robustness — 2× in this paper · explained in 5 episodes
Alignment — 1× in this paper · explained in 1 episode
Approximate Normalized Levenshtein Similarity — 1× in this paper
Benchmarking — 1× in this paper · explained in 3 episodes
Character Error Rate — 1× in this paper
Data Sovereignty — 1× in this paper · explained in 2 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generalization — 1× in this paper · explained in 13 episodes
Graphics Processing Unit — 1× in this paper
Low-Rank Adaptation (LoRA) — 1× in this paper · explained in 9 episodes
Quantization — 1× in this paper · explained in 12 episodes
Resource Constraints — 1× in this paper · explained in 3 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The study presents a comparative analysis of eight lightweight Vision-Language Models (VLMs) to perform Optical Character Recognition (OCR) and structured JSON extraction across three university heritage collections, demonstrating that carefully adapted models can offer a sustainable, private alternative to commercial black-box systems for archival digitization.

Model Evaluation and Datasets

The research benchmarks eight open-source lightweight VLMs (≤ 7B parameters) on records from three specialized university collections: the Erlangen-Prehistoric dataset, the Pforzheim Jewelry dataset, and the Regensburg Scalalogy dataset Table 3 summarizes zero-shot results for all eight models across three datasets The Erlangen-Prehistoric dataset involves processing 308 pages of the historic “Inventarbuch der Anthropologisch Prähistorischen Sammlung” The Pforzheim Jewelry dataset comprises 590 digitized inventory cards from the historical teaching aids collection at the Pforzheim University, School of Design The Regensburg Scalalogy dataset consists of 164 survey forms from the Scalalogy Collections of the Friedrich Mielke Institute (FMIS) at OTH Regensburg.

Constraint-Aware Prompting and Evaluation Metrics

A key aspect of the evaluation is a constraint-aware protocol that respects diverse model interfaces, as some models only accept fixed OCR-style prompts while others allow custom prompts for structured JSON extraction The metrics used to measure extraction fidelity and structured-output quality include Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS∗), and mean Average Precision F1 (mAP-F1) CER is computed against a fixed canonical ground-truth JSON string, while mAP-F1 requires valid JSON output. The evaluation protocol tracks JSON validity rate, which stays in the range of ≈ 95 %–100 % for models like Gemma-3 and Phi-3.5-Vision across settings.

Practical Adaptation Strategies

To assess practical improvements, controlled additional studies were run on a baseline fine-tuning pipeline to test the independent impact of several strategies These studies include (i) hyperparameter optimization, (ii) classical image preprocessing, and (iii) multi-stage training. For example, Study 2 evaluated whether classical image preprocessing improves model performance compared to using raw images, which showed a mixed effect depending on the model and dataset Study 3 investigated whether curriculum-based multi-stage training improves convergence and final performance, observing that it consistently benefits Gemma-3-4B across all three datasets.

Fine-Tuning and Cross-Dataset Transfer

The study compares single-dataset fine-tuning with a single multi-dataset checkpoint trained on the union of all training sets The effect of multi-dataset fine-tuning is model-dependent; for instance, Gemma-3 improves across all three collections, suggesting better cross-schema robustness. Conversely, Qwen2.5-VL shows signs of negative transfer on Regensburg Scalalogy when using the multi-dataset checkpoint.

Conclusion and Guidance

Instruction-tuned general-purpose VLMs like Qwen2.5-VL, Phi-3.5 Vision, and Gemma-3 are identified as the strongest candidates for reliable structured extraction due to their instruction following capabilities Fine-tuning is highlighted as the most effective adaptation strategy for achieving strong local performance on a single consumer GPU The findings suggest that deployment decisions should consider whether an institution needs one general checkpoint across collections or separate dataset-specific checkpoints for maximum accuracy. Future work is suggested to explore constrained decoding for valid JSON and layout-aware field binding. The paper concludes that reliable extraction depends not only on text recognition but also on instruction following, field binding, and stable schema-conformant decoding.

Improvements for AI systems

textbfInstruction Following via Constraint-Aware Prompting Architecture: Improving Model Versatility for Diverse Architectures. The system can now reliably extract structured JSON from models with different interfaces by utilizing a constraint-aware evaluation protocol that accounts for diverse model interfaces (fixed vs. custom prompting) in zero-shot settings. This allows deployment flexibility across the eight evaluated VLMs without needing to know their exact prompting requirements beforehand, as they are tested under their specific supported interface.
textbfAdaptive Fine-Tuning Strategy: Optimizing Resource Allocation for Institutional Deployment. The system can now select the optimal fine-tuning approach based on the required robustness, as it can evaluate single-dataset versus multi-dataset fine-tuning to understand cross-collection transfer and deployment trade-offs. This directly informs whether an institution should use one general checkpoint or separate dataset checkpoints to maximize accuracy for their specific collection needs.
textbf Performance Enhancement via Targeted Post-Training: Quantifying the Impact of Specific Adaptations on Accuracy. The system can now selectively apply targeted improvements by "testing the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and Contrast Limited Adaptive Histogram Equalization (CLAHE)), and (iii) multi-stage training." This allows for a data-driven decision on whether to invest computational resources in these specific techniques based on empirically measured gains.
textbf Robust Structured Output Validation: Ensuring Usability for Downstream Systems. The system can now guarantee the output is usable by strict downstream parsers by utilizing metrics like JSON validity and parsing policy, ensuring that if parsing fails, we compare the raw prediction text directly against the canonical ground-truth JSON string, reflecting downstream usability requirements. This prevents deploying models that generate syntactically incorrect JSON.
textbf Model Selection Guidance: Identifying High-Fidelity Candidates for Archival Work. The system can now provide actionable recommendations by selecting the top three models, Phi-3.5-Vision, Qwen2.5-VL-7B, and Gemma-3-4B, based on their demonstrated ability to maintain near-saturated JSON validity (≈95 %–100 %) across datasets. This guides users toward models with the strongest potential for reliable structured extraction.