Document-as-Image Representations Fall Short for Scientific Retrieval

arXiv:2604.18508 · cs.IR, cs.AI, cs.CL · Submitted 2026-04-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Document-as-Image Representations Fall Short for Scientific Retrieval".

Jane: The gist: document-as-image representations are consistently suboptimal for scientific retrieval, especially in dense documents where evidence is distributed across structured sources like text, tables, and figures.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up "Document-as-Image Representations Fall Short for Scientific Retrieval", the paper is basically arguing that we need to move away from just embedding rendered pages as images for scientific searching. They specifically introduce ArXivDoc built on LaTeX sources to test this, and their main finding is that document-as-image representations are consistently suboptimal, especially with longer documents.

Jane: It really boils down to the fact that critical evidence in science papers isn't just in the rendered image; it's distributed across text and tables and figures, so a representation that treats it all as one image just misses most of the signal.

Lu: The authors stress that LaTeX provides direct access to those structured elements like sections tables figures and equations which allows for controlled query construction based on specific evidence types. This shifts the focus from just looking at the page layout to understanding the underlying structure.

Meng: From a production standpoint, this suggests that if you're building an AI system for scientific search, you can't just rely on a single image embedding. You need something that understands how text and figures relate to each other structurally.

Lalam: The paper's core contribution is showing that interleaved text image representations outperform document-as-image approaches without requiring any additional training, which is a significant finding because it suggests combining modalities is inherently more robust than relying on just rendered pages.

Tom: So the implication for us listening right now is that if you are working on scientific retrieval, you should probably stop thinking of documents as just images and start thinking about how text and those other structured elements work together.

Jane: That's right, because the authors suggest extending ArXivDoc to support queries that require combining evidence across multiple parts of a document or even across documents can really advance retrieval in these scientific settings.

Conclusion: Tom: So we’re wrapping up this talk on "Document-as-Image Representations Fall Short for Scientific Retrieval." Basically, the paper is showing that treating scientific documents just as images of a page isn't working well anymore.

Jane: Right, so if you think about it, most of the big models out there were trained on those images, right? But this study builds a new benchmark called ArXivDoc using actual LaTeX source code.

Lu: That’s the core idea—LaTeX lets you see the structure directly, like where the tables and figures are placed, instead of just looking at how it looks on a screen.

Meng: And what they found is that when you use those image-based representations, performance gets worse as the document gets longer. That's a real concern for any system trying to handle long papers.

Lalam: The paper shows that text-only representations actually do surprisingly well for queries about figures too, because the text around the figure gives away what’s in it.

Tom: I’m interested in how they compare those different ways of representing the data, especially seeing how interleaved text and image stuff performs without needing any special extra training.

Jane: That seems pretty important for us trying to build systems that actually understand scientific evidence, not just the visual layout.

Lu: It points out a gap between how these complex scientific documents are actually built in LaTeX and how the current retrieval models were designed to learn from images.

Meng: If we’re deploying something at scale, we need to know what breaks when you move from that training data to real-world traffic, which this benchmark helps us test.

Lalam: The authors suggest that maybe training models directly on these interleaved text and image documents is a promising way forward for getting better results.

Tom: So the authors are essentially saying the current standard approach of document-as-image is consistently suboptimal, and they’re suggesting we look at structured sources like LaTeX as a better way to build those embeddings.

Jane: Exactly. It’s about moving beyond just seeing a picture of a page and starting to see how text and structure actually work together in science.

Tom: And that opens up the idea of building more effective representations for these dense, multimodal scientific papers across the board.

Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh, Bhuwan Dhingra

Duke University

cs.IR, cs.AI, cs.CL

Submitted: 2026-04-20

Updated: 2026-10-13

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: The gist: document-as-image representations are consistently suboptimal for scientific retrieval, especially in dense documents where evidence is distributed across structured sources like text,

Key concepts

Document-as-Image Representations: These models embed documents by treating them like photographs of a rendered page. This approach is often used but fails in scientific contexts because critical evidence is spread across structured elements like tables and equations, which are lost when only the visual image is considered.
ArXivDoc Benchmark: This new benchmark uses the original LaTeX source code of scientific papers instead of PDFs or images. This allows for controlled queries targeting specific structured elements (like sections or figures), providing a richer dataset for testing different document representations.
Multimodal Representations: These are embedding methods that combine different data types, such as text and images, into a single vector representation. The study finds that combining text and image information effectively is better than using just one type of representation alone for scientific retrieval.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes
Papers — 1× in this paper · explained in 3 episodes

Summary

The gist: document-as-image representations are consistently suboptimal for scientific retrieval, especially in dense documents where evidence is distributed across structured sources like text, tables, and figures.

Introduction to the Problem

Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Existing benchmarks for scientific document retrieval treat documents as images of pages, implicitly favoring such representations. This paradigm is not well-suited for text-rich multimodal scientific documents where critical evidence is distributed across structured sources including text, tables, and figures.

Introducing ArXivDoc Benchmark

To study this setting, the authors introduce ArXivDoc a new benchmark constructed from the underlying LaTeX sources of scientific papers. Unlike PDF or image-based representations, LaTeX provides direct access to structured elements such as sections, tables, figures, equations enabling controlled query construction grounded in specific evidence types. The benchmark contains 8,210 full-length scientific papers spanning substantially more pages than prior multimodal retrieval benchmarks. Queries are generated filtered and verified through a multi-stage pipeline using gpt-5.2 and manually verified by expert human annotators.

Comparison of Representations

The paper systematically compares text-only, image-based, and multimodal representations across both single-vector and multi-vector retrieval models. The results show that document-as-image representations are consistently suboptimal especially as document length increases. Text-based representations are most effective even for figure-based queries by leveraging captions and surrounding context. Furthermore interleaved text+image representations outperform document-as-image approaches without requiring specialized training.

Key Findings on Retrieval Performance

The main results indicate several key findings regarding representation effectiveness. Text alone is surprisingly competitive for figure-based queries as text-only models achieve strong performance on figure-based queries. Interleaved representations outperform document-as-image despite no dedicated training. Single-vector models outperform late interaction ones despite much smaller index sizes across both text and multimodal settings.

Analysis of Retrieval Mechanisms

The analysis reveals why certain representations succeed or fail in retrieval tasks. For figure-based queries, text retrieval relies on surrounding textual descriptions such as references, explanations, and caption content rather than direct access to the figure itself. This suggests that text retrieval relies on surrounding textual descriptions that encode information in text form.

Conclusion and Future Directions

The paper concludes by presenting ArXivDoc as a benchmark for scientific document retrieval built from raw LaTeX sources. The authors observe that interleaved text+image representations outperform document-as-image approaches even without explicit training suggesting that training embedding models on interleaved scientific documents is a promising direction. Moreover this points to a gap between how scientific documents are structured and how current retrieval models are trained. The authors suggest extending ArXivDoc to support queries that require combining evidence across multiple parts of a document or across documents can further advance retrieval in scientific settings.

Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g., retrieving a specific page versus the entire document Modality indicates the default representation of the corpus ArXivDoc is the only large-scale open-domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity e.g.

Improvements for AI systems

Enhanced Scientific Retrieval Accuracy by Leveraging Structured Source Access: The improved system can utilize ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers, which provides direct access to structured elements (e.g., sections, tables, figures, equations), enabling controlled query construction grounded in specific evidence types.
Superior Performance on Multimodal Queries through Interleaved Representations: The system can leverage the finding that interleaved text–image representations outperform document-as-image approaches without requiring specialized training, suggesting that embedding models should be applied to these inputs directly to preserve the document’s native structure and align text with corresponding figures.
Robustness Across Document Lengths: By adopting text-based representations, the system will overcome the degradation observed in other methods, as research shows that As more text is added to a page, document-as-image representations degrade faster than text-based representations.

Abstract

Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing benchmarks for scientific document retrieval, such as ArXivQA and ViDoRe, treat documents as images of pages, implicitly favoring such representations. In this work, we argue that this paradigm is not well-suited for text-rich multimodal scientific documents, where critical evidence is distributed across structured sources, including text, tables, and figures. To study this setting, we introduce ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers. Unlike PDF or image-based representations, LaTeX provides direct access to structured elements (e.g., sections, tables, figures, equations), enabling controlled query construction grounded in specific evidence types. We systematically compare text-only, image-based, and multimodal representations across both single-vector and multi-vector retrieval models. Our results show that: (1) document-as-image representations are consistently suboptimal, especially as document length increases; (2) text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context; and (3) interleaved text+image representations outperform document-as-image approaches without requiring specialized training.