KoViDoRe: A Benchmark for Korean Visual Document Retrieval

arXiv:2608.20840 · cs.IR, cs.CV · Submitted 2026-08-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "KoViDoRe: A Benchmark for Korean Visual Document Retrieval".

Jane: The gist: KoViDoRe introduces a benchmark for Korean visual document retrieval that explicitly models multi-page evidence aggregation, showing that current models struggle with this complex task in Korean settings,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: The paper introduces "KoViDoRe: A Benchmark for Korean Visual Document Retrieval," and the authors are Yongbin Choi, Yongwoo Song, Mujeen Sung, all from Kyung Hee University.

Tom: They’re building this benchmark using publicly available Korean documents that have a really diverse set of layouts—things like tables and figures—which is key for testing real-world retrieval systems >

Lu: And they describe their data curation pipeline as a multistage process involving structured document parsing, synthetic query generation, and relevance mapping to make sure the data is high quality >

Meng: I'm interested in that query generation part; they use both summary-based and context-based strategies to create queries based on both the text and the visuals found during preprocessing >

Lalam: Plus they generate queries directly from local document context using page-level or local multi-page context windows, which should help capture that localized understanding of the document structure >

Tom: And they clean up everything with a relevance mapping step where they compare signals from the generation stage against a separate mapping stage to filter out inconsistent pairs >

The paper's summary: Jane: So what’s the core suggestion here is that we need to shift our focus away from single-page retrieval and towards multi-page evidence aggregation, which the authors see as a necessary change in how we evaluate these systems >

Tom: They argue that models should be able to identify a coherent set of pages instead of just pulling one relevant page, especially when you’re looking at things like detailed reports or manuals >

Lu: They point out that before this, most reasoning was confined to a single page or a very small context window, but this benchmark pushes the limits by demanding reasoning over multiple pages >

Meng: So the improvement they are highlighting is really testing if late-interaction models and dual-encoder approaches can actually handle this aggregation task effectively when dealing with Korean data >

Lalam: The paper notes that even though vision-language models are getting better at understanding text and visuals, the datasets they used for training and evaluation are still heavily concentrated in English and European languages >

Tom: So the main takeaway is addressing that lack of dedicated Korean benchmarks, which limits how we can actually test if retrieval models can work in realistic Korean document scenarios >

The paper's improvements: Jane: To summarize the improvements, the paper demonstrates that current multimodal retrieval models struggle with this multi-page evidence aggregation setting specifically in Korean contexts because they haven't been trained on language-specific data >

Tom: The main finding is that training on Korean-specific data can significantly improve performance, showing how important those language-specific supervision resources are for this type of task >

Lu: They also release a large-scale training dataset called KoVDR Train Public, which consists of over three hundred ten thousand query-page pairs derived from Korean visual documents >

Meng: That dataset is curated with filtering to remove low-quality queries, which is important because running models on noisy data at scale can be tricky when you’re shipping them to production >

Lalam: Fine-tuning two late-interaction retrieval models, colSmol and colqwen2 v1 point 0, on this KoVDR Train Public dataset consistently improved performance across both of them >

Tom: So the implication is that having Korean-specific training data combined with large-scale public training data generally yields the best overall performance for these Korean visual document retrieval tasks >

Conclusion: Jane: So to wrap up, KoViDoRe is this new benchmark for Korean visual document retrieval that really focuses on multi-page evidence aggregation, and it clearly shows that current models are struggling with that complexity >

Tom: It seems like the dataset construction pipeline they built was pretty detailed—they used structured document parsing to get page and element level markdown, then used both summary based and context based strategies for query generation >

Lu: I think the ambition there is really about how much reasoning these models can do across different sections of a document, not just looking at one isolated chunk of text >

Meng: From a deployment standpoint, having that dataset is useful because it gives us something concrete to train on, but I always wonder if that complex query generation pipeline translates cleanly to real traffic where you don't have perfect pre-processed structure >

Lalam: I see the point in their relevance mapping—comparing signals from the generation stage against a separate mapping stage to filter out inconsistent pairs and get better annotations >

Tom: And the results across those four domains, cybersecurity, energy, economic, and human resources show that performance gap between model scales is still there, but it also shows that even strong models hit a wall on certain subsets >

Jane: That’s what I found interesting—the economic and human resources domains consistently scored lower because of the complex document structures they mentioned earlier >

Lu: Yeah, the complexity of those structures seems to be a real bottleneck, indicating that just having more data isn't enough if it doesn't account for how information is actually distributed across pages >

Meng: For us, the implication is that we need models designed specifically to handle those multi-page aggregation tasks efficiently without getting bogged down in every single piece of evidence >

Lalam: And this brings us to KoVDR Train Public, which is that large-scale training dataset with over three hundred ten thousand query-page pairs they released specifically for fine-tuning late interaction models >

Tom: That fine-tuning showed consistent performance improvements on both colSmol and colqwen2 v1 point zero which suggests Korean specific supervision really helps the models learn the right retrieval patterns >

Jane: It’s a big step forward because it gives researchers a unified benchmark and a training resource for anyone trying to build retrieval systems for Korean visual documents >

Lu: The limitation they pointed out is that they rely on parsed markdown representations for query generation, which could introduce noise if the initial parsing isn't perfect >

Meng: That means future work needs to focus on integrating vision-language models directly into the query generation process to capture more of the visual information, not just what's already structured text >

Lalam: I think that’s where things are headed, moving beyond just structured text toward more integrated multimodal understanding in those Korean contexts >

Tom: So that’s KoViDoRe and KoVDR Train Public for now. Next up, we're going to look at how Ball Differential Privacy is trying to handle data reconstruction noise in other areas.

Kyung Hee University

cs.IR, cs.CV

Submitted: 2026-08-21

Updated: 2026-10-15

Comments: v2: title updated

Code: https://github.com/whybe-choi/kovidore-benchmark

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist: KoViDoRe introduces a benchmark for Korean visual document retrieval that explicitly models multi-page evidence aggregation, showing that current models struggle with this complex task in

Key concepts

KoViDoRe Benchmark: This is a new evaluation tool specifically designed for assessing how well AI models can retrieve information from complex Korean visual documents. Unlike previous benchmarks that only check single pages, KoViDoRe forces models to find evidence spread across many pages, simulating real-world document retrieval challenges.
Multi-stage Data Curation Pipeline: This is a detailed process used to create the KoViDoRe dataset. It involves several steps: parsing documents into structured data, creating diverse synthetic queries (both summary and context-based), and mapping relevance with human checks. This ensures the dataset accurately reflects real document complexity.
Ko-VDR Train Public: This is a large training dataset of 310,226 query-page pairs derived from Korean visual documents. It was created to help researchers fine-tune retrieval models specifically for Korean visual documents, showing that language-specific data significantly improves model performance.
Multi-hop Reasoning Queries: These are complex queries included in the dataset that require models to perform multi-step thinking to find an answer. They test a model's ability to aggregate information from different parts of a document or across multiple pages, which is crucial for handling complex Korean documents.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes

Summary

The gist: KoViDoRe introduces a benchmark for Korean visual document retrieval that explicitly models multi-page evidence aggregation, showing that current models struggle with this complex task in Korean settings, and it releases a large-scale training dataset to support model development.

KoViDoRe Benchmark Introduction

KoViDoRe is introduced as a benchmark for Korean visual document retrieval designed to evaluate multi-page retrieval performance in realistic document settings, moving beyond single-page retrieval tasks We introduce KoViDoRe, a benchmark for Korean visual document retrieval that explicitly models this setting The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures We develop a multistage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and contextbased strategies, and relevance mapping with human verification Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types

Dataset Curation Pipeline

The dataset construction pipeline is inspired by ViDoRe V3 but adapted to better reflect the characteristics of Korean document ecosystems and real-world data sources The overall pipeline design is inspired by ViDoRe V3 (Loison et al., 2026), and is adapted to better reflect the characteristics of Korean document ecosystems and real-world data sources

Document Collection: Documents are collected from publicly available sources, prioritizing those under the Korea Open Government License (KOGL) Type 1 and 2, ensuring authentic document sources
Document Processing and Parsing: Each PDF is split into pages, and Upstage Document Parse is employed to extract structured representations at the page level, providing both page-level markdown and element-level markdown for fine-grained components like tables and figures
Query Generation: A synthetic query generation pipeline based on SolarPro33 is used, conditioning queries on document content including textual and visual information extracted during preprocessing This involves two types of summaries: single-section summaries to capture global relationships and cross-section summaries constructed by sampling multiple single-section summaries to capture cross-sectional relationships Context-based generation also generates queries directly from local document context using page-level or local multi-page context windows
Relevance Mapping and Filtering: Relevance mapping is performed at the page level to identify supporting evidence, followed by consistency-based and rule-based filtering to improve annotation quality

Query Diversity and Annotations

To promote diversity, a query formulation scheme from ViDoRe V3 is adopted, controlling query format and type during generation to include multi-hop reasoning, numerical comparison, and aggregation-based queries The dataset includes various query types such as multi-hop reasoning, numerical comparison, and aggregation-based queries Relevance mapping involves comparing relevance signals from the generation stage with those from the additional mapping stage to retain only consistent pairs The process includes human verification where annotators perform a final review of query-page relevance annotations and refine queries through rephrasing when they are unnatural or ambiguous Statistics show that the average number of relevant pages per query ranges from 2.55 to 3.30 across domain-specific subsets, indicating complex information needs

Experimental Results and Findings

Experiments evaluate Korean visual document retrieval as a ranking task over document pages using nDCG@10, and performance is reported across four domains: Cybersecurity, Energy, Economic, and Human Resources Table 2 shows a clear performance gap between model scales, with larger models generally achieving higher scores Analysis reveals that domains such as Economic and Human Resources consistently show lower scores, suggesting that retrieving relevant information in these settings is especially challenging due to complex document structures and distributed information Furthermore, even strong retrieval models achieve relatively limited performance on several subsets, indicating that the benchmark introduces challenges like complex layouts and information distributed across multiple pages

Ko-VDR Train Public Dataset

To address the limitations identified, Ko-VDR Train Public is curated as a large-scale training dataset consisting of 310,226 query-page pairs derived from Korean visual documents Data quality is ensured through consistency-based and rule-based filtering to remove low-quality queries, such as those with excessive keyword enumeration or those that directly reveal answer content Training involves fine-tuning two late-interaction retrieval models, colSmol-500M and colqwen2-v1.0, using the colpali engine framework Results Table 3 show that fine-tuning on this dataset consistently improves performance across both models, demonstrating the effectiveness of Korean-specific supervision Training with both private and public data combined generally yields the best overall performance across most subsets, highlighting the complementary nature of Korean-specific supervision and large-scale public training data

Conclusion and Limitations

KoViDoRe emphasizes queries that require retrieving and integrating information distributed across multiple pages, showing that current multimodal retrieval models struggle with this setting in Korean contexts The work further demonstrates that training on Korean-specific data can substantially improve retrieval performance, highlighting the importance of language-specific training resources Limitations include reliance on parsed markdown representations for query generation and the potential noise from imperfect alignment between generation and mapping stages Future work plans to incorporate vision-language models into the query generation process to better capture visual information, and explore model designs specifically suited for structured documents The research provides a unified benchmark and training resource for Korean visual document retrieval

Acknowledgments

This work was supported by the IITP(Institute of Information & Communications Technology Planning & Evaluation)-ITRC(Information Technology Research Center) grant funded by the Korea government(Ministry of Science and ICT) (IITP-2026-RS-2024-00438239) The paper concludes with a hope that KoViDoRe and Ko-VDR Train Public will facilitate future research on Korean visual documents retrieval

References

Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. 2025. Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16776–16809, Vienna, Austria. Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024. M3docrag: Multimodal retrieval is what you need for multi-page multi-document understanding. arXiv preprint arXiv:2411.04952. Kuicai Dong, Yujing Chang, Derrick Goh Xin Deik, Dexun Li, Ruiming Tang, and Yong Liu. 2025. MMDocIR: Benchmarking multimodal retrieval for long documents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30971–31005, Suzhou, China. Association for Computational Linguistics. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. ColPali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.

Improvements for AI systems

Improved query generation by leveraging a multi-stage pipeline combining summary-based and context-based strategies to capture both global document relationships and local content, as described in Section 3.3: Combining both summary-based and context-based queries allows the dataset to cover a broader spectrum of retrieval scenarios.
Enhanced relevance mapping using a hybrid approach that compares relevance signals from generation and mapping stages, ensuring higher annotation quality by retaining only consistent pairs rather than relying solely on automatic filtering.
Development of retrieval models capable of multi-page evidence aggregation, addressing the limitation where current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types.

Abstract

Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. However, existing benchmarks remain largely centered on English and provide limited coverage of Korean visual documents with complex structures. Furthermore, most existing Korean resources primarily evaluate single-page retrieval, failing to capture realistic scenarios that require evidence aggregation across multiple pages. To address these gaps, we introduce KoViDoRe, a benchmark for Korean visual document retrieval. The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures. We develop a multi-stage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and context-based strategies, and relevance mapping with human verification. Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types. Motivated by this finding, we further curate a large-scale training dataset, Ko-VDR Train Public, to support the development of retrieval models tailored to Korean visual documents. Together, KoViDoRe and Ko-VDR Train Public provide a unified benchmark and training resource for Korean visual document retrieval.