VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

arXiv:2602.00122 · cs.CV, cs.AI, cs.MM · Submitted 2026-01-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents".

Jane: The gist The VDE Bench is a rigorously human annotated and evaluated benchmark specifically designed to assess the performance of image editing models on bilingual Chinese-English and complex visual document…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper today, "VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents." It seems like they're tackling a specific problem where models can change text in images but it has to keep the style and background correct.

Jane: Exactly. The big idea here is that most image editing stuff focuses on simple things, but dense documents with lots of text, especially in Chinese or English, are really hard to get right while keeping everything else intact.

Lu: Yeah, they propose this benchmark because they see a gap where existing methods struggle with complex text and non-Latin scripts like Chinese. They feel like current research is mostly focused on English and sparse images, so it doesn't cover the real difficulty of dense documents or those other writing systems.

Meng: So what exactly are they testing? It sounds like they're trying to see if these image editing models can actually perform precise local changes within a document structure without messing up the layout.

Tom: Right. The paper introduces VDE Bench, which is this evaluation resource for instruction-guided editing of Chinese and English document images. They’ve got three thousand five hundred six samples in total, with one thousand five hundred seventy-eight text-region edits and nearly two thousand table-region edits.

Jane: That's a lot of data to work with. What makes this benchmark unique is how they pair the editing instructions and the reference images with annotated regions and structured OCR outputs. It’s not just an image; it’s instructions, targets, and expected text recognition results all together.

Lu: The evaluation protocol they use is pretty thorough; they combine region-level measurements with full-page measurements to check for textual correctness, structural fidelity, and visual content preservation. They focus on how the local edits interact with the rest of the document context.

Meng: From a deployment standpoint, I'm curious about how robust this evaluation is. Can we trust these scores when we move these models into production environments where things get messy?

Tom: Well, they used a pipeline for construction that involved collecting diverse documents, generating editing instructions using Gemini3-Pro or Qwen3-VL-235B-A22B Instruct, and then human review by fifteen annotators scoring things like instruction compliance and subjective quality.

Jane: That human element is key. They used a multi dimensional scoring system, including IoU based region verification and some subjective filtering to make sure the edits are actually good. Then they ran structured OCR processing with PaddleOCR-VL on both the edited image and the modified regions to check for discrepancies against ground truth outputs.

Lu: When looking at the evaluation protocol itself, they measure spatial localization accuracy alongside textual content correctness using OCR metrics, and style preservation is used as a reference indicator of output quality. They also have metrics like IoU for spatial localization, CDM for character distance metric, BLEU-four for linguistic fluency, and a TEDS-like metric for structural integrity <ref:2602.00122#pg1>.

Paper summary: Meng: I wonder how much effort goes into generating those structured OCR outputs at scale. That sounds like it would be a significant computational overhead if you wanted to run this on high volume.

Lalam: My analysis of the literature shows that VDE Bench is designed to diagnose document-specific editing failures, which is important because general benchmarks don't always capture these specific types of errors in complex layouts. I’m quite confident in the direction they are heading because they are focusing on the interaction between local edits and their surrounding document context.

Tom: And it seems like that focus is paying off, especially when you look at some preliminary results where Qwen achieves the best performance in the Global-image Setting across most OCR metrics, with a TEDS score of zero point nine zero two. That’s a solid number to start with.

Jane: What's interesting is what they found during their verification stage: localized quality doesn't automatically mean global consistency, and for instance, FireRed collapsed on global text modify with an IoU of zero point zero five two when testing that specific task. That’s a real warning about what models might miss.

Lu: That finding is important because it highlights that even if an edit looks perfect locally, the overall document structure might still be broken, which is a major issue when you're dealing with dense text documents.

Meng: So for someone deploying these models, this suggests we need to test them not just on isolated edits but on how they handle the entire flow of document modification. What would break at scale is usually those subtle structural issues that only show up when the whole page is considered.

Tom: Exactly, and that brings us to the next part of our discussion, which is what this benchmark means for the broader field. The paper aims to provide a comprehensive framework for systematically evaluating the direct editing capabilities of image editing models on complex text documents.

Jane: It sets a rigorous standard because it aligns its evaluation protocol with human judgment, which makes it a trustworthy way to assess performance on multilingual and densely textual documents. It moves beyond simple accuracy checks to look at modification, format preservation, and layout consistency all at once.

Lu: The implication is that for anyone building an AI system that needs to handle visual documents—think legal contracts or technical manuals—this benchmark gives them a reliable way to diagnose where the model fails in terms of recognition, modification, format preservation, and layout consistency.

Meng: If we use this as a guide, we can start designing our own stress tests that specifically target those structural integrity issues they've highlighted. It helps us know exactly what kind of failure mode to worry about when we deploy these models in production.

Tom: So, to wrap up the summary of VDE Bench: it’s a rigorously human annotated and evaluated benchmark specifically designed to assess the performance of image editing models on bilingual Chinese-English and complex visual document editing tasks.

Paper summary: Jane: It’s definitely a detailed resource that pairs instructions with reference images, edited regions, and OCR outputs, allowing researchers to see exactly how well models handle modifying text within tight document structures.

Lu: The construction process itself shows the level of effort involved—collecting data, generating instructions using large AI models like Gemini3-Pro or Qwen3-VL-235B-A22B Instruct, and then getting fifteen human annotators to score it multi dimensionally.

Meng: I'm still concerned about the cost of running this kind of evaluation if you want to scale up the testing environment. It’s a lot of manual work upfront, and we need to see if that pipeline can be automated effectively before we seriously consider using it for high volume testing.

Tom: The results show that while some models are strong in specific areas, like Qwen achieving a TEDS score of zero point nine zero two in the Global-image Setting, there are clear patterns where performance degrades noticeably when moving between languages, particularly on Chinese compared to Latin-script English documents.

Jane: That cross-lingual pattern is quite telling for anyone building global systems; it shows that the models aren't uniformly capable across different scripts, which is something we have to keep in mind when designing our user interfaces or document processing tools.

Lu: And the authors are transparent about where they stop working, noting that VDE Bench is currently focused primarily on Chinese and English documents and doesn't cover other languages like Korean or Japanese. That’s a limitation you have to keep in mind if you are building a system for those specific document types.

Tom: So, the VDE Bench provides this comprehensive framework for systematically evaluating direct editing capabilities on complex text documents, assessing performance in recognition, modification, format preservation, and layout consistency. It’s a very useful tool for anyone studying how image editing AI performs on real-world document tasks.

Jane: For someone listening who doesn't read the paper, this means you can see exactly what kind of errors an image editing model makes when it tries to edit text in a complicated document structure, and you know whether it’s failing because it misreads the characters or because the layout is falling apart.

Lu: The ultimate implication is that for developers building these kinds of tools, this benchmark offers a trustworthy framework for evaluating performance on multilingual and densely textual documents. It sets a high bar for what successful document editing looks like.

Meng: So, it’s a way to ground the abstract idea of "visual editing" into concrete metrics related to text fidelity and structural integrity, which is something I think is really helpful for moving beyond just looking at pretty pictures.

Tom: That’s the core idea: moving from subjective visual quality to objective measures on textual correctness and layout consistency in a challenging domain. The VDE Bench gives us that objective language.

Conclusion: Tom: So, we’re wrapping up VDE Bench, which is this new benchmark for testing how image editing models handle dense text in documents.

Jane: Right, so it’s not just about making a picture look nice anymore; they are systematically checking if these models can actually change the words inside a complicated layout while keeping everything else structurally sound.

Tom: The authors, they used this resource to evaluate instruction-guided editing on both Chinese and English documents. They put together three thousand five hundred six samples in total, with almost two thousand table region edits included.

Jane: And what’s interesting is their evaluation protocol; they don't just look at if the text looks right, they measure spatial localization accuracy against textual correctness using OCR metrics like Character Distance Metric and BLEU-four.

Tom: They also track visual content preservation using a TEDS-like metric for structural integrity. It’s trying to see how well the local edits interact with the whole page context.

Jane: The conclusion is that this benchmark gives us a way to diagnose exactly where these models fail, whether it’s misreading characters or just letting the document layout collapse around them.

Tom: It sets a pretty high bar for testing image editing on complex text, especially for multilingual documents like Chinese and English.

Jane: And it shows that we need to move past just looking at pretty images and start measuring how well they handle the real complexity of documents in production settings.

Lu: So, if you're building something that needs to manipulate visual information, VDE Bench gives you a rigorous framework based on human judgment for assessing text recognition and layout consistency.

Meng: It means we can start stress testing our models against these specific failure modes rather than just running general benchmarks on random images.

Tom: Exactly, it’s a trustworthy way to see if an image editing model is actually capable of doing the job in a real document workflow.

Jane: This leads us to thinking about what happens next—how do we use this kind of structured evaluation to guide the next generation of these models?

UCAS 2CASIA Tencent CMU WashU SJTU XDU

cs.CV, cs.AI, cs.MM

Submitted: 2026-01-27

Updated: 2026-10-15

Code: https://github.com/zion-zion-zion/VDE-Bench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist The VDE Bench is a rigorously human annotated and evaluated benchmark specifically designed to assess the performance of image editing models on bilingual Chinese-English and complex visual

Key concepts

VDE Bench: A comprehensive evaluation resource built to systematically measure the ability of image editing models to perform precise modifications on complex visual documents. It pairs editing instructions with reference images and provides structured annotations for text regions and tables.
Instruction-Guided Editing: The process where an AI model modifies an image based on a natural language command given by a user. This task is challenging because the model must perform local changes within a dense document while maintaining the original text style and layout context.
OCR Text-Level Evaluation: A metric used to assess the accuracy of the text recognition output after an image is edited. It compares the recognized text in both original and modified regions against ground truth data to ensure that textual content modifications were performed correctly.
Structural Fidelity: This refers to how well the editing process preserves the overall layout, arrangement, and visual integrity of a document. It measures whether localized text changes have negatively impacted the surrounding document structure or background context.

Terminology

Benchmark — 4× in this paper · explained in 2 episodes
Accuracy — 1× in this paper · explained in 1 episode
Consistency — 1× in this paper · explained in 3 episodes
Papers — 1× in this paper · explained in 3 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

The gist The VDE Bench is a rigorously human annotated and evaluated benchmark specifically designed to assess the performance of image editing models on bilingual Chinese-English and complex visual document editing tasks.

Motivation

Image editing models have continuously advanced, enabling users to iteratively manipulate image content through natural language, which greatly enhances the flexibility and interactivity of visual creation However, an important yet underexplored research direction remains dense visual document image editing, which involves modifying textual content within images while faithfully preserving the original text style and background context Existing methods primarily focus on English scenarios and images with relatively sparse text, and thus cannot adequately address dense, structurally complex documents or non-Latin scripts such as Chinese

Guiding Principles and Task Definition

To systematically evaluate the usability of different image editing models on complex text documents, VDE Bench is proposed as a rigorously human evaluated benchmark designed to diagnose the performance of image editing models on complex text document editing tasks VDE Bench evaluates the quality of editing operations performed on dense text documents from two complementary dimensions: OCR text-level evaluation and image visual-level evaluation The task is formally defined as producing an edited document D′ along with its corresponding edited regions ei, and an OCR system is executed to obtain the recognition output Oi, which are then assessed for discrepancy against the ground truth

VDE Bench Construction

Constructing VDE Bench requires extensive manual annotation and processing, utilizing a pipeline consisting of four stages: (1) collecting diverse documents across multiple categories and languages; (2) generating editing instructions via Gemini3-Pro / Qwen3-VL-235B-A22B Instruct and producing edited images with Nano Banana Pro; (3) human review and annotation by 15 annotators with multi dimensional scoring, IoU based region verification, and subjective quality filtering; and (4) structured OCR processing with PaddleOCR-VL The final benchmark contains 942 high quality instruction editing samples with rich structural annotations

Evaluation Protocol and Results

The evaluation framework simultaneously considers spatial localization accuracy and textual content correctness (OCR metrics), while style preservation serves as a reference indicator of the output image quality Key metrics include IoU for spatial localization and CDM (Character Distance Metric), BLEU-4 for linguistic fluency, and TEDS-like metric for structural integrity Table 8 shows that Qwen achieves the best performance in the Global-image Setting across most OCR metrics, with a TEDS score of 0.902

Conclusion

VDE Bench introduces a comprehensive benchmark for systematically evaluating the direct editing capabilities of image editing models on complex text documents, assessing performance in text recognition, modification, format preservation, and layout consistency The evaluation protocol is highly aligned with human judgement This framework provides a trustworthy and rigorous framework for evaluating image editing models on multilingual and densely textual documents

How it works

Data Collection Stage: Documents were sourced from Omni Doc Bench Layout and Document Types Diversity: Layout elements are extracted using a normalization method to compute document layout density, highlighting distribution patterns specific to different document types Manual Review Stage: Annotators score samples based on Instruction Compliance, Modification Authenticity, and Subjective Satisfaction Final Verification: After image-level annotations are completed, PaddleOCR-VL is applied to perform OCR recognition on both the entire generated image and the modified regions Furthermore, different instruction types pose varying levels of difficulty; text-level edits are generally easier than structural ones The global evaluation shows that localized quality does not imply global consistency; for instance, FireRed collapses on global Text Modify (IOU 0.052) Finally, cross-lingual performance shows a consistent pattern where every model degrades noticeably on Chinese compared to Latin-script English documents

Limitation

VDEBench is currently focused primarily on Chinese and English documents and does not cover other languages such as Korean or Japanese <ref:2602.

Improvements for AI systems

Improve instruction constraint handling by utilizing strong constraint property of the constructed editing instructions to mitigate performance degradation under weakly constrained conditions, as shown in Figure 8. This allows models to achieve better performance across all metrics and exhibit higher alignment with the expected results when instructions are tightly constrained.
Enhance model robustness against language interference by analyzing language composition has a measurable impact on instruction ambiguity, specifically addressing the elevated ambiguity of mixed-language instructions observed in Table 6. This enables systems to better handle code-switching overhead, where the interleaving of two writing systems introduces additional parsing complexity.
Develop specialized structural editing capabilities by focusing on models robust to layout changes, such as LongCat and Qwen's performance on Table Structure Edit under global settings. This allows AI systems to achieve a superior preservation of tree-structured layout under a localized view, as indicated by their high TEDS scores in this setting.
Implement a comprehensive evaluation framework that simultaneously measures both OCR-level text accuracy and image-level visual fidelity, assessing textual content correctness and image-based metrics to provide a comprehensive reflection of model performance in real-world document editing scenarios.
Improve spatial localization precision by employing a greedy distance-based matching algorithm where the Euclidean distance is defined as d(i, j) = q(cgt x - cpred x) squared + (cgt y - cpred y) squared, ensuring that bounding box predictions are matched based on the nearest predicted block.
Refine text modification accuracy by integrating multiple metrics—"CDM verifies character-level precision of the edits; BLEU-4 measures linguistic fluency and semantic fidelity via n-gram overlap; and the TEDS-like metric leverages tree edit distance to assess structural and layout integrity—to evaluate low-level spatial perception to high-level semantic understanding."
Improve visual style preservation assessment by utilizing a suite of metrics, including SSIM, CLIP, PSNR, and LPIPS, which together characterize style preservation across four perceptual and structural dimensions, allowing for a more nuanced assessment than relying on single image-level metrics.