Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

arXiv:2605.11307 · cs.CV, cs.LG · Submitted 2026-05-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation".

Jane: The gist Vision2Code introduces a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation, which matters because it provides a reproducible testbed for measuring, diagnosing,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, wrapping up this discussion on Vision2Code, Tom, what's the main message we should take away about this work?

Tom: The main thing is that by moving to a multi-domain benchmark with dataset-specific rubrics, they are creating a much more accurate way to measure how well vision language models can reconstruct structure into executable code.

Lu: It moves us away from the narrow visual scopes of older benchmarks and tackles things like dense text layout in documents or precise depth ordering in three dee scenes simultaneously <ref:2605.11307#pg1>.

Meng: For practical application, having this evaluation framework means we have a way to systematically diagnose where an AI is failing when it tries to turn a picture into functioning code.

Tom: And the authors suggest that this approach is more aligned with human judgment because it focuses on the specific visual information that must be preserved for each task.

Jane: It really shows that relying on generic similarity metrics doesn't capture the nuanced semantic requirements of structured image reconstruction.

Lu: They are testing how models handle things like label-to-object binding in geometry or edge directionality in graphs, which are specific structural challenges.

Tom: So, Vision2Code gives us a way to push these models toward better understanding the underlying structure rather than just generating a plausible-looking image.

Jane: It's an important step for making sure that when we evaluate these vision language models, we're measuring what actually matters in terms of code generation quality.

Tom: That’s it for this look at Vision2Code: a multi-domain benchmark for evaluating image-to-code generation.

Conclusion: Tom: So we’ve seen how Vision2Code is setting up this whole new system for testing image-to-code models that you can actually trust to work across different visual tasks.

Jane: It really boils down to them creating a benchmark where you don't need some pre-written code as a reference anymore, right?

Lu: Exactly. They’re looking at fifteen totally different types of images—everything from charts and graphs to complex three dee scenes—and they want to see if the model can turn those visuals into working programs.

Meng: The point is that these old benchmarks were too narrow, so this new one tries to prove the AI can do it in many places, not just one or two specific areas.

Tom: And what they found is that when you use these specific rubrics for each type of image, the final scores actually match human ratings way better than just looking at how similar the generated code looks to the original picture.

Jane: That’s a big deal because it tells us that just having a visually okay output isn't enough; it has to be structurally and semantically correct for what you asked of it.

Lu: They even showed that for things like charts and graphs, the model is significantly better than in other domains when they use this detailed scoring method.

Meng: From an engineering side, knowing where the weaknesses are across all these domains helps us know exactly what we need to train next to make those models more reliable.

Tom: It makes you wonder what this means for how we actually use AI to build things that rely on visual understanding.

Jane: It suggests that for any complex visual task, you need a way to measure the specific details being reconstructed, not just a general similarity score.

Lu: So it’s not just about making pretty pictures anymore; it’s about ensuring the underlying data structure is preserved correctly when generating code. **(Sound of traffic fades out)**

Duke University

cs.CV, cs.LG

Submitted: 2026-05-11

Updated: 2026-10-09

Comments: Project page: https://image2code.github.io/vision2code/

Project page: https://image2code.github.io/vision2code

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist Vision2Code introduces a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation, which matters because it provides a reproducible testbed for

Key concepts

Vision2Code Benchmark: This is a comprehensive testbed containing 2,169 examples from 15 datasets covering six visual domains: charts/plots, geometry, graphs, scientific imagery, documents, and 3D scenes. It allows researchers to measure image-to-code generation performance without needing paired source code for evaluation.
Dataset-Specific Rubrics: Instead of generic scoring rules, these rubrics are tailored to each dataset's requirements. For example, a chart rubric might heavily weight 'Numeric information' and 'Structure and layout,' while a spatial scene rubric focuses on 'Depth ordering.' These tailored categories enforce strict penalties for domain-specific failures.
Rendered-Code Evaluation: The model generates code, which is then actually executed by the system (defaulting to Python/Matplotlib). The resulting rendered image is then compared against the original source image using the dataset-specific rubric. This process tests whether the generated code successfully produces a visually accurate output.
Correlation with Human Ratings: The paper demonstrates that scores derived from this rigorous, domain-specific evaluation method correlate much more strongly with human ratings than generic metrics like simple pixel similarity or embedding-based scores. This validates the framework as a reliable way to assess VLM performance.

Terminology

Benchmark — 4× in this paper · explained in 2 episodes
Reconstruction Error — 1× in this paper · explained in 2 episodes
Vision-Language Model (VLM) — 1× in this paper · explained in 3 episodes

Summary

The gist Vision2Code introduces a reference-code-free benchmark and evaluation framework for multi-domain image-to-code generation, which matters because it provides a reproducible testbed for measuring, diagnosing, and improving image-to-code generation across diverse visual domains without relying on paired executable reference code.

Vision2Code Benchmark Scope

Vision2Code contains 2,169 test examples from 15 source datasets that span charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes It spans six visual domains: charts and plots, geometry, graphs, science imagery, documents, and 3D spatial scenes. The benchmark contains 2,169 manually filtered test examples, along with a 539-example test-mini subset for lower-cost evaluation.

Evaluation Framework

The evaluation process involves several key steps to ensure rigorous measurement

Rendered-code evaluation begins by executing the code generated by the model, defaulting to Python with Matplotlib and NumPy for the main benchmark.
Dataset-specific rubric uses four weighted categories tailored to each dataset, scoring outputs on a 0–5 scale.
Deterministic guardrails cap the final score if severe semantic mistakes are detected, ensuring that severe semantic mistakes cannot receive high ratings.
Validation protocol compares the dataset-specific rubric against human judgments and automatic baselines, showing that it aligns better with human ratings than generic rubrics or embedding-similarity baselines.

Domain Coverage and Rubrics

The six domains are chosen to expose different reconstruction challenges

** Charts and plots probe axis scales, tick spacing, and the binding between series colors and legend entries. 2. Geometry probes vertex and angle labeling, where labels are spatially adjacent to but not coincident with the objects they name. 3. Graphs probe node identity, edge connectivity, and edge directionality. 4. Scientific figures probe specialized notation where small visual differences carry the semantics. 5. Documents probe dense OCR with column structure, table cells, and reading order. 6. Spatial scenes probe object identity, depth ordering, and precise spatial relations in 3D. The domains together span structured quantitative graphics, symbolic diagrams, text-heavy layouts, and 3D scenes. The dataset-specific rubrics are built by identifying the visual information that must be preserved for that dataset. For instance, for ChartQA, the rubric includes categories like Numeric information (.30), Structure and layout (.30), and Text, labels, and legibility (.25). The critical categories are treated as critical semantic categories with higher weights to enforce strict penalties for failures. For SpatialVLM-QA, the rubric emphasizes Object shape and identity (.25), Spatial relations (.35), 3D depth rendering (.25), Color and scene completeness (.15). The cap profiles are designed to prevent high final scores when a key semantic failure is present, such as wrong geometric relations or ambiguous label association in Geometry3K. The generic rubric baseline uses four equal-weight categories for every dataset, testing whether the improvement comes from dataset-specific criteria or just the rater model. The generic rubric reaches Pearson/Spearman correlations of 0.723/0.703, compared with 0.626/0.633 for a generic rubric. The dataset-specific rubric is the most aligned automatic metric, reaching Pearson/Spearman correlations of 0.723/0.703. The embedding similarity baselines are weaker because they reward global visual resemblance and often miss localized semantic errors. For instance, for ChartQA on test-mini, the Image+text cosine-similarity scaled score is 3.980. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703 compared with 0.626/0.633 for a generic rubric. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.5-9B on test-mini, the Cos img+txt scaled score is 3.626. The dataset-specific rubric reaches Pearson/Spearman correlations of 0.723/0.703. The dataset-specific rubric is the most aligned automatic metric. The cosine baselines are substantially weaker, even though their mean scores are high. For Qwen3.

Improvements for AI systems

Bold header: Domain-Specific Evaluation Framework Implementation

Vision2Code introduces a domain-aware evaluation framework that renders generated code, scores outputs with a VLM rater and dataset-specific rubrics while applying deterministic guardrails for severe semantic failures. This allows models to be rigorously tested against the specific structural requirements of each visual domain (e.g., axes and legends for charts in ChartQA) rather than relying on generic metrics.

Bold header: Evaluator-Guided Model Fine-Tuning Pipeline

The paper shows that evaluator-filtered model outputs can serve as training data to improve image-tocode capability. Specifically, fine-tuning on first-stage pairs (I0, C1) selected by filters over (R1, R2) improves Qwen3.5-9B from 1.60 to 1.86 on test-mini. This enables iterative self-improvement of image-to-code models without requiring expensive paired source programs.

Bold header: Enhanced Error Diagnosis via Render Diagnostics

The system reports render-success diagnostics that separate code execution failures from reconstruction quality. This allows researchers to distinguish between a model's inability to write executable code (e.g., Syntax/truncation or Missing dependency) and its failure in visual understanding (e.g., Shape/3D error or semantic reconstruction errors).

Bold header: Robustness Against Spatial Reconstruction Failures

The results indicate that models remain weak on spatial scenes, and the evaluation framework includes caps like Cap at 2.5 for wrong geometric relations or ambiguous label association in Geoperception and Geometry3K rubrics. This forces models to learn precise 3D spatial relations necessary for high scores in challenging domains like SpatialVLM-QA.

Bold header: Automated Comparison Against Human Judgments

The framework validates that the dataset-specific rubric aligns better with human judgments than a generic rubric or embedding-similarity baselines, reaching Pearson/Spearman correlations of 0.723/0.703 compared to lower scores for generic and embedding metrics. This ensures the benchmark's scoring protocol is grounded in real human perceptual standards.

Bold header: Adaptability Across Programming Toolchains

The evaluation protocol is code-language agnostic: any executable artifact whose rendering produces an image is admissible, demonstrated by successful testing with Excalidraw JSON and standalone LaTeX, proving the framework can be extended to non-Python targets.

Abstract

We study whether vision-language models (VLMs) can reconstruct structured, graphics-based images by writing executable code to generate them. Existing benchmarks on image-to-code generation are generally limited in the domains they cover, rely on reference source code for evaluation, and do not provide metrics to quantify domain-specific issues. To address this, we introduce Vision2Code, a multi-domain, reference-code-free benchmark containing 539 test examples from 15 different datasets, spanning charts and plots, geometry, graphs, scientific diagrams, documents and 3D spatial scenes. Evaluation is done by executing the generated programs and comparing their rendered outputs to the source images using dataset- and domain-specific rubrics for a VLM judge. We find that the final scores obtained in this manner have substantially higher correlation with human ratings than those obtained via generic VLM rubrics or standard pixel-based, embedding-based or perceptual similarity metrics. We further demonstrate the reliability of our evaluation under repeated sampling from the VLM judges, varying rubric weights, and in the presence of controlled errors. Our results highlight the variability across domains of existing image-to-code generation models, with accuracy on charts and graphs being significantly higher than other domains. Lastly, we demonstrate the utility of our scoring method for rejection sampling of generated programs for fine-tuning, improving Qwen3.5-9B by 16% and Phi-4-MM by 28% compared to their base checkpoints. Our code and data are publicly available at https://image2code.github.io/vision2code/.