Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition

This paper addresses the synthetic-to-real gap in Optical Chemical Structure Recognition (OCSR), where systems perform well on synthetic images but poorly on real documents. The authors state: "OCSR appears nearly solved on synthetic images, but remains difficult on real documents. The starting recognizer Qwen2.5-VL-7B in this work achieves over 91% accuracy on synthetic images, but below 16% on three real-world benchmarks: ACS, CLEF-IP, and USPTO."

The authors fine-tuned 21 recognizers on mixtures of synthetically rendered structures and labeled real depictions drawn from patents, journal figures, and hand-drawn collections. The experiments varied the Vision Language Model (VLM) base, the fraction of real training data, and the vision-tower adaptation strategy.

Key findings:

1. Real labeled training images are the most effective way to close the synthetic-to-real gap. The authors state: Adding labeled real training images makes the largest difference. For Qwen2.5-VL, ACS exact match rises from 0.15 with no real data to 0.37 at 9.5% real data and 0.46 at 50.2%. A controlled experiment across three base models shows the same trend.

2. Vision LoRA does not always help. The authors report: On synthetic-only Qwen2.5-VL-7B, extending LoRA to the vision tower and connector changes ACS exact match by 0.000 (p = 1.00). However, It substantially helps InternVL3-8B (+22.8 to +34.6 pt) and modestly helps GLM-4.1V-9B (+1.0 to +9.6 pt), showing that its value depends on the base model.

3. Base-model choice matters most when real data are scarce. The authors state: The performance spread between bases falls from 0.212 with no real data to 0.060 at 70% real data, and their ranking changes as more real data are added.

4. The strongest overall model is GLM-4.1V-9B, trained on a mixture of synthetic and real depictions with LoRA applied to the language decoder, vision tower, and connector. It achieves 0.96 exact match on clean renders and 0.49, 0.65, 0.84, and 0.76 on ACS, CLEF-IP, UOB, and USPTO, respectively.

Experimental design: The central controlled experiment crosses three pretrained VLM bases (Qwen2.5-VL-7B, InternVL3-8B, GLM-4.1V-9B) with six real-data fractions (0%, 5%, 10%, 20%, 40%, 70%), producing 18 freshly fine-tuned cells. An approximately 496k-sample mixture size, 10k-step budget, supervised fine-tuning objective, and light vision-LoRA surface are held fixed.

Synthetic scaling results: Increasing the number of synthetic training molecules from 200k to 7M produces no monotonic improvement on the three available evaluation splits. Additional rendered-only interventions (verifiable RDKit reward, renderer diversity, larger trainable vision surface) provide either small, inconsistent, or negative changes.

Real-data dose series: An independent Qwen2.5-VL-7B series shows that with LM-only LoRA (vision encoder and connector frozen), When 9.5% real data is introduced, exact match reaches 0.372 on ACS, 0.608 on CLEF-IP, 0.818 on UOB, and 0.698 on USPTO. ACS exact match increases further to 0.459 at 50.2% real data.

Controlled sweep results: "Mean real-document exact match increases sharply between 0% and 5% real data and continues to increase through 70%. The first increment is the largest: Qwen2.5-VL-7B increases from 0.239 to 0.526, InternVL3-8B from 0.027 to 0.476, and GLM-4.1V-9B from 0.161 to 0.539. The real-document improvement is accompanied by a decline in rendered-clean exact match: Between 0% and 70% real data, the rendered-clean score decreases from 0.784 to 0.680 for Qwen, from 0.844 to 0.706 for InternVL, and from 0.816 to 0.718 for GLM."

Base rankings change: Qwen has the highest real-document average at 0% real data, while GLM leads at every nonzero fraction. InternVL moves from the weakest base at 0% to the second strongest from 20% onward.

Vision adaptation findings: For Qwen2.5-VL-7B with synthetic-only data, vision-plus-connector LoRA provides no positive real-document effect. For GLM-4.1V-9B with real depictions, light vision LoRA increases exact match from 0.486 to 0.547 on ACS, from 0.586 to 0.630 on CLEF-IP, from 0.866 to 0.876 on UOB, and from 0.684 to 0.780 on USPTO. For InternVL3-8B, the gains are much larger: Exact match increases from 0.227 to 0.505 on ACS, from 0.240 to 0.578 on CLEF-IP, from 0.616 to 0.844 on UOB, and from 0.380 to 0.726 on USPTO.

Comparison with published recognizers: Specialist recognizers remain strongest on several real-document sources, whereas the tuned GLM checkpoint is strongest on the displayed rendered conditions. No recognizer leads across all six conditions. OCSRGlyph performs best on patent-derived test sets (0.850 on CLEF-IP, 0.928 on USPTO). The 70%-real GLM checkpoint achieves the highest ACS (0.598) and UOB (0.898) scores. Paired McNemar tests place the 70%-real GLM checkpoint level with the strongest released system on ACS (p=0.66) and UOB (p=0.12).

Cross-task probes: For handwritten image-to-LaTeX recognition, InternVL3 obtains edit similarity 0.829, Qwen2.5-VL 0.815, and GLM-4.1V 0.737. For chart-to-table conversion, InternVL3 obtains 0.606, Qwen2.5-VL 0.560, and GLM-4.1V 0.524. The authors conclude: Base-model performance is therefore task-dependent: a base that transfers well to molecular depictions is not necessarily the strongest base for other image-to-structure tasks.

Conclusion: The authors state: Real depictions are the most effective lever examined for OCSR under document-level distribution shift. They recommend: representative real data should be acquired first, the pretrained base should be selected at the intended real-data scale, and the vision path should be adapted only when supported by a matched comparison.

Improvements to AI systems:

1. Implement a real-data-aware training curriculum. Instead of relying solely on synthetic data, the improved system will prioritize acquiring and mixing labeled real-world examples into the training set, with the first 5% of real data providing the largest performance jump (e.g., Qwen2.5-VL-7B real-document exact match rises from 0.239 to 0.526). The system will dynamically adjust the real-to-synthetic ratio based on target-domain benchmarks, up to 70% real data, while monitoring and accepting a controlled trade-off in clean-render performance (e.g., from 0.784 to 0.680 for Qwen).

2. Make vision-tower adaptation conditional on base-model evidence. The improved system will not blindly apply vision LoRA. It will run a matched-pair ablation (e.g., with and without vision LoRA on a small validation set) before full training. For bases like Qwen2.5-VL-7B, it will skip vision LoRA entirely (since it yields zero gain, p=1.00), while for InternVL3-8B it will enable it to capture large gains (e.g., ACS exact match from 0.227 to 0.505) and for GLM-4.1V-9B it will use light vision LoRA for moderate gains (e.g., USPTO from 0.684 to 0.780).

3. Select the base model at the intended real-data scale, not in a data-scarce setting. The improved system will evaluate candidate bases (Qwen2.5-VL-7B, InternVL3-8B, GLM-4.1V-9B) across multiple real-data fractions (0%, 5%, 10%, 20%, 40%, 70%) and choose the leader at the target fraction. For example, it will prefer Qwen when real data is absent, but switch to GLM at any nonzero fraction (GLM leads at 5% through 70%), and recognize that InternVL becomes second-best only from 20% onward. This prevents suboptimal base selection based on synthetic-only performance.

4. Avoid synthetic-data scaling as a primary lever. The improved system will cap synthetic training data at around 200k molecules and instead allocate compute to collecting real depictions, because increasing synthetic samples to 7M yields no monotonic improvement on real benchmarks. It will also deprioritize renderer diversity and verifiable reward interventions, as these provide small, inconsistent, or negative effects.

5. Use task-specific base-model routing for multi-domain image-to-structure tasks. The improved system will maintain a portfolio of fine-tuned bases and route inputs by task type. For molecular structure recognition, it will use the GLM-4.1V-9B checkpoint (trained with 70% real data and vision LoRA) for best overall real-document performance (ACS 0.598, UOB 0.898). For handwritten LaTeX, it will route to InternVL3 (edit similarity 0.829 vs. GLM’s 0.737). For chart-to-table conversion, it will route to InternVL3 (0.606 vs. GLM’s 0.524). This avoids forcing a single base to handle all tasks.

6. Integrate a hybrid recognizer selection for patent-heavy workloads. The improved system will combine the tuned GLM checkpoint with a specialist recognizer like OCSRGlyph, using a confidence-based or domain-rule-based switch. It will use OCSRGlyph for patent-derived test sets (CLEF-IP 0.850, USPTO 0.928) and GLM for ACS and UOB, since no single recognizer leads across all six conditions.

What the improved AI system can do:

• Achieve up to 3.2× higher real-document exact match (e.g., from 0.15 to 0.46 on ACS with Qwen) by incorporating labeled real images, without sacrificing clean-render accuracy beyond a controlled, predictable margin.

• Automatically decide whether to adapt the vision tower based on empirical evidence, avoiding wasted compute and potential degradation on bases like Qwen2.5-VL-7B.

• Deliver state-of-the-art OCSR performance on real-world benchmarks: 0.598 exact match on ACS, 0.898 on UOB, 0.780 on USPTO, and 0.630 on CLEF-IP, using the tuned GLM-4.1V-9B with 70% real data and vision LoRA.

• Match the strongest published specialist recognizers on ACS (p=0.66) and UOB (p=0.12) while remaining competitive on patent-derived data when paired with OCSRGlyph.

• Generalize across multiple image-to-structure tasks by routing to the best-performing base per task, achieving 0.829 edit similarity on handwritten LaTeX and 0.606 on chart-to-table conversion.

• Reduce data collection costs by focusing on a small but representative real-data set (first 5% yields the largest gain) rather than scaling synthetic data, which provides diminishing or no returns.