COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training".
Jane: Here's the gist of the paper. Optical chemical structure recognition (OCSR) translates molecular images into machine-readable representations like SMILES strings
1: and remains challenging in real-world documents due to inexhaustible variations in chemical structures, shorthand conventions, and visual noise <ref:2604.23546#pg2> Most existing deep-learning-based approaches rely on teacher forcing with token-level Maximum Likelihood Estimation (MLE) which suffers from exposure bias and hinders optimization towards molecular-level evaluation criteria such as chemical validity and structural similarity <ref:2604.23546#pg3> Here we introduce Minimum Risk Training (MRT) to OCSR and propose COMO, a closed-loop framework that mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives by iteratively sampling and evaluating the model’s own predictions <ref:2604.23546#pg4> Experiments on ten benchmarks including synthetic and real-world chemical diagrams from patent and scientific literature demonstrate that COMO substantially outperforms existing rule-based and learning-based methods with less training data <ref:2604.23546#pg2> Pruning redundant reasoning tokens COMO is a closed-loop optical molecule recognition framework built with Minimum Risk Training (MRT) which mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives by iteratively sampling and evaluating the model’s own predictions <ref:2604.23546#pg2> This framework retains the encoder–decoder backbone while augmenting purely token-level MLE with a joint MLE–MRT training strategy <ref:2604.23546#pg2> The reward function is modular, combining three complementary signals: chemical validity, exact string match, and an interchangeable structural similarity term <ref:2604.23546#pg2> This design offers three practical advantages including closing the exposure-bias loop by computing gradients over samples drawn from the model’s own predictive distribution rather than teacher-forced prefixes <ref:2604.23546#pg2> How the training signal is built The Minimum Risk Training (MRT) loss is defined as LMRT(θ) = E ŷ∼pθ(·x)
2604.235468: which minimizes the expected cost under the model’s own sampling distribution <ref:2604.23546#pg9> The reward function R(ˆy, y∗) is decomposed into three terms capturing finer aspects of prediction quality: R(ˆy, y∗) = wv · 1
valid: + ws · Sim(ˆy, y∗) · 1
valid: + we · 1
exact: <ref:2604.23546#pg10> The similarity component Sim is modular and can be instantiated as edit distance, fingerprint-based Tanimoto similarity, or a learned visual similarity score from a separate image encoder Where the gains come from COMO achieves state-of-the-art exact match accuracy across all ten OCSR benchmarks with particularly large improvements on challenging real-world datasets including JPO (+10.2 pp), UOB (+3.6 pp), ACS (+16.3 pp), and WildMol-10K (+0.3 pp) Ablation studies further confirm that the gains stem from the training mechanism rather than from additional data, as replacing MRT steps with identically scheduled MLE steps on the same real-world images results in performance degradation on 5 out of 10 benchmarks Furthermore, applying MRT to a structurally distinct model, SwinOCSR, yields accuracy gains of 2–20× on real-world benchmarks, confirming that MRT is a model-agnostic training principle for OCSR COMO achieves the highest exact-match accuracy across all 10 benchmarks, with both variants delivering comparable results (within 2 percentage points of each other), suggesting that MRT is robust to the choice of similarity measurement The performance improvement on JPO, USPTO, and ACS is marginal compared to MRT results (lower half of Table II) This pattern indicates that MRT is especially effective at bridging the domain gap between synthetic training images and real-world test images, which cannot be achieved by MLE alone, even when given the same additional data The gains range from modest on synthetic data (+1.1 on Indigo, +2.7 on ChemDraw) to substantial on real-world data (+5.9 on CLEF, +12.2 on JPO, +10.1 on UOB, +14.5 on ACS, +10.7 on WildMol-10K) The work is the first effort to introduce MRT into end-to-end OCSR Page 1 --- COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training Zhuoqi Lyu1 and Qing Ke1, Department of Data Science, College of Computing, City University of Hong Kong, Hong Kong, China (Dated: April 28, 2026 Abstract Optical chemical structure recognition (OCSR) translates molecular images into machinereadable representations like SMILES strings or molecular graphs <ref:2604.23546#pg2> Most existing deep-learning-based approaches rely on teacher forcing with token-level Maximum Likelihood Estimation (MLE) which suffers from exposure bias, as models are trained under ground-truth prefixes but must condition on their own previous predictions during inference <ref:2604.23546#pg3> Moreover, token-level MLE objectives hinder the optimization towards molecular-level evaluation criteria such as chemical validity and structural similarity <ref:2604.23546#pg2> Here we introduce Minimum Risk Training (MRT) to OCSR and propose COMO (Closed-loop Optical Molecule recOgnition), a closed-loop framework that mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives, by iteratively sampling and evaluating the model’s own predictions <ref:2604.23546#pg2> Page 2 --- I.
Tom: INTRODUCTION Optical chemical structure recognition (OCSR) aims to convert molecular structure images into machine-readable representations such as linear notations like SMILES strings
1: It is a fundamental component for large-scale chemical information extraction, enabling the digitization of compounds from patents
2, 3: and scientific literature
4: Although recent deep-learning-based OCSR systems have substantially improved over rule-based pipelines by framing the task as image-to-sequence generation or graph prediction, accurate recognition from real-world documents remains challenging <ref:2604.23546#pg2> As presented in Fig.
Jane: 1, models must cope with complex ring structures, substituent abbreviations (e.g., “Et”, “Me”), condensed formulas (e.g., “COOH”), R-group placeholders (e.g., “R1”, “L”), as well as cluttered surrounding text and visual artifacts introduced by scanning and document layout <ref:2604.23546#pg2> These factors create a substantial distribution gap between the clean, synthetic images commonly used for training and the heterogeneous molecular depictions encountered in real documents, causing models to produce predictions that are structurally plausible but chemically incorrect <ref:2604.23546#pg2> A central reason for this failure is the mismatch between how current encoder-decoder OCSR models are trained and how they perform inference <ref:2604.23546#pg2> Most modern systems rely on an autoregressive sequence decoder trained with teacher forcing, which minimizes token-level cross-entropy under ground-truth prefixes <ref:2604.23546#pg2> While effective for learning local syntactic patterns, this Maximum Likelihood Estimation (MLE) objective introduces two compounding limitations: • Exposure bias: Under teacher forcing, the decoder is trained to condition on goldstandard prefixes, whereas at inference time, it must condition on its own previous outputs rather than ground-truth prefixes
5, 6: <ref:2604.23546#pg2> Consequently, errors made early in decoding can propagate and accumulate in ways never encountered during training <ref:2604.23546#pg2> This issue is especially severe for SMILES generation: a single incorrect atom type, misplaced parentheses, or erroneous ring-closure digit can render an otherwise correct molecular structure chemically invalid, turning a near-miss into a complete failure <ref:2604.23546#pg2> • Objective function mismatch: The token-level cross-entropy objective is structurally incapable of optimizing the molecular-level metrics, including chemical validity, Tanimoto similarity, and exact structural match, which the evaluation actually measures <ref:2604.23546#pg2> Page 3 --- CLEF Input Image log p = -34.4 R = 0.41 Greedy Prediction log p = -35.0 R = 1.00 Candidate 1 log p = -54.7 R = 0.42 Candidate 2 log p = -55.0 R = 0.54 Candidate 3 JPO log p = -11.8 R = 0.46 log p = -12.6 R = 0.50 log p = -12.7 R = 0.38 ACS log p = -8.9 R = 0.42 log p = -9.4 R = 0.31 log p = -11.1 R = 0.42 log p = -12.2 R = 1 <ref:2604.23546#pg4> WildMol-10K log p = -25.8 R = 0.44 log p = -47.0 R = 0.41 log p = -52.1 R = 0.44 log p = -66.4 R = 1 <ref:2604.23546#pg4> FIG.
Tom: 1 Examples of predictions made by an MLE-trained model The second column presents greedy-search predictions ˆyi, with the corresponding log probabilities log p(ˆyix) (larger indicates higher probability) annotated below In all shown examples, the greedy predictions fail (highlighted in red), with atom-level discrepancies explicitly highlighted in the skeletal formulas In the third to fifth columns, we display three prediction candidates generated using multinomial sampling (N = 32, temperature = 0.5) We show that the correct predictions (shown in green) can lie within the MLE model’s search space By applying meaningful similarity measurements (Tanimoto similarity is used in this figure) between the candidate and the ground truth, we can compute a reward R (annotated below) that serves as a guiding signal to optimize the decoding distribution backward When fed the same images into our trained COMO model, it always hits the correct predictions with the highest R Page 4 --- FIG.
Title and authors: Tom: So we're talking about the "COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training" paper. The authors are Zhuoqi Lyu and Qing Ke from the Department of Data Science at City University of Hong Kong.
Jane: That's a pretty specific team for such a deep learning problem. What does that title actually tell us about what they did?
Lu: It tells us that they are taking their existing optical molecule recognition system and adding this closed-loop training mechanism, the Minimum Risk Training, to fix the problems we just talked about.
Tom: So it's not just a tweak; it's a whole new way of training the model by making it check its own work against real molecular quality metrics <ref:2604.23546#pg1>.
Jane: And what does this mean for the world? If they can fix the chemical validity issue, what kind of applications do we see?
Lu: It means we can move beyond just making things look plausible to actually ensuring the molecules are chemically correct and structurally sound, which is a huge leap for drug discovery or any chemical information pipeline <ref:2604.23546#pg2>.
Tom: I wonder if this makes it easier to build systems that handle messy, real-world documents, like those from old patents or scientific journals.
Jane: That's right, because they are specifically addressing the visual noise and shorthand conventions that confuse models when they look at real documents <ref:2604.23546#pg2>.
Lu: The paper shows this framework retains the original encoder-decoder backbone but augments the token-level training with a joint MLE and MRT strategy <ref:2604.23546#pg2>.
The paper's summary: Tom: So, to summarize what COMO does, it uses Minimum Risk Training to directly optimize for molecule-level objectives that aren't differentiable, like chemical validity and structural similarity <ref:2604.23546#pg1>.
Jane: That’s the core idea: they’re not just optimizing for how well the model predicts the next token; they are making it optimize for whether the whole final molecule is chemically valid or structurally similar to the target <ref:2604.23546#pg2>.
Lu: The reward function they use is modular, combining three signals: chemical validity, an exact string match, and a structural similarity term that can be swapped out for different kinds of similarity measures <ref:2604.23546#pg1>.
Tom: That modularity is interesting because it means they aren't locked into one definition of "good" structure or "good" representation.
Jane: And the way they build the training signal is key here; they define the Minimum Risk Training loss as minimizing the expected cost under the model’s own sampling distribution <ref:2604.23546#pg9>.
Lu: They decompose that reward into three parts: a weight for validity, a weight for similarity if it's valid, and a weight for exact matches <ref:2604.23546#pg10>.
The paper's improvements: Tom: What are the actual results they show? I want the numbers. How much better is this compared to just using standard MLE training?
Jane: They tested it on ten different benchmarks, including synthetic and real-world chemical diagrams from patents and literature <ref:2604.23546#pg2>.
Lu: The results show that COMO substantially outperforms existing rule-based and learning-based methods, even when they use less training data than previous approaches <ref:2604.23546#pg2>.
Tom: That's good, but how much better are we talking? What kind of improvements on those real-world datasets?
Jane: On ten benchmarks, they saw big jumps. For example, on JPO, they improved by ten point two percent, and on ACS, they saw a jump of sixteen point three percent <ref:2604.23546#pg2>.
Lu: They also did ablation studies that confirm these gains come from the training mechanism itself and not just having more data <ref:2604.23546#pg2>.
Conclusion: Tom: So, to wrap up, the main implication is that this closed-loop training mechanism makes a massive difference when moving from synthetic training images to real-world test images <ref:2604.23546#pg1>.
Jane: It bridges that gap where standard MLE just can't reach because it doesn't understand chemical validity or structural similarity <ref:2604.23546#pg1>.
Lu: The paper shows that MRT is model-agnostic; they tested it on a different model called SwinOCSR and saw accuracy gains of two to twenty percent on real-world benchmarks <ref:2604.23546#pg2>.
Tom: That's compelling, because if it works across different architectures like that, it suggests this is a really robust training principle for optical molecule recognition two thousand six hundred four point zero two five four three.
Jane: So the final picture is that COMO achieves the highest exact-match accuracy across all ten benchmarks, and both variants perform very similarly, within two percentage points of each other <ref:2604.23546#pg1>.
Lu: It's interesting that on some datasets like JPO and ACS, the performance improvement is smaller compared to other datasets like CLEF or UOB <ref:2604.23546#pg1>.
Meng: From an engineering standpoint, it’s good to know that the gains come from the training itself, not just feeding it more data on the same images <ref:2604.23546#pg2>.
Tom: And for deployment at scale, if we can use this MRT approach instead of just relying on token-level MLE, that sounds like a cleaner way to build systems that actually work in the real world two thousand six hundred four point zero two five four three.
Zhuoqi Lyu, Qing Ke
City University of Hong Kong
cs.CV, cs.AI, cs.LG
Submitted: 2026-04-26
Updated: 2026-11-10
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Optical chemical structure recognition (OCSR) translates molecular images into machine-readable representations like SMILES strings [1] and remains challenging in real-world documents due to…
Key concepts
- Exposure Bias
- This occurs when a model is trained using teacher forcing, meaning it learns to condition on correct ground-truth prefixes during training. However, at inference time, the model must rely on its own previous predictions. This mismatch causes errors made early in decoding to accumulate and propagate incorrectly.
- Minimum Risk Training (MRT)
- MRT is a training strategy that minimizes expected cost by directly optimizing over molecule-level objectives instead of relying solely on token-level MLE. It achieves this by iteratively sampling the model's own predictions, evaluating them using non-differentiable metrics, and updating parameters to favor higher-quality candidates.
- Reward Function Decomposition
- The reward function is split into three parts: chemical validity (is the structure chemically correct?), structural similarity (how close is it to the ground truth?), and exact match (is it identical?). This modular design allows the system to optimize for multiple, complex molecular quality criteria simultaneously.
- Closed-Loop Framework
- COMO creates a loop where the model generates predictions, evaluates them against real molecule metrics (the reward), and uses that feedback to retrain itself. This cycle closes the gap between training data and real-world evaluation, ensuring the model learns from its own performance signals.
Terminology
Summary
Optical chemical structure recognition (OCSR) translates molecular images into machine-readable representations like SMILES strings [1] and remains challenging in real-world documents due to inexhaustible variations in chemical structures, shorthand conventions, and visual noise <ref:2604. p. 023546#pg2> Most existing deep-learning-based approaches rely on teacher forcing with token-level Maximum Likelihood Estimation (MLE) which suffers from exposure bias and hinders optimization towards molecular-level evaluation criteria such as chemical validity and structural similarity <ref:2604. p. 023546#pg3> Here we introduce Minimum Risk Training (MRT) to OCSR and propose COMO, a closed-loop framework that mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives by iteratively sampling and evaluating the model’s own predictions <ref:2604. p. 023546#pg4> Experiments on ten benchmarks including synthetic and real-world chemical diagrams from patent and scientific literature demonstrate that COMO substantially outperforms existing rule-based and learning-based methods with less training data <ref:2604. p. 023546#pg2>
Pruning redundant reasoning tokens
COMO is a closed-loop optical molecule recognition framework built with Minimum Risk Training (MRT) which mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives by iteratively sampling and evaluating the model’s own predictions <ref:2604.23546#pg2> This framework retains the encoder–decoder backbone while augmenting purely token-level MLE with a joint MLE–MRT training strategy <ref:2604. p. 323546#pg2> The reward function is modular, combining three complementary signals: chemical validity, exact string match, and an interchangeable structural similarity term <ref:2604. p. 323546#pg2> This design offers three practical advantages including closing the exposure-bias loop by computing gradients over samples drawn from the model’s own predictive distribution rather than teacher-forced prefixes <ref:2604. p. 323546#pg2>
How the training signal is built
The Minimum Risk Training (MRT) loss is defined as LMRT(θ) = E ŷ∼pθ(·x) [2604.235468] which minimizes the expected cost under the model’s own sampling distribution <ref:2604. p. 723546#pg9> The reward function R(ˆy, y∗) is decomposed into three terms capturing finer aspects of prediction quality: R(ˆy, y∗) = wv · 1[valid] + ws · Sim(ˆy, y∗) · 1[valid] + we · 1[exact] <ref:2604. p. 723546#pg10> The similarity component Sim is modular and can be instantiated as edit distance, fingerprint-based Tanimoto similarity, or a learned visual similarity score from a separate image encoder
Where the gains come from
COMO achieves state-of-the-art exact match accuracy across all ten OCSR benchmarks with particularly large improvements on challenging real-world datasets including JPO (+10.2 pp), UOB (+3. p. 56 pp), ACS (+16. p. 23 pp), and WildMol-10K (+0. p. 23 pp) Ablation studies further confirm that the gains stem from the training mechanism rather than from additional data, as replacing MRT steps with identically scheduled MLE steps on the same real-world images results in performance degradation on 5 out of 10 benchmarks Furthermore, applying MRT to a structurally distinct model, SwinOCSR, yields accuracy gains of 2–20× on real-world benchmarks, confirming that MRT is a model-agnostic training principle for OCSR
Conclusion
COMO achieves the highest exact-match accuracy across all 10 benchmarks, with both variants delivering comparable results (within 2 percentage points of each other), suggesting that MRT is robust to the choice of similarity measurement The performance improvement on JPO, USPTO, and ACS is marginal compared to MRT results (lower half of Table II) This pattern indicates that MRT is especially effective at bridging the domain gap between synthetic training images and real-world test images, which cannot be achieved by MLE alone, even when given the same additional data The gains range from modest on synthetic data (+1.1 on Indigo, +2. p. 47 on ChemDraw) to substantial on real-world data (+5. p. 09 on CLEF, +12. p. 22 on JPO, +10. p. 21 on UOB, +14. p. 55 on ACS, +10. p. 27 on WildMol-10K) The work is the first effort to introduce MRT into end-to-end OCSR
--- Page 1 ---
COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training Zhuoqi Lyu1 and Qing Ke1, Department of Data Science, College of Computing, City University of Hong Kong, Hong Kong, China (Dated: April 28, 2026 Abstract Optical chemical structure recognition (OCSR) translates molecular images into machinereadable representations like SMILES strings or molecular graphs <ref:2604.23546#pg2> Most existing deep-learning-based approaches rely on teacher forcing with token-level Maximum Likelihood Estimation (MLE) which suffers from exposure bias, as models are trained under ground-truth prefixes but must condition on their own previous predictions during inference <ref:2604. p. 023546#pg3> Moreover, token-level MLE objectives hinder the optimization towards molecular-level evaluation criteria such as chemical validity and structural similarity <ref:2604. p. 023546#pg2> Here we introduce Minimum Risk Training (MRT) to OCSR and propose COMO (Closed-loop Optical Molecule recOgnition), a closed-loop framework that mitigates exposure bias by directly optimizing over molecule-level, non-differentiable objectives, by iteratively sampling and evaluating the model’s own predictions <ref:2604. p. 023546#pg2>
--- Page 2 ---
I. INTRODUCTION Optical chemical structure recognition (OCSR) aims to convert molecular structure images into machine-readable representations such as linear notations like SMILES strings [1] It is a fundamental component for large-scale chemical information extraction, enabling the digitization of compounds from patents [2, 3] and scientific literature [4] Although recent deep-learning-based OCSR systems have substantially improved over rule-based pipelines by framing the task as image-to-sequence generation or graph prediction, accurate recognition from real-world documents remains challenging <ref:2604. p. 123546#pg2> As presented in Fig. p. 1 1, models must cope with complex ring structures, substituent abbreviations (e. p. 1g., “Et”, “Me”), condensed formulas (e. p. 1g., “COOH”), R-group placeholders (e. p. 1g., “R1”, “L”), as well as cluttered surrounding text and visual artifacts introduced by scanning and document layout <ref:2604. p. 123546#pg2> These factors create a substantial distribution gap between the clean, synthetic images commonly used for training and the heterogeneous molecular depictions encountered in real documents, causing models to produce predictions that are structurally plausible but chemically incorrect <ref:2604. p. 123546#pg2> A central reason for this failure is the mismatch between how current encoder-decoder OCSR models are trained and how they perform inference <ref:2604. p. 123546#pg2> Most modern systems rely on an autoregressive sequence decoder trained with teacher forcing, which minimizes token-level cross-entropy under ground-truth prefixes <ref:2604. p. 123546#pg2> While effective for learning local syntactic patterns, this Maximum Likelihood Estimation (MLE) objective introduces two compounding limitations: • Exposure bias: Under teacher forcing, the decoder is trained to condition on goldstandard prefixes, whereas at inference time, it must condition on its own previous outputs rather than ground-truth prefixes [5, 6] <ref:2604. p. 123546#pg2> Consequently, errors made early in decoding can propagate and accumulate in ways never encountered during training <ref:2604. p. 123546#pg2> This issue is especially severe for SMILES generation: a single incorrect atom type, misplaced parentheses, or erroneous ring-closure digit can render an otherwise correct molecular structure chemically invalid, turning a near-miss into a complete failure <ref:2604. p. 123546#pg2> • Objective function mismatch: The token-level cross-entropy objective is structurally incapable of optimizing the molecular-level metrics, including chemical validity, Tanimoto similarity, and exact structural match, which the evaluation actually measures <ref:2604. p. 123546#pg2>
--- Page 3 ---
CLEF Input Image log p = -34.4 R = 0.41 Greedy Prediction log p = -35. p. 20 R = 1.00 Candidate 1 log p = -54. p. 27 R = 0.42 Candidate 2 log p = -55. p. 20 R = 0.54 Candidate 3 JPO log p = -11. p. 28 R = 0.46 log p = -12. p. 26 R = 0.50 log p = -12. p. 27 R = 0.38 ACS log p = -8. p. 29 R = 0.42 log p = -9. p. 24 R = 0.31 log p = -11. p. 21 R = 0.42 log p = -12. p. 22 R = 1 <ref:2604. p. 023546#pg4> WildMol-10K log p = -25. p. 28 R = 0.44 log p = -47. p. 20 R = 0.41 log p = -52. p. 21 R = 0.44 log p = -66. p. 24 R = 1 <ref:2604. p. 023546#pg4> FIG. p. 1 1 Examples of predictions made by an MLE-trained model The second column presents greedy-search predictions ˆyi, with the corresponding log probabilities log p(ˆyix) (larger indicates higher probability) annotated below In all shown examples, the greedy predictions fail (highlighted in red), with atom-level discrepancies explicitly highlighted in the skeletal formulas In the third to fifth columns, we display three prediction candidates generated using multinomial sampling (N = 32, temperature = 0. p. 25) We show that the correct predictions (shown in green) can lie within the MLE model’s search space By applying meaningful similarity measurements (Tanimoto similarity is used in this figure) between the candidate and the ground truth, we can compute a reward R (annotated below) that serves as a guiding signal to optimize the decoding distribution backward When fed the same images into our trained COMO model, it always hits the correct predictions with the highest R
--- Page 4 ---
FIG. 1 both illustrates these failures and points toward a remedy Across several representative real-world examples, greedy decoding from a purely MLE-trained model yields incorrect predictions However, when the same model is allowed to generate multiple candidates by sampling, the correct structure frequently exists already within its search space The problem is therefore not a capacity failure: the model can represent the correct answer The problem is that MLE has concentrated probability mass on a suboptimal decoding path If one could evaluate these candidates against a molecular-level similarity signal and redistribute probability mass toward the chemically superior ones, the training objective would directly align with the evaluation criterion—closing the loop between prediction quality and parameter update
--- Page 5 ---
measures. These metrics are inherently discrete and non-differentiable, and hence inaccessible to standard MLE training Fig. p. 3 1 both illustrates these failures and points toward a remedy Across several representative real-world examples, greedy decoding from a purely MLE-trained model yields incorrect predictions However, when the same model is allowed to generate multiple candidates by sampling, the correct structure frequently exists already within its search space The problem is therefore not a capacity failure: the model can represent the correct answer The problem is that MLE has concentrated probability mass on a suboptimal decoding path If one could evaluate these candidates against a molecular-level similarity signal and redistribute probability mass toward the chemically superior ones, the training objective would directly align with the evaluation criterion—closing the loop between prediction quality and parameter update
--- Page 6 ---
Motivated by this observation, we propose COMO, a Closed-loop Optical Molecule recOgnition framework built with Minimum Risk Training (MRT) [7, 8] which provides a principled way to directly optimize discrete, non-differentiable metrics We instantiate it here for molecular recognition As shown in Fig. 2, COMO retains the encoder-decoder backbone while augmenting purely token-level MLE with a joint MLE–MRT training strategy During MRT, the model samples candidate SMILES from its own predictive distribution, evaluates them using a molecular-level reward, and updates its parameters to favor candidates with higher reward This design allows the model to learn from both fully supervised synthetic data and weakly supervised real-world images throughout training
--- Page 7 ---
A. Architecture COMO employs an encoder–decoder architecture for image-to-graph molecular recognition [11] Given an input molecular image I, the model predicts a 2D molecular graph G = (A, B), where A = [ai = (li, xi, yi)] is the set of atoms The bond type set T includes single, double, triple, aromatic, and wedge bonds a. p. 5 Image encoder. p. 3 An ImageNet-pretrained Swin-B backbone [12] encodes the 384×384 input into spatial feature maps b. p. 6 Sequence decoder. p. 1 We employ a 6-layer Transformer decoder with 8 attention heads to autoregressively generate a chartok coords token sequence S A = [l1, xˆ1, yˆ1,. p. 6.., ln, xˆn, yˆn] During training, the decoder is optimized via teacher forcing with label-smoothed cross-entropy (ϵ=0. p. 61) c. Bond predictor. p. 4 For each atom ai, its hidden state hai at the last coordinate token in the decoder output serves as the atom representation A two-layer MLP with GELU activation takes the concatenation [hai; haj] and classifies the bond type bi,j among seven classes (no bond, single, double, triple, aromatic, solid-wedge, dashed-wedge) Bidirectional predictions bi,j and bj i are symmetrized at inference time to improve accuracy Stereochemistry is determined by applying RDKit rules over the predicted graph and coordinates d. p. 6 MolDiscriminator. p. 3 In addition to the recognition model described above, we train a separate MolDiscriminator—a Siamese image-embedding network—that will later serve as a visual reward signal during minimum risk training (The following subsection)
--- Page 8 ---
B. Minimum Risk Training Under the standard MLE training object, the decoder is trained via teacher forcing, which introduces an exposure bias: at inference time, the model conditions on its own (potentially erroneous) predictions rather than ground-truth prefixes More importantly, the MLE objective is based on token-level crossentropy and cannot directly optimize molecular-level metrics, such as Tanimoto similarity It also cannot leverage real-world images that lack atom-coordinate annotations, since MLE requires fully tokenized supervision To address these limitations, we adopt Minimum Risk Training (MRT) [7, 8] which minimizes the expected cost under the model’s own sampling distribution We further decompose the reward into three terms that capture progressively finer aspects of prediction quality: R(ˆy, y∗) = wv · 1[valid] + ws · Sim(ˆy, y∗) · 1[valid] + we · 1[exact]
--- Page 9 ---
Algorithm 1 ComputeMRTLoss Input: Encoded features h ∈ RB′×d×H′×W′ Ground-truth SMILES y∗ b B' b=1, model θ, frozen MolDiscriminator g, sample size N, temperature τ, sharpness α, reward weights wv, ws, we Output: MRT loss LMRT // Phase 1: Sample candidates 1: yˆb i N i=1 ← SampleDecode(hb; τ; θ) for each b ▷ multinomial decoding // Phase 2: Compute rewards 2: validi, exacti ← Canonicalize(ˆyi, y∗ b) ▷ RDKit → validity & exact match 3: si ← Sim(ˆyi, y∗ b) · 1[validi] ▷ See Eq. 3 for Sim variants // Phase 3: Teacher-forced log-probabilities 4: log p(ˆyi hb) ← PT i t=1 log pθ(ˆyi,t yˆ1, yˆ2,. p. 8..
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Closed-loop training mechanism that mitigates exposure bias by
directly optimizing over molecule-level, non-differentiable objectives
instead of relying on token-level Maximum Likelihood Estimation (MLE). This allows the model todirectly close the exposure-bias loop: the distribution the model optimizes during training matches the distribution it operates under at inference.
-
Modular reward function with three variants—string-level edit distance, fingerprint-based Tanimoto similarity, and a visual similarity score from a MolDiscriminator—enables reward computation
without requiring for atom-coordinate annotations.
This allows the system to optimize forchemical validity, structural similarity, and exact match directly.
-
Joint training strategy that interleaves MLE and MRT steps during training. This design allows the model to
learn from both fully supervised synthetic data and weakly supervised real-world images throughout training,
leading to performance gains on real-world datasets that are not achievable by MLE alone. -
Model-agnostic application of Minimum Risk Training (MRT) across different architectures, such as SwinOCSR, yielding
2–20× accuracy gains on real-world benchmarks.
This confirms MRT is amodel-agnostic training principle for OCSR systems.
Sources
- Sequence Level Training with Recurrent Neural Networks
- Self-critical Sequence Training for Image Captioning
- OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs