CompleteRXN: Toward Completing Open Chemical Reaction Databases

arXiv:2605.00222 · cs.LG, physics.chem-ph · Submitted 2026-04-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "CompleteRXN: Toward Completing Open Chemical Reaction Databases".

Jane: Here's the gist of the paper. Chemical reaction datasets such as USPTO suffer from substantial incompleteness, frequently missing byproducts, co-reactants, and stoichiometric coefficients <ref:2605.00222#pg2> This paper introduces CompleteRXN, a large-scale supervised benchmark for reaction completion under realistic missing-data conditions <ref:2605.00222#pg2> The constrained encoder–decoder model achieves strong performance and consistently outperforms its unconstrained version and an algorithmic baseline <ref:2605.00222#pg8> Dataset Construction The CompleteRXN dataset is constructed by mapping incomplete USPTO records to counterparts derived from an atom-balanced mechanistic dataset This process yields realistic input-output pairs for systematic evaluation The construction involves merging multi-step mechanistic reactions from the FlowER dataset into a single reaction and mapping specific USPTO SMILES to these counterparts This results in approximately 200k aligned incomplete (USPTO) to atom-balanced (FlowER) reaction equations Benchmark Design and Splits The benchmark evaluates models under three settings: random splits, mechanism-aware group splits, and extreme out-of-distribution (OOD) splits Random splits are noted to overestimate performance due to template-level redundancy in reaction datasets Group splits are constructed by representing reactions with DRFP fingerprints, grouping structurally similar reactions by fingerprint similarity, and assigning entire groups to train or test sets The extreme OOD split increases difficulty by selecting test groups that are both distant from the training data in fingerprint space and highly incomplete Model Performance Comparison The constrained reaction balancer (CRB) shows the best performance across all evaluation settings, consistently outperforming the unconstrained baseline (RB) CRB achieves high accuracy on random splits, reaching 99.20% equivalence accuracy and 87.

Title and authors: Tom: So we're looking at this paper called CompleteRXN: Toward Completing Open Chemical Reaction Databases. It’s from Delft University of Technology and it tackles a really big problem with chemical reaction datasets that are just incomplete, missing byproducts, co-reactants, and stoichiometric coefficients.

Jane: That sounds frustrating because those missing pieces make the data unreliable for things like modeling how processes work or assessing sustainability. So what does this paper actually propose to fix that?

Tom: Basically, they’ve introduced CompleteRXN as a large scale supervised benchmark for reaction completion under these realistic missing-data conditions. They built this dataset by mapping incomplete records from USPTO patents to atom balanced reactions from other mechanistic datasets.

Lu: What I find interesting is their construction process; they map specific patent SMILES to these atom balanced counterparts derived from the FlowER dataset, which results in about two hundred thousand aligned incomplete to atom balanced reaction equations.

Meng: two hundred thousand is a lot of data, but I wonder how much of that actually reflects what you’d see when you’re trying to deploy a model in a real industrial setting where data quality is usually messy.

Lalam: My literature search shows that large scale datasets often inherit biases from the data mining steps, and this work seems to directly address those limitations by creating a high quality template curated set.

Jane: And they set up three different ways to test models: random splits, mechanism aware group splits, and even extreme out of distribution splits. That suggests they’re really trying to see how robust these models are when the data gets harder or weirder.

Tom: They found that their constrained encoder decoder model does very well across all these settings, and it consistently outperforms its unconstrained version and even an algorithmic baseline. The constrained reaction balancer shows the best performance overall.

Lu: The paper details the improvement they made to the decoding strategy, which is a constrained decoding strategy for sequence to sequence models that enforces atom balance during generation by masking invalid tokens based on implied atom counts. This stops the model from switching to products too early and forces it to complete only when the reaction is atom and charge balanced.

Title and authors: Meng: I like that idea of forcing it to be balanced, but what are the practical costs of using this constrained decoding method? Does it slow down inference by a lot when you’re running these models at scale?

Tom: The paper shows that this constraint actually leads to better accuracy on random splits, achieving nine point two zero percent equivalence accuracy and eighty-seven point eight. It really proves that constraining the decoder helps guide the model toward chemically correct solutions.

Jane: That number is pretty high, but I wonder how reliable those results are when you move them out of a controlled environment and into something less structured? What about testing against the more challenging splits they designed, like the mechanism aware group splits?

Lalam: The paper mentions that their evaluation protocols include challenging out of distribution splits and mechanism aware group splits using differential reaction fingerprints, which tests generalization across different reaction mechanisms and increasing levels of incompleteness.

Lu: And on page three, they show a comparison in table one where their constrained model achieves exact match accuracy on atom balanced reactions and equivalence match accuracy on ground truth origin, while the method-independent constraint model does not achieve those exact matches.

Meng: So what's the practical difference between achieving an exact match and just an equivalence match when you’re trying to predict a complete reaction?

Jane: It means that even if the final product isn't written exactly as in the ground truth, it’s chemically equivalent, which is a useful metric for many applications. But the paper also points out a limitation where their evaluation protocol used mo...

Tom: They pointed out that their high precision proxy for ground truth validation comes from cross method agreement between their constrained reaction balancer and SynRBL, which shows a nine point nine five percent accuracy on the extreme out of distribution split. That’s a solid number for reliability.

Title and authors: Lu: The paper also mentions that they have an expert-curated dataset focused on diversity, complexity preference for highly unbalanced reactions, and coverage of underrepresented ones, going beyond just completion to include correction of errors. This is important because it moves the task from just completing a reaction to correcting errors too.

Meng: So the dataset isn't just a repository of incomplete data anymore; it’s something that actually challenges models to fix mistakes too. That sounds like a more useful training target for production systems than just filling in blanks.

Jane: Exactly, and this whole effort on CompleteRXN is about building this kind of high quality, template curated data instead of relying on synthetic corruption or small manual validations which you mentioned earlier. It’s about learning from real examples structured around mechanistic chemistry.

Tom: So to wrap up, the main implication is that supervised learning models can perform very reliably when given a large scale dataset specifically designed to handle realistic missing data patterns and chemical imbalances. The Constrained Reaction Balancer really seems to be the best approach they found for robust performance under increasing task difficulty.

Lu: The paper shows that while methods relying on synthetic generation don't reflect real missing data patterns, this work, by mapping USPTO records to atom balanced reactions, provides a major advance in reflecting realistic incomplete reaction scenarios.

Meng: From an engineering standpoint, the benchmark design itself is valuable because it forces us to test our models against distribution shift and increasing incompleteness systematically rather than just checking if they pass on a simple random split.

Lalam: The paper’s conclusion emphasizes that this framework moves beyond just completion to address correction too, which is a real-world need when dealing with messy chemical data. CompleteRXN provides a large scale supervised dataset of aligned incomplete and atom balanced reactions, enabling supervised learning for reaction completion under realistic missing data conditions derived from USPTO data mapped to mechanistic reactions.

Jane: It really shows how structured curation and careful benchmarking can significantly improve the performance of AI models when they are applied to complex scientific domains like chemistry. This paper is a big step in making these models more applicable outside of clean lab environments.

The paper's summary: Tom: So, to recap, this paper introduces CompleteRXN as a big benchmark for reaction completion that handles missing data really realistically by mapping patent records to balanced chemical reactions from other datasets.

Jane: That’s a lot of mapping and cleaning data just to get this one set of examples, Tom. What does that actually mean for the models we’re training?

Lu: It means the models get trained on high quality templates rather than just guessing based on corrupted or messy synthetic data, which is a big deal because real-world chemical data is always messy.

Meng: I’m wondering how many of those two hundred thousand examples are actually useful for someone trying to deploy something in a factory setting? Is it all just academic exercise?

Tom: The paper shows that the constrained reaction balancer performs best across all their testing settings, beating its unconstrained version and even a simple algorithmic baseline.

Jane: It’s not just about accuracy on random splits, though. They test these models with group splits based on chemical fingerprints and even extreme out of distribution scenarios where the data is really weird.

Lu: That OOD split is interesting because it tests how much the model actually learns about reaction mechanisms versus just memorizing patterns from the training set.

Meng: So, they found that forcing a constraint on the decoding process—making sure it stays balanced while generating tokens—actually leads to better performance across all these tough tests.

Tom: Exactly. The constrained decoder masks invalid tokens based on implied atom counts, which stops the model from switching to products too early and forces it to finish only when the reaction is perfectly balanced, meaning atoms and charges match up.

Jane: So this constraint makes a huge difference in getting a chemically correct output instead of just an incomplete sequence. That's what we need for anything chemical.

Lu: And they have this high precision proxy for validation coming from cross method agreement with another method called SynRBL, showing nine point nine five percent accuracy on that extreme out of distribution split.

Tom: nine point nine five percent on the hardest test case is a pretty strong number when you're dealing with these kinds of complex chemical systems.

Meng: That high reliability metric is something I could use if I had a way to apply this to my production pipeline without needing ground truth for every single run.

Jane: It suggests that instead of just hoping the AI fills in the blanks, we can build a system that actively guides it toward chemically sound solutions using these structural constraints.

Lu: The dataset construction itself is key here; they are taking messy USPTO records and aligning them to atom balanced reactions from FlowER data, which reflects a more realistic incomplete scenario than what’s usually available.

Tom: So, the big picture is that we can move away from just training models on whatever noisy data we find and start building benchmarks that reflect the actual challenges of chemical databases.

Jane: This moves the goal from just reaction completion to actually correcting errors too, which is a much bigger hurdle for real-world applications.

The paper's improvements: Tom: So, we’re looking at what the authors suggest they did to improve this benchmark setup for reaction completion.

Jane: What’s the main shift in strategy they propose beyond just building a bigger dataset?

Lu: They focused on making sure the dataset itself is highly diverse and complex. They prioritized reactions that are not just simple, but those that have many missing pieces or are structurally unbalanced.

Meng: So, it's not enough to just get two hundred thousand examples; they’re curating the data to be harder and more representative of real chemical problems.

Tom: Right. They specifically focused on diversity and complexity, preferring highly unbalanced reactions because that’s where models usually struggle most.

Jane: And they mentioned a new goal: moving beyond just completion to include error correction in the training task itself. That’s a big step for practical use.

Lu: They suggest the dataset should actually train models to fix mistakes too, not just fill in blanks, because that’s what happens in real chemistry when data is incomplete.

Tom: It means we're moving away from just being completion engines and towards systems that can actively correct errors while they complete a reaction.

Jane: That makes the training target much more useful for real-world scenarios, especially if you’re dealing with messy experimental results or old patent data where mistakes are common.

Meng: If models learn to correct errors instead of just hallucinating completions, the deployment cost might actually drop because we spend less time validating every single output from those systems.

Lu: They frame it as a joint task of completion and correction, which addresses a major limitation in current completion benchmarks where models only ever focus on finishing the sequence.

Tom: That’s good because I’ve seen how models sometimes just keep appending nonsense to finish the string, even if it makes no chemical sense at all.

Jane: So this paper is suggesting a way to make the learning process smarter by forcing the AI to think about fixing what it sees, not just finishing what it can see.

Lu: It’s about making sure that when these models are used in production, they aren't just guessing the next atom but actually verifying if that next step makes chemical sense.

Tom: So it’s about adding a layer of self-correction to the sequence generation process, which should stabilize the output more than just pure sequence prediction.

Jane: It sounds like they’re trying to bridge the gap between perfect theoretical solutions and messy experimental reality, which is exactly where most AI applied in science ends up.

Conclusion: Tom: So to wrap up, CompleteRXN is this big supervised benchmark for reaction completion that uses realistic missing data from patent records mapped onto balanced chemical reactions from other datasets.

Jane: It’s a lot of work to build this set up, Tom, but it gives us a really concrete way to see if AI can actually handle messy chemistry in the real world.

Lu: The main point is that supervised learning models can perform very reliably when they are given a large scale dataset specifically designed to handle realistic missing data patterns and chemical imbalances.

Meng: From an engineering side, the benchmark design itself is valuable because it forces us to test our models against distribution shift and increasing incompleteness systematically rather than just checking if they pass on a simple random split.

Lalam: I’m confident that this framework moves beyond just completion to actually address correction too, which is a real-world need when dealing with messy chemical data.

Tom: The Constrained Reaction Balancer really seems to be the best approach they found for robust performance under increasing task difficulty, consistently outperforming the unconstrained baseline.

Jane: It really shows how structured curation and careful benchmarking can significantly improve the performance of AI models when they are applied to complex scientific domains like chemistry.

Lu: The fact that their high precision proxy for validation comes from cross method agreement on those extreme out of distribution splits is a strong indicator of reliability for deployment.

Meng: That nine point nine five percent accuracy number on the hardest test case is something I could actually use if I had a way to apply this to my production pipeline without needing ground truth for every single run.

Lalam: The paper’s conclusion emphasizes that this framework provides a large scale supervised dataset of aligned incomplete and atom balanced reactions, enabling supervised learning for reaction completion under realistic missing data conditions derived from USPTO data mapped to mechanistic reactions.

Tom: So the implication is that we can stop relying on synthetic corruption or small manual validations and start learning from high quality, template curated targets instead.

Jane: It’s a big step in making these models more applicable outside of clean lab environments where the data is always going to be imperfect.

Lu: We need to keep pushing the idea that we can build benchmarks that reflect the actual challenges of chemical databases instead of just checking if they pass on a simple random split.

Meng: The dataset isn't just a repository anymore; it’s something that actually challenges models to fix mistakes too, which is a much more useful training target for production systems.

Delft University of Technology

cs.LG, physics.chem-ph

Submitted: 2026-04-30

Updated: 2026-11-06

Comments: Accepted at NeurIPS 2026 (Evaluations and Datasets Track). This version matches the camera-ready paper and includes expanded appendices and the paper checklist

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Chemical reaction datasets such as USPTO suffer from substantial incompleteness, frequently missing byproducts, co-reactants…

Key concepts

CompleteRXN Dataset
A large dataset created by mapping incomplete USPTO reaction records onto atom-balanced mechanistic counterparts. This allows researchers to train models on realistic, high-quality input-output pairs derived from established chemical reaction mechanisms.
Constrained Decoding Strategy
A method used during model generation where the decoder is restricted to only produce tokens that maintain atom balance. This forces the model to complete a reaction only when it is chemically correct, preventing premature product side completion.
Mechanism-Aware Group Splits
A benchmark splitting technique that groups reactions based on structural similarity using Differential Reaction Fingerprints (DRFP). This tests model generalization by ensuring the test sets contain structurally diverse reactions, simulating real-world data distribution shifts.
Constrained Reaction Balancer (CRB)
The specific model architecture used in the study. It uses a constrained decoding strategy to guide sequence-to-sequence generation, consistently achieving superior performance over unconstrained baselines on all evaluation settings.

Terminology

Summary

Chemical reaction datasets such as USPTO suffer from substantial incompleteness, frequently missing byproducts, co-reactants, and stoichiometric coefficients <ref:2605. p. 0

Where the paper says it

weber@tudelft. nl Abstract Chemical reaction datasets such as USPTO suffer from substantial incompleteness, frequently missing byproducts, co-reactants, and stoichiometric coefficients.

00222#pg2> This paper introduces CompleteRXN, a large-scale supervised benchmark for reaction completion under realistic missing-data conditions <ref:2605. p. 0
Where the paper says it

This limits their applicability and reliability in downstream applications. Here, we introduce CompleteRXN, a large-scale supervised benchmark for reaction completion under realistic missing-data conditions.

00222#pg2> The constrained encoder–decoder model achieves strong performance and consistently outperforms its unconstrained version and an algorithmic baseline <ref:2605. p. 9
Where the paper says it

9 Conclusion We introduce a dataset and benchmark for completing incomplete chemical reaction equations and use it to compare two learned and one algorithmic approach. The constrained encoder–decoder model achieves strong performance and consistently outperforms its unconstrained version and an algorithmic baseline, although all methods degrade as reactions become more incomplete and more dissimilar from the training data.

00222#pg8>

Dataset Construction

The CompleteRXN dataset is constructed by mapping incomplete USPTO records to counterparts derived from an atom-balanced mechanistic dataset This process yields realistic input-output pairs for systematic evaluation The construction involves merging multi-step mechanistic reactions from the FlowER dataset into a single reaction and mapping specific USPTO SMILES to these counterparts This results in approximately 200k aligned incomplete (USPTO) to atom-balanced (FlowER) reaction equations

Benchmark Design and Splits

The benchmark evaluates models under three settings: random splits, mechanism-aware group splits, and extreme out-of-distribution (OOD) splits Random splits are noted to overestimate performance due to template-level redundancy in reaction datasets Group splits are constructed by representing reactions with DRFP fingerprints, grouping structurally similar reactions by fingerprint similarity, and assigning entire groups to train or test sets The extreme OOD split increases difficulty by selecting test groups that are both distant from the training data in fingerprint space and highly incomplete

Model Performance Comparison

The constrained reaction balancer (CRB) shows the best performance across all evaluation settings, consistently outperforming the unconstrained baseline (RB) CRB achieves high accuracy on random splits, reaching 99.20% equivalence accuracy and 87. p. 0

Where the paper says it

On our CompleteRXN benchmark, the CRB achieves high performance across splits of increasing difficulty, reaching 99. 20% equivalence accuracy on the random split and 91.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Constrained decoding strategy for sequence-to-sequence models enforces atombalance during generation by masking invalid tokens based on implied atom counts, which prevents switching to product side too early and forces the model to complete only when the reaction is atom and charge balanced. This allows for the prediction of atoms on the product side that are not on the completed reactant side.

  2. The CompleteRXN benchmark provides a large-scale supervised dataset of aligned incomplete and atom-balanced reactions, enabling supervised learning for reaction completion under realistic missing-data conditions derived from USPTO data mapped to mechanistic reactions. This allows models to learn from high-quality, template-curated targets rather than relying on synthetic corruption or small manual validation.

  3. Evaluation protocols include challenging out-of-distribution splits and mechanism aware group splits using Differential Reaction Fingerprints (DRFP) to test generalization across reaction mechanisms and increasing levels of incompleteness, ensuring robust performance assessment beyond random splits. This systematically tests model robustness against distribution shift and increasing reaction incompleteness.

  4. The Constrained Reaction Balancer (CRB) consistently outperforms the unconstrained baseline (RB) across all evaluation settings, achieving higher accuracy on random, group-based, and extreme OOD splits by effectively guiding decoding toward chemically correct solutions. This demonstrates that constrained beam search improves robustness under increasing task difficulty.

  5. A high-precision proxy for ground truth validation is cross-method agreement between CRB and SynRBL, which shows high precision (e.g., 99.95% on the extreme OOD split), indicating a small but highly reliable subset of completions where both methods agree on chemically equivalent solutions. This can be used as a high-precision selection strategy when applying models to data without ground truth.

  6. The proposed expert-curated dataset focuses on diversity, complexity (preference for highly unbalanced reactions), and coverage of underrepresented and out-of-distribution reactions, going beyond pure completion to include correction of errors. This allows for the joint task of reaction completion and correction, addressing the limitation that models only address completion but not correction in real-world practice.

Abstract

Chemical reaction datasets such as USPTO suffer from substantial incompleteness, frequently missing byproducts, co-reactants, and stoichiometric coefficients. This limits their applicability and reliability in downstream applications. Here, we introduce CompleteRXN, a large-scale supervised benchmark for reaction completion under realistic missing-data conditions. We construct a dataset of aligned incomplete and atom-balanced reactions by mapping USPTO records to curated mechanistic reactions. We evaluate representative baselines, including a novel encoder-decoder reaction completion model with constrained decoding, the Constrained Reaction Balancer (CRB), and a recent algorithmic method, SynRBL. On our CompleteRXN benchmark, the CRB achieves high performance across splits of increasing difficulty, reaching 99.28% equivalence accuracy on the random split and 91.71% on the extreme out-of-distribution split. SynRBL produces many balanced and chemically plausible completions, but with lower accuracy on the benchmark test splits. Across all methods, performance degrades with increasing incompleteness. We observe a substantial drop when evaluating on reactions outside the benchmark (full uncurated USPTO), highlighting the gap between benchmark performance and practical robustness and motivating future work. Data, code and models can be found at: https://github.com/Intelligent-molecular-systems/CompleteRxn-Benchmarking.

Sources

Related papers