Molecular notation can act as a shortcut in AI benchmark evaluation
summary
The gist
Chemical language models (CLMs) are increasingly used for molecular design and property prediction, but hidden differences in molecular notation can distort how AI models are evaluated…
In short
Chemical language models (CLMs) are often evaluated using inconsistent molecular notations like SMILES, which can mislead AI performance assessments. The study found that notation inconsistencies systematically perturb model representations and degrade structural translation accuracy. While property prediction showed robustness, confounding artifacts in certain datasets suggest notation can spuriously correlate with labels, highlighting a need for standardized preprocessing.
Key concepts
- Canonicalization
- This is the process of converting a molecule's string representation (like SMILES) into a single, unique standard form. Different software tools can produce different canonical forms, which creates inconsistencies that confuse AI models during training and testing.
- Latent Representations
- These are the internal numerical codes an AI model creates to understand molecular structure. The study showed that inconsistent input notations (Raw vs. Standardized SMILES) lead to different latent representations, proving that the notation directly shapes how the model 'sees' a molecule.
- Confounding Artifacts
- These are misleading correlations where a specific notational detail, like a Kekulé/Aromatic mismatch in certain datasets, appears to predict an outcome (like class labels) when it actually does not reflect true chemical properties. These artifacts inflate performance scores artificially.
- Molecular Translation Model (MTM)
- This task involves converting one molecular string representation into another. The study found that raw SMILES caused lower translation accuracy compared to standardized versions, indicating that inconsistent notation makes it harder for the model to accurately map structures between different formats.
Terminology used across episodes
This episode discusses
- Molecular notation can act as a shortcut in AI benchmark evaluation · Paper Radio
- Language models in molecular discovery
- A Survey of Machine Learning for Big Code and Naturalness
The paper
Molecular notation can act as a shortcut in AI benchmark evaluation · Read on arXiv
Laboratory of Molecular Pharmacokinetics, Graduate School of Pharmaceutical Sciences, The University of Tokyo · Division of Genome Safety Science, National Institute of Health Sciences · Division of Cellular and Molecular Toxicology, National Institute of Health Sciences · The Institute of Statistical Mathematics (ISM), Research Organization of Information and Systems
High predictive accuracy does not necessarily mean that an AI model relies on information relevant to its scientific task. Molecular prediction illustrates this concern: the same molecule can be represented by different SMILES strings, potentially altering model behavior. We combined a systematic PubMed literature survey with experiments on chemical language models. SMILES preparation could not be identified from articles or Supplementary Information in 71 of 174 studies (40.8%). Across six benchmarks, 58.7% of identity-preserving input pairs used different SMILES strings. Notation differences changed reconstruction and latent representations, while separately trained classifiers often achieved similar scores. In two datasets, notation was associated with labels and changed predictions of a fixed model, showing how benchmark scores can reward notational shortcuts. Balancing notation during classifier training reduced prediction differences but sometimes lowered performance on original inputs. These findings support evaluating predictive performance alongside consistency under identity-preserving changes in input representation.
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "Molecular notation can act as a shortcut in AI benchmark evaluation".
Marcus: Here's the gist of the paper. Chemical language models (CLMs) are increasingly used for molecular design and property prediction, but hidden differences in molecular notation can distort how AI models are evaluated, creating misleading confidence Literature Survey of Current Practices The systematic survey of 264 CLM publications found that about half did not specify their canonicalization procedure, which limits transparency and reproducibility Among the remaining studies, RDKit and OpenBabel were most commonly used, though version information was almost never reported This lack of reporting matters because preprocessing directly shapes model behavior, unlike other domains where inconsistent nomenclature mainly affects metadata Dataset-Level Analysis of Notational Inconsistencies The study investigated how notational inconsistencies appear in benchmark datasets like MoleculeNet and the Therapeutics Data Commons (TDC) Standardization collapsed many redundant encodings, showing that identical molecules were represented by multiple SMILES variants Stereochemical annotations were frequently missing, with about half of enantiomeric pairs lacking R/S labels in MoleculeNet Because canonicalization via RDKit cannot restore information that is absent in the original strings, these omissions propagate directly into model training and evaluation Impact on Structural Comprehension The Molecular Translation Model (MTM) showed consistently lower translation accuracy for Raw SMILES across MoleculeNet datasets compared to Standardized SMILES Latent space analyses confirmed this instability, showing clear separation between embeddings derived from Raw and Standardized SMILES Furthermore, inconsistent stereochemical annotations caused frequent mispredictions of “@”/“@@” tokens in the translation task Counterintuitive Robustness in Property Prediction In sharp contrast to translation tasks, property prediction tasks did not show the expected degradation in performance when models were evaluated with Raw versus Standardized SMILES This stability was explained by downstream feature selection, where restricting the representation to the most predictive features markedly compressed the distance between Raw and Standardized SMILES embeddings However, closer inspection revealed confounding artifacts in ClinTox and BBBP datasets where specific notational variants correlated spuriously with class labels, inflating AUROC scores Key Contributions The key contributions of this study include a comprehensive literature survey showing that nearly half of recent CLM publications omit canonicalization details The analysis demonstrated that inconsistent SMILES representations systematically perturb latent representations and degrade structural reconstruction in translation tasks The research identified confounding artifacts in ClinTox and BBBP, where notation correlated spuriously with labels and inflated AUROC scores Finally, the study highlights that canonicalization variability is an evaluation-level vulnerability rather than a theoretical flaw of SMILES itself Materials & Methods The model combined a Transformer encoder with a variational autoencoder (VAE) following the architecture validated in previous work Randomized SMILES generated by shifting the starting atom were used as inputs, and Canonical SMILES were used as outputs Training was performed for 200,000 steps with the AdamW optimizer Evaluation For structural comprehension, translation accuracy was computed using perfect reconstruction accuracy and partial (character-level) reconstruction accuracy metrics For property prediction, performance was reported as area under the ROC curve (AUROC) on held-out test sets Detection of confounding artifacts involved computing a running-sum statistic along the ranked ROC list to measure the cumulative excess of Kekulé/Aromatic compounds Figures and Tables Figure 1 summarizes the literature survey on canonicalization practices in CLM studies Figure 2 illustrates the extent of SMILES notational inconsistencies in public datasets Figure 3 shows the impact of notation inconsistencies on structural comprehension, including t-SNE visualizations of latent representations Figure 4 demonstrates counterintuitive robustness in property prediction tasks and confounding artifacts in ClinTox and BBBP Table S1 describes the types of SMILES used in this study, including Raw, Standardized, 3D-added, and 3D-removed SMILES Table S4 shows the proportion of Kekulé/Aromatic notational mismatch in each dataset Table S6 shows the association between stereochemical annotations and aromatic ring features in the training dataset Related Work Chemical Language Models are broadly categorized into Graph Neural Networks (GNNs) and chemical language models (CLMs) that treat molecular structures as strings like SMILES A major limitation of SMILES lies in the absence of a publicly disclosed standard for canonicalization, leading to divergent Canonical SMILES across different software packages RDKit is an open-source Python library widely used in cheminformatics, and its MolToSmiles function offers a canonical option Supplementary Notes The impact of stereochemical annotations on MTM representations was examined by adding or removing 3D stereochemical information The study found that the absence of stereochemical cues may impair the model’s ability to resolve complex topologies, suggesting such annotations support structural priors in MTMs Additional limitations include task scope and evaluation metrics, which suggest future work should broaden the scope of evaluation Future Directions Future work should broaden the scope of evaluation to build a more comprehensive picture by examining regression settings and large-scale pretraining tasks Extending the analysis to larger or proprietary datasets would be valuable to assess generalizability across different domains The authors conclude that ensuring CLMs capture genuine chemical principles requires explicit documentation of preprocessing pipelines, including canonicalization tools and versions Acknowledgements We thank all those who contributed to the construction of the following data sets employed in the present study such as PubChem and MoleculeNet The author(s) used ChatGPT in order to improve the English grammar and phrasing during this work Funding This work was supported by AMED under Grant Number JP22mk0101250h, 23ak0101199h0001, and 25ak0101199h0003, by MHLW under Grant Number 21KD24KD24H2, and JST BOOST under Grant Number JPMJBY24H2 Author Contributions Tadahaya Mizuno was responsible for Conceptualization, Resources, Supervision, Project administration, and Funding acquisition Yosuke Kikuchi and Yasuhiro Yoshikai were responsible for Methodology and Software Competing Interests The authors declare that they have no conflicts of interest Data and Materials Availability Code and models are available at https://github.
Title and authors: Ines: So we're looking at this paper by Yosuke Kikuchi and Tadahaya Mizuno titled "Molecular notation can act as a shortcut in AI benchmark evaluation". It seems to be digging into how the way we write molecular strings, like SMILES, can actually mess with how well these chemical language models perform on tests.
Marcus: Right. It’s about this idea that hidden differences in molecular notation can distort how we evaluate these models, creating misleading confidence in their results. It’s not just about whether a model gets the right answer for one specific molecule; it's about the whole system of checking if the model is actually learning chemistry or just learning to recognize patterns in weird strings.
Yuki: And from my perspective as a population geneticist, this is interesting because when we look at sequence and structure data, even small variations in how we encode that information can affect how we understand the underlying biological reality of that molecule.
Ines: Exactly. The authors found a pretty big problem in the literature itself. They surveyed two hundred sixty-four chemical language model publications and discovered about half of them didn't actually specify their canonicalization procedure—that’s how they standardized those molecular strings before feeding them to the AI, which limits transparency and reproducibility > <ref:2505.07139#pg3>
Marcus: And that lack of reporting is a major hurdle for us. Most studies rely on tools like RDKit or OpenBabel, but they rarely report the specific version they used. The authors point out that this preprocessing step shapes how the model behaves in a way that other data domains don't, which makes it way harder to trust the performance claims >
Yuki: It suggests that if we want to build reliable tools for understanding molecules, we can't just look at the final number; we have to look at exactly what went into getting those strings standardized in the first place.
Ines: Right. Then they took it further by looking at real datasets like MoleculeNet and the Therapeutics Data Commons, and they found that this notation inconsistency really collapsed a lot of redundant encodings >
Marcus: They showed that identical molecules could be represented by multiple SMILES variants, and stereochemical annotations were frequently missing in some places, with about half of the enantiomeric pairs lacking R or S labels in MoleculeNet >
Title and authors: Yuki: That makes sense. If you’re trying to study how a certain class of compounds evolves or behaves, having those structural details consistent is crucial for drawing any meaningful conclusions about the biology >
Ines: Because when you can't even get the basic structure right because of missing labels, that information simply doesn't make it into the model’s training and evaluation process correctly >
Marcus: And then they used a molecular translation framework to test this, and they showed that for models trying to translate Raw SMILES, accuracy was consistently lower compared to standardized SMILES latent space analyses >
Yuki: So, the raw string is inherently noisier for the model than the cleaned-up version. That points to how much structural context is lost when you start with less standardized data >
Ines: The paper also looked at property prediction tasks, and it’s counterintuitive because performance didn't degrade as much when they compared Raw versus Standardized SMILES in those specific tasks >
Marcus: They explained that this stability happens because of downstream feature selection; when you restrict the representation to only the most predictive features, that compression between the two notation types actually reduces the distance between their latent embeddings >
Yuki: But then they found some confusing artifacts in datasets like ClinTox and BBBP where specific notational variants seemed to correlate spuriously with class labels, which inflated their area under the ROC curve scores >
Ines: That’s a real concern. It means that sometimes the notation itself might be providing a shortcut to get a high score, independent of the true chemical features we are trying to measure >
Marcus: So what they suggest is that canonicalization variability is more of an evaluation-level vulnerability than some fundamental flaw in the SMILES system itself, which is a good distinction to make >
Yuki: It shifts the focus from just "is this model good?" to "how reliable are our benchmarks when we don't standardize our inputs?"
Ines: The authors suggest concrete improvements. They propose that we need to explicitly report all preprocessing tools, versions, and settings so downstream users can distinguish between true chemical insights and unreported preprocessing decisions >
Marcus: That’s a practical call to action. And for property prediction models specifically, they recommend implementing task-specific feature selection methods by analyzing the latent vectors from both Raw and Standardized SMILES inputs >
Title and authors: Yuki: And I think that ties back to what we talked about earlier—we need robust translation models trained on consistent notation, maybe even three dee-removed SMILES, to minimize those errors related to stereochemical context > <ref:2505.07139#pg1>
Ines: Exactly. And they also suggest incorporating anomaly detection mechanisms specifically targeting notational variants that correlate spuriously with class labels, like those Kekulé or aromatic mismatches in datasets such as ClinTox and BBBP >
Marcus: They want to see how models perform across different software toolkits, testing them against various RDKit versions to ensure they aren't overly reliant on the specific canonical SMILES generated by just one toolkit >
Yuki: It’s about building a system where the model is invariant to those specific rules, so it generalizes better when we move from one chemical ecosystem to another >
Ines: So, to wrap up this paper, the main implication is that if we don't explicitly document our notation pipeline, we are introducing a source of error in how we judge AI performance on molecular data >
Marcus: And they are suggesting concrete ways to fix that by standardizing reporting and improving how models handle input variation so their scores actually reflect real chemical understanding rather than just string manipulation >
Yuki: It sounds like the path forward is less about fixing the core language model architecture and more about making our testing and evaluation process much more rigorous regarding data preparation >
Ines: So, to summarize this paper on "Molecular notation can act as a shortcut in AI benchmark evaluation," they found that inconsistent SMILES representations systematically perturb latent representations and degrade structural reconstruction in translation tasks, while also noting spurious correlations with labels in certain property prediction datasets >
Marcus: The authors suggest we need explicit documentation of preprocessing pipelines, implementing task-specific feature selection for property prediction models, developing translation models trained on consistent notation variants like three dee-removed SMILES to handle stereochemistry better, and using anomaly detection to catch notational variants that spuriously correlate with class labels > <ref:2505.07139#pg1>
Yuki: The big picture here is that ensuring chemical language models capture genuine chemical principles requires explicit documentation of preprocessing pipelines, including canonicalization tools and versions >
Ines: It’s about moving past just getting a high AUROC score on a benchmark and instead building systems where we can actually trust what those scores are telling us about the actual chemistry >
The paper's summary: Ines: So we've looked at how inconsistent molecular notation can mess up AI evaluation benchmarks, and now we're going to dig into what the authors actually found in their main summary of this paper.
Marcus: Right. Basically, they surveyed a bunch of chemical language model publications and found that about half of them didn't even bother saying which tool they used to standardize those molecular strings before feeding them to the AI.
Ines: That lack of reporting is a huge deal because it means we can't really tell if a performance difference we see is actually due to genuine chemical understanding or just because one team used RDKit version three and another used version four.
Marcus: Exactly. They pointed out that when you look at real datasets, like MoleculeNet, these notation inconsistencies basically collapse a lot of redundant molecular representations. Identical molecules show up as several different strings with missing stuff like stereochemical labels for about half the pairs.
Ines: That missing information directly feeds into the model's training and evaluation process, which is bad for structural tasks where you need that precise spatial detail.
Marcus: The study showed that when testing molecular translation models, those using raw SMILES strings performed worse than those using standardized ones, proving this instability in the latent space representations.
Ines: But there was a twist; for property prediction tasks, the performance didn't drop as much when comparing raw versus standardized strings.
Marcus: That stability comes from feature selection. The model ends up only picking the most predictive features, and that compression between raw and standardized strings actually shrinks the gap between their latent embeddings.
Ines: However, they also found something concerning in datasets like ClinTox and BBBP where specific notation variants seemed to correlate with class labels spuriously, boosting their accuracy scores artificially.
Marcus: So the authors conclude that this isn't a flaw in SMILES itself, but rather a vulnerability at the evaluation level, meaning we need much better documentation for preprocessing pipelines.
Ines: They’re pushing for us to report every tool and version used so downstream users can actually distinguish between what is true chemical insight and what’s just unreported string manipulation.
Marcus: And they want models that are invariant to specific canonicalization rules, testing them across different software toolkits like various RDKit versions, so they don't get stuck on one toolkit's way of generating the string.
Ines: It sounds like the next step is less about building a better core model and more about making our evaluation process much more rigorous regarding how we prepare the data before it even hits the AI.
Marcus: So, if we want to trust these AI predictions for molecular design, we need to start treating data preparation as a critical part of the science itself.
The paper's improvements: Ines: So we’re looking at what the authors propose to fix these notation issues in their paper on molecular AI evaluation.
Marcus: They suggest three main things to move toward more reliable systems and stop relying on just one type of string representation.
Ines: First, they want us to explicitly report every single preprocessing step we take, naming the exact tools and versions used so we can tell if a performance gap is from real chemistry or just sloppy notation handling.
Marcus: That makes sense for cohort analysis because you have to know exactly what batch effects might be hidden in the string formatting itself before you even look at the statistics.
Ines: Second, for property prediction models, they advocate for task-specific feature selection methods using those latent vectors from both raw and standardized strings.
Marcus: That’s an engineering solution to mitigate that representational instability we talked about earlier by isolating only the features that actually stay predictive across different string formats.
Ines: And third, for molecular translation, they recommend training or evaluating models using a consistent notation variant, like three dee-removed SMILES, specifically to minimize those errors caused by missing stereochemical context.
Marcus: It’s about building translation models that are robust to the specific way a molecule is written on paper so they don't fail just because someone used an inconsistent labeling convention.
Ines: They also push for anomaly detection mechanisms during evaluation, specifically targeting those notation variants, like Kekulé or aromatic mismatches, that seem to correlate spuriously with class labels in datasets like ClinTox and BBBP.
Marcus: That’s the way to catch those shortcuts where the notation itself provides a high score independent of the actual chemical features we are trying to measure.
Ines: They conclude by saying that for real progress, we need models that are invariant to specific canonicalization rules, testing them across different software toolkits like various RDKit versions.
Marcus: It’s about ensuring the model doesn't become overly dependent on the specific way one toolkit generates the string, which helps with generalization across different cheminformatics ecosystems.
Ines: The main implication here is that we need to change our focus from just getting a high score on a benchmark to building systems where we can actually trust what those scores tell us about the actual chemistry.
Marcus: It’s a call for more rigor in data preparation and more transparency from the people developing these language models.
Ines: But there's still room for future work, like extending this analysis to larger or proprietary datasets to see if these issues hold up in bigger, messier real-world scenarios.
Conclusion: Tom: So we're wrapping up our look at "Molecular notation can act as a shortcut in AI benchmark evaluation" and what this means for how we test these molecular AI systems.
Ines: Basically, they showed that hidden differences in how molecular strings are written—like SMILES—can actually distort our evaluations, creating a false sense of confidence.
Marcus: They highlighted that inconsistent notation systematically messes up the latent representations the models learn, especially in tasks where structure matters, like translation.
Yuki: From a population genetics view, this means if our benchmarks aren't standardized, we might be drawing conclusions about molecular biology that just reflect how a specific string was written rather than the underlying chemical reality of the molecule.
Ines: Exactly. The authors pointed out that while property prediction tasks were more stable because they only look at the most predictive features, there were still those confusing artifacts in datasets like ClinTox and BBBP where notation seemed to trick the model into scoring high.
Marcus: So, what it changes for us is that we can't just trust a benchmark score without knowing exactly what went into preparing that input data.
Yuki: It puts pressure on us to demand better documentation from the groups building these models, forcing them to be more explicit about their preprocessing pipelines.
Ines: The authors suggest reporting every tool and version used so we can separate true chemical insight from unreported string manipulation.
Marcus: They are pushing for models that are invariant to specific canonicalization rules, testing them across different software toolkits to ensure they aren't just tuned for one specific way of writing the string.
Yuki: It’s about building systems that generalize better across different chemical ecosystems, which is important when we think about how diverse our biological samples are.
Ines: This study on "Molecular notation can act as a shortcut in AI benchmark evaluation" is a big reminder that our evaluation process needs to be just as disciplined as the model training itself.
Marcus: It’s about moving past just getting a high score and instead demanding accountability for how that score was achieved.
Yuki: Next time, we'll be looking at papers on tissue graphs and how spatial context affects how AI understands biological systems.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models