Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty
cs.LG
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/MatthiasHBusch/LLMsUncertaintyMolecules
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored.
Terminology
Abstract
Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored. In this work we evaluate the capability of 7 frontier LLMs to predict molecular properties together with a self-reported uncertainty from in-context examples, and compare them with classical baselines trained on the same molecules. We use 4 datasets and screen each for contamination of the training data of the LLMs: three are clean, and the fourth, ESOL, is memorised by most of the models and serves as a comparison. On the clean datasets every LLM improves with the context, at 30 examples the most accurate one beats every baseline. The requested reasoning effort changes accuracy little. The self-reported uncertainty ranks the prediction errors weakly, with Spearman rank correlations of up to 0.46, about the level reached by the uncertainties of the baselines. Combining uncertainty and prediction, on lipophilicity the most certain half of the best LLM is the most accurate at small context sizes; on BACE and antiviral, and on lipophilicity at large context sizes, the best LLMs are on par with the best baseline. On the contaminated ESOL dataset the LLMs show no improvement with the context size, as expected for a memorised dataset, and their stated uncertainty is smaller on the molecules whose value they recall.
Sources
- Many-Shot In-Context Learning
- Molecular D'ej`a Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
- Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Selective Classification for Deep Neural Networks
- GAUCHE: A Library for Gaussian Processes in Chemistry
- TabPFN-3: Technical Report
- What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
- Regression with Large Language Models for Materials and Molecular Property Prediction
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- DataComp-LM: In search of the next generation of training sets for language models
- Teaching Models to Express Their Uncertainty in Words
- Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
- MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
- Bayesian Optimization of Catalysis With In-Context Learning
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
- Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
- 2 OLMo 2 Furious
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning