General-purpose foundation models surpass specialized ones on chemical property prediction tasks
physics.chem-ph, cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/TheorChemGroup/FewShot4Chem
License: http://creativecommons.org/licenses/by/4.0/
The gist: Foundation models (FMs) learn reusable representations from large collections of unlabeled data.
Terminology
Abstract
Foundation models (FMs) learn reusable representations from large collections of unlabeled data. In chemistry the corresponding data mainly consists of molecular structures, which are abundant, whereas experimental measurements are scarce. Chemical FMs such as ChemBERTa and MoLFormer were therefore trained on diverse molecular structures to enable fine-tuning on small experimental datasets (less than 100 datapoints in most realistic scenarios) and subsequent property prediction. At the same time, general-purpose FMs such as GPT-6 Astra and Claude Opus 5.5 were trained on virtually all publicly available data, including knowledge of the humankind on molecular properties and laws governing them. Here we compare accuracies of chemical and general-purpose FMs on tasks from various areas of chemistry. Strikingly, general-purpose GPT-6 Astra outperforms the currently eligible chemical FMs on five physicochemical and quantum-chemical regression tasks when only 60 experimental datapoints are available. Only when provided 10x datapoints do the best chemical FMs reach performance of GPT-6 Astra: on 1,000 and 10,000 points they show performance on par with Astra on 60 and 1,000 points, respectively. Thus, counterintuitively, frontier general-purpose FMs are better suited for guiding chemical research in data-scarce areas than specialized chemical FMs, suggesting that learning relevant primitives (e.g. chemical structures) does not substitute training on relevant humankind knowledge.
Sources
- ChemBERTa-2: Towards Chemical Foundation Models
- On the Opportunities and Risks of Foundation Models
- Training language models to follow instructions with human feedback
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Human Language Shapes the Emerging Physical Reasoning in Large Language Models
- Do Language Models Dream of Binding Molecules? Benchmarking LLMs under Spatial Constraints
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Strategies for Pre-training Graph Neural Networks
- Deep Learning Foundation Models from Classical Molecular Descriptors
- TabPFN-3.5: Technical Report
- Universal Language Model Fine-tuning for Text Classification
- LoRA: Low-Rank Adaptation of Large Language Models
- TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
- Why Large Language Models Fail at Tabular Prediction
- Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data
Related papers
- A System-Independent Metadynamics Strategy for Reactive Training Data: Application to Gas-Phase Organic Reactions
- A Machine Learning Model for the Chemistry of a Solvated Electron
- Compression of virtual spaces in transcorrelated methods via singular value decomposition: application to the G2 set
- Charge Transfer with a Spin. II: A Framework for Diabatization which Localizes Charge and Spin
- Diabatic Seam Space Sampling for Hydrogen Tunneling Systems with Nuclear-Electronic Orbital Theory
- Augmented Roothaan-Hall Hessian Applied to Spin-Restricted Open-Shell Density-Functional Theory