Quantifying out-of-distribution data severity with Wasserstein distances: benchmarking large language models for alloy property prediction
cs.LG, cond-mat.mtrl-sci
Submitted: 2026-11-02
Updated: 2026-11-11
Comments: 62 pages, 11 figures. Revised version
Code: https://github.com/Jack-Liu0227/BERT_ML
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: General-purpose Large Language Models (LLMs) increasingly rival domain-specific pipelines on scientific prediction, but their performance under distribution shift is untested.
Terminology
Abstract
General-purpose Large Language Models (LLMs) increasingly rival domain-specific pipelines on scientific prediction, but their performance under distribution shift is untested. Across six alloy and crystal-structure datasets, we benchmark LLMs against traditional, BERT-based, TabPFN, and text-regression baselines under random cross-validation and six out-of-distribution (OOD) protocols shifting input and target distributions, with severity quantified via a standardized Wasserstein distance. We isolate the LLM's advantage over k-nearest-neighbor (k-NN) retrieval: it beats k-NN on all eight tasks at a fixed 5% reference budget, on seven of eight when independently optimized, and in 72.5% of 120 OOD task--protocol configurations against an optimized baseline. No model family is uniformly best; the LLM is often strongest under severe shift, and error increases with target-space but not representation-space severity. Self-reported LLM confidence tracks both shift and error, more consistently than a density-only control from the same retrieved samples, and a matched-cohort check finds no substantial evidence of memorization.
Sources
- LoRA: Low-Rank Adaptation of Large Language Models
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Information fusion strategy integrating pre-trained language model and contrastive learning for materials knowledge mining
- A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools
- DeepSeek-V3 Technical Report
- 34 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discovery
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Are Reasoning Models More Prone to Hallucination?
- Optuna: A Next-generation Hyperparameter Optimization Framework
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis