Quantifying out-of-distribution data severity with Wasserstein distances: benchmarking large language models for alloy property prediction

arXiv:2611.03767 · cs.LG, cond-mat.mtrl-sci · Submitted 2026-11-02 · Read on arXiv

cs.LG, cond-mat.mtrl-sci

Submitted: 2026-11-02

Updated: 2026-11-11

Comments: 62 pages, 11 figures. Revised version

Code: https://github.com/Jack-Liu0227/BERT_ML

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: General-purpose Large Language Models (LLMs) increasingly rival domain-specific pipelines on scientific prediction, but their performance under distribution shift is untested.

Terminology

Abstract

General-purpose Large Language Models (LLMs) increasingly rival domain-specific pipelines on scientific prediction, but their performance under distribution shift is untested. Across six alloy and crystal-structure datasets, we benchmark LLMs against traditional, BERT-based, TabPFN, and text-regression baselines under random cross-validation and six out-of-distribution (OOD) protocols shifting input and target distributions, with severity quantified via a standardized Wasserstein distance. We isolate the LLM's advantage over k-nearest-neighbor (k-NN) retrieval: it beats k-NN on all eight tasks at a fixed 5% reference budget, on seven of eight when independently optimized, and in 72.5% of 120 OOD task--protocol configurations against an optimized baseline. No model family is uniformly best; the LLM is often strongest under severe shift, and error increases with target-space but not representation-space severity. Self-reported LLM confidence tracks both shift and error, more consistently than a density-only control from the same retrieved samples, and a matched-cohort check finds no substantial evidence of memorization.

Sources

Related papers