ZEBRA-Prop: A Zero-Shot Embedding-Based Rapid and Accessible Regression Model for Materials Properties
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.
Kai: We're looking at the paper: "ZEBRA-Prop: A Zero-Shot Embedding-Based Rapid and Accessible Regression Model for Materials Properties".
Mira: Here's the gist of the paper. Large language models exhibit substantial potential across diverse scientific disciplines, including materials science The framework employs MatTPUSciBERT, an LLM specialized for materials science, to enhance predictive capability ZEBRA-Prop constitutes a scalable and computationally efficient framework for materials property prediction and supports accelerated materials discovery, particularly under limited computational resources.
Kai: Model Architecture The property prediction model is constructed through the following four stages: (1) the crystal structure is first expressed as a textual representation composed of several sentences, each describing the structure from a different perspective (2) each sentence is then individually converted into an embedding vector using materials-science-specific LLMs (3) these embeddings are then integrated through a learnable weighted integration and (4) the resulting aggregated embedding is fed into a multilayer perceptron (MLP) to predict the target property value.
Mira: ZEBRA-Prop keeps the parameters of the LLM frozen during training, whereas LLM-Prop fine-tunes the entire LLM.
Kai: Text Preprocessing Strategies The proposed model adopts a different text preprocessing strategy from that of LLMProp Specifically, chemical formulas are converted into simplified chemical formulas, and numerical values are transformed into integerized representations.
Title and authors: Kai: Let's go over the title of this paper, "ZEBRA-Prop: A Zero-Shot Embedding-Based Rapid and Accessible Regression Model for Materials Properties," and who came up with it. It’s Ryoma Yamamoto, Akira Takahashi, Kei Terayama, Yu Kumagai, Fumiyasu Oba.
Mira: And what that title tells us is that the core idea is using zero-shot embeddings to create a rapid and accessible regression model for materials properties. That accessibility part is key because they are targeting making this tool usable by a broader range of scientists in this field.
Lev: I mean, "zero-shot embedding-based" suggests the model doesn't need task-specific training data for every single property it tries to predict, which is a massive simplification over traditional machine learning approaches that usually demand labeled datasets for each target property.
Kai: And the authors are from institutions like the MDX Research Center and KISTEC, suggesting this work is coming from a strong base in materials science research environments.
Mira: That background definitely helps explain why they’re using specialized models like MatTPUSciBERT; you need experts to know what kind of domain-specific LLMs are actually relevant for the task.
Lev: I mean, the whole point seems to be that this framework doesn't require a huge fine-tuning budget to get started, which is a major hurdle when you're working on real hardware and want to test things out quickly.
Kai: So it’s about moving away from the idea that every new property requires a custom fine-tuned model, and instead offering this unified framework that can handle multiple properties efficiently.
Mira: That seems like the central thesis: using a zero-shot embedding approach combined with smart integration to achieve rapid access and accessibility for materials property prediction.
Lev: And from an engineering standpoint, if you need a quick baseline prediction without spending weeks fine-tuning, this is way more practical for getting initial data points or exploring new material candidates.
Kai: So the authors are proposing a solution that leverages the power of existing specialized LLMs to do the heavy lifting without demanding intensive adaptation work from the user.
The paper's summary: Mira: Now, let's look at what they actually summarize in this paper about ZEBRA-Prop. They focus on presenting ZEBRA-Prop as an extension of LLM-Prop, but with a key difference being that it eliminates the need for task-specific fine-tuning.
Kai: They detail the core four stages of the prediction process: first turning the crystal structure into text descriptions from different perspectives, then converting each sentence into embeddings using materials science LLMs, then integrating those embeddings through a learnable weighting mechanism, and finally feeding that aggregated embedding into a multilayer perceptron for the final property value.
Lev: That pipeline sounds complex because of all those steps—structure to text, sentence to embedding, weighted integration, then MLP regression—but the summary makes it sound like it’s a unified system for handling complexity.
Mira: It is a unified system because it addresses the context-length issues of existing LLM-Prop by using that learnable weighting mechanism to integrate diverse textual representations effectively.
Kai: So they are essentially saying that you don't have to rely on one monolithic text input; you can combine several inputs intelligently.
Lev: And they use MatTPUSciBERT specifically for materials science, which is supposed to enhance the quality of those initial sentence embeddings right from the start.
Mira: Exactly, and they show that this combination of specialized LLMs and weighted integration overcomes the limitations inherent in previous approaches, leading to better predictive accuracy for things like band gap and dielectric constants.
Kai: And they mention that the data used spans two datasets: a large TextEdge dataset with about one hundred forty thousand entries and a smaller in-house set from the Materials Project database of about two thousand entries <ref:2603.26060#pg2,from the Materials Project database>.
Lev: That’s a good scale to show that it works both on large, public literature data and on more curated, high-quality internal data derived from first-principles calculations.
Mira: The summary really highlights that the performance is close to LLM-Prop for the band gap and dielectric constants, but with this improved integration leading to better results.
The paper's improvements: Kai: So what are the specific improvements they claim in ZEBRA-Prop that set it apart from its predecessor? They focus on eliminating fine-tuning and reducing computational cost.
Mira: The main improvement is that ZEBRA-Prop eliminates the need for task-specific fine-tuning of the LLM, which directly reduces computational cost and enables much faster model training.
Lev: That’s a huge practical gain because it means we don't have to spend weeks or months adapting a massive language model just to get it ready for one new material property.
Kai: And they achieve this speedup by having about eighty percent of the total training time dedicated just to converting textual inputs into those embedding representations.
Mira: That’s because the embeddings can be reused across different prediction tasks, and only about twenty percent is spent on training the weighting mechanism and the MLP regression model.
Lev: This efficiency is what allows ZEBRA-Prop to run without needing high-performance GPUs or specialized hardware for training, which is a huge win for accessibility.
Kai: So they’re showing that you can get comparable performance without paying the high training price tag associated with fine-tuning LLMs.
Mira: They also point out that the text preprocessing strategy is quite effective at handling chemical formulas and integerizing numerical values to make the inputs cleaner and better aligned for language models.
Lev: And they also show that this approach helps improve prediction quality on structurally diverse phases, specifically mentioning that it substantially improves the accuracy of band gap prediction when unstable or metastable phases are included in the dataset.
Conclusion: Kai: So wrapping up, ZEBRA-Prop is an efficient regression model for materials properties that uses zero-shot embeddings to get good results with comparable accuracy to fine-tuned models.
Mira: It’s essentially a framework where you combine multiple textual descriptions using a learnable weighting mechanism to get better predictions than relying on just one description.
Lev: The big practical improvement is the training speed, achieving about a ninety-five percent reduction in training time compared to LLM-Prop without needing heavy hardware for the core training phase.
Kai: So, this paper presents ZEBRA-Prop as a powerful way to predict material properties rapidly and accessibly by leveraging zero-shot embeddings and smart aggregation of textual information.
Mira: It’s about showing that you can achieve strong predictive performance by combining diverse textual inputs in a way that optimizes the overall prediction outcome.
Lev: To weigh in, I think the key takeaway is that this method drastically lowers the barrier to entry for using these sophisticated LLMs for property prediction because of that training efficiency gain.
Kai: So we’re wrapping up on this discussion about ZEBRA-Prop, a framework that offers a much faster and more accessible path to material property prediction without needing massive fine-tuning effort.
Materials and Structures Laboratory, Institute of Integrated Research, Institute of Science Tokyo · Graduate School of Medical Life Science, Yokohama City University · MDX Research Center for Element Strategy, Institute of Integrated Research, Institute of Science Tokyo · Institute for Materials Research, Tohoku University · Kanagawa Institute of Industrial Science and Technology (KISTEC)
cond-mat.mtrl-sci
Submitted: 2026-03-27
Updated: 2026-03-27
DOI: 10.1002/aisy.70592
Code: https://github.com/obagroup/ZEBRA-Prop
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 83/100
The gist: Large language models exhibit substantial potential across diverse scientific disciplines, including materials science The framework employs MatTPUSciBERT…
Key concepts
- MatTPUSciBERT
- This is a materials science-specific large language model designed for material science tasks. It is used in ZEBRA-Prop because it understands the language and structure of materials better than general models, leading to higher quality initial embeddings for property prediction.
- Weighted Integration Mechanism
- Instead of simply averaging different textual descriptions, this mechanism learns how to weigh each sentence's embedding. This allows the model to intelligently balance complementary or redundant information from multiple perspectives, leading to more robust and accurate predictions across diverse material structures.
- Zero-Shot Embedding-Based Model
- This means the model can make predictions on new materials it hasn't seen during training, relying solely on the embeddings generated from its text representations. It doesn't need task-specific fine-tuning for every new material type, making it fast and accessible for discovery.
- Text Preprocessing Strategies
- This involves transforming raw structural data into a format LLMs understand well. This includes simplifying chemical formulas by expanding elements and converting numerical values into integers through scaling and rounding, which helps the model better interpret the input text.
Terminology
Summary
Large language models exhibit substantial potential across diverse scientific disciplines, including materials science The framework employs MatTPUSciBERT, an LLM specialized for materials science, to enhance predictive capability ZEBRA-Prop constitutes a scalable and computationally efficient framework for materials property prediction and supports accelerated materials discovery, particularly under limited computational resources. p. 1
Model Architecture
The property prediction model is constructed through the following four stages: (1) the crystal structure is first expressed as a textual representation composed of several sentences, each describing the structure from a different perspective (2) each sentence is then individually converted into an embedding vector using materials-science-specific LLMs (3) these embeddings are then integrated through a learnable weighted integration and (4) the resulting aggregated embedding is fed into a multilayer perceptron (MLP) to predict the target property value. ZEBRA-Prop keeps the parameters of the LLM frozen during training, whereas LLM-Prop fine-tunes the entire LLM. p. 5
Text Preprocessing Strategies
The proposed model adopts a different text preprocessing strategy from that of LLMProp Specifically, chemical formulas are converted into simplified chemical formulas, and numerical values are transformed into integerized representations. The conversion to simplified chemical formulas expands all elements and their corresponding quantities and replaces the original representation with a simplified chemical formula in which elements are listed sequentially Numerical values are transformed through scaling and subsequent rounding to integers, while preserving the original numerical information. p. 4
Computational Efficiency
To systematically compare models from the perspective of training cost, ZEBRA-Prop achieves an approximately 95% reduction in training time compared to LLM-Prop This significant reduction in the training time of ZEBRA-Prop is achieved by avoiding fine-tuning of the LLM. Approximately 80% of the total training time is devoted to converting textual inputs into embedding representations using an LLMbased text encoder, while the remaining 20% is spent on training the weighting mechanism and the MLP regression model. p. 5 Due to this reduction in training time, ZEBRA-Prop does not rely on large-scale computational resources or specialized hardware such as high-performance GPUs. p. 5
Performance Analysis
The predictive performance of ZEBRA-Prop is close to that of LLM-Prop for the band gap, formation energy, and dielectric constants (both electronic and ionic contributions) The integration of diverse textual descriptions through weighted aggregation allowed the model to overcome the context-length limitations inherent to LLMs and further improved predictive accuracy. The learned weights are not necessarily proportional to the predictive accuracy of the corresponding individual sentences The weighted integration mechanism does not simply prioritize individually accurate descriptions but instead balances complementary and redundant information to optimize overall prediction performance. p. 9
Domain Specialization
All domain-specific LLMs considered in this study, including SciBERT, MatBERT, MatSciBERT, and MatTPUSciBERT, consistently outperform the general-purpose BERT across all target properties on both the in-house and TextEdge datasets. This improvement is observed for every evaluated physical property, indicating that continual pretraining on domain-relevant corpora consistently improves the quality of text embeddings for property prediction.
Conclusion
In this study, we proposed an efficient LLM-based material property prediction model that does not require fine-tuning and demonstrated its effectiveness through empirical validation It achieved a promising balance between computational efficiency and predictive accuracy, showing approximately a 95% reduction in training time compared with LLM-Prop while maintaining comparable predictive performance. The predictive performance of ZEBRA-Prop is expected to further improve with ongoing advances in foundation LLMs and domain-specific LLMs, as well as through the enhancement of input text quality and diversity. p. 3
Data Availability
The implementation of ZEBRA-Prop is available at: https://github.com/obagroup/ZEBRA-Prop The in-house dataset used for evaluation is publicly available at: https://github. p. 3 p. 3com/oba-group/ZEBRA-dataset. p. 4 p. 4
Acknowledgments
This work was supported by KAKENHI (Grant Numbers: JP24K08562 and JP24H00376) from the Japan Society for the Promotion of Science, Data Creation and Utilization Type Material Research and Development Project (Grant Number: JPMXP1122683430) and Design & Engineering by Joint Inverse Innovation for Materials Architecture Project from the Ministry of Education, Culture, Sports, Science, and Technology, Japan. Computations were carried out using the TSUBAME4. p. 10 supercomputer at Institute of Science Tokyo. p. 0
References
-
Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient estimation of word representations in vector space
-
Wolf, T. et al. Transformers: State-of-the-art natural language processing
-
Baroni, S. & Resta, R. Ab initio calculation of the macroscopic dielectric constant in silicon
-
Zhu, J. et al. LLM-mambaformer: Integrating mamba and transformer for crystalline solids properties prediction
-
Grattafiori, A. et al. The Llama 3 herd of models
-
Ward, L., Agrawal, A., Choudhary, A., & Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials
-
MatTPUSciBERT
-
MatSciBERT
-
Tang, Y. et al. MatterChat: A Multi-Modal LLM for Material Science
-
Ward, L., et al. Matminer: An open source toolkit for materials data mining
-
Deml, A. M., O’Hayre, R., Wolverton, C. & Stevanović, V. Predicting density functional theory total energies and enthalpies of formation of metal-nonmetal compounds by linear regression
-
Kresse, G. & Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set
-
Anisimov, V. I., VI, Zaanen, J. & Andersen, O. K. Band theory and Mott insulators: Hubbard U instead of Stoner I
-
Hinuma, Y., Kumagai, Y., Tanaka, I., & Oba, F. Band alignment of oxides by learnable structural-descriptor-aided neural network and transfer learning
-
Kiyohara, S., Shibui, C., Bae, S. & Kumagai, Y. Machine-learning prediction of charged-defect formation energies from crystal structures
-
Seko, A. et al. Prediction of low-thermal-conductivity compounds with firstprinciples anharmonic lattice-dynamics calculations and Bayesian optimization
-
Ward, L., Agrawal, A., Choudhary, A., & Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials
-
Seko, A. et al. Prediction of low-thermal-conductivity compounds with firstprinciples anharmonic lattice-dynamics calculations and Bayesian optimization
-
Rajan, K. Materials informatics
-
Butler, K. T., Davies, D. W., Cartwright, H., Isayev, O. & Walsh, A general-purpose machine learning framework for predicting properties of inorganic materials
-
Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties
-
Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals
-
Choudhary, K. & DeCost, B. Atomistic Line Graph Neural Network for improved materials property predictions
-
Ward, L., et al. Matminer: An open source toolkit for materials data mining
-
Breiman, L. Random forests
-
Ganose, A. M. & Jain, A., Robocrystallographer: automated crystal structure text descriptions and analysis
-
Foppiano, L. & Suarez, P. MatTPUSciBERT
-
Tian, S., et al. Steel design based on a large language model
47488-47529 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), San Diego, California, USA, 2024)
- Wolf, T. et al. Transformers: State-of-the-art natural language processing
537–5315 (Association for Computational Linguistics, Stroudsburg, PA, USA, 2019)
Supporting Information
Table S1 shows the classes of matminer used to generate features for constructing the input crystal structure descriptions in this study Table S5 provides examples of crystal structure descriptions for BaTiO3 Figure S4 shows the relationship between single-sentence prediction performance (R2 on the test fold) and the learned weights obtained from ZEBRA-Prop trained using all descriptions for the band gap, formation energy, electronic contribution to the dielectric constant (log10 scale), and ionic contribution to the dielectric constant (log10 scale) on the in-house dataset Figure S5 shows the relationship between single-sentence prediction performance (R2 on the test fold) and the learned weights obtained from ZEBRA-Prop trained using all descriptions for the band gap and formation energy on the TextEdge dataset. Table 7 presents the dependence of ZEBRA-Prop prediction accuracy on text preprocessing strategies on the in-house dataset Figure S9 shows a hexbin map plot of the predicted values with respect to values of the TextEdge dataset for the test set, where color indicates sample count per hexagonal bin. p. 3 Figure S11 shows performance obtained from a systematic hyperparameter optimization via grid search for ZEBRA-Prop on (a) the band gap and (b) the formation energy on the in-house dataset. p. 6 Figure S12 shows performance obtained from a systematic hyperparameter optimization via grid search for ZEBRA-Prop on (a) the in-house and (b) TextEdge datasets. p. 1
Hyperparameter Optimization
Based on the grid search results presented in Figure S8-S12, a learning rate of 0.001 and a batch size of 64 were selected, as they provided high predictive performance while maintaining computational efficiency. p. 5 The reported values correspond to the mean test-set MAE across the test folds in 5-fold cross-validation for the band gap, formation energy, and dielectric constants. p. 7
Data Partitioning
The data were partitioned using 5-fold cross-validation with random shuffling The data were divided into five approximately equal-sized folds, with any remainder distributed evenly so that fold sizes differed by at most one sample. For each cross-validation iteration, one fold was designated as the test set, the subsequent fold (in cyclic order) was used as the validation set, and the remaining three folds were combined to form the training set. p. 7 The same crossvalidation splits were used for all models to guarantee fair comparison. p. 8
Implementation Details
We implemented ZEBRA-Prop in Python (version 3.12.12) using the following libraries: transformers51 (version 5. p. 30.0), PyTorch52 (version 2.10.0), matminer31 (version 0. p. 410.0), and pymatgen53 (version 2025.10.7). The transformers and PyTorch codes were used for model construction and training, while the matminer and pymatgen codes were employed for materials data processing and feature extraction. All evaluations were conducted using the same software environment to ensure reproducibility. p. 5
Conflict of Interest
The authors declare no conflict of interest.
Data Availability Statement
The implementation of ZEBRA-Prop is available at: https://github.com/obagroup/ZEBRA-Prop The in-house dataset used for evaluation is publicly available at: https://github.com/oba-group/ZEBRA-dataset.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI systems can predict materials properties rapidly without requiring task-specific fine-tuning of large language models by using ZEBRAProp, which
eliminates fine-tuning, thereby reducing computational cost and enabling rapid model training.
-
The system can handle context length limitations inherent in current LLMs more effectively by
integrating multiple embeddings through a learnable weighting mechanism,
allowing it to process diverse textual representations without truncation. -
The AI system can leverage domain-specific knowledge effectively by utilizing specialized LLMs such as MatTPUSciBERT, which
enhance predictive capability
in the absence of task-specific fine-tuning. -
The model can be made more flexible in incorporating diverse information by using a framework where
users can design their own textual descriptions based on domain knowledge, without being constrained by predefined descriptor sets.
-
The system's training efficiency is significantly improved, as ZEBRAProp achieves
approximately 95% reduction in training time compared to LLM-Prop
because it avoids fine-tuning the LLM and embedding vectors can be reused across different prediction tasks. -
The system can process numerical information better by employing text preprocessing strategies that include
integerization of numerical values
and converting chemical formulas intosimplified representations,
which are designed toenhance numerical reasoning by aligning numerical inputs with the operational characteristics of language models.
-
The system can aggregate complementary information across multiple textual perspectives more accurately through a weighted summation mechanism, as shown in Figure 3, which demonstrates that
incorporating all descriptions consistently achieves higher prediction accuracy than any single description setting.
-
The AI system can perform better on structurally diverse phases by using the
weighted integration mechanism,
which is shown tosubstantially improve the accuracy of band gap prediction
when the dataset includes unstable and metastable phases. -
The system's backbone can be improved by utilizing domain-specific language models, as all tested domain-specific LLMs
consistently outperform the general-purpose BERT across all target properties on both the in-house and TextEdge datasets.
-
Experimental data can be incorporated into property prediction by extending ZEBRAProp beyond crystal structures to
directly utilize laboratory notebooks, synthesis records, and other experimental documents as model inputs.
Sources
Related papers
- Structural contribution to light-induced gap suppression in Ta 2 NiSe 5
- Anatomy of fast current-induced skyrmion motion in synthetic antiferromagnets
- Magnetic control of an exciton-polariton condensate in a van der Waals magnet
- High-pressure elastic properties of GeO2 polymorphs up to 120 GPa
- Decoupling Strain-Rate Sensitivity and Deformation Length Scale Effects in Neutron-Irradiated Tungsten: A Coupled Nano-Indentation, HR-EBSD and Crystal Plasticity Study
- Magnetism modulation in Co3Sn2S2 by current-assisted domain wall motion