Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

The gist The authors demonstrate that standard random data splits in machine learning models for perovskite solar cells systematically leak information through massive internal duplication, which rewards mere recall rather than testing true physical generalisation

How it works

The paper introduces a provenance-grouped protocol to evaluate machine learning models on the Perovskite Database by establishing a hierarchy of grouped splits that progressively isolate duplicate devices, publications, and institutions This protocol is designed to separate recall from generalisation to new devices

The hierarchy of splits includes:

1. Random split where Devices are assigned independently, as in earlier work on this archive.

2. Level 1, feature-grouped where All duplicates of a device are assigned to the same set.

3. Level 2, publication-grouped where All devices from a single publication are assigned to the same set.

4. Level 3, institution-grouped where All devices from a single institution are assigned to the same set, ensuring no test device shares its institution with a training device.

5. Temporal split where Publications up to 2016 form the training set, 2017 the validation set, and 2018 onward the test set.

Progressive elimination of data leakage

The necessity of this grouped protocol is apparent when examining the archive’s massive internal duplication, noting that Most devices in the curated archive have a duplicate. The authors show how each level removes one more route by which a test device can be predicted from related training devices. For example, Level 3 ensures no test device shares a duplicate, a publication or an institution with the training set.

The thirteen-model benchmark

A common benchmark of thirteen models is established across every split to set the state-of-the-art under each kind of generalisation. These models include six tree-based models, six neural models and a gradient-boosted baseline with fixed settings. The results show a critical regime shift where while tree ensembles dominate random splits by recognising familiar recipes, their ranking collapses on genuinely unseen institutions and future discoveries, where composition-aware neural architectures excel.

Feature importance: from labels to physical properties

Feature importance analysis quantitatively identifies the absorber composition, the device architecture and the fabrication process as the inputs that carry the predictive information. The analysis reveals that the device architecture and fabrication methodology dictate final efficiency just as strongly as the absorber chemistry. Specifically, at level 3, removing these three groups of inputs raises the test MAE by 0.091%, 0.082%, and 0.074% respectively when removed. This indicates that the fabrication process, the device stack, and the absorber composition are all highly predictive.

Conclusion

The paper concludes that random splits actively mislead the field by mistaking memorization for physical understanding. By auditing evaluation on the Perovskite Database, they revealed a critical regime shift where neural architectures significantly outperform tree-based models in true extrapolation to unseen institutions and future discoveries. The authors recommend adopting four fundamental practices, including evaluating models on level 3 or the temporal split for claims of generalisation. This framework empowers the community to separate memorisation from true physical discovery

Data Curation and Split Protocols

The Perovskite Database was curated by selecting features that dictate physical performance, keeping only records that are completely populated across them. The 24 descriptive fields were expanded to capture recipes comprehensively. The protocol defines the hierarchy of splits based on three types of recurring relationships: an identical description (duplicates), a shared publication, and a shared institution.

Test MAE (%) on the replicated evaluation setup of the published state-of-the-art [16] over ten random splits, given as the mean and the standard deviation (SD) over splits.

The test MAE values for models vary significantly across splits; for instance, Lookup table 2.934 ± 0.015 0.048 on a random split is compared against the reported state-of-the-art value of 2.936% [16]. The recall controls also show a clear separation, as the error rises immediately from 2.643% on the random split to 3.793% at Level 1.

Model Mean MAE ± SE SD

The performance of models changes dramatically across splits; for instance, CatBoost (native) +0.039 ± 0.018 +0.080 ± 0.027-0.158 ± 0.18 - - across Random, Level 1, Level 2, and Level 3. The ranking correlation shows that a model’s rank on a random split carries no information about its rank on new institutions.

Test MAE (%) of each model on the two temporal splits

The performance difference between architectures is most pronounced in the temporal split, where every neural model improves on the gradient-boosted baseline, by 0.19% to 0.27%, and is more accurate than every tree-based model. This massive advantage stems directly from the bounds of the models’ predictions.

Feature reliance via permutation importance on every split.

Shuffling inputs among test devices reveals how the model relies on different features, showing that layer names functioning as categorical laboratory identifiers. At level 3, shuffling the device stack and fabrication process raises the error by 0.454% and 0.345%, proving that its dominance on looser splits is heavily driven by recall, with layer names functioning as categorical laboratory identifiers. The study confirms that the device stack loses its predictive value fastest: its error penalty collapses by 65.8% from the random split to level 3 against 49.8% for the fabrication process and 51.1% for the composition.

Uncertainty estimation.

Uncertainties are given as ± one standard error, with the standard error being the width of the 95% interval of a paired bootstrap (20,000 resamples, seed 0) divided by 3.92. This rigorous uncertainty estimation ensures that comparisons are valid when assessing true advances.

Data and code availability.

A reproducibility release is provided, which includes the row-level assignments of every split and of the ten further partitions, the publication and OpenAlex-resolved institution of every device. This ensures that this is sufficient to place a new model on the same protocol without retraining ours.

Supporting Information

The supporting information details the 19 core fields and provides a breakdown of which archive field(s) are used to construct each descriptive field, such as Absorber Composition as weights of pure compounds. The table shows that Deposition procedure (text) derived Perovskite deposition procedure and Perovskite deposition solvents, joined as text.

Field Missing

The supporting information lists the percentage of records missing values for various fields, such as Bandgap 24.9%. The table shows that Deposition procedure 0.3% is missing in the curated set. The analysis confirms that Full device description 100.0% (10,752) 100.0% (10,752).

Split

The final table summarizes what is kept together by each split; for instance, under the Temporal split, a The publication date is one of the inputs, so duplicates cannot cross a date boundary. Level 3 shows that no test device shares a duplicate, a publication or an institution with the training set. The temporal split assigns publications by date (shade).

Recall controls.

The recall controls predict by recalling training devices rather than by learning how efficiency depends on the inputs. The lookup table predicts the median PCE of the training devices with identical inputs, and the training median when there is none. The nearest-neighbour model predicts the mean PCE of the k training devices closest to the test device in the baseline’s inputs.

Models.

The comparison includes XGBoost, LightGBM, CatBoost [27] with out-of-fold target encoding and with its native categorical handling. The suite also includes complex models like SciBERT+attention neural and the gradient-boosted baseline.

**Benchmark settings.

1. The system will implement a provenance-grouped split protocol to evaluate machine learning models on perovskite solar cells, ensuring that the leakage at each level is measured independently. This allows researchers to separate recall from generalisation to new devices.

2. The improved system will transition from relying on random splits to using the hierarchy of grouped splits (identical description, shared publication, shared institution) to rigorously test physical discovery. This addresses the finding that a model’s rank on a random split carries no information about its rank on new institutions.

3. For predicting future discoveries (the temporal split), the system will prioritize neural architectures because they demonstrate superior extrapolation capabilities: every neural model improves on the gradient-boosted baseline, by 0.19% to 0.27%, and is more accurate than every tree-based model.

4. The system will incorporate feature importance analysis to guide model optimization by prioritizing physical inputs: true physical discovery requires treating the composition, the device architecture, and the fabrication methodology as equally critical pillars that must be co-engineered.

5. The system will utilize a comparison strategy where tree ensembles can serve as standard, rigorous baselines against which new architectures are evaluated, preventing reliance on models that merely memorize the past.