Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials
summary
The gist
Machine-learned interatomic potentials are deployed for high-throughput materials screening without formal reliability guarantees…
In short
Machine-learned interatomic potentials (MLIPs) are used for fast materials screening but lack formal reliability guarantees. Proof-Carrying Materials (PCM) addresses this by using adversarial testing, envelope refinement, and Lean 4 formal proofs to create falsifiable safety certificates. This allows users to verify the safety of MLIP predictions against high-fidelity data like Density Functional Theory (DFT).
Key concepts
- Adversarial Falsification for Compositional Safety Certificates
- This involves systematically testing a material's composition space using six different strategies (like random, grid, or LLMs). The goal is to actively search for chemical combinations where the MLIP prediction significantly disagrees with the more accurate DFT calculations. This identifies specific 'failure regions' in material chemistry.
- Envelope Refinement with Bootstrap Confidence Intervals
- Instead of just checking if a material passes or fails, this method tightens the safety claim. It uses counterexamples to build a 'safety envelope' around successful evaluations, calculated using 95% confidence intervals. This provides a statistically robust boundary for when an MLIP prediction is considered reliable.
- Formal Certification via Lean 4 Proofs
- The refined safety envelope is translated into formal mathematical proofs written in the Lean 4 proof assistant. These proofs incorporate explicit physical axioms, providing machine-checkable evidence that the material's safety claim holds true, which traditional methods cannot offer.
- Prospective Validation for Predictive Intervention
- A gradient-boosted classifier is trained on features discovered during PCM testing to predict failures on entirely new materials. This system achieves high accuracy (AUC ROC of 0.938) and allows users to proactively identify potential failures before they are synthesized or tested experimentally.
Terminology used across episodes
This episode discusses
- Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials · Paper Radio
- Lack of differentiability of semigroups associated to delayed abstract thermoelastic systems
- Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
- Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)
- DARWIN 1.5: Large Language Models as Materials Science Adapted Learners
- Possibility Frames and Forcing for Modal Logic
- Concrete Problems in AI Safety
- Are LLMs Ready for Real-World Materials Discovery?
- Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models
- UMA: A Family of Universal Models for Atoms
- DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
- Lean Copilot: Large Language Models as Copilots for Theorem Proving in Lean
The paper
Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials · Read on arXiv
Abhinaba Basu, Pavan Chakraborty
Indian Institute of Information Technology Allahabad · National Institute of Electronics and Information Technology
Machine-learned interatomic potentials (MLIPs) are deployed for high-throughput materials screening without formal reliability guarantees. We show that a single MLIP used as a stability filter misses 93% of density functional theory (DFT)-stable materials (recall 0.07) on a 25,000-material benchmark. Proof-Carrying Materials (PCM) closes this gap through three stages: adversarial falsification across compositional space, bootstrap envelope refinement with 95% confidence intervals, and Lean 4 formal certification. Auditing CHGNet, TensorNet and MACE reveals architecture-specific blind spots with near-zero pairwise error correlations (r <= 0.13; n = 5,000), confirmed by independent Quantum ESPRESSO validation (20/20 converged; median DFT/CHGNet force ratio 12x). A risk model trained on PCM-discovered features predicts failures on unseen materials (AUC-ROC = 0.938 +/- 0.004) and transfers across architectures (cross-MLIP AUC-ROC 0.70; feature importance r = 0.877). In a thermoelectric screening case study, PCM-audited protocols discover 62 additional stable materials missed by single-MLIP screening - a 25% improvement in discovery yield.
Transcript
Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.
Kai: We're looking at the paper: "Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials".
Mira: Here's the gist of the paper. Machine-learned interatomic potentials (MLIPs) are deployed for high-throughput materials screening without formal reliability guarantees, and this paper introduces Proof-Carrying Materials (PCM) to close this gap through adversarial falsification, envelope refinement, and Lean 4 formal certification
Title and authors: Kai: So we're looking at this paper now titled Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials by Abhinaba Basu and Pavan Chakraborty, and it’s about how these machine-learned interatomic potentials, MLIPs, are being used for materials screening right now without any formal proof of reliability.
Mira: Yeah, it tackles that exact issue. It shows that just using a single MLIP as a filter misses ninety-three percent of density functional theory or DFT stable materials on a twenty-five thousand material benchmark, which is a pretty big gap to fill.
Lev: And this paper introduces Proof-Carrying Materials PCM as the way to close that gap by using adversarial falsification and formal certification rather than just relying on the MLIP output alone.
Kai: Right, so instead of assuming the MLIP is good, they are building a system where you have to prove its safety claim before you can trust it for real materials design.
Mira: Exactly. The core idea is framing the MLIP reliability as a falsifiable safety claim, which means we need a world model like the MLIP, some bounds on acceptable error, and then a verifier that can check those bounds with machine-checked proofs.
Lev: That sounds like moving from just prediction to actual verification. The paper outlines this through three stages: adversarial falsification across compositional space, bootstrap envelope refinement with ninety-five percent confidence intervals, and finally Lean four formal certification.
Kai: Adversarial falsification is where they use six different strategies—random, heuristic, grid, lattice sampling, Sobol sequences, and even large language models—to actively probe for failure regions in the material chemistry.
Mira: They found that these adversarial tests can reveal architecture-specific blind spots in the MLIPs. Specifically, they saw near-zero pairwise error correlations of less than zero point one three when testing CHGNet, TensorNet, and MACE architectures across five thousand materials.
Lev: That finding is significant because it suggests that the problem isn't just any material; it points to specific chemical combinations or structural motifs that the MLIP struggles with on a hardware level.
Kai: And then they move to envelope refinement, where they use those counterexamples—the failures found in stage one—to tighten the safety claim using bootstrap ninety-five percent confidence intervals from those failures.
Mira: That refines the safety claim by shrinking the bounds around passing evaluations into a tighter envelope, which gives us a better idea of what is actually safe.
Lev: But they also say that these confidence intervals widen uniformly across compositional space, which means while it tightens the bounds on *where* it works, it doesn't automatically tell you *which* specific chemistries are driving those failures.
Kai: That’s a key point because if the envelope widens uniformly, you still don't know where to spend your time looking for high-risk materials.
Mira: And that leads them into Lean four formal certification, where they take that refined envelope and compile it into Lean four proofs with explicit physical axioms.
Lev: That gives you machine-checkable evidence of safety, which is something structured safety cases currently lack when you're relying on these MLIPs for high-throughput screening.
Kai: So the output isn't just a prediction anymore; it’s a Safety Certificate Proof plus the envelope and those discovery counterexamples.
Mira: And they also built a risk model trained on those PCM-discovered features to predict failures on materials they haven't seen yet, achieving an area under the ROC curve of zero point nine three eight and a precision at twenty percent of one point zero zero zero.
Lev: That prospective validation part is interesting because it transforms auditing into something you can use to predict failures on new materials before you even run a full DFT calculation on them.
Kai: And they also identified that the architecture and the training pipeline determine which chemistries appear problematic, which means users can select the optimal auditing strategy for their specific MLIP deployment.
Mira: They also developed a mechanism for targeted DFT budget allocation using this risk model, prioritizing materials that need DFT validation based on a risk quintile to achieve a thirty-four percent improvement in DFT efficiency.
Lev: Their cross-MLIP transfer risk modeling is another part, where they train one MLIP risk model and use it to predict failures on different architectures because of feature importance correlations reaching zero point eight seven seven.
Kai: So the main idea here is that by moving from a black-box prediction to a formal, falsifiable safety certificate process, we get much better guidance for deploying these MLIPs in materials screening.
Mira: It shifts the focus from just achieving accuracy on a benchmark to actually understanding and quantifying the risks associated with using these potentials in practice.
Lev: The paper's limitation is that while they provide marginal coverage—ninety-five percent of all materials fall within the prediction interval—they admit that this method cannot answer the deployment question which specific compositions are unreliable and why.
Kai: So it doesn't pinpoint the drivers of failure, like heavy elements or large unit cells, though it helps you know where to look based on the architecture.
Mira: It’s a lot of machinery to build—the adversarial testing, the envelope refinement with those bootstrap intervals, and compiling everything into Lean four proofs—but it shows a path toward making MLIPs more trustworthy for high-throughput discovery.
Lev: For real hardware running this, the overhead lands in how much computation is needed for that adversarial probing and then generating those confidence intervals across thousands of materials.
Kai: Still, the fact that they can use a risk model trained on those PCM features to predict failures on unseen materials with an AUC ROC of zero point nine three eight is a concrete result we can actually use.
Mira: It’s about turning the uncertainty into a quantifiable risk metric that helps guide subsequent experimental or computational steps.
Lev: If you're looking at this for real hardware, you have to factor in the cost of generating those counterexamples and then running those formal proofs, but it’s a step toward making the screening process more efficient overall.
Kai: So to wrap up on Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials, they've shown how to close the gap between MLIP prediction and actual reliability by providing machine-checkable safety certificates.
Mira: It’s a framework that moves us toward a world model, a safety specification, and a verifier for these potentials.
Lev: It shows how to use those results to guide targeted DFT allocation, improving efficiency by thirty-four percent in some cases.
Kai: The work on the risk model trained on PCM features is particularly interesting because it allows for prospective validation against unseen materials.
Mira: It's a practical step toward making these high-throughput pipelines more responsible when they are screening materials for applications like lead-free perovskite solar cells or other candidates.
Lev: For those of us building the actual hardware, the implication is that we need to account for the computational cost of this verification process if we want to use it on every single candidate.
Kai: It’s about using this framework to ensure that when a material screening pipeline says something is safe, there's a formal mathematical justification attached.
The paper's summary: Kai: So, this paper is about taking these machine-learned interatomic potentials, those MLIPs we use for materials screening without any formal reliability proof, and building something called Proof-Carrying Materials to close that gap.
Mira: Right. It moves beyond just having a prediction from the MLIP and tries to build a system where you have to prove the safety claim before you can trust it for real material design, using adversarial testing and then formal mathematical proofs.
Lev: The paper sets up this three-stage process: first, adversarial falsification across all possible material chemistries; second, envelope refinement using ninety-five percent confidence intervals based on those failures; and third, Lean four formal certification of that refined envelope.
Kai: Adversarial falsification means they’re using six different ways to poke the MLIP—random testing, heuristic searching, grid sampling, lattice sampling, Sobol sequences—and even large language models—to actively hunt for failure regions where the MLIP gives a wrong answer compared to our gold standard DFT.
Mira: They found that these adversarial tests can actually reveal specific architectural blind spots in the MLIPs; they saw near-zero pairwise error correlations of less than zero point one three when testing things like CHGNet and TensorNet across five thousand materials.
Lev: That’s a big deal because it suggests the problem isn't just any material; it points to specific chemical combinations or structural motifs that the MLIP struggles with on a hardware level.
Kai: Then they use those counterexamples found in stage one to tighten the safety claim by refining bounds into a "safety envelope" using bootstrap ninety-five percent confidence intervals derived from those failures.
Mira: That refines the safety claim by shrinking the bounds around passing evaluations into a tighter envelope, which gives you a better idea of what is actually safe, but they also note that these confidence intervals widen uniformly across compositional space.
Lev: That means while it tightens the bounds on *where* it works, it doesn't automatically tell you *which* specific chemistries are driving those failures.
Kai: But they use those PCM-discovered features to train a gradient-boosted classifier to predict failures on materials they haven't seen yet, hitting an AUC ROC of zero point nine three eight and a precision at twenty percent of one point zero zero zero.
Mira: That prospective validation part is interesting because it transforms auditing into something you can use to predict failures on new materials before you even run a full DFT calculation on them.
Lev: They also identified that the architecture and the training pipeline determine which chemistries appear problematic, which means users can select the optimal auditing strategy for their specific MLIP deployment.
Kai: And they built a risk model trained on those PCM features to prioritize which materials most need DFT validation, achieving a thirty-four percent improvement in DFT efficiency by guiding the budget to the highest-risk quintile.
Mira: That’s really smart because it turns uncertainty into a quantifiable risk metric that helps guide subsequent experimental or computational steps, making the screening process more efficient overall.
Lev: The paper's limitation is that while they provide marginal coverage—ninety-five percent of all materials fall within the prediction interval—they admit this method cannot answer which specific compositions are unreliable and why; it doesn't pinpoint the drivers of failure.
Kai: So it doesn't tell you exactly what chemical causes the failure, though it helps you know where to look based on the architecture; that’s a trade-off.
Mira: It’s a lot of machinery to build—the adversarial testing, the envelope refinement with those bootstrap intervals, and compiling everything into Lean four proofs—but it shows a path toward making MLIPs more trustworthy for high-throughput discovery.
Lev: For real hardware running this, you have to factor in the computation needed for that adversarial probing and then running those formal proofs across thousands of materials.
Kai: Still, the fact that they can use a risk model trained on PCM features to predict failures on unseen materials with an AUC ROC of zero point nine three eight is a concrete result we can actually use.
Mira: It’s about turning the uncertainty into a quantifiable risk metric that helps guide subsequent experimental or computational steps for things like lead-free perovskite solar cells.
Lev: If you're building the actual hardware, the implication is that you need to account for the cost of this verification process if you want to use it on every single candidate.
Kai: It’s about using this framework to ensure that when a material screening pipeline says something is safe, there’s a formal mathematical justification attached.
The paper's improvements: Tom: So, we're looking at how they actually improve the original Proof-Carrying Materials system by adding more layers of verification and risk management.
Kai: They take those initial findings—the adversarial testing and the envelope refinement—and they formalize them into a Lean four proof with explicit physical axioms, which is a big step for making it machine-checkable.
Mira: That’s what’s important; it moves us from just having a statistical claim to having verifiable evidence that links the MLIP prediction back to real physics, which is what we need for high-stakes materials screening.
Lev: They also introduced prospective validation, where they train a gradient-boosted classifier on those PCM features to predict failures on materials they haven't seen yet, achieving an AUC ROC of zero point nine three eight.
Kai: That predictive step is powerful because it transforms the auditing process into something you can use for proactive intervention before you even start running expensive DFT calculations.
Mira: And they also cracked a few architectural blind spots, showing that the architecture and the training pipeline determine which chemistries are problematic, allowing users to choose an optimal auditing strategy for their specific MLIP.
Lev: They also have this targeted DFT budget allocation tool, using a risk model to prioritize which materials need DFT validation—that's a thirty-four percent improvement in DFT efficiency by focusing on the highest-risk quintile.
Kai: Plus, they developed cross-MLIP transfer risk modeling; you can train a risk model on one MLIP and use it to predict failures on different architectures because of feature importance correlations reaching zero point eight seven seven.
Mira: So they’ve built a whole system that not only proves the safety of an MLIP but also tells you how to efficiently spend your DFT budget and how to spot weaknesses across different potential models.
Lev: The limitation they flag is that this method still doesn't pinpoint the exact chemical drivers of failure; it tells you where to look based on architecture, which is a necessary trade-off for this level of formal certification.
Kai: It’s about turning prediction into a guided action plan, showing users how to manage their computational resources more intelligently when screening materials.
Mira: This framework moves us toward a world model, a safety specification, and a verifier for these potentials, giving us something concrete to work with instead of just relying on the raw output from the MLIP.
Lev: If you're looking at this for real hardware, you have to factor in the computational cost of generating those counterexamples and then running those formal proofs across thousands of materials.
Conclusion: Kai: So to wrap up, Proof-Carrying Materials provides a framework for taking those unverified MLIP predictions and turning them into formal, falsifiable safety certificates using adversarial testing and Lean four proofs.
Mira: It’s about building that world model—the MLIP—and then rigorously checking the safety claim against known physical axioms through the envelope refinement process.
Lev: That prospective validation step is key because it lets you use those risk models to predict failures on new materials, which is where the real value for error-correction hardware comes in.
Kai: It’s a shift from just hoping an MLIP works well on a benchmark to having mathematically grounded evidence about its failure regions.
Mira: The implication is that we can start treating these machine-learned potentials not as black boxes, but as tools that require a specific level of formal verification before they can be trusted for high-throughput screening.
Lev: For us building error correction systems, it means having a way to quantify the risk associated with using different MLIP architectures across different material classes.
Kai: We’re moving toward making these screening pipelines more responsible by forcing them to prove what they claim about material safety before we commit computational resources to actual synthesis or experimental testing.
Mira: It’s a lot of machinery to build, but it establishes a standard for how we should approach the reliability of MLIPs in materials science.
Lev: I just want to say that while the formal certification is powerful, the overhead of running those adversarial probes and generating all those confidence intervals across thousands of materials is something you really have to account for when you're building the actual screening hardware.
More episodes
- 2610.32678-Probing the limits of materials design via a comprehensive crystal database of 14.6 million structures with 140 properties
- 2610.33595-Native-oxide-driven layer asymmetry, two-step switching and exchange bias in Co/Ir/Co trilayers
- 2503.19572-Spin models from nonlinear cellular automata
- 2606.23350-Irreversibility Enhances Quantum-Enhanced Markov-Chain Monte Carlo
- 2608.02744-Two-Parameter Ansatz for the Violation of Eigenstate Thermalization
- 2210.00806-Low- and high-energy localization landscapes for tight-binding Hamiltonians in two-dimensional lattices
- 2409.09625-Selective switch for band-edge alignments in pentagonal CdSeTe monolayer: Atom-valley locking
- 2606.20168-Norms, overlaps and Yangian descendants for the Haldane-Shastry spin chain
- 2512.19503-Strain-induced splitting of the CCDW-NCCDW phase transition in 1T-TaS 2
- 2307.12638-Strong localization of microwaves beyond two dimensions in aperiodic Vogel spirals