Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning".
Jane: Here's the gist of the paper. SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models, achieving high accuracy and data efficiency by building rotational symmetry directly into the architecture.
Tom: The proposed structural pruning method substantially reduces inference cost while retaining higher accuracy than an independently trained small model.
Jane: Pruning redundant reasoning tokens The paper introduces a structural pruning method for SO(3) equivariant atomistic foundation models to bridge the gap between model accuracy and inference efficiency.
Tom: The pruning is applied along the channel and order dimensions, with each irreducible representation kept or removed as a complete block, thereby retaining SO(3) equivariance.
Title and authors: Tom: So, to recap, this paper is about taking these big SO(three) equivariant atomistic foundation models and making them much smaller by cutting down their structure, without losing that essential rotational symmetry that keeps the physics correct.
Jane: Right, so instead of just randomly chopping up the model like you might do with a regular neural network, they are pruning along specific dimensions—the channel and the order dimensions—and they keep entire irreducible representations as solid blocks.
Lu: Exactly. They're using this structural approach because it’s designed to preserve that SO(three) equivariance exactly, no matter what rotation you apply to the atomic configuration. It keeps things physically consistent across different setups.
Meng: So, I mean, if you can cut down the model size without breaking the symmetry, that’s a huge win for deployment because we don't have to run these massive models on expensive hardware just to get an inference result.
Tom: That’s the core idea. And they show this isn't just about making it smaller; they show that this pruning method actually keeps accuracy higher than a small model you train from scratch and cuts the pre-training compute cost by two point five times to four times.
Jane: The results on Matbench Discovery are pretty strong, showing that the pruned version performs better for downstream tasks when you fine-tune it compared to a standard, unpruned model.
Lu: They use this four-stage framework—calibration, pruning, retraining, and then fine-tuning—to get these compact specialized models ready for real work. And they use a first order Taylor expansion on the potential energy surface sensitivity to decide which parts of the network are most important.
Tom: That sensitivity calculation is what drives the importance score. It’s based on how much removing a feature block messes up the energy surface, and they make sure that score is SO(three) invariant so it doesn't care about rotations.
Jane: And this consistency across rotations is what makes it generalizable, because it works whether you are looking at an atom configuration in one orientation or another. It validates the method across different equivariant architectures like SevenNet and eSCN too.
Meng: From my side, what I need to see is how much simpler the fine-tuning process is once you have this compact base model ready. The fact that it reduces energy and force errors by over seventy percent in those fine-tuning runs sounds very promising for accelerating our own production pipelines.
Lu: It really shows how you can systematically bridge that gap between needing these incredibly accurate foundation models and needing them to be computationally feasible for actual scientific discovery.
Tom: And they also point out a limitation, which is important—the method requires you to specify the target architecture before it starts pruning, so they're looking into using neural architecture search to make that step automatic.
Jane: So, while it’s a very solid way to compress these models now, the next big challenge for this field is getting that pruning process to be fully automated and discovery-driven.
Lu: That's exactly what they’re heading toward; making the search for optimal pruning configurations automatic instead of you having to pick them beforehand.
Meng: For us in production, that automation would be huge because it cuts down on manual tuning, which is a real pain at scale when you’re dealing with these complex representations.
Tom: So, if we can automate the search for the best blocks to prune while keeping physics intact and hitting those compute savings—that’s where the real efficiency gain comes from.
The paper's summary: Tom: So, we're talking about how they suggest we can stack this pruning method on top of other compression techniques to get even better results when deploying these models.
Jane: Right, so they say you don't have to choose just one way to compress the model; you can combine structural pruning with something like quantization or knowledge distillation.
Lu: It sounds like a really smart idea because it means practitioners can tailor the compression strategy exactly to what their deployment constraints are—like memory limits versus inference speed.
Meng: I mean, if you can layer them on top of each other, we could potentially see huge gains. They claim up to nine point six times combined throughput improvement on an RTX three thousand ninety when using standard floating-point precision with both techniques applied together.
Tom: That’s a big number. It means that for production pipelines, you don't have to stick to just one compression trick; you can layer them on top of each other without the techniques conflicting with each other.
Jane: It's about complementarity, so if structural pruning saves memory and quantization shrinks the numbers further, you get that massive performance boost.
Lu: The authors are pushing this idea that it’s not just about one technique but how you compose different compression methods to hit the best possible efficiency targets.
Tom: And they're saying this flexibility is key because different deployments have different bottlenecks—some need to save memory, others need faster inference, and they can use these methods together for those specific needs.
Jane: It takes the guesswork out of deployment a bit, so you have tools at your disposal to optimize for the exact environment you're working in.
Meng: That flexibility is what I care about because it lets us be smarter about our hardware choices and how we configure the model for real traffic versus just running a quick test.
Lu: And this moves the discussion away from finding one perfect compression method and toward finding the optimal combination of techniques for a specific application.
Tom: So, to wrap up, they're suggesting that structural pruning is a great starting point, but combining it with quantization or knowledge distillation opens up a whole new level of optimization possibilities for deployment.
Jane: It really shows that we’re moving toward more sophisticated strategies where practitioners can freely compose these techniques according to their specific deployment constraints.
Meng: That sounds like something that will actually make things much more practical when we take these models out of the lab and into the field.
The paper's improvements: Tom: So we're wrapping up on Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning. Basically, they show how to cut down these giant atomistic foundation models using structural pruning based on physical sensitivity while keeping that rotational symmetry intact.
Jane: That's right, so the main implication is that you can have much smaller models for simulating atoms without sacrificing the accuracy needed for serious science.
Lu: They did a lot of work showing how this method works across different equivariant architectures like SevenNet and eSCN, which proves it’s a general technique.
Meng: I just keep thinking about the cost; if we can get models that are four times cheaper to pre-train and still fine-tune better than scratch models, that makes shipping these things to users way more feasible.
Lalam: I'm confident in this direction; it directly addresses the efficiency bottleneck for atomistic simulations, making high accuracy accessible on less compute.
Tom: It does sound like a solid piece of work because they put concrete numbers behind their claims about accuracy and cost reduction.
Jane: And they’ve given us a clear path forward by showing how to combine this pruning with other techniques like quantization for even better performance.
Lu: The real future direction, though, is that automation; right now, you still have to tell the system what kind of architecture you want to prune before it starts working.
Meng: Yeah, I agree; if we could let neural architecture search automatically find those optimal blocks to prune instead of having us pick them manually, that would be a real game-changer for production.
Lalam: I'm sure the literature will focus on that automation because it turns this powerful method into something truly scalable for any new model.
Tom: So, to summarize, Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning gives us a way to shrink these models structurally while maintaining physical correctness through an SO(three) invariant importance criterion <ref:2605.08885#pg1>.
Jane: It's a strong method that bridges the gap between accuracy and efficiency for atomistic foundation models.
Meng: For us, it means moving from massive pre-training runs to manageable fine-tuning steps with much lower costs and better performance margins.
Conclusion: Tom: So we're wrapping up on "Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning." Basically, they show how to cut down these giant atomistic foundation models using structural pruning based on physical sensitivity while keeping that rotational symmetry intact.
Jane: That's right, so the main implication is that you can have much smaller models for simulating atoms without sacrificing the accuracy needed for serious science.
Lu: They did a lot of work showing how this method works across different equivariant architectures like SevenNet and eSCN, which proves it’s a general technique.
Meng: I just keep thinking about the cost; if we can get models that are four times cheaper to pre-train and still fine-tune better than scratch models, that makes shipping these things to users way more feasible.
Lalam: I'm confident in this direction; it directly addresses the efficiency bottleneck for atomistic simulations, making high accuracy accessible on less compute.
Tom: It does sound like a solid piece of work because they put concrete numbers behind their claims about accuracy and cost reduction.
Jane: And they’ve given us a clear path forward by showing how to combine this pruning with other techniques like quantization for even better performance.
Lu: The real future direction, though, is that automation; right now, you still have to tell the system what kind of architecture you want to prune before it starts working.
Meng: Yeah, I agree; if we could let neural architecture search automatically find those optimal blocks to prune instead of having us pick them manually, that would be a real game-changer for production.
Lalam: I'm sure the literature will focus on that automation because it turns this powerful method into something truly scalable for any new model.
Tom: So, to summarize, "Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning" gives us a way to shrink these models structurally while maintaining physical correctness through an SO(three) invariant importance criterion.
Jane: It's a strong method that bridges the gap between accuracy and efficiency for atomistic foundation models.
Lu: This paper provides a blueprint for systematic compression, focusing on structural soundness rather than just random weight removal.
Meng: For us, it means moving from massive pre-training runs to manageable fine-tuning steps with much lower costs and better performance margins.
Chen Wang, Siyu Hu, *Guangming Tan, *Weile Jia
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
cs.LG
Submitted: 2026-05-09
Updated: 2026-11-06
Comments: Accepted as NeurIPS 2026 Oral paper
Code: https://github.com/NVIDIA/cuEquivariance
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models…
Key concepts
- SO(3) Equivariant Models
- These are AI models designed to respect the rotational symmetry of 3D space. They ensure that if you rotate the input atomic structure, the model's output transforms in a predictable, consistent way. This architectural feature is crucial for accurately modeling physical systems like molecules.
- Structural Pruning
- This technique removes entire blocks of features or irreducible representations from the model's architecture. The method uses a specific importance score derived from energy surface sensitivity to decide which parts are least important, effectively shrinking the model while preserving the essential rotational symmetry.
- Importance Criterion I(t)k,l
- This is a mathematical score used to rank tokens for pruning. It is based on how sensitive the potential energy surface is when a feature block is removed. Because this criterion is SO(3) invariant, it makes pruning decisions consistent regardless of how the atomic configuration is rotated.
- Four-Stage Framework
- This systematic process involves calibration, where initial data sets are prepared; pruning, where redundant tokens are removed using the importance score; retraining to adapt the model to the reduced structure; and finally, fine-tuning for specific tasks. This sequence ensures a compact and highly accurate specialized model is created.
Terminology
Summary
SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models, achieving high accuracy and data efficiency by building rotational symmetry directly into the architecture. p. 0 Compact SO(3) Equivariant Atomistic Foundation
Models via Structural Pruning
Chen Wang1,2 Siyu Hu1,2,∗ Guangming Tan1,2,∗ Weile Jia1,2,∗
1State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
{wangchen24z, husiyu, tgm, jiaweile}@ict. cn
Abstract
SO(3) equivariant graph neural networks have become the dominant paradigm
for atomistic foundation models, achieving high accuracy and data efficiency by
building rotational symmetry directly into the architecture. The pruning is applied along the channel and order
dimensions, with each irreducible representation kept or removed as a complete
block, thereby retaining SO(3) equivariance. Starting from a large checkpoint,
the pruned model substantially reduces the inference cost while retaining higher
accuracy than an independently trained small model.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper introduces a structural pruning method for SO(3) equivariant atomistic foundation models to bridge the gap between model accuracy and inference efficiency. The pruning is applied along the channel and order dimensions, with each irreducible representation kept or removed as a complete block, thereby retaining SO(3) equivariance. p. 0 In this paper, we propose a structural
pruning method for SO(3) equivariant atomistic foundation models to bridge this
accuracy-efficiency gap. The pruning is applied along the channel and order
dimensions, with each irreducible representation kept or removed as a complete
block, thereby retaining SO(3) equivariance. Our key contributions are as follows:
• We introduce a structural pruning method for higher-order SO(3) equivariant tensors across
both channel and order dimensions while preserving SO(3) equivariance. • We propose a four-stage framework to compress foundation models for downstream finetuning: (1) calibration, (2) pruning, (3) retraining, and (4) fine-tuning.Where the paper says it
Where the paper says it
How the training signal is built
The importance criterion is based on the sensitivity of the potential energy surface using a first-order Taylor expansion to approximate the impact of removing a feature block. The final importance score I(t)k,l is defined as an aggregation over the representation dimension m and all atoms j, averaged over a calibration dataset D. p. 3 We aggregate this importance score over the representation dimension m and all atoms j, averaged
over a calibration dataset D. The final importance score I
(t)
k,l is defined as:
I
(t)
k,l = ER,Z∼D
X
j,m
∂E
∂h(t)
j,klm
· h
(t)
j,klm
. This criterion is SO(3) invariant (see Appendix B. 3 for proof), ensuring that pruning decisions are
consistent under rotations of the atomic configuration. This criterion is SO(3) invariant (see Appendix B. 3 for proof), ensuring that pruning decisions are
consistent under rotations of the atomic configuration. This criterion is SO(3) invariant (see Appendix B. 3 for proof), ensuring that pruning decisions are
consistent under rotations of the atomic configuration.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The structural alignment step involves Tensor Product Reconfiguration and Weight Slicing to ensure physical alteration of the model structure to match the reduced feature dimensions. The importance criterion is SO(3) invariant (see Appendix B. p. 4 This criterion is SO(3) invariant (see Appendix B. 3 for proof), ensuring that pruning decisions are
consistent under rotations of the atomic configuration. 4 (Appendix B. 4), higher order l corresponds to higher angular frequencies.Where the paper says it
Where the paper says it
Efficiency and Generalizability
The pruned MACE-MP model outperforms official models trained from scratch on 7 of 9 metrics on the Matbench Discovery leaderboard. In terms of efficiency, compressed MACE-MP and MACE-OFF models contain 1. p. 0 The pruned MACE-MP model
outperforms the official from-scratch trained small model on 7 of 9 metrics on the
Matbench Discovery leaderboard. In terms of efficiency, compressed MACE-MP
and MACE-OFF models contain 1. In terms of efficiency, compressed MACE-MP
and MACE-OFF models contain 1. 5× to 4× fewer parameters and require 2. 5× to 4× fewer parameters and require 2. 5×
to 4× less pre-training compute than training a small model from scratch. 5×
to 4× less pre-training compute than training a small model from scratch. For
downstream applications, fine-tuning the pruned model reduces energy and force
errors by 70. For
downstream applications, fine-tuning the pruned model reduces energy and force
errors by 70. 4% compared to training task-specific models from scratch
across eight representative downstream datasets. 4% compared to training task-specific models from scratch
across eight representative downstream datasets. We demonstrate that the method
generalizes to other SO(3) equivariant architectures (SevenNet, eSCN) and can be
combined with quantization and knowledge distillation for further gains. This translates directly to lower training cost. The
full pipeline (calibration, pruning, and retraining on
25% of MPtrj) requires approximately 181 GPUhours on an A800, compared to ∼750 GPU-hours
for pre-training a model from scratch, confirming
a 4× reduction in training cost (Appendix D.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The core principle of our method, evaluating the importance of coupled irreducible representations, is broadly applicable to any equivariant architecture that represents features as sums of SO(3) irreducible representations. A current limitation is that the target pruned architecture must be specified in advance. p. 8 The core principle of our method, evaluating the importance of coupled irreducible representations,
is broadly applicable to any equivariant architecture that represents features as sums of SO(3)
irreducible representations, as validated on MACE, SevenNet, and eSCN. A current limitation is that
the target pruned architecture must be specified in advance. A current limitation is that
the target pruned architecture must be specified in advance. Incorporating neural architecture search
to automatically discover optimal configurations is a promising direction for future work, enabling
more flexible compression tailored to specific deployment constraints.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Structural pruning of SO(3) equivariant atomistic foundation models reduces inference cost by 1.5 times to 4 times while retaining higher accuracy than independently trained small models on Matbench Discovery leaderboard metrics. This allows for substantial reduction in computational overhead for practical deployment and training costs, as shown by a
2.5x–4x cheaper
result in training cost compared to scratch-trained models. -
The proposed four-stage framework (calibration, pruning, retraining, and fine-tuning) enables the creation of compact specialized models that serve as superior initializations for downstream tasks, reducing energy and force errors by 70.1% and 34.4% compared to training from scratch across eight representative downstream datasets.
-
The method allows for compounding gains by combining structural pruning with quantization or knowledge distillation, achieving up to a 9.6 times combined throughput improvement on an RTX 3090 when using FP32 precision with both techniques applied together. This complementarity means practitioners can freely compose these techniques according to their deployment constraints without conflict.
-
The importance criterion I(t)k,l is SO(3) invariant, ensuring that pruning decisions are consistent under rotations of the atomic configuration and that the resulting pruned model preserves SO(3) equivariance across other architectures like SevenNet and eSCN. This consistency validates the method's generalizability across different equivariant models.
-
The framework enables species-specific compression by allowing parameters corresponding to chemical elements not present in the target downstream dataset to be explicitly removed prior to fine-tuning, resulting in a highly compact model specialized for both geometric structure and chemical composition of the target system.
Abstract
SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models, achieving high accuracy and data efficiency by building rotational symmetry directly into the architecture. Yet the computational cost of their higher-order tensor operations creates a tough trade-off between model accuracy and inference efficiency. In this paper, we propose a structural pruning method for SO(3) equivariant atomistic foundation models to bridge this accuracy-efficiency gap. The pruning is applied along the channel and order dimensions, with each irreducible representation kept or removed as a complete block, thereby retaining SO(3) equivariance. Starting from a large checkpoint, the pruned model substantially reduces the inference cost while retaining higher accuracy than an independently trained small model. The pruned MACE-MP model outperforms the official from-scratch trained small model on 7 of 9 metrics on the Matbench Discovery leaderboard. In terms of efficiency, compressed MACE-MP and MACE-OFF models contain 1.5 times to 4 times fewer parameters and require 2.5 times to 4 times less pre-training compute than training a small model from scratch. For downstream applications, fine-tuning the pruned model reduces energy and force errors by 70.1% and 34.4% compared to training task-specific models from scratch across eight representative downstream datasets. We demonstrate that the method generalizes to other SO(3) equivariant architectures (SevenNet, eSCN) and can be combined with quantization and knowledge distillation for further gains.
Sources
- A foundation model for atomistic materials chemistry
- UMA: A Family of Universal Models for Atoms
- Orb-v3: atomistic simulation at scale
- Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
- e3nn: Euclidean Neural Networks
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Scalable Foundation Interatomic Potentials via Message-Passing Pruning and Graph Partitioning
- High-performance training and inference for deep equivariant interatomic potentials
- Graph atomic cluster expansion for foundational machine learning interatomic potentials
- A Graph Neural Network for the Era of Large Atomistic Models
- MatterSim: A Deep Learning Atomistic Model Across Elements, Temperatures and Pressures
- A recipe for scalable attention-based MLIPs: unlocking long-range accuracy with all-to-all node attention
- MatRIS: Toward Reliable and Efficient Pretrained Machine Learning Interatomic Potentials
- Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models
- The Open Molecules 2025 (OMol25) Dataset, Evaluations, and Models
- An open dataset of gas adsorption in deformable metal-organic frameworks for direct air capture
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians
- Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis