Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning
summary
The gist
SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models…
In short
The paper proposes a structural pruning method for SO(3) equivariant atomistic foundation models to reduce inference costs while maintaining high accuracy. By removing redundant reasoning tokens based on an SO(3)-invariant importance criterion, the method creates smaller, more efficient models that outperform independently trained small models and significantly lower pre-training compute requirements.
Key concepts
- SO(3) Equivariant Models
- These are AI models designed to respect the rotational symmetry of 3D space. They ensure that if you rotate the input atomic structure, the model's output transforms in a predictable, consistent way. This architectural feature is crucial for accurately modeling physical systems like molecules.
- Structural Pruning
- This technique removes entire blocks of features or irreducible representations from the model's architecture. The method uses a specific importance score derived from energy surface sensitivity to decide which parts are least important, effectively shrinking the model while preserving the essential rotational symmetry.
- Importance Criterion I(t)k,l
- This is a mathematical score used to rank tokens for pruning. It is based on how sensitive the potential energy surface is when a feature block is removed. Because this criterion is SO(3) invariant, it makes pruning decisions consistent regardless of how the atomic configuration is rotated.
- Four-Stage Framework
- This systematic process involves calibration, where initial data sets are prepared; pruning, where redundant tokens are removed using the importance score; retraining to adapt the model to the reduced structure; and finally, fine-tuning for specific tasks. This sequence ensures a compact and highly accurate specialized model is created.
Terminology used across episodes
This episode discusses
- Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning · Paper Radio
- A foundation model for atomistic materials chemistry
- UMA: A Family of Universal Models for Atoms
- Orb-v3: atomistic simulation at scale
- Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
- e3nn: Euclidean Neural Networks
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Scalable Foundation Interatomic Potentials via Message-Passing Pruning and Graph Partitioning
- High-performance training and inference for deep equivariant interatomic potentials
- Graph atomic cluster expansion for foundational machine learning interatomic potentials
- A Graph Neural Network for the Era of Large Atomistic Models
- MatterSim: A Deep Learning Atomistic Model Across Elements, Temperatures and Pressures
- A recipe for scalable attention-based MLIPs: unlocking long-range accuracy with all-to-all node attention
- MatRIS: Toward Reliable and Efficient Pretrained Machine Learning Interatomic Potentials
- Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models
- The Open Molecules 2025 (OMol25) Dataset, Evaluations, and Models
- An open dataset of gas adsorption in deformable metal-organic frameworks for direct air capture · Paper Radio
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians
- Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields
The paper
Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning · Read on arXiv
Chen Wang, Siyu Hu, *Guangming Tan, *Weile Jia
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models, achieving high accuracy and data efficiency by building rotational symmetry directly into the architecture. Yet the computational cost of their higher-order tensor operations creates a tough trade-off between model accuracy and inference efficiency. In this paper, we propose a structural pruning method for SO(3) equivariant atomistic foundation models to bridge this accuracy-efficiency gap. The pruning is applied along the channel and order dimensions, with each irreducible representation kept or removed as a complete block, thereby retaining SO(3) equivariance. Starting from a large checkpoint, the pruned model substantially reduces the inference cost while retaining higher accuracy than an independently trained small model. The pruned MACE-MP model outperforms the official from-scratch trained small model on 7 of 9 metrics on the Matbench Discovery leaderboard. In terms of efficiency, compressed MACE-MP and MACE-OFF models contain 1.5 times to 4 times fewer parameters and require 2.5 times to 4 times less pre-training compute than training a small model from scratch. For downstream applications, fine-tuning the pruned model reduces energy and force errors by 70.1% and 34.4% compared to training task-specific models from scratch across eight representative downstream datasets. We demonstrate that the method generalizes to other SO(3) equivariant architectures (SevenNet, eSCN) and can be combined with quantization and knowledge distillation for further gains.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning".
Jane: Here's the gist of the paper. SO(3) equivariant graph neural networks have become the dominant paradigm for atomistic foundation models, achieving high accuracy and data efficiency by building rotational symmetry directly into the architecture.
Tom: The proposed structural pruning method substantially reduces inference cost while retaining higher accuracy than an independently trained small model.
Jane: Pruning redundant reasoning tokens The paper introduces a structural pruning method for SO(3) equivariant atomistic foundation models to bridge the gap between model accuracy and inference efficiency.
Tom: The pruning is applied along the channel and order dimensions, with each irreducible representation kept or removed as a complete block, thereby retaining SO(3) equivariance.
Title and authors: Tom: So, to recap, this paper is about taking these big SO(three) equivariant atomistic foundation models and making them much smaller by cutting down their structure, without losing that essential rotational symmetry that keeps the physics correct.
Jane: Right, so instead of just randomly chopping up the model like you might do with a regular neural network, they are pruning along specific dimensions—the channel and the order dimensions—and they keep entire irreducible representations as solid blocks.
Lu: Exactly. They're using this structural approach because it’s designed to preserve that SO(three) equivariance exactly, no matter what rotation you apply to the atomic configuration. It keeps things physically consistent across different setups.
Meng: So, I mean, if you can cut down the model size without breaking the symmetry, that’s a huge win for deployment because we don't have to run these massive models on expensive hardware just to get an inference result.
Tom: That’s the core idea. And they show this isn't just about making it smaller; they show that this pruning method actually keeps accuracy higher than a small model you train from scratch and cuts the pre-training compute cost by two point five times to four times.
Jane: The results on Matbench Discovery are pretty strong, showing that the pruned version performs better for downstream tasks when you fine-tune it compared to a standard, unpruned model.
Lu: They use this four-stage framework—calibration, pruning, retraining, and then fine-tuning—to get these compact specialized models ready for real work. And they use a first order Taylor expansion on the potential energy surface sensitivity to decide which parts of the network are most important.
Tom: That sensitivity calculation is what drives the importance score. It’s based on how much removing a feature block messes up the energy surface, and they make sure that score is SO(three) invariant so it doesn't care about rotations.
Jane: And this consistency across rotations is what makes it generalizable, because it works whether you are looking at an atom configuration in one orientation or another. It validates the method across different equivariant architectures like SevenNet and eSCN too.
Meng: From my side, what I need to see is how much simpler the fine-tuning process is once you have this compact base model ready. The fact that it reduces energy and force errors by over seventy percent in those fine-tuning runs sounds very promising for accelerating our own production pipelines.
Lu: It really shows how you can systematically bridge that gap between needing these incredibly accurate foundation models and needing them to be computationally feasible for actual scientific discovery.
Tom: And they also point out a limitation, which is important—the method requires you to specify the target architecture before it starts pruning, so they're looking into using neural architecture search to make that step automatic.
Jane: So, while it’s a very solid way to compress these models now, the next big challenge for this field is getting that pruning process to be fully automated and discovery-driven.
Lu: That's exactly what they’re heading toward; making the search for optimal pruning configurations automatic instead of you having to pick them beforehand.
Meng: For us in production, that automation would be huge because it cuts down on manual tuning, which is a real pain at scale when you’re dealing with these complex representations.
Tom: So, if we can automate the search for the best blocks to prune while keeping physics intact and hitting those compute savings—that’s where the real efficiency gain comes from.
The paper's summary: Tom: So, we're talking about how they suggest we can stack this pruning method on top of other compression techniques to get even better results when deploying these models.
Jane: Right, so they say you don't have to choose just one way to compress the model; you can combine structural pruning with something like quantization or knowledge distillation.
Lu: It sounds like a really smart idea because it means practitioners can tailor the compression strategy exactly to what their deployment constraints are—like memory limits versus inference speed.
Meng: I mean, if you can layer them on top of each other, we could potentially see huge gains. They claim up to nine point six times combined throughput improvement on an RTX three thousand ninety when using standard floating-point precision with both techniques applied together.
Tom: That’s a big number. It means that for production pipelines, you don't have to stick to just one compression trick; you can layer them on top of each other without the techniques conflicting with each other.
Jane: It's about complementarity, so if structural pruning saves memory and quantization shrinks the numbers further, you get that massive performance boost.
Lu: The authors are pushing this idea that it’s not just about one technique but how you compose different compression methods to hit the best possible efficiency targets.
Tom: And they're saying this flexibility is key because different deployments have different bottlenecks—some need to save memory, others need faster inference, and they can use these methods together for those specific needs.
Jane: It takes the guesswork out of deployment a bit, so you have tools at your disposal to optimize for the exact environment you're working in.
Meng: That flexibility is what I care about because it lets us be smarter about our hardware choices and how we configure the model for real traffic versus just running a quick test.
Lu: And this moves the discussion away from finding one perfect compression method and toward finding the optimal combination of techniques for a specific application.
Tom: So, to wrap up, they're suggesting that structural pruning is a great starting point, but combining it with quantization or knowledge distillation opens up a whole new level of optimization possibilities for deployment.
Jane: It really shows that we’re moving toward more sophisticated strategies where practitioners can freely compose these techniques according to their specific deployment constraints.
Meng: That sounds like something that will actually make things much more practical when we take these models out of the lab and into the field.
The paper's improvements: Tom: So we're wrapping up on Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning. Basically, they show how to cut down these giant atomistic foundation models using structural pruning based on physical sensitivity while keeping that rotational symmetry intact.
Jane: That's right, so the main implication is that you can have much smaller models for simulating atoms without sacrificing the accuracy needed for serious science.
Lu: They did a lot of work showing how this method works across different equivariant architectures like SevenNet and eSCN, which proves it’s a general technique.
Meng: I just keep thinking about the cost; if we can get models that are four times cheaper to pre-train and still fine-tune better than scratch models, that makes shipping these things to users way more feasible.
Lalam: I'm confident in this direction; it directly addresses the efficiency bottleneck for atomistic simulations, making high accuracy accessible on less compute.
Tom: It does sound like a solid piece of work because they put concrete numbers behind their claims about accuracy and cost reduction.
Jane: And they’ve given us a clear path forward by showing how to combine this pruning with other techniques like quantization for even better performance.
Lu: The real future direction, though, is that automation; right now, you still have to tell the system what kind of architecture you want to prune before it starts working.
Meng: Yeah, I agree; if we could let neural architecture search automatically find those optimal blocks to prune instead of having us pick them manually, that would be a real game-changer for production.
Lalam: I'm sure the literature will focus on that automation because it turns this powerful method into something truly scalable for any new model.
Tom: So, to summarize, Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning gives us a way to shrink these models structurally while maintaining physical correctness through an SO(three) invariant importance criterion <ref:2605.08885#pg1>.
Jane: It's a strong method that bridges the gap between accuracy and efficiency for atomistic foundation models.
Meng: For us, it means moving from massive pre-training runs to manageable fine-tuning steps with much lower costs and better performance margins.
Conclusion: Tom: So we're wrapping up on "Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning." Basically, they show how to cut down these giant atomistic foundation models using structural pruning based on physical sensitivity while keeping that rotational symmetry intact.
Jane: That's right, so the main implication is that you can have much smaller models for simulating atoms without sacrificing the accuracy needed for serious science.
Lu: They did a lot of work showing how this method works across different equivariant architectures like SevenNet and eSCN, which proves it’s a general technique.
Meng: I just keep thinking about the cost; if we can get models that are four times cheaper to pre-train and still fine-tune better than scratch models, that makes shipping these things to users way more feasible.
Lalam: I'm confident in this direction; it directly addresses the efficiency bottleneck for atomistic simulations, making high accuracy accessible on less compute.
Tom: It does sound like a solid piece of work because they put concrete numbers behind their claims about accuracy and cost reduction.
Jane: And they’ve given us a clear path forward by showing how to combine this pruning with other techniques like quantization for even better performance.
Lu: The real future direction, though, is that automation; right now, you still have to tell the system what kind of architecture you want to prune before it starts working.
Meng: Yeah, I agree; if we could let neural architecture search automatically find those optimal blocks to prune instead of having us pick them manually, that would be a real game-changer for production.
Lalam: I'm sure the literature will focus on that automation because it turns this powerful method into something truly scalable for any new model.
Tom: So, to summarize, "Compact SO(three) Equivariant Atomistic Foundation Models via Structural Pruning" gives us a way to shrink these models structurally while maintaining physical correctness through an SO(three) invariant importance criterion.
Jane: It's a strong method that bridges the gap between accuracy and efficiency for atomistic foundation models.
Lu: This paper provides a blueprint for systematic compression, focusing on structural soundness rather than just random weight removal.
Meng: For us, it means moving from massive pre-training runs to manageable fine-tuning steps with much lower costs and better performance margins.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation