Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

arXiv:2610.04093 · cs.LG, cs.DC · Submitted 2026-10-02 · Read on arXiv

cs.LG, cs.DC

Submitted: 2026-10-02

Updated: 2026-10-02

Code: https://github.com/JafarBadour/Snack-SNN

Terminology

• Sparsity — 1× in this paper · explained in 4 episodes

Sources

• Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
• Truly Sparse Neural Networks at Scale
• Sparse Networks from Scratch: Faster Training without Losing Performance
• The State of Sparsity in Deep Neural Networks
• Dynamic Sparse Training with Structured Sparsity
• PopSparse: Accelerated block sparse matrix multiplication on IPU
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
• BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
• Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
• LLaMA: Open and Efficient Foundation Language Models
• To prune, or not to prune: exploring the efficacy of pruning for model compression

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks