Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity
cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
License: http://creativecommons.org/licenses/by/4.0/
The gist: Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging.
Terminology
Abstract
Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pruning step small enough for sparse adaptation to compensate, allowing the adapted model to provide end-to-end gradients that guide the next pruning decision without model-sized optimizer state. Across six models from 125M to 13B parameters, ETAPS achieves lower perplexity than ELSA at every evaluated sparsity from 50% to 90%, with perplexity reductions of up to 42%. ETAPS is also far cheaper: at 90% sparsity, it prunes LLaMA2-7B on a single H200, where ELSA uses four, with 5.0 times smaller total persistent state and 49% fewer GPU-hours. Together, our findings point to a promising direction for achieving high quality in demanding regimes such as extreme sparsity at substantially lower cost.
Sources
- Scaling Sparse Fine-Tuning to Large Language Models
- Fast and Effective Weight Update for Pruned Large Language Models
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Gemma 2: Improving Open Language Models at a Practical Size
- The Llama 3 Herd of Models
- PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
- LoRA: Low-Rank Adaptation of Large Language Models
- One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
- How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark
- SAFE: Finding Sparse and Flat Minima to Improve Pruning
- The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
- MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity
- ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
- Pointer Sentinel Mixture Models
- Accelerating Sparse Deep Neural Networks
- LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
- BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
- Don't Be Greedy, Just Relax! Pruning LLMs via Frank-Wolfe
- Llama 2: Open Foundation and Fine-Tuned Chat Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents