AIRE-Prune: Asymptotic Impulse-Response Energy for State Pruning in State Space Models
cs.LG, cs.SY, eess.SY
Submitted: 2026-01-31
Updated: 2026-01-31
Journal ref: International Conference on Learning Representations (ICLR), 2026
Code: https://github.com/falcon-arrow/AIRE-Prune
License: http://creativecommons.org/licenses/by/4.0/
The gist: State space models (SSMs) often sacrifice capacity, search space, or stability to offset the memory and compute costs of large state dimensions.
Terminology
Abstract
State space models (SSMs) often sacrifice capacity, search space, or stability to offset the memory and compute costs of large state dimensions. We introduce a structured post-training pruning method for SSMs -- AIRE-Prune (Asymptotic Impulse-Response Energy for State PRUN(E)) -- that reduces each layer's state dimension by directly minimizing long-run output-energy distortion. AIRE-Prune assigns every state a closed-form asymptotic impulse-response energy-based score, i.e., the total impulse-response energy it contributes over an infinite horizon (time), and normalizes these scores layer-wise to enable global cross-layer comparison and selection. This extends modal truncation from single systems to deep stacks and aligns pruning with asymptotic response energy rather than worst-case gain. Across diverse sequence benchmarks, AIRE-Prune reveals substantial redundancy in SISO and MIMO SSMs with average pruning of 60.8%, with average accuracy drop of 0.29% without retraining, while significantly lowering compute. Code: https://github.com/falcon-arrow/AIRE-Prune.
Sources
- The State of Sparsity in Deep Neural Networks
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- ListOps: A Diagnostic Dataset for Latent Tree Learning
- State-Free Inference of State-Space Models: The Transfer Function Approach
- Simplified State Space Layers for Sequence Modeling
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
- To prune, or not to prune: exploring the efficacy of pruning for model compression
Related papers
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning
- Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine