OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
cs.LG
Submitted: 2026-09-09
Updated: 2026-09-29
Comments: Work in progress, revisions ongoing
License: http://creativecommons.org/licenses/by/4.0/
The gist: Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements.
Terminology
Abstract
Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33 times faster search and 1.55 times inference speedup. Codes will be available after acceptance.
Sources
- GPT-4 Technical Report
- gpt-oss-120b & gpt-oss-20b Model Card
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
- Evaluating Large Language Models Trained on Code
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3 Technical Report
- The Llama 3 Herd of Models
- Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
- Mixtral of Experts
- HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space
- Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Pointer Sentinel Mixture Models
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Unveiling Super Experts in Mixture-of-Experts Large Language Models
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- Qwen3 Technical Report
- MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
- Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks