Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
cs.LG, cs.AI
Submitted: 2026-08-08
Updated: 2026-10-10
Comments: 25 pages, 6 figures, 11 tables. Code: https://github.com/ianKa1/MoE_pruning/tree/main
Code: https://github.com/ianKa1/MoE_pruning
License: http://creativecommons.org/licenses/by/4.0/
The gist: Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert.
Terminology
Abstract
Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning. We test whether lightweight adaptation can recover this signal. We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced 2 router change, and prune the least-changed experts in one shot. On Mixtral-8 times 7B-Instruct (44.83% MMLU-Pro), router-only LoRA trains 0.002% of parameters and outperforms all-module LoRA at matched rank with half the experts removed (27.54% vs. 24.42%); signal quality declines as adaptation spreads to attention and expert weights. Accuracy improves monotonically with LoRA rank, reaching 28.76%. IA3, which leaves router weights frozen, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. Router-guided MMLU-Pro accuracy decays quasi-linearly rather than collapsing, remains nearly 1.8 times that of magnitude-based or random pruning at maximal compression, and reduces memory by 49% and per-token latency by 37%. At 25% compression, retention is competitive with methods using full activation statistics. The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed while random pruning falls to single digits. Router sensitivity under lightweight fine-tuning therefore makes provably motivated expert pruning practical at scale.
Sources
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
- Task-Specific Expert Pruning for Sparse Mixture-of-Experts
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- QLoRA: Efficient Finetuning of Quantized LLMs
- LoRA: Low-Rank Adaptation of Large Language Models
- What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models
- Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
- Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
- Mixtral of Experts
- REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
- Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
- SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks