Pruning and Distilling Mixture-of-Experts into Dense Language Models
summary
The gist
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory…
In short
The work systematically converts a trained Mixture-of-Experts (MoE) model into a standard dense architecture by scoring and selecting experts, grouping them, and distilling knowledge. The key finding is that using a diversity-aware scoring method called DO-ACP yields the best results across various models, significantly outperforming other methods.
Key concepts
- Mixture-of-Experts (MoE)
- MoE is a model architecture where different 'experts' are trained to handle different types of data or tasks. During inference, only a few experts are activated for any given input, making the model efficient but requiring all expert parameters to be loaded into memory.
- Diversity-aware Scoring (DO-ACP)
- This is a novel method used to score experts by considering both their individual importance and how different they are from each other. It aims to select experts that maximize both quality and variety, leading to better overall model performance than simpler scoring techniques.
- Knowledge Distillation
- This process involves training a smaller 'student' model to mimic the behavior of a larger, more complex 'teacher' model. In this study, it is used after converting the MoE to dense form to recover quality lost during the conversion process.
Terminology used across episodes
This episode discusses
- Pruning and Distilling Mixture-of-Experts into Dense Language Models · Paper Radio
- gpt-oss-120b & gpt-oss-20b Model Card
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
- Task-Specific Expert Pruning for Sparse Mixture-of-Experts
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Distilling the Knowledge in a Neural Network
- REAM: Merging Improves Pruning of Experts in LLMs
- Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
- Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
- REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
- Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
- SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
- Ministral 3
- Pointer Sentinel Mixture Models
- MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
- Compact Language Models via Pruning and Knowledge Distillation
- Expert Merging in Sparse Mixture of Experts with Nash Bargaining
- GLU Variants Improve Transformer
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- LLM Pruning and Distillation in Practice: The Minitron Approach
The paper
Pruning and Distilling Mixture-of-Experts into Dense Language Models · Read on arXiv
Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho
KRAFTON AI · KAIST
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after 4B-token distillation at 1.6x faster training wall-clock speed.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Pruning and Distilling Mixture-of-Experts into Dense Language Models".
Jane: Here's the gist of the paper. Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment The first systematic framework for converting a trained MoE into a standard fully dense architecture involves experts being scored, selected, and grouped into a dense Fully Connected Network (FFN) and refined by knowledge distillation from the MoE teacher This work introduces the first end-to-end study of converting a trained MoE language model into a fully dense architecture Systematic Framework and Evaluation The authors present the first systematic framework for converting a trained MoE language model into a standard fully dense architecture through three steps: experts are scored by importance, the top-scoring experts are selected, grouped and merged within groups, and then concatenated into a standard dense FFN with down-projection matrices scaled to preserve output magnitude They evaluate 7 scoring methods, 5 grouping methods, and 2 magnitude scaling options across a range of selected expert counts on Qwen3-30BA3B, yielding 350 configurations The key finding is that the choice of scoring method is the most impactful decision, with the novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ∼4B-token distillation at 1.6× faster training wall-clock speed Expert Selection and Diversity The authors introduce a diversity-aware expert selection criterion based on the Gram log-determinant that jointly maximizes expert importance and mutual diversity This criterion, called DO-ACP, achieves the best accuracy across all configurations and all three models The authors show that selecting diverse experts without merging is the best strategy, as frequency-based scoring selects redundant experts and benefits from merging The D-Optimal selection (DO) criterion maximizes the log-determinant of an importance-weighted kernel, which captures both individual quality and pairwise redundancy Theoretical results prove that the greedy D-Optimal selection is near-optimal with a standard greedy guarantee Performance Comparison and Distillation The MoE-to-dense pipeline outperforms dense-to-dense pruning by +6.3 pp after ∼4B-token distillation at 1.6× faster training wall-clock speed The best configuration, DO-ACP at K=8, achieves the highest average accuracy across all 35 configurations (43.41%) on Qwen3-30B-A3B Distillation is performed using forward KL divergence between the teacher and student output distributions to recover quality lost during compression Furthermore, expanding teacher routing to activate k' > k experts improves quality by +0.70 pp when k'=16 Cross-Model Validation and Generalization The findings are validated across DeepSeek-V2-Lite and GPT-OSS-20B, showing that the advantage of diversity-aware scoring diminishes with smaller expert pools The best configuration uses pure pruning (K=k) on all three models, confirming that this strategy is robust across architectures The advantage of MoE-to-dense over random FFN initialization on GPT-OSS (+3.3 pp with 32 experts) is smaller than on Qwen3 (+10.7 pp with 128 experts), suggesting effectiveness depends on expert pool size The paper concludes that the best recipe is scoring with DO-ACP, retaining exactly the top-k experts per layer, and distilling with forward KL This approach achieves an average accuracy of 43.41% on Qwen3-30B-A3B The overall results point to a simple recipe: scoring with DO-ACP, retaining exactly the top-k experts per layer, and distilling with forward KL Limitations The limitations include the fact that when merging is used (K > k), the framework ties merge weights to selection scores, and decoupling these may improve such configurations The benefit of our method over random FFN initialization on GPT-OSS (+3.3 pp with 32 experts) is smaller than on Qwen3 (+10.
Paper summary: Lu: So wrapping up the "Pruning and Distilling Mixture-of-Experts into Dense Language Models" paper, the authors present a systematic framework for converting MoE models into standard dense architectures by focusing on expert scoring, selection, grouping them in a dense fully connected network, and then distilling that result from the original MoE teacher.
Meng: The main point is that they found a specific recipe: use the DO-ACP scoring method to select experts, keep exactly the top k experts per layer, and distill using forward KL divergence between the teacher and student outputs.
Tom: That means if you have an MoE model, you don't just throw away all the routing; you systematically convert it into a dense network by intelligently choosing which parts of that expert structure to keep.
Jane: The big picture is that this technique gives us a concrete path for taking these dominant but memory-heavy MoE models and making them usable in environments where loading every single parameter isn't feasible, all while trying to maintain high accuracy.
Lalam: They also pointed out a limitation: when you use merging, meaning you keep more than k experts, the framework ties those merge weights directly to the selection scores. The authors suggest that decoupling those two might actually lead to better configurations in the future.
Lu: And another thing they noted is that their method's advantage over just randomly initializing a dense fully connected network on GPT OSS 20B was smaller than what they saw on Qwen3, suggesting the benefit of this specific method really depends on the size of the expert pool you start with <ref:2605.28207#pg1>.
Conclusion: Tom: So we're wrapping up this look at "Pruning and Distilling Mixture-of-Experts into Dense Language Models." It boils down to taking these big mixture-of-experts models, which are great for performance but huge in size, and converting them into standard dense networks.
Jane: That’s right. The authors show you the framework they used: first scoring experts by importance, then picking the best ones, grouping them into a standard fully connected network that we call an FFN, and finally using knowledge distillation from the original expert model to refine that new dense network.
Lu: What's interesting is their systematic approach. They tested three hundred fifty different setups across several models like Qwen and DeepSeek-V2-Lite to see what works best for scoring and grouping.
Meng: From a deployment standpoint, the real win they found was that their method beats standard pruning by about six point three percent in accuracy, even when you do that distillation step at a faster wall clock speed. That’s tangible performance gain.
Lalam: The specific recipe they landed on is scoring with this diversity-aware method called DO-ACP, keeping exactly the top k experts per layer, and using forward KL divergence for the distillation process.
Tom: So what does this actually mean for us? It gives us a way to make these massive MoE models more manageable without losing too much smarts. The naive question is, can we really get that accuracy back when we force it into a dense structure?
Jane: The answer is yes, they show it works. But there are caveats. They note that the benefit of their diversity-aware scoring drops off when you start with smaller expert groups.
Lu: And another thing they flag is that if you choose to merge experts—keeping more than k—the framework ties those merge weights too closely to the selection scores, which they think could be improved later.
Meng: So it’s a good path forward for production if we can nail that DO-ACP scoring. But how robust is this conversion method when you move beyond just comparing models?
Tom: We've seen its success across Qwen and DeepSeek-V2-Lite, but the paper does show that the advantage over just randomly setting up a dense network is smaller on GPT OSS 20B compared to Qwen <ref:2605.28207#pg1>.
Jane: So the takeaway is a recipe: use DO-ACP scoring, stick to exactly top k experts per layer, and use forward KL distillation for quality recovery. But we need to keep an eye on those scaling limitations as we try this out at scale.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI