Pruning and Distilling Mixture-of-Experts into Dense Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Pruning and Distilling Mixture-of-Experts into Dense Language Models".
Jane: Here's the gist of the paper. Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment The first systematic framework for converting a trained MoE into a standard fully dense architecture involves experts being scored, selected, and grouped into a dense Fully Connected Network (FFN) and refined by knowledge distillation from the MoE teacher This work introduces the first end-to-end study of converting a trained MoE language model into a fully dense architecture Systematic Framework and Evaluation The authors present the first systematic framework for converting a trained MoE language model into a standard fully dense architecture through three steps: experts are scored by importance, the top-scoring experts are selected, grouped and merged within groups, and then concatenated into a standard dense FFN with down-projection matrices scaled to preserve output magnitude They evaluate 7 scoring methods, 5 grouping methods, and 2 magnitude scaling options across a range of selected expert counts on Qwen3-30BA3B, yielding 350 configurations The key finding is that the choice of scoring method is the most impactful decision, with the novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ∼4B-token distillation at 1.6× faster training wall-clock speed Expert Selection and Diversity The authors introduce a diversity-aware expert selection criterion based on the Gram log-determinant that jointly maximizes expert importance and mutual diversity This criterion, called DO-ACP, achieves the best accuracy across all configurations and all three models The authors show that selecting diverse experts without merging is the best strategy, as frequency-based scoring selects redundant experts and benefits from merging The D-Optimal selection (DO) criterion maximizes the log-determinant of an importance-weighted kernel, which captures both individual quality and pairwise redundancy Theoretical results prove that the greedy D-Optimal selection is near-optimal with a standard greedy guarantee Performance Comparison and Distillation The MoE-to-dense pipeline outperforms dense-to-dense pruning by +6.3 pp after ∼4B-token distillation at 1.6× faster training wall-clock speed The best configuration, DO-ACP at K=8, achieves the highest average accuracy across all 35 configurations (43.41%) on Qwen3-30B-A3B Distillation is performed using forward KL divergence between the teacher and student output distributions to recover quality lost during compression Furthermore, expanding teacher routing to activate k' > k experts improves quality by +0.70 pp when k'=16 Cross-Model Validation and Generalization The findings are validated across DeepSeek-V2-Lite and GPT-OSS-20B, showing that the advantage of diversity-aware scoring diminishes with smaller expert pools The best configuration uses pure pruning (K=k) on all three models, confirming that this strategy is robust across architectures The advantage of MoE-to-dense over random FFN initialization on GPT-OSS (+3.3 pp with 32 experts) is smaller than on Qwen3 (+10.7 pp with 128 experts), suggesting effectiveness depends on expert pool size The paper concludes that the best recipe is scoring with DO-ACP, retaining exactly the top-k experts per layer, and distilling with forward KL This approach achieves an average accuracy of 43.41% on Qwen3-30B-A3B The overall results point to a simple recipe: scoring with DO-ACP, retaining exactly the top-k experts per layer, and distilling with forward KL Limitations The limitations include the fact that when merging is used (K > k), the framework ties merge weights to selection scores, and decoupling these may improve such configurations The benefit of our method over random FFN initialization on GPT-OSS (+3.3 pp with 32 experts) is smaller than on Qwen3 (+10.
Paper summary: Lu: So wrapping up the "Pruning and Distilling Mixture-of-Experts into Dense Language Models" paper, the authors present a systematic framework for converting MoE models into standard dense architectures by focusing on expert scoring, selection, grouping them in a dense fully connected network, and then distilling that result from the original MoE teacher.
Meng: The main point is that they found a specific recipe: use the DO-ACP scoring method to select experts, keep exactly the top k experts per layer, and distill using forward KL divergence between the teacher and student outputs.
Tom: That means if you have an MoE model, you don't just throw away all the routing; you systematically convert it into a dense network by intelligently choosing which parts of that expert structure to keep.
Jane: The big picture is that this technique gives us a concrete path for taking these dominant but memory-heavy MoE models and making them usable in environments where loading every single parameter isn't feasible, all while trying to maintain high accuracy.
Lalam: They also pointed out a limitation: when you use merging, meaning you keep more than k experts, the framework ties those merge weights directly to the selection scores. The authors suggest that decoupling those two might actually lead to better configurations in the future.
Lu: And another thing they noted is that their method's advantage over just randomly initializing a dense fully connected network on GPT OSS 20B was smaller than what they saw on Qwen3, suggesting the benefit of this specific method really depends on the size of the expert pool you start with <ref:2605.28207#pg1>.
Conclusion: Tom: So we're wrapping up this look at "Pruning and Distilling Mixture-of-Experts into Dense Language Models." It boils down to taking these big mixture-of-experts models, which are great for performance but huge in size, and converting them into standard dense networks.
Jane: That’s right. The authors show you the framework they used: first scoring experts by importance, then picking the best ones, grouping them into a standard fully connected network that we call an FFN, and finally using knowledge distillation from the original expert model to refine that new dense network.
Lu: What's interesting is their systematic approach. They tested three hundred fifty different setups across several models like Qwen and DeepSeek-V2-Lite to see what works best for scoring and grouping.
Meng: From a deployment standpoint, the real win they found was that their method beats standard pruning by about six point three percent in accuracy, even when you do that distillation step at a faster wall clock speed. That’s tangible performance gain.
Lalam: The specific recipe they landed on is scoring with this diversity-aware method called DO-ACP, keeping exactly the top k experts per layer, and using forward KL divergence for the distillation process.
Tom: So what does this actually mean for us? It gives us a way to make these massive MoE models more manageable without losing too much smarts. The naive question is, can we really get that accuracy back when we force it into a dense structure?
Jane: The answer is yes, they show it works. But there are caveats. They note that the benefit of their diversity-aware scoring drops off when you start with smaller expert groups.
Lu: And another thing they flag is that if you choose to merge experts—keeping more than k—the framework ties those merge weights too closely to the selection scores, which they think could be improved later.
Meng: So it’s a good path forward for production if we can nail that DO-ACP scoring. But how robust is this conversion method when you move beyond just comparing models?
Tom: We've seen its success across Qwen and DeepSeek-V2-Lite, but the paper does show that the advantage over just randomly setting up a dense network is smaller on GPT OSS 20B compared to Qwen <ref:2605.28207#pg1>.
Jane: So the takeaway is a recipe: use DO-ACP scoring, stick to exactly top k experts per layer, and use forward KL distillation for quality recovery. But we need to keep an eye on those scaling limitations as we try this out at scale.
Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho
KRAFTON AI · KAIST
cs.CL, cs.AI, cs.LG
Submitted: 2026-05-27
Updated: 2026-11-10
Comments: 44 pages, 8 figures, 28 tables. Accepted at NeurIPS 2026 (Oral)
Code: https://github.com/krafton-ai/moe-to-dense
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 95/100
The gist: Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory…
Key concepts
- Mixture-of-Experts (MoE)
- MoE is a model architecture where different 'experts' are trained to handle different types of data or tasks. During inference, only a few experts are activated for any given input, making the model efficient but requiring all expert parameters to be loaded into memory.
- Diversity-aware Scoring (DO-ACP)
- This is a novel method used to score experts by considering both their individual importance and how different they are from each other. It aims to select experts that maximize both quality and variety, leading to better overall model performance than simpler scoring techniques.
- Knowledge Distillation
- This process involves training a smaller 'student' model to mimic the behavior of a larger, more complex 'teacher' model. In this study, it is used after converting the MoE to dense form to recover quality lost during the conversion process.
Terminology
Summary
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment The first systematic framework for converting a trained MoE into a standard fully dense architecture involves experts being scored, selected, and grouped into a dense Fully Connected Network (FFN) and refined by knowledge distillation from the MoE teacher This work introduces the first end-to-end study of converting a trained MoE language model into a fully dense architecture
Systematic Framework and Evaluation
The authors present the first systematic framework for converting a trained MoE language model into a standard fully dense architecture through three steps: experts are scored by importance, the top-scoring experts are selected, grouped and merged within groups, and then concatenated into a standard dense FFN with down-projection matrices scaled to preserve output magnitude They evaluate 7 scoring methods, 5 grouping methods, and 2 magnitude scaling options across a range of selected expert counts on Qwen3-30BA3B, yielding 350 configurations The key finding is that the choice of scoring method is the most impactful decision, with the novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ∼4B-token distillation at 1. p. 0 Under a controlled comparison at matched parameter count,
MoE-to-dense outperforms dense-to-dense pruning by +6. 3 pp in average downstream accuracy after
∼4B-token distillation at 1.Where the paper says it
Expert Selection and Diversity
The authors introduce a diversity-aware expert selection criterion based on the Gram log-determinant that jointly maximizes expert importance and mutual diversity This criterion, called DO-ACP, achieves the best accuracy across all configurations and all three models The authors show that selecting diverse experts without merging is the best strategy, as frequency-based scoring selects redundant experts and benefits from merging The D-Optimal selection (DO) criterion maximizes the log-determinant of an importance-weighted kernel, which captures both individual quality and pairwise redundancy Theoretical results prove that the greedy D-Optimal selection is near-optimal with a standard greedy guarantee
Performance Comparison and Distillation
The MoE-to-dense pipeline outperforms dense-to-dense pruning by +6.3 pp after ∼4B-token distillation at 1. p. 0 Under a controlled comparison at matched parameter count,
MoE-to-dense outperforms dense-to-dense pruning by +6. 3 pp in average downstream accuracy after
∼4B-token distillation at 1. 3 pp in average downstream accuracy after
∼4B-token distillation at 1. 6× faster training wall-clock speed.Where the paper says it
Where the paper says it
Cross-Model Validation and Generalization
The findings are validated across DeepSeek-V2-Lite and GPT-OSS-20B, showing that the advantage of diversity-aware scoring diminishes with smaller expert pools The best configuration uses pure pruning (K=k) on all three models, confirming that this strategy is robust across architectures The advantage of MoE-to-dense over random FFN initialization on GPT-OSS (+3.3 pp with 32 experts) is smaller than on Qwen3 (+10. p. 1 Pruning and Distilling Mixture-of-Experts into Dense Language Models
Table 1 MoE compression methods and their design choices. Prior work produces smaller MoE models, while
ours is the first to target a fully dense architecture. Pruning and Distilling Mixture-of-Experts into Dense Language Models
Table 1 MoE compression methods and their design choices. Prior work produces smaller MoE models, while
ours is the first to target a fully dense architecture.Where the paper says it
Where the paper says it
Conclusion and Recipe
The paper concludes that the best recipe is scoring with DO-ACP, retaining exactly the top-k experts per layer, and distilling with forward KL This approach achieves an average accuracy of 43.41% on Qwen3-30B-A3B The overall results point to a simple recipe: scoring with DO-ACP, retaining exactly the top-k experts per layer, and distilling with forward KL
Limitations
The limitations include the fact that when merging is used (K > k), the framework ties merge weights to selection scores, and decoupling these may improve such configurations The benefit of our method over random FFN initialization on GPT-OSS (+3.3 pp with 32 experts) is smaller than on Qwen3 (+10.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Systematic framework for converting trained Mixture-of-Experts into a standard fully dense architecture by scoring experts, selecting, grouping, and concatenating them into a dense FFN with appropriate magnitude scaling. This allows for memory-constrained deployment without the fundamental limitation of loading all expert parameters.
-
Diversity aware expert selection via Gram log determinant (DO-ACP) which jointly captures expert importance and mutual diversity to maximize the log determinant of an importance weighted kernel, achieving the best accuracy across all configurations and models evaluated.
-
MoE-to-dense conversion outperforms dense-to-dense pruning by plus 6.3 pp in average downstream accuracy after approximately 4B token distillation at 1.6x faster training wall clock speed, suggesting a more efficient initialization strategy for subsequent knowledge distillation steps compared to dense teacher pruning.
-
Knowledge distillation from the MoE teacher recovers quality lost during compression, and testing expanded teacher routing to activate k prime greater than k experts improves quality by plus 0.70 pp on Qwen3-30B-A3B, providing a way to leverage non activated experts for richer knowledge transfer.
-
The system can achieve the lowest total catastrophic failure rate (54.5%) while maintaining the lowest knowledge error rate (4.2%) on MMLU chain of thought zero shot reasoning, indicating that diversity aware expert selection preserves factual knowledge more effectively than dense-to-dense pruning or frequency based expert selection.
-
The pipeline is robust across different MoE architectures, including Qwen3-30B-A3B (128 experts), DeepSeek-V2-Lite (64+2 experts), and GPT-OSS-20B (32 experts), demonstrating that the best configuration uses pure pruning with exactly K equal to k.
-
The system demonstrates superior performance on knowledge intensive tasks, reaching 49.2% accuracy on humanities compared to 18.5% for dense-to-dense pruning and 19.2% for frequency based scoring, suggesting that diversity aware expert selection captures richer factual and cultural knowledge more effectively than other methods.
Abstract
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after 4B-token distillation at 1.6x faster training wall-clock speed.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
- Task-Specific Expert Pruning for Sparse Mixture-of-Experts
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Distilling the Knowledge in a Neural Network
- REAM: Merging Improves Pruning of Experts in LLMs
- Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
- Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
- REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
- Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
- SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
- Ministral 3
- Pointer Sentinel Mixture Models
- MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
- Compact Language Models via Pruning and Knowledge Distillation
- Expert Merging in Sparse Mixture of Experts with Nash Bargaining
- GLU Variants Improve Transformer
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- LLM Pruning and Distillation in Practice: The Minitron Approach
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data