Prune, Don't Merge: Expert Compression of Mixture-of-Experts LLMs on Multilingual Production Tasks
cs.CL, cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Comments: 38 pages, 8 figures, 15 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Mixture-of-Experts language models are cheap to run per token but expensive to hold in GPU memory, which drives the cost of serving them in production.
Terminology
Abstract
Mixture-of-Experts language models are cheap to run per token but expensive to hold in GPU memory, which drives the cost of serving them in production. One-shot expert compression shrinks them after a single calibration pass. Its two current state-of-the-art methods are REAP for expert pruning, which deletes the least salient experts, and REAM for expert merging, which merges them into the most salient ones. Both have been evaluated only on academic benchmarks. We compare the two on five multilingual production tasks of a web-hosting company, from classification to the generation of marketing HTML email templates, with two 30B-parameter models (Qwen3-30B-A3B-Instruct-2507 and the hybrid Mamba-attention Nemotron-3.5-Lightning) at 25% and 50% of experts removed and the same calibration data. Pruning wins: on the same rows, REAP is significantly better in 16 of 20 comparisons and REAM in 2. At 25%, REAP keeps 93.6% of Qwen3-30B's generative quality and REAM 62.5%, although on benchmarks the two are within a point for the same model. The damage concentrates in the output contract rather than the content: 89.8-95.2% of each significant loss on email templates comes from hard-gate failures such as invalid JSON, broken mobile layouts and runaway generation, and most failed non-English domain descriptions copy the prompt's English opening phrase. Cross-perplexity and logit divergence, two diagnostics from the REAP paper, also track production quality, as does the rate of repetition loops; n-gram diversity does not. We close with practical guidance for compressing MoE models for production.
Sources
- What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning
- Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems
- Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
- Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
- Accuracy is Not All You Need
- PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
- Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
- REAM: Merging Improves Pruning of Experts in LLMs
- Beware of Calibration Data for Pruning Large Language Models
- REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
- FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
- FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
- Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction
- How Does Quantization Affect Multilingual LLMs?
- ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models
- Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data