FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment

arXiv:2606.27866 · cs.LG · Submitted 2026-06-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment".

Paper summary: Tom: So, wrapping up this talk on FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment. The paper is about taking a single large pretrained MoE model and learning a family of deployable subnetworks across different budgets without needing separate checkpoints for each one.

Jane: They achieved this by ranking expert channels, letting each expert learn a discrete pruning action based on an increasing cost function during training, and using an optional mid budget fine tuning stage to create a shared set of weights that can be transferred between budget settings.

Lu: The core contribution is the importance-ordered FFN layout which creates a shared nested weight space that enables learning those coherent family action masks <ref:2606.27866#pg3>. This means it learns structured allocation rather than just one global prune ratio.

Meng: For deployment, this translates to being able to serve the largest possible model at any given time while still respecting the specific constraints of the serving environment, which simplifies our infrastructure a lot by reducing checkpoint management.

Tom: And they showed that on Qwen2-57B-A14B, they retain about ninety nine point eight percent performance at fifty percent pruning budget and about ninety two point nine percent at eighty percent budget <ref:2606.27866#pg3>. That's a solid number for flexible deployment.

Jane: The implication is that instead of being locked into one compression level, we can have a single foundation that adapts its structure to the demand, which is what this paper points toward for future system level co design and dynamic budget adaptation strategies <ref:2606.27866#pg1>.

Conclusion: Tom: So we're wrapping up on FlexMoE, which is about learning one pruning strategy that works across many different model sizes or budgets.

Jane: That's right, it’s this framework that takes a single large model and lets you create a family of deployable subnetworks instead of just one fixed version.

Lu: The authors are looking at how to make MoE models more flexible for real deployment by linking the pruning action to different budget constraints.

Meng: From an engineering standpoint, the big deal here is that you don't need a whole new training run every time you want a smaller or larger model; you just change one setting.

Tom: Exactly, it’s about making deployment much smoother by learning those structured cuts instead of just guessing where to prune.

Jane: The paper title, FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment, really gets to the heart of what they did.

Lu: They’re using ranking and then letting each expert learn a discrete pruning action based on an increasing cost function during training.

Meng: That part about the importance-ordered FFN layout is key; it sets up this shared weight space so all those different budget settings can work together coherently.

Tom: And they showed that this method beats current MoE compression baselines, maintaining high performance even at significant pruning levels like fifty percent or eighty percent.

Jane: It means you get a much more adaptable model for serving, and we need to see how this flexibility translates into real-world inference costs across different deployment scenarios.

Meng: That’s the million-dollar question for me; if it works across budgets, what does that mean for the cost of running models on edge devices or in data centers?

Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You

National University of Singapore

cs.LG

Submitted: 2026-06-26

Updated: 2026-11-18

Comments: Accepted at NeurIPS 2026 (poster)

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models The authors study a different setting…

Key concepts

Expert FFN Channels Ranking
The method ranks the Feed-Forward Network (FFN) channels within each expert based on estimated importance using a first-order Taylor saliency score. This ranking is crucial because it establishes an importance order that guides the subsequent pruning process, ensuring that more critical parts of the model are preserved during budget reduction.
Discrete Slice Action Learning
Each expert learns a specific retention ratio from a predefined set of discrete options using Straight-Through Gumbel-Softmax. This allows the model to decide which channels to keep or prune, optimizing a balance between maintaining high quality and minimizing computational cost during training.
Nested Expert Weight Space
By ordering the expert FFN layout based on importance scores, FlexMoE creates a shared nested weight space. This structure enables the system to learn a coherent family of budget-specific action masks, meaning the learned pruning strategy is structured across different resource constraints.
Kernel Co-design
A customized kernel co-design optimizes deployment by grouping experts based on their retained width and using grouped GEMMs. This mitigates performance bottlenecks caused by fragmented small-shape execution during inference, making budget switching feasible at runtime.

Terminology

Summary

Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models The authors study a different setting: converting a large pretrained MoE LLM into a nested family of deployable subnetworks across budgets Our method first ranks expert FFN channels by their importance, then lets each expert learn a discrete action to prune its channels Overall, our framework surpasses recent MoE compression baselines. p. 0

FlexMoE Framework and Pipeline

The FlexMoE framework converts one pretrained MoE model into a family of materializable subnetworks across budgets The pipeline first reranks expert FFN channels by estimated importance to enable top-retained channel slicing Then each expert learns a retention ratio from a predefined discrete ratio set continuously in a training loop with increasing pruning pressure, allowing a single training run to yield a family of action masks across multiple budgets Optionally, we further perform a single mid-budget fine-tuning stage to recover degraded model quality and transfer the recovered model to other unseen budgets This results in a “train-once, deploy-many” pruning pipeline.

How the training signal is built

The method involves several steps to generate the action masks and budget ratios 1) Expert FFN channels are ranked by estimating a first-order Taylor saliency, which is computed on a small ranking set Brank 2) A permutation matrix Pl,e is obtained by sorting the estimated saliency scores in descending order to reorder the expert FFN hidden dimension 3) Each expert learns a discrete slice action from a predefined set A = (r1,..., rK) using Straight-Through Gumbel-Softmax [Jang et al. p. 4, 2017] The optimization objective is Laction = Lqual + λcost(t) C(p, q) − β(t) H(p), which balances quality preservation and cost The action logits are initialized to the thickest action (full model retained), and during optimization, we gradually increase λcost(t) so that actions under training will move from weaker to stronger channel pruning. p. 4

Where the gains come from

The gains stem from several architectural and deployment optimizations 1) The importance-ordered FFN layout yields a shared nested expert weight space, which is crucial for learning a coherent family of budget-specific action masks 2) A customized kernel co-design mitigates bottlenecks by bucket experts by retained width and use grouped GEMMs to reduce per-expert fragmented small-shape execution 3) The single mid-budget fine-tuning stage transfers well to unseen higher and lower budgets, providing a robust shared recovered model

Performance and Deployment Analysis

Experimental results show that FlexMoE surpasses strong MoE compression baselines while offering a substantially more flexible deployment On Qwen2-57B-A14B, the pruned subnet retains about 99.8% of the base performance at 50% pruning budget and still preserves about 92. p. 19% at 80% budget The offline pruned subnetworks deliver real throughput gains at their fixed budget, and a kernelized co-design makes runtime budget switching feasible. p. 8 The middle budget (40%) recovery point provides the best global trade-off between coverage and specialization across the whole budget family The learned action distributions are clearly non-uniform across depth, indicating that FlexMoE does not merely recover a global prune ratio, but learns structured budget allocation. p. 7

Limitations and Future Works

While our main focus is MoE structure search and static expert parameter pruning, the multi-budget shared-weight family produced by FlexMoE opens a promising direction for future work on stronger system-level co-design and dynamic budget-adaptation strategies for deployment and online serving The kernelized co-design implementation details show that the reordered layout allows obtaining all requested weights by performing only 1 direct prefix slicing. This reduces the online scheduling cost from repeated tensor assembly The authors hope this work provides a new perspective of MoE model structure search and a practical foundation for budget-adaptive MoE model deployment and inference. p. 8

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Budget-adaptive model deployment: The system can convert a large pretrained Mixture-of-Experts (MoE) LLM into a family of materializable subnetworks across various budgets without maintaining separate compressed checkpoints for each target budget.

  2. One for all pruning pipeline: A single action training run generates a series of action masks from high to low budgets, enabling the system to prune 50% of routed expert parameters on Qwen2-57B-A14B while retaining approximately 99.8% of base performance without fine-tuning.

  3. Real-time online budget adjustment: By using a kernelized co-design, the system supports online clipping where the server keeps the channel-ranked full base model in memory and adjusts the inference budget at runtime by specifying budget-specific action masks, allowing for real throughput gains while maintaining service continuity.

  4. Shared weight recovery across budgets: A single mid-budget recovery fine-tune produces a shared recovered set of weights that is transferable to unseen higher and lower budgets, enabling train-once, deploy-many pruning pipelines where the recovered model is adaptable to all masks across upstream and downstream budgets.

  5. Structured expert specialization capture: The system learns discrete slice actions per expert by optimizing an action loss function that includes a quality preservation term and a cost objective C(p, q), resulting in learned action distributions that show layer-wise and expert-wise structure rather than random budget convergence.

Sources

Related papers