How to Score Experts for One-Shot MoE Expert Pruning: A Systematic Formulation and Selection Principle

summary

The gist

Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool…

In short

The paper proposes a unified mathematical formulation for one-shot pruning of Mixture-of-Experts (MoE) models based on three factors: routing frequency, gate weighting, and activation strength. This leads to a selection principle: task-agnostic pruning should use averaged, gate-free activation scores, while task-specific pruning can retain routing and gate information when calibration data matches the target capability.

Key concepts

Unified Scoring Formulation S(b, α, β)
This is a mathematical formula used to score experts for removal. It balances three variables: b (routing frequency retention), α (gate weight exponent), and β (activation strength exponent). By adjusting these parameters, the formulation allows researchers to systematically choose which expert pruning criteria are best for their specific deployment goal.
Task-Agnostic Pruning
This approach focuses on selecting experts based on criteria that are consistent across different tasks. The paper suggests favoring routing-token-averaged, gate-free activation scores in this setting. This method aims to be robust regardless of the specific task being performed.
Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN)
These are two new, strong task-agnostic criteria identified by the study. They measure the norm of expert activations across routed tokens. These metrics consistently performed well, achieving top-two average ranks and improving overall model performance by up to 8.8 percentage points over existing baselines.

Terminology used across episodes

This episode discusses

The paper

How to Score Experts for One-Shot MoE Expert Pruning: A Systematic Formulation and Selection Principle · Read on arXiv

Zongfang Liu, Jinghui Zhang, Zijian Ma, Guangyi Chen, †Xin Yuan

Zhejiang University · Westlake University · –Mohamed bin Zayed University of Artificial Intelligence Institute of Artificial Intelligence Research Center at Carnegie Mellon University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "How to Score Experts for One-Shot MoE Expert Pruning".

Jane: Here's the gist of the paper. Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage The unified formulation provides a systematic view for choosing expert-pruning scoring criteria under different deployment objectives Pruning redundant reasoning tokens The paper introduces a unified formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength This formulation yields a criteria selection principle: task-agnostic pruning should favor routedtoken-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gateweight information Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN) Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the taskagnostic setting, obtain the top-two average ranks, and improve average performance by up to 8.8 points over the strongest baseline How the training signal is built The damage measure for expert removal is defined by separating the output change into a removed-expert contribution and a rerouting residual A scalar damage score can be obtained by measuring the norm of the output perturbation, which is written as Dbj:= PMt=1 1

j ∈ Et: gj,t∥fj,t∥2 The unified scoring formulation is defined as Sj (b, α, β):= 1/Nb j PM t=1 1

j ∈ Et: gα j,t∥fj,t∥β 2 Here b determines whether routing frequency is retained and α and β are exponents for the gate weight and activation strength, respectively Where the gains come from The principle derived suggests that task-agnostic pruning should favor gatefree activation scores averaged over routed tokens, while task-specific pruning can benefit from retaining routing-frequency and gate-weight information The Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN) are identified as the strongest overall choices in the taskagnostic setting, achieving the top-two average ranks and improving average performance by up to 8.8 percentage points over the strongest prior baseline Task-specific pruning benefits from retaining routing-frequency, gate magnitude, and activation strength jointly when calibration data are aligned with the target capability Model performance comparison Experiments on four MoE models and 16 diverse benchmarks show that the proposed criteria deliver more balanced overall performance On Qwen3, MAN/MSAN improve the overall average while also giving the strongest or near-strongest results on coding, math, and multiple-choice evaluation The loading memory analysis shows that at a 50% pruning ratio, expert pruning reduces loading memory by ∼50% across the evaluated MoE models This validates it as a practical way to address memory usage, which is the main bottleneck in MoE deployment Limitations The unified formulation covers commonly used one-shot expert-pruning criteria including Frequency, SEER, EAN, and REAP However, not all criteria can be represented in the same form; for example, MoNE uses routing-frequency-weighted activation variance across routed tokens which is not fully captured by S(b, α, β) Furthermore, the study focuses on expert pruning where experts are removed according to importance scores rather than expert merging The principle suggests that routing-related signals introduce calibration dependence and should therefore be suppressed for taskagnostic pruning while retained when the calibration data are target-aligned The unified formulation provides a compact and empirically supported basis for choosing one-shot expert-pruning criteria under different deployment requirements The paper concludes that the unified formulation decomposes scoring criteria into routing frequency, gate weighting, and activation strength This formulation exposes a criteria selection principle: task-agnostic pruning should suppress calibration-specific routing effects by using routed-token-averaged, gate-free activation criteria, whereas task-specific pruning can benefit from routing-frequency and gateweight signals when calibration data are aligned with the target capability This perspective explains why routing-only criteria such as Frequency and SEER are sensitive to calibration distributions Overall, our results provide a compact and empirically supported basis for choosing one-shot expert-pruning criteria under different deployment requirements while keeping the resulting method simple and broadly applicable in practice References

Title and authors: Tom: So we're looking at this paper about how to score experts for one-shot MoE expert pruning. It’s called "How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle".

Jane: What does that actually mean for someone who just hears the name?

Tom: Basically, they've taken all those different ways people score experts—like frequency or gate weighting—and put them into one single mathematical framework. It’s trying to figure out which scoring rules are best depending on what you actually want to do with your model when you prune it.

Lu: The core idea is that one-shot pruning is a trade-off, so different experts might be important for different tasks, and this formulation helps decide how to balance those needs.

Meng: But if we have to store the full expert pool just to use it, we need a good way to cut things down without losing too much performance. So what's the main problem they are trying to solve here?

Jane: Well, the authors are saying that instead of just picking one specific scoring rule, there isn't one universal best answer for pruning experts. They introduce three core factors: routing frequency, gate weighting, and activation strength—the paper calls this their unified formulation.

Tom: Right. So they aren't just proposing a new number; they’re proposing a system for picking the right number based on whether you care more about how often an expert is used or how confident the router is when it picks that expert.

Lu: They suggest that if you want to prune in a way that works across many different tasks, you should favor criteria based on averaged activation scores from routed tokens, without looking at the gate weight directly.

Jane: That sounds like a practical choice for deployment where you don't know the exact task beforehand. But what about when we *do* know the task?

Tom: When you know the task, they say you can be smarter and keep information about how often an expert is routed and how strongly the gate weights are involved. That’s a different way to look at expert importance.

Meng: So if I'm deploying this on a production server, what does that mean for my cost? Does this unified formulation help me choose a pruning method that saves the most memory without tanking the performance?

Jane: It does. They found two new task agnostic criteria, Mean Activation Norm and Mean Squared Activation Norm, and these were consistently very strong in the task agnostic setting. They even showed that these two criteria improved average performance by up to eight point eight over their strongest baseline across four different models and sixteen different benchmarks.

Tom: Eight point eight is a solid number when you’re talking about performance improvement on coding, math, and multiple-choice tasks specifically. That’s not just theoretical; they tested it against real capabilities.

Title and authors: Lu: The damage measure they use separates the output change into what the removed expert contributed and what the rerouting residual is, which gives them a way to calculate a scalar damage score using exponents alpha and beta for gate weight and activation strength.

Jane: That’s how they define their scoring function, S(b, alpha, beta), which lets you control how much weight you put on routing frequency versus those other factors when choosing your pruning metric.

Meng: I'm a bit concerned about the calibration dependence they mentioned earlier. If task agnostic pruning suppresses routing-related signals, does that mean we lose some of the fine-grained control we usually have over which experts to keep for specific scenarios?

Tom: That’s the trade-off they are highlighting. If you use routing frequency criteria alone, it gets too sensitive to how the model is calibrated on its training data. The paper suggests that when your calibration data actually matches your target capability, keeping those routing signals can be beneficial for task-specific pruning.

Lu: So the principle they derive is this: task agnostic pruning should use averaged activation scores from routed tokens without gate weight, while task specific pruning benefits from keeping both routing frequency and gate weight information when the calibration data aligns with the target capability.

Jane: That gives you a clear direction for your decision-making. It moves away from just picking one heuristic and toward selecting criteria based on your deployment goals.

Tom: So, to wrap up this part, this paper on "How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle" boils down to giving you a systematic way to choose your pruning scoring criteria. It separates things into routing frequency, gate weighting, and activation strength.

Lu: The key result they found is that task agnostic pruning should favor criteria based on routed token averaged, gate free activation scores.

Jane: And for those who are doing task specific pruning, they can benefit from retaining routing frequency and gate weight information when the calibration data is aligned with the target capability.

Meng: From my side, it means we have a concrete way to reduce memory by about fifty percent at a fifty percent pruning ratio across all these models, which validates this approach as practical for deployment.

Tom: Exactly. The Mean Activation Norm and Mean Squared Activation Norm are shown to be the strongest task agnostic choices, improving performance by up to eight point eight percentage points over the strongest baseline they tested.

Lu: The paper also notes a limitation: not every existing pruning criterion can be represented in that same unified form, for instance, MoNE uses routing frequency weighted activation variance across routed tokens which isn't fully captured by their formula.

Jane: And another thing they pointed out is that the principle suggests suppressing routing-related signals for task agnostic pruning because those signals introduce calibration dependence.

Tom: So the whole point is a compact and empirically supported basis for choosing your one-shot expert pruning criteria under different deployment requirements, which should keep things simple and applicable in practice.

The paper's summary: Tom: So, what we're looking at now is just how they put all those different ways of scoring experts into one big mathematical framework, a unified formulation for one-shot MoE expert pruning.

Jane: That means instead of picking just one rule, they’ve built a system that lets you choose the best rule based on your specific goal for deploying the model.

Tom: Exactly. The main takeaway is this selection principle: if you want to prune experts across different tasks without knowing which task you're doing, you should favor criteria based on averaged activation scores from routed tokens, completely ignoring the gate weight information.

Lu: That’s because those routing-related signals cause calibration issues, and averaging them out makes the pruning method more robust when it's task agnostic.

Meng: But if you actually know which task you’re running—say, a specific coding benchmark—then keeping information about how often an expert is routed and how much weight the router gives it can actually help.

Jane: Right, so for those specific tasks, you get to keep the routing frequency and gate weight data in your scoring, provided your calibration data matches that target capability.

Tom: And they found two new criteria they call Mean Activation Norm and Mean Squared Activation Norm that are consistently strong when you're just doing task agnostic pruning.

Lu: Those two, MAN and MSAN, they actually improve the average performance by up to eight point eight percentage points over the strongest baseline across four different MoE models and sixteen different benchmarks.

Meng: Eight point eight is a solid improvement when you’re talking about real performance gains on coding and math tasks specifically. That’s not just theoretical; it shows how this method can actually help reduce the memory footprint by about fifty percent at a fifty percent pruning ratio across all these models.

Jane: So, for someone listening who doesn't read papers, it means they can stop guessing which expert to cut and start using a system that tells them exactly what to look for based on whether they are being general or specific with their task.

Tom: It’s about making the decision process systematic instead of just relying on some old heuristic like the simple frequency score alone.

Lu: The limitation they pointed out is that not every existing scoring method can be put into this exact unified form; for example, one method uses routing frequency weighted activation variance across routed tokens, and their formula doesn't quite capture that specific structure.

Jane: And they also flagged that because routing signals introduce calibration dependence, you have to be careful—you shouldn't keep those signals if you’re doing task agnostic pruning.

Tom: So the whole point is a compact way to choose your one-shot expert pruning criteria based on whether you prioritize general performance or task-specific accuracy when dealing with memory constraints.

Jane: It gives us a clear path for choosing which way to prune, depending on what we need from the model in deployment.

The paper's improvements: Tom: So, what we’re hearing now is how they actually suggest improving existing scoring methods for this pruning task using their unified formulation.

Jane: They take all these old rules—like Frequency or SEER—and show how the new scoring function, S of b alpha beta, lets you control exactly how much weight you put on routing frequency versus gate weighting and activation strength.

Lu: The real improvement is that this framework gives a clear rule for selection: if your goal is general performance across many tasks, use averaged activation scores from routed tokens without looking at the gate weight at all.

Meng: That’s nice because it means you don't have to manually tweak dozens of scoring parameters; you just pick the right formula based on whether you need task-agnostic or task-specific results.

Jane: And for those specific tasks, they say you can actually keep routing frequency and gate weight information if your calibration data lines up with that target capability.

Tom: That means when you’re doing something very specialized, like complex math evaluation, you get to use richer signals to make a smarter cut.

Lu: They also identified Mean Activation Norm and Mean Squared Activation Norm as the strongest overall choices for the general task agnostic setting, and those metrics improved performance by up to eight point eight percentage points compared to the best prior baseline.

Meng: Eight point eight is a solid number when you’re talking about real performance gains on coding and math tasks specifically. That shows this method isn't just theoretical; it actually helps your model perform better under resource constraints.

Jane: So, what this changes for people listening is that they get a way to choose pruning criteria that are explicitly designed for the deployment environment you’re working in, whether you want broad applicability or deep task-specific optimization.

Tom: It moves us away from just using one heuristic and toward a systematic decision process based on three measurable factors: frequency, gate weight, and activation strength.

Lu: The paper also makes it clear that routing signals introduce calibration dependence, so for general pruning, you should suppress those signals to keep things stable.

Jane: So the big improvement is getting this compact mathematical basis for choosing one-shot pruning criteria under whatever deployment requirements you have in mind.

Conclusion: Tom: So, to wrap up on "How to Score Experts for One-Shot MoE Expert Pruning: A Systematic Formulation and Selection Principle," they’ve given us a unified system for picking pruning rules based on routing frequency, gate weighting, and activation strength.

Jane: That means they’ve successfully shown how to systematically choose which scoring criteria to use depending on whether you need general pruning or task-specific pruning.

Lu: The core idea is that task agnostic work should favor averaged activation scores from routed tokens without looking at the gate weight directly, because routing signals cause calibration problems.

Meng: For deployment, this means we have a compact and empirically supported basis for choosing one-shot expert pruning criteria under different requirements.

Jane: It’s a practical tool that lets us reduce memory usage without totally losing performance across those various MoE models they tested on sixteen different benchmarks.

Tom: And they showed that the Mean Activation Norm and Mean Squared Activation Norm are the strongest task agnostic choices, actually improving average performance by up to eight point eight percentage points over the prior strongest baseline.

Lu: They also flagged a limitation: not every existing pruning criterion fits into their unified form perfectly, like MoNE’s variance score.

Meng: That’s important because it tells us that if you use a method they didn't design for, you might have to do extra work to fit it into their framework.

Jane: And they cautioned that routing-related signals introduce calibration dependence, so we have to be careful not to keep those signals when doing general pruning.

Tom: It boils down to this: task agnostic pruning suppresses those routing effects for stability, while task specific pruning can retain them if your calibration data aligns with the target capability.

Lu: So, it’s a compact and empirically supported way to select the right one-shot expert-pruning criteria based on your deployment needs.

Jane: That gives us a concrete starting point for deciding how we want to manage the memory bottleneck in AI systems.

Tom: It’s interesting how they managed to distill so much complexity into three core factors—routing frequency, gate weight, and activation strength—into one usable structure.

Lu: It shows that even when dealing with sparse models, there’s still a systematic view for making those hard trade-offs.

Meng: For production engineering, it means we can finally select a pruning method with confidence instead of just picking the one that looks best on a paper.

Jane: Exactly. It moves us toward using these criteria to make deployment decisions based on clear objectives rather than guesswork.

Tom: So that’s our take on "How to Score Experts for One-Shot MoE Expert Pruning: A Systematic Formulation and Selection Principle."

Lu: We’re looking forward to seeing how this formulation helps when we start tackling more complex, multi-turn reasoning tasks where expert selection needs to be even more dynamic.

More episodes

← Home