ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

summary

The gist

In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR…

In short

ELDR introduces an expert-locality-aware decode router for prefill–decode disaggregated Mixture of Experts (MoE) serving. It uses a compact 'expert signature' derived from prefill activations to predict which experts requests will use during decoding. This allows requests with similar patterns to share experts, reducing per-step activation and improving performance across different batch sizes.

Key concepts

Expert Signature
This is a compact representation of a request’s prefill activations, summarizing how many tokens each expert received. The signature is designed so that the distance between two signatures predicts the overlap in expert usage during decoding.
Expert Locality
This concept exploits the correlation between related requests. The paper argues that experts used by similar requests are likely to be used together during decoding, meaning locality is a predictable axis for routing decisions.
Prefix Cache Coherence
ELDR maintains an expert-signature cache synchronized with the KV cache at block granularity. This ensures that the request's signature remains accurate regardless of whether it's a partial hit or a full eviction, minimizing overhead.
Locality-Band Routing
The online routing strategy uses the expert signature to select decode workers. It favors workers whose precomputed centroids are close to the request's best match (locality) while falling back to load balancing if multiple workers are similar.

Terminology used across episodes

This episode discusses

The paper

ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving · Read on arXiv

Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon

KAIST · Microsoft Research · Shanghai Xingyunzhili Artificial Intelligence Institute

Equal load does not mean equal decode latency in prefill-decode (PD) disaggregated mixture-of-experts (MoE) serving. In bandwidth-bound decode, more distinct experts mean more weight loads and higher latency. We present ELDR, a lossless, expert-locality-aware decode router. Prefill signatures predict decode expert usage. Balanced K-means fits per-decoder centroids with background refitting. Locality-band routing selects the least-loaded worker among close matches. A block-granular cache keeps signatures exact under prefix caching. We evaluate ELDR in vLLM on up to 40 GPUs. Across three MoE models on 12 public task benchmarks and multilingual WildChat, it reduces median time-per-output-token (TPOT) by 5.4-14.5% on average across request rates and up to 17.8% over the best of four load-balancing baselines, preserving model outputs and token-to-expert assignments.

DOI: 10.1145/3828523.3857077

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving".

Jane: Here's the gist of the paper. In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving The gap is large: on Qwen3-30B-A3B, growing the active-expert count from 16 to 128 raises MoE-layer latency 4.7× at fixed batch size, while batch size at fixed active-expert count barely moves it Expert locality is therefore not only structured but visible at exactly the moment the router must act Expert Locality as a Predictable Routing Axis ELDR turns each request’s prefill activations into a compact expert signature whose distances predict decode-time expert overlap, so requests with nearby signatures share experts at decode This approach exploits the fact that expert selection is correlated across related requests, not just sparse within a token The prefill phase has already pushed the prompt through the same gates that will route decode, and the two agree Expert locality is therefore not only structured but visible at exactly the moment the router must act Where Gains Come From The primary mechanism for performance improvement is reducing the number of distinct experts activated per decode step Fig.

Tom: 4 quantifies this: same-domain batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches across batch sizes 32–128 ELDR reduces the perstep active-expert count by 22.0% on average across decode batch sizes compared to RR This reduction is achieved by grouping requests whose signatures are close, so that the distinct experts activated across a group are fewer than under naive, similarity-agnostic routing How the Training Signal is Built The expert signature summarizes each request’s prefill-time expert activations into a compact per-request representation, the expert signature The raw material is cheap to collect at prefill: a per-layer histogram of how many tokens each expert receives, available as a by-product of routing The signature is designed so that the distance between two signatures predicts how much the two requests overlap in their decodetime expert usage Signature quality is then the rank correlation between signature pair-distance and decode-pattern pair-distance, over random calibration pairs Pruning Redundant Reasoning Tokens The routing decision is split along an information boundary where locality, which experts colocated requests share, is an aggregate property visible only across many requests; load, which decode’s non-MoE work imposes per colocated request, is instantaneous and visible only at the decision ELDR splits the decision along that information boundary Offline, balanced K-means partitions signature space into one locality region per decode worker, capturing structure without inheriting workload skew Online, locality-band routing keeps the workers whose centroids fall within a similarity band of the request’s best match and routes to the least loaded among them Prefix Cache Coherence ELDR keeps an expert-signature cache co-indexed with the KV cache at block granularity, which keeps signatures exact across partial hits, full hits, and evictions, so ELDR composes with prefix caching at negligible cost A request’s KV cache spans a block set B (r), so its signature is the sum over those blocks, sr = Í b∈ B (r) sig

b: , exact regardless of which request populated each block The signature cache inherits the KV cache’s lifecycle—a reused block resets its signature, a cached prefix carries its signature, and an eviction reclaims both stores together—so ELDR keeps no prefix-tree, eviction, or sharing state of its own Decode Clustering and Routing ELDR routes the request to one of K decode workers under two objectives: 1) Locality—colocated requests should share experts—is what the signature is built to enable 2) Load balance is the constraint routing cannot ignore Offline, ELDR produces one centroid per decode worker by clustering the calibration signatures into K groups such that signatures within a group are close (locality) and groups are balanced in size Online routing favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied Decode routing in PD-disaggregated MoE serving optimizes only decode-worker load ELDR adds a second axis, expert locality: a prefill-derived signature drives load-tolerant routing that stays coherent with the prefix cache, losslessly lowering median and tail TPOT across models, workloads, and topologies References

Paper summary: Tom: To wrap up on ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving, it adds a second axis to decode routing that focuses on expert locality derived from prefill activations <ref:2607.00466#pg1>.

Jane: Essentially, they take the structure of which experts are used during prefill and turn it into a compact signature that predicts how much overlap will happen in the decoding phase <ref:2607.00466#pg2>.

Lu: The key is that this signature is built from prefill data, which is cheap to collect at the beginning of the process, and they use that to partition decode workers offline based on locality regions <ref:2607.00466#pg2>.

Meng: For me, what's interesting is how they handle the online routing by favoring locality when a signature strongly points to one worker but falling back to load balancing if several workers are nearly tied <ref:2607.00466#pg1>.

Tom: The paper shows that this approach results in a twenty-two percent reduction in the per-step active-expert count on average compared to baseline methods, which is what drives the speedup <ref:2607.00466#pg3>.

Jane: So for someone just listening, it means that instead of just balancing how many requests are waiting at a worker right now, you're also telling the system where those requests belong based on how much they share in their expert usage <ref:2607.00466#pg2>.

Lu: It's about decoupling the routing decision; you have one axis for locality—which is aggregate and visible across many requests—and a second axis for load, which is instantaneous and only visible at the moment of the decision <ref:2607.00466#pg2>.

Tom: The authors point out that this combination allows them to achieve better performance across models, workloads, and topologies than systems that only focus on load balancing <ref:2607.00466#pg1>.

Jane: And one limitation they mention is that the locality routing can overload popular domains if you rely too heavily on it <ref:2607.00466#pg2>.

Meng: So, while it’s a solid improvement for latency, we still have to be careful about those domain-specific imbalances when deploying this at scale <ref:2607.00466#pg3>.

Tom: It’s an architecture that uses prefill data to predict decode behavior, and they show a measurable gain in reduced expert activation per step across different batch sizes <ref:2607.00466#pg3>.

Conclusion: Tom: So, we're wrapping up on ELDR, which is Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving. The main thing here is that they’ve added a second layer to how those decode workers pick their tasks.

Jane: That second layer is about expert locality, which means they look at how much experts are shared between requests during the prefill phase and use that to guide the decoding step. I was wondering what it actually does for someone just listening.

Lu: It turns every request into a compact signature based on those prefill activations. This signature predicts exactly how much overlap you'll have when it comes to using those experts during decoding.

Meng: From an engineering standpoint, this is cool because the core mechanism is grouping requests whose signatures are close together. That means fewer distinct experts get activated per decode step for a batch of related tasks.

Tom: And the numbers back that up—they show this reduces the active expert count by twenty-two percent on average across different batch sizes compared to previous routing methods.

Jane: So, when you think about how a system handles millions of requests at once, it’s not just about balancing the number of workers; it’s about making sure related tasks stay together on the same worker.

Lu: The paper shows this is achieved by splitting the decision: one axis for locality and one axis for load, which is what your specific worker's current workload is.

Meng: I gotta ask, what does that mean for deployment? Does it make things more predictable when you're running a massive system with thousands of different models?

Tom: It makes the routing decisions much smarter about where the work goes based on data similarity, not just blindly spreading requests out.

Jane: And this works even while keeping the prefix cache coherent, which is a big win because it keeps things fast without adding a lot of extra overhead to memory management.

Lu: The authors conclude that this method gives you load-tolerant routing that stays perfectly synchronized with the prefix cache, lowering those tail performance errors across different models and setups.

Meng: So it’s about getting better results without completely rewriting the whole serving stack, just adding this smart routing layer on top.

Tom: Exactly. It's a way to get more work done per step by being smarter about which experts are used together during the decode phase.

More episodes

← Home