ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

arXiv:2607.00466 · cs.DC · Submitted 2026-07-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving".

Jane: Here's the gist of the paper. In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving The gap is large: on Qwen3-30B-A3B, growing the active-expert count from 16 to 128 raises MoE-layer latency 4.7× at fixed batch size, while batch size at fixed active-expert count barely moves it Expert locality is therefore not only structured but visible at exactly the moment the router must act Expert Locality as a Predictable Routing Axis ELDR turns each request’s prefill activations into a compact expert signature whose distances predict decode-time expert overlap, so requests with nearby signatures share experts at decode This approach exploits the fact that expert selection is correlated across related requests, not just sparse within a token The prefill phase has already pushed the prompt through the same gates that will route decode, and the two agree Expert locality is therefore not only structured but visible at exactly the moment the router must act Where Gains Come From The primary mechanism for performance improvement is reducing the number of distinct experts activated per decode step Fig.

Tom: 4 quantifies this: same-domain batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches across batch sizes 32–128 ELDR reduces the perstep active-expert count by 22.0% on average across decode batch sizes compared to RR This reduction is achieved by grouping requests whose signatures are close, so that the distinct experts activated across a group are fewer than under naive, similarity-agnostic routing How the Training Signal is Built The expert signature summarizes each request’s prefill-time expert activations into a compact per-request representation, the expert signature The raw material is cheap to collect at prefill: a per-layer histogram of how many tokens each expert receives, available as a by-product of routing The signature is designed so that the distance between two signatures predicts how much the two requests overlap in their decodetime expert usage Signature quality is then the rank correlation between signature pair-distance and decode-pattern pair-distance, over random calibration pairs Pruning Redundant Reasoning Tokens The routing decision is split along an information boundary where locality, which experts colocated requests share, is an aggregate property visible only across many requests; load, which decode’s non-MoE work imposes per colocated request, is instantaneous and visible only at the decision ELDR splits the decision along that information boundary Offline, balanced K-means partitions signature space into one locality region per decode worker, capturing structure without inheriting workload skew Online, locality-band routing keeps the workers whose centroids fall within a similarity band of the request’s best match and routes to the least loaded among them Prefix Cache Coherence ELDR keeps an expert-signature cache co-indexed with the KV cache at block granularity, which keeps signatures exact across partial hits, full hits, and evictions, so ELDR composes with prefix caching at negligible cost A request’s KV cache spans a block set B (r), so its signature is the sum over those blocks, sr = Í b∈ B (r) sig

b: , exact regardless of which request populated each block The signature cache inherits the KV cache’s lifecycle—a reused block resets its signature, a cached prefix carries its signature, and an eviction reclaims both stores together—so ELDR keeps no prefix-tree, eviction, or sharing state of its own Decode Clustering and Routing ELDR routes the request to one of K decode workers under two objectives: 1) Locality—colocated requests should share experts—is what the signature is built to enable 2) Load balance is the constraint routing cannot ignore Offline, ELDR produces one centroid per decode worker by clustering the calibration signatures into K groups such that signatures within a group are close (locality) and groups are balanced in size Online routing favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied Decode routing in PD-disaggregated MoE serving optimizes only decode-worker load ELDR adds a second axis, expert locality: a prefill-derived signature drives load-tolerant routing that stays coherent with the prefix cache, losslessly lowering median and tail TPOT across models, workloads, and topologies References

Paper summary: Tom: To wrap up on ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving, it adds a second axis to decode routing that focuses on expert locality derived from prefill activations <ref:2607.00466#pg1>.

Jane: Essentially, they take the structure of which experts are used during prefill and turn it into a compact signature that predicts how much overlap will happen in the decoding phase <ref:2607.00466#pg2>.

Lu: The key is that this signature is built from prefill data, which is cheap to collect at the beginning of the process, and they use that to partition decode workers offline based on locality regions <ref:2607.00466#pg2>.

Meng: For me, what's interesting is how they handle the online routing by favoring locality when a signature strongly points to one worker but falling back to load balancing if several workers are nearly tied <ref:2607.00466#pg1>.

Tom: The paper shows that this approach results in a twenty-two percent reduction in the per-step active-expert count on average compared to baseline methods, which is what drives the speedup <ref:2607.00466#pg3>.

Jane: So for someone just listening, it means that instead of just balancing how many requests are waiting at a worker right now, you're also telling the system where those requests belong based on how much they share in their expert usage <ref:2607.00466#pg2>.

Lu: It's about decoupling the routing decision; you have one axis for locality—which is aggregate and visible across many requests—and a second axis for load, which is instantaneous and only visible at the moment of the decision <ref:2607.00466#pg2>.

Tom: The authors point out that this combination allows them to achieve better performance across models, workloads, and topologies than systems that only focus on load balancing <ref:2607.00466#pg1>.

Jane: And one limitation they mention is that the locality routing can overload popular domains if you rely too heavily on it <ref:2607.00466#pg2>.

Meng: So, while it’s a solid improvement for latency, we still have to be careful about those domain-specific imbalances when deploying this at scale <ref:2607.00466#pg3>.

Tom: It’s an architecture that uses prefill data to predict decode behavior, and they show a measurable gain in reduced expert activation per step across different batch sizes <ref:2607.00466#pg3>.

Conclusion: Tom: So, we're wrapping up on ELDR, which is Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving. The main thing here is that they’ve added a second layer to how those decode workers pick their tasks.

Jane: That second layer is about expert locality, which means they look at how much experts are shared between requests during the prefill phase and use that to guide the decoding step. I was wondering what it actually does for someone just listening.

Lu: It turns every request into a compact signature based on those prefill activations. This signature predicts exactly how much overlap you'll have when it comes to using those experts during decoding.

Meng: From an engineering standpoint, this is cool because the core mechanism is grouping requests whose signatures are close together. That means fewer distinct experts get activated per decode step for a batch of related tasks.

Tom: And the numbers back that up—they show this reduces the active expert count by twenty-two percent on average across different batch sizes compared to previous routing methods.

Jane: So, when you think about how a system handles millions of requests at once, it’s not just about balancing the number of workers; it’s about making sure related tasks stay together on the same worker.

Lu: The paper shows this is achieved by splitting the decision: one axis for locality and one axis for load, which is what your specific worker's current workload is.

Meng: I gotta ask, what does that mean for deployment? Does it make things more predictable when you're running a massive system with thousands of different models?

Tom: It makes the routing decisions much smarter about where the work goes based on data similarity, not just blindly spreading requests out.

Jane: And this works even while keeping the prefix cache coherent, which is a big win because it keeps things fast without adding a lot of extra overhead to memory management.

Lu: The authors conclude that this method gives you load-tolerant routing that stays perfectly synchronized with the prefix cache, lowering those tail performance errors across different models and setups.

Meng: So it’s about getting better results without completely rewriting the whole serving stack, just adding this smart routing layer on top.

Tom: Exactly. It's a way to get more work done per step by being smarter about which experts are used together during the decode phase.

Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon

KAIST · Microsoft Research · Shanghai Xingyunzhili Artificial Intelligence Institute

cs.DC

Submitted: 2026-07-01

Updated: 2026-11-06

Comments: 15 pages. Accepted to ACM SIGOPS ATC 2026. Updated to the final conference version

DOI: 10.1145/3828523.3857077

Code: https://github.com/deepseek-ai/EPLB

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR…

Key concepts

Expert Signature
This is a compact representation of a request’s prefill activations, summarizing how many tokens each expert received. The signature is designed so that the distance between two signatures predicts the overlap in expert usage during decoding.
Expert Locality
This concept exploits the correlation between related requests. The paper argues that experts used by similar requests are likely to be used together during decoding, meaning locality is a predictable axis for routing decisions.
Prefix Cache Coherence
ELDR maintains an expert-signature cache synchronized with the KV cache at block granularity. This ensures that the request's signature remains accurate regardless of whether it's a partial hit or a full eviction, minimizing overhead.
Locality-Band Routing
The online routing strategy uses the expert signature to select decode workers. It favors workers whose precomputed centroids are close to the request's best match (locality) while falling back to load balancing if multiple workers are similar.

Terminology

Summary

In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving The gap is large: on Qwen3-30B-A3B, growing the active-expert count from 16 to 128 raises MoE-layer latency 4. p. 0

Where the paper says it

ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving Sangjin Choi∗ KAIST Daejeon, Korea Sukmin Cho∗ KAIST Daejeon, Korea Yifan Xiong Microsoft Research Beijing, China Ziyue Yang† Shanghai Xingyunzhili Artificial Intelligence Institute Shanghai, China Youngjin Kwon‡✉ KAIST Daejeon, Korea Peng Cheng Microsoft Research Redmond, WA, USA Abstract In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates.

7× at fixed batch size, while batch size at fixed active-expert count barely moves it Expert locality is therefore not only structured but visible at exactly the moment the router must act

Expert Locality as a Predictable Routing Axis

ELDR turns each request’s prefill activations into a compact expert signature whose distances predict decode-time expert overlap, so requests with nearby signatures share experts at decode This approach exploits the fact that expert selection is correlated across related requests, not just sparse within a token The prefill phase has already pushed the prompt through the same gates that will route decode, and the two agree Expert locality is therefore not only structured but visible at exactly the moment the router must act

Where Gains Come From

The primary mechanism for performance improvement is reducing the number of distinct experts activated per decode step Fig. 4 quantifies this: same-domain batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches across batch sizes 32–128 ELDR reduces the perstep active-expert count by 22. p. 3

Where the paper says it

4 Opportunity: Expert-Locality-Aware Decode Routing In PD-disaggregated MoE serving, decode-step latency is governed not only by per-worker load but by request composition: requests that dispatch to overlapping experts activate fewer distinct experts per step than an equally sized batch of unrelated requests. 4 quantifies this: same-domain batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches across batch sizes 32–128.

0% on average across decode batch sizes compared to RR This reduction is achieved by grouping requests whose signatures are close, so that the distinct experts activated across a group are fewer than under naive, similarity-agnostic routing

How the Training Signal is Built

The expert signature summarizes each request’s prefill-time expert activations into a compact per-request representation, the expert signature The raw material is cheap to collect at prefill: a per-layer histogram of how many tokens each expert receives, available as a by-product of routing The signature is designed so that the distance between two signatures predicts how much the two requests overlap in their decodetime expert usage Signature quality is then the rank correlation between signature pair-distance and decode-pattern pair-distance, over random calibration pairs

Pruning Redundant Reasoning Tokens

The routing decision is split along an information boundary where locality, which experts colocated requests share, is an aggregate property visible only across many requests; load, which decode’s non-MoE work imposes per colocated request, is instantaneous and visible only at the decision ELDR splits the decision along that information boundary Offline, balanced K-means partitions signature space into one locality region per decode worker, capturing structure without inheriting workload skew Online, locality-band routing keeps the workers whose centroids fall within a similarity band of the request’s best match and routes to the least loaded among them

Prefix Cache Coherence

ELDR keeps an expert-signature cache co-indexed with the KV cache at block granularity, which keeps signatures exact across partial hits, full hits, and evictions, so ELDR composes with prefix caching at negligible cost A request’s KV cache spans a block set B (r), so its signature is the sum over those blocks, sr = Í b∈ B (r) sig[b], exact regardless of which request populated each block The signature cache inherits the KV cache’s lifecycle—a reused block resets its signature, a cached prefix carries its signature, and an eviction reclaims both stores together—so ELDR keeps no prefix-tree, eviction, or sharing state of its own

Decode Clustering and Routing

ELDR routes the request to one of K decode workers under two objectives: 1) Locality—colocated requests should share experts—is what the signature is built to enable 2) Load balance is the constraint routing cannot ignore Offline, ELDR produces one centroid per decode worker by clustering the calibration signatures into K groups such that signatures within a group are close (locality) and groups are balanced in size Online routing favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied

Conclusion

Decode routing in PD-disaggregated MoE serving optimizes only decode-worker load ELDR adds a second axis, expert locality: a prefill-derived signature drives load-tolerant routing that stays coherent with the prefix cache, losslessly lowering median and tail TPOT across models, workloads, and topologies

References

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Expert locality as a predictable routing axis: The system can reduce decode latency by grouping requests with similar prefill activations to share experts at decode time, as same-domain decode batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches.

  2. Expert-locality-aware routing: The router will build a quality selected expert signature, partition the signature space with balanced K means, and route using a locality band that balances expert locality against live load.

  3. Prefix cache coherent signatures: The system can maintain exact signatures across partial hits, full hits, and evictions by keeping a block granular signature cache co-indexed with the KV cache without recomputation or change to caching.

  4. Signature design optimization: The system will use the count idf metric for the expert signature because it downweights common experts and amplifies the rare ones that distinguish workloads, maximizing rank correlation with decode-time overlap as measured by Spearman rank correlation rho.

  5. Offline cluster balance improvement: The system will use Hungarian balanced K means instead of vanilla K means to ensure that each centroid takes at most N/K points, which recovers both median and tail TPOT reductions by reducing load imbalance effects on the tail latency.

  6. Online routing adaptation: The locality-band routing will adapt its width by using a parameter tau, allowing it to favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied.

Abstract

Equal load does not mean equal decode latency in prefill-decode (PD) disaggregated mixture-of-experts (MoE) serving. In bandwidth-bound decode, more distinct experts mean more weight loads and higher latency. We present ELDR, a lossless, expert-locality-aware decode router. Prefill signatures predict decode expert usage. Balanced K-means fits per-decoder centroids with background refitting. Locality-band routing selects the least-loaded worker among close matches. A block-granular cache keeps signatures exact under prefix caching. We evaluate ELDR in vLLM on up to 40 GPUs. Across three MoE models on 12 public task benchmarks and multilingual WildChat, it reduces median time-per-output-token (TPOT) by 5.4-14.5% on average across request rates and up to 17.8% over the best of four load-balancing baselines, preserving model outputs and token-to-expert assignments.

Sources

Related papers