ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving".
Jane: Here's the gist of the paper. In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving The gap is large: on Qwen3-30B-A3B, growing the active-expert count from 16 to 128 raises MoE-layer latency 4.7× at fixed batch size, while batch size at fixed active-expert count barely moves it Expert locality is therefore not only structured but visible at exactly the moment the router must act Expert Locality as a Predictable Routing Axis ELDR turns each request’s prefill activations into a compact expert signature whose distances predict decode-time expert overlap, so requests with nearby signatures share experts at decode This approach exploits the fact that expert selection is correlated across related requests, not just sparse within a token The prefill phase has already pushed the prompt through the same gates that will route decode, and the two agree Expert locality is therefore not only structured but visible at exactly the moment the router must act Where Gains Come From The primary mechanism for performance improvement is reducing the number of distinct experts activated per decode step Fig.
Tom: 4 quantifies this: same-domain batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches across batch sizes 32–128 ELDR reduces the perstep active-expert count by 22.0% on average across decode batch sizes compared to RR This reduction is achieved by grouping requests whose signatures are close, so that the distinct experts activated across a group are fewer than under naive, similarity-agnostic routing How the Training Signal is Built The expert signature summarizes each request’s prefill-time expert activations into a compact per-request representation, the expert signature The raw material is cheap to collect at prefill: a per-layer histogram of how many tokens each expert receives, available as a by-product of routing The signature is designed so that the distance between two signatures predicts how much the two requests overlap in their decodetime expert usage Signature quality is then the rank correlation between signature pair-distance and decode-pattern pair-distance, over random calibration pairs Pruning Redundant Reasoning Tokens The routing decision is split along an information boundary where locality, which experts colocated requests share, is an aggregate property visible only across many requests; load, which decode’s non-MoE work imposes per colocated request, is instantaneous and visible only at the decision ELDR splits the decision along that information boundary Offline, balanced K-means partitions signature space into one locality region per decode worker, capturing structure without inheriting workload skew Online, locality-band routing keeps the workers whose centroids fall within a similarity band of the request’s best match and routes to the least loaded among them Prefix Cache Coherence ELDR keeps an expert-signature cache co-indexed with the KV cache at block granularity, which keeps signatures exact across partial hits, full hits, and evictions, so ELDR composes with prefix caching at negligible cost A request’s KV cache spans a block set B (r), so its signature is the sum over those blocks, sr = Í b∈ B (r) sig
b: , exact regardless of which request populated each block The signature cache inherits the KV cache’s lifecycle—a reused block resets its signature, a cached prefix carries its signature, and an eviction reclaims both stores together—so ELDR keeps no prefix-tree, eviction, or sharing state of its own Decode Clustering and Routing ELDR routes the request to one of K decode workers under two objectives: 1) Locality—colocated requests should share experts—is what the signature is built to enable 2) Load balance is the constraint routing cannot ignore Offline, ELDR produces one centroid per decode worker by clustering the calibration signatures into K groups such that signatures within a group are close (locality) and groups are balanced in size Online routing favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied Decode routing in PD-disaggregated MoE serving optimizes only decode-worker load ELDR adds a second axis, expert locality: a prefill-derived signature drives load-tolerant routing that stays coherent with the prefix cache, losslessly lowering median and tail TPOT across models, workloads, and topologies References
Paper summary: Tom: To wrap up on ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving, it adds a second axis to decode routing that focuses on expert locality derived from prefill activations <ref:2607.00466#pg1>.
Jane: Essentially, they take the structure of which experts are used during prefill and turn it into a compact signature that predicts how much overlap will happen in the decoding phase <ref:2607.00466#pg2>.
Lu: The key is that this signature is built from prefill data, which is cheap to collect at the beginning of the process, and they use that to partition decode workers offline based on locality regions <ref:2607.00466#pg2>.
Meng: For me, what's interesting is how they handle the online routing by favoring locality when a signature strongly points to one worker but falling back to load balancing if several workers are nearly tied <ref:2607.00466#pg1>.
Tom: The paper shows that this approach results in a twenty-two percent reduction in the per-step active-expert count on average compared to baseline methods, which is what drives the speedup <ref:2607.00466#pg3>.
Jane: So for someone just listening, it means that instead of just balancing how many requests are waiting at a worker right now, you're also telling the system where those requests belong based on how much they share in their expert usage <ref:2607.00466#pg2>.
Lu: It's about decoupling the routing decision; you have one axis for locality—which is aggregate and visible across many requests—and a second axis for load, which is instantaneous and only visible at the moment of the decision <ref:2607.00466#pg2>.
Tom: The authors point out that this combination allows them to achieve better performance across models, workloads, and topologies than systems that only focus on load balancing <ref:2607.00466#pg1>.
Jane: And one limitation they mention is that the locality routing can overload popular domains if you rely too heavily on it <ref:2607.00466#pg2>.
Meng: So, while it’s a solid improvement for latency, we still have to be careful about those domain-specific imbalances when deploying this at scale <ref:2607.00466#pg3>.
Tom: It’s an architecture that uses prefill data to predict decode behavior, and they show a measurable gain in reduced expert activation per step across different batch sizes <ref:2607.00466#pg3>.
Conclusion: Tom: So, we're wrapping up on ELDR, which is Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving. The main thing here is that they’ve added a second layer to how those decode workers pick their tasks.
Jane: That second layer is about expert locality, which means they look at how much experts are shared between requests during the prefill phase and use that to guide the decoding step. I was wondering what it actually does for someone just listening.
Lu: It turns every request into a compact signature based on those prefill activations. This signature predicts exactly how much overlap you'll have when it comes to using those experts during decoding.
Meng: From an engineering standpoint, this is cool because the core mechanism is grouping requests whose signatures are close together. That means fewer distinct experts get activated per decode step for a batch of related tasks.
Tom: And the numbers back that up—they show this reduces the active expert count by twenty-two percent on average across different batch sizes compared to previous routing methods.
Jane: So, when you think about how a system handles millions of requests at once, it’s not just about balancing the number of workers; it’s about making sure related tasks stay together on the same worker.
Lu: The paper shows this is achieved by splitting the decision: one axis for locality and one axis for load, which is what your specific worker's current workload is.
Meng: I gotta ask, what does that mean for deployment? Does it make things more predictable when you're running a massive system with thousands of different models?
Tom: It makes the routing decisions much smarter about where the work goes based on data similarity, not just blindly spreading requests out.
Jane: And this works even while keeping the prefix cache coherent, which is a big win because it keeps things fast without adding a lot of extra overhead to memory management.
Lu: The authors conclude that this method gives you load-tolerant routing that stays perfectly synchronized with the prefix cache, lowering those tail performance errors across different models and setups.
Meng: So it’s about getting better results without completely rewriting the whole serving stack, just adding this smart routing layer on top.
Tom: Exactly. It's a way to get more work done per step by being smarter about which experts are used together during the decode phase.
Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon
KAIST · Microsoft Research · Shanghai Xingyunzhili Artificial Intelligence Institute
cs.DC
Submitted: 2026-07-01
Updated: 2026-11-06
Comments: 15 pages. Accepted to ACM SIGOPS ATC 2026. Updated to the final conference version
Code: https://github.com/deepseek-ai/EPLB
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR…
Key concepts
- Expert Signature
- This is a compact representation of a request’s prefill activations, summarizing how many tokens each expert received. The signature is designed so that the distance between two signatures predicts the overlap in expert usage during decoding.
- Expert Locality
- This concept exploits the correlation between related requests. The paper argues that experts used by similar requests are likely to be used together during decoding, meaning locality is a predictable axis for routing decisions.
- Prefix Cache Coherence
- ELDR maintains an expert-signature cache synchronized with the KV cache at block granularity. This ensures that the request's signature remains accurate regardless of whether it's a partial hit or a full eviction, minimizing overhead.
- Locality-Band Routing
- The online routing strategy uses the expert signature to select decode workers. It favors workers whose precomputed centroids are close to the request's best match (locality) while falling back to load balancing if multiple workers are similar.
Terminology
Summary
In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill; this paper introduces ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving The gap is large: on Qwen3-30B-A3B, growing the active-expert count from 16 to 128 raises MoE-layer latency 4. p. 0 ELDR: Expert-Locality-Aware Decode Routing for
PD-Disaggregated MoE Serving
Sangjin Choi∗
KAIST
Daejeon, Korea
Sukmin Cho∗
KAIST
Daejeon, Korea
Yifan Xiong
Microsoft Research
Beijing, China
Ziyue Yang†
Shanghai Xingyunzhili
Artificial Intelligence Institute
Shanghai, China
Youngjin Kwon‡✉
KAIST
Daejeon, Korea
Peng Cheng
Microsoft Research
Redmond, WA, USA
Abstract
In prefill–decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing
decode routers balance only load; for mixture-of-experts
(MoE) models this is incomplete: equally loaded workers can
differ in latency, since each decode step loads the weights of
every distinct expert its batch activates.Where the paper says it
Expert Locality as a Predictable Routing Axis
ELDR turns each request’s prefill activations into a compact expert signature whose distances predict decode-time expert overlap, so requests with nearby signatures share experts at decode This approach exploits the fact that expert selection is correlated across related requests, not just sparse within a token The prefill phase has already pushed the prompt through the same gates that will route decode, and the two agree Expert locality is therefore not only structured but visible at exactly the moment the router must act
Where Gains Come From
The primary mechanism for performance improvement is reducing the number of distinct experts activated per decode step Fig. 4 quantifies this: same-domain batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches across batch sizes 32–128 ELDR reduces the perstep active-expert count by 22. p. 3 4 Opportunity: Expert-Locality-Aware Decode
Routing
In PD-disaggregated MoE serving, decode-step latency is
governed not only by per-worker load but by request composition: requests that dispatch to overlapping experts activate
fewer distinct experts per step than an equally sized batch
of unrelated requests. 4 quantifies this: same-domain
batches activate 17–21% (task) and 3–10% (language) fewer
experts per step than mixed-domain batches across batch
sizes 32–128.Where the paper says it
How the Training Signal is Built
The expert signature summarizes each request’s prefill-time expert activations into a compact per-request representation, the expert signature The raw material is cheap to collect at prefill: a per-layer histogram of how many tokens each expert receives, available as a by-product of routing The signature is designed so that the distance between two signatures predicts how much the two requests overlap in their decodetime expert usage Signature quality is then the rank correlation between signature pair-distance and decode-pattern pair-distance, over random calibration pairs
Pruning Redundant Reasoning Tokens
The routing decision is split along an information boundary where locality, which experts colocated requests share, is an aggregate property visible only across many requests; load, which decode’s non-MoE work imposes per colocated request, is instantaneous and visible only at the decision ELDR splits the decision along that information boundary Offline, balanced K-means partitions signature space into one locality region per decode worker, capturing structure without inheriting workload skew Online, locality-band routing keeps the workers whose centroids fall within a similarity band of the request’s best match and routes to the least loaded among them
Prefix Cache Coherence
ELDR keeps an expert-signature cache co-indexed with the KV cache at block granularity, which keeps signatures exact across partial hits, full hits, and evictions, so ELDR composes with prefix caching at negligible cost A request’s KV cache spans a block set B (r), so its signature is the sum over those blocks, sr = Í b∈ B (r) sig[b], exact regardless of which request populated each block The signature cache inherits the KV cache’s lifecycle—a reused block resets its signature, a cached prefix carries its signature, and an eviction reclaims both stores together—so ELDR keeps no prefix-tree, eviction, or sharing state of its own
Decode Clustering and Routing
ELDR routes the request to one of K decode workers under two objectives: 1) Locality—colocated requests should share experts—is what the signature is built to enable 2) Load balance is the constraint routing cannot ignore Offline, ELDR produces one centroid per decode worker by clustering the calibration signatures into K groups such that signatures within a group are close (locality) and groups are balanced in size Online routing favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied
Conclusion
Decode routing in PD-disaggregated MoE serving optimizes only decode-worker load ELDR adds a second axis, expert locality: a prefill-derived signature drives load-tolerant routing that stays coherent with the prefix cache, losslessly lowering median and tail TPOT across models, workloads, and topologies
References
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Expert locality as a predictable routing axis: The system can reduce decode latency by grouping requests with similar prefill activations to share experts at decode time, as
same-domain decode batches activate 17–21% (task) and 3–10% (language) fewer experts per step than mixed-domain batches.
-
Expert-locality-aware routing: The router will build a quality selected expert signature, partition the signature space with balanced K means, and route using a locality band that balances expert locality against live load.
-
Prefix cache coherent signatures: The system can maintain exact signatures across partial hits, full hits, and evictions by keeping a block granular signature cache co-indexed with the KV cache without recomputation or change to caching.
-
Signature design optimization: The system will use the count idf metric for the expert signature because it
downweights common experts
andamplifies the rare ones that distinguish workloads,
maximizing rank correlation with decode-time overlap as measured by Spearman rank correlation rho. -
Offline cluster balance improvement: The system will use Hungarian balanced K means instead of vanilla K means to ensure that
each centroid takes at most N/K points,
which recovers both median and tail TPOT reductions by reducing load imbalance effects on the tail latency. -
Online routing adaptation: The locality-band routing will adapt its width by using a parameter tau, allowing it to
favors locality when the signature singles out one worker, and falls back to load-aware balancing when several workers are nearly tied.
Abstract
Equal load does not mean equal decode latency in prefill-decode (PD) disaggregated mixture-of-experts (MoE) serving. In bandwidth-bound decode, more distinct experts mean more weight loads and higher latency. We present ELDR, a lossless, expert-locality-aware decode router. Prefill signatures predict decode expert usage. Balanced K-means fits per-decoder centroids with background refitting. Locality-band routing selects the least-loaded worker among close matches. A block-granular cache keeps signatures exact under prefix caching. We evaluate ELDR in vLLM on up to 40 GPUs. Across three MoE models on 12 public task benchmarks and multilingual WildChat, it reduces median time-per-output-token (TPOT) by 5.4-14.5% on average across request rates and up to 17.8% over the best of four load-balancing baselines, preserving model outputs and token-to-expert assignments.
Sources
- Program Synthesis with Large Language Models
- Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3 Technical Report
- MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
- Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection
- OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling
- Let's Verify Step by Step
- Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
- Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
- Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
- gpt-oss-120b & gpt-oss-20b Model Card
- Splitwise: Efficient generative LLM inference using phase splitting
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- Preble: Efficient Distributed Prompt Scheduling for LLM Serving
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends
- AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition