AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts".
Paper summary: Tom: So we're looking at this paper called AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts.
Jane: It looks like the main problem they're tackling is that when you have agentic reinforcement learning rollouts, you have prefill work and decode work happening at the same time, and they share the same accelerators.
Lu: Right, so those asynchronous trajectories introduce this persistent mix of prefill work and decode work because other trajectories are still decoding while new ones are being prefetched zero <ref:2609.00993#pg2>. And that co-location on shared hardware means that prefill can mess with the latency-sensitive decode work and just slow down the whole rollout >
Meng: But typically, when we look at these systems in practice, we separate prefill and decode to avoid that interference, which means you need separate device pools and you have to transfer the KV cache between them zero <ref:2609.00993#pg2>. So if you share devices for speed, how do you handle this persistent mix of prefill and decode?
Tom: That's what AInfer-PD claims it solves by extending in-place multiplexing to these distributed Mixture of Experts, or MoE, rollouts one <ref:2609.00993#pg1>. Basically, they want to keep sharing the model weights and the KV cache on the same devices while coordinating how prefill and decode happen together >
Jane: And their core idea is that they coordinate the order of those prefill and decode collectives across all the different ranks in a way that keeps everything safe during concurrent execution two <ref:2609.00993#pg2>. They give each of those two DeepEP paths, which handle prefill and decode, independent communication states >
Lu: The design coordinates the collective order across ranks and separates the mutable state of those two paths so they can run concurrently without stepping on each other's toes three <ref:2609.00993#pg2>. They retain shared model weights and KV storage while managing prefill and decode on the same devices >
Meng: But how do they manage that coordination when things are happening in different orders? I mean, if you have intersecting collectives, how do you make sure they don't clash? What specific mechanism does AInfer-PD use to handle those conflicting communication paths?
Paper summary: Tom: Well, the paper talks about establishing one cross-rank order for each pair of intersecting prefill and decode collectives >
Jane: It seems like they use a distributed turnstile to establish the same host enqueue order on every rank for that specific pair, and then a selective stream route to realize that order for the conflicting device kernels four. That sounds like a pretty clever way to enforce discipline across different devices >
Lu: And it’s not just about ordering those collectives; they also add phase-owned communication domains within one process level runtime five. This is key because it isolates the mutable protocol state, which keeps the normal prefill path and the low latency decode path from interfering with each other's specific state management >
Meng: So if we think about deployment, this sounds like it solves a real headache for large MoE systems that combine attention operations with distributed expert execution two <ref:2609.00993#pg2>. But what’s the actual performance gain they are seeing when they test this against the baseline? What did they measure?
Tom: They showed reductions in rollout makespan of seven point one to twenty-two point five percent versus the same engine with multiplexing disabled >
Jane: That's a pretty big chunk, especially when you compare it to SGLang, where they see reductions of up to thirty-two point nine percent across single node prefill-intensive profiles one. The two node profiles also show reductions between eighteen point zero and thirty-five point three percent >
Lu: They also found that adding fine grained segment boundaries adds another eight point six to nineteen point eight percent over whole epoch asynchronous enqueue in the measured crossed topology seven. That suggests you can get even more speedup by breaking up the prefill work into smaller pieces >
Meng: If I'm shipping this to production, I need to know what these segment boundaries mean for my hardware utilization and latency spikes. Does this method create a predictable pattern of communication overhead? What are the practical costs of managing these turnstiles across many ranks?
Tom: The paper suggests that AInfer-PD shortens the critical path without needing a second model instance or any KV transfer, which is significant because it keeps everything on one set of devices >
Paper summary: Jane: So for someone who only listens to this show, the main point is that you can run prefill and decode at the same time on shared hardware with MoE models and get much faster rollout times by carefully ordering how those two phases communicate >
Lu: The limitation they mention is that this in-place approach still requires separate device pools if you need to totally disaggregate your resources, or if you can't tolerate any KV transfer at all nine. It's not a perfect solution for every single hardware setup >
Meng: I see. So it’s very focused on the in-place scenario where sharing devices and state is the main goal, rather than a full separation of concerns nine. What about the deepEP state isolation? Is that really just about protocol state or something more substantial?
Lu: The isolation specifically targets mutable protocol state, not things like NIC bandwidth, caches, or unreserved memory on the small processing units >
Tom: It’s important to remember they are coordinating P and D on the same devices while keeping the model weights and KV cache shared across that same pool >
Jane: So AInfer-PD is essentially a way to manage the communication plumbing between prefill and decode when you're running large MoE models on a single set of GPUs, without having to move your state around ten.
Lu: Exactly. They order the registered collectives and give DeepEP phase owned state while keeping one model and KV replica, which is a key design choice for their in-place system >
Meng: Okay, so the big picture is that this could mean much faster agentic RL rollouts on existing hardware without having to buy extra GPUs or manage complex offloading schemes ten.
Tom: That's the core claim. It shortens the rollout critical path significantly across those single node prefill-intensive profiles where you have persistent P and D coexistence >
Jane: We'll leave you there. We’ve been talking about AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts one <ref:2609.00993#pg1,AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE>.
Conclusion: Tom: So we’ve been talking about AInfer-PD, and now we're coming to the end of this discussion about it.
Jane: It’s really interesting how they tackle that persistent problem of prefill and decode phases interfering when they share hardware in distributed Mixture of Experts systems.
Tom: Yeah, so they are basically saying you can keep using one set of devices for both parts of the rollout without having to move your data around constantly.
Lu: Exactly, it’s about coordinating the communication between those two different execution paths across all the ranks in a way that keeps them safe from clashing.
Meng: So what does that actually mean for me when I'm trying to deploy this at scale? Does it just mean less waiting time on my end?
Jane: It means they’ve measured real results, and the rollout time shrinks by up to twenty point five percent compared to systems where you turn off this multiplexing feature.
Tom: And they did even better when comparing it against SGLang, showing a reduction of nearly thirty point three percent on two-node setups.
Lu: The core mechanism is ordering the intersecting collectives and giving those two different execution paths their own independent communication states so they can run at the same time.
Jane: But there are still trade-offs, and the authors mention that this in-place approach still needs separate device pools if you want to totally separate your resources for other reasons.
Meng: So it’s not a magic fix for every single deployment scenario, just an optimization for when sharing devices is the main constraint.
Tom: It shortens the critical path significantly without needing a second model instance or any extra data transfer between devices.
Jane: That’s what they’re claiming, and it really changes how we think about running large MoE models where prefill and decode are constantly happening together.
Ant Group, China
cs.DC
Submitted: 2026-09-01
Updated: 2026-11-17
Comments: 13 pages, 10 figures
Code: https://github.com/deepseekai/DeepEP
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: Rollout generation often dominates the wall-clock time of large-scale reinforcement learning (RL) [1] and AInfer-PD reduces fixed-workload rollout completion time by 7.1–22.5% relative to the same…
Key concepts
- Prefill-Decode Interference
- In agentic RL rollouts, prefill work (P) and decode work (D) often happen simultaneously on shared accelerators. This coexistence causes interference because P can disrupt the latency-sensitive D phase by issuing conflicting communication requests or sharing mutable protocol states, leading to longer total rollout completion times.
- In-Place P/D Multiplexing
- This technique allows both prefill and decode phases to share the same model weights and KV cache on the same devices. AInfer-PD manages this by coordinating communication order across ranks and ensuring that P and D paths maintain independent communication states, enabling safe concurrent execution.
- Phase-Owned Communication Domains
- DeepEP faces state ownership issues when P and D run concurrently. AInfer-PD addresses this by creating separate communication domains within the runtime for each phase. This ensures that the prefill path uses one set of communication rules while the decode path uses another, preventing conflicts between them.
- Ordered Phase-Specific Collectives
- The system establishes a specific cross-rank order for intersecting P/D collectives. By using a distributed turnstile and selective stream routing, AInfer-PD ensures that conflicting communication requests are ordered correctly across ranks, allowing the decode phase to proceed while prefill segments execute in controlled segments.
Terminology
Summary
Rollout generation often dominates the wall-clock time of large-scale reinforcement learning (RL) [1] and AInfer-PD reduces fixed-workload rollout completion time by 7. p. 01–22.5% relative to the same AInfer engine with P/D multiplexing disabled and by 24. p. 08–32.9% relative to SGLang [1]. p. 0
The core problem addressed is the interference between prefill (P) and decode (D) phases in agentic RL rollouts when they share accelerators [1]. p. 0
Asynchronous trajectories introduce new prefill (P) work while other trajectories remain in decode (D), making P/D coexistence a persistent property of the rollout rather than a one-time prompt-ingestion event [1]. p. 0 On shared accelerators, persistent P/D coexistence can make prefill interfere with latency-sensitive decode and prolong rollout completion [1]. p. 0 In practical implementations, P and D can issue intersecting collectives in inconsistent cross-rank orders; DeepEP’s P and D paths also share mutable protocol state [1]. p. 0
AInfer-PD introduces a solution by extending in-place P/D multiplexing to distributed MoE rollouts [1]. p. 0
The design coordinates P/D collective order across ranks and gives the two DeepEP paths independent communication state, making crossed ADP/ATP and DeepEP paths safe for concurrent P/D execution [2]. p. 0 The system retains shared model weights and KV storage while coordinating P and D on the same devices [2]. p. 0
Design Overview
AInfer-PD extends in-place P/D multiplexing within one distributed MoE model, where both phases share weights, KV cache, and devices [3]. A rank-shared scheduler controls phase admission, while the communication layer orders intersecting ADP/ATP collectives and separates the mutable state of DeepEP’s normal and low-latency paths [3]. p. 2
In-Place P/D Execution
At each engine iteration, a rank-shared planner admits ready D first and uses the remaining batch and KV capacity for P [3]. A single P execution typically spans the duration of several D iterations, so AInfer-PD divides P into segments delimited by backend-provided communication boundaries [4]. p. 3 In each turnstile round, all ranks first enqueue one D iteration in a consistent order and then admit one P segment [4]. p. 3
Ordering Phase-Specific Collectives
AInfer-PD establishes one cross-rank order for each registered pair of intersecting P/D collectives, ordering only the conflicting pair while segment releases let D advance through a long P execution [4]. This is achieved by using a distributed turnstile to establish the same host-enqueue order on every participating rank and a selective stream route to realize that order for the conflicting device kernels [4]. p. 3
Concurrent DeepEP Communication
DeepEP introduces a separate state-ownership problem, and AInfer-PD adds phase-owned communication domains within one process-level runtime [5]. P uses the normal path and D keeps the low-latency path while P is active, with AInfer-PD splitting the normal dispatch after expert notification before P submits data movement [5]. p. 3
Evaluation Results
AInfer-PD reduces rollout makespan by 7.1–22.5% versus the same engine with multiplexing disabled and by 24. p. 0 p. 08–32.9% versus SGLang across repeated single-node prefill-intensive profiles [1]. p. 1 The two-node profiles yield reductions of 18. p. 1 p. 10–35.3% and 18.3–31.8%, respectively [7]. p. 0 p. 0 Fine-grained segment boundaries add 8. p. 6 p. 66–19.8% over whole-epoch asynchronous enqueue in the measured crossed topology [7]. p. 0 p. 0
Discussion and Limitations
AInfer-PD targets rollout workers that keep one model and KV state on the same devices while P and D coexist, noting that disaggregation remains preferable when separate phase capacity and KV transfer are acceptable [9]. In-place collective ordering is needed only when P/D paths intersect and ranks can expose them in different orders [9]. p. 2 The turnstile is not tied to ADP/ATP but applies when phase-specific paths traverse intersecting collective groups in different orders and expose safe insertion boundaries [9]. p. 1 State isolation isolates mutable protocol state, not NIC bandwidth, caches, or unreserved SM capacity [5]. p. 4
Conclusion
AInfer-PD orders the registered collectives and gives DeepEP phase-owned state while retaining one model and KV replica [10]. Across the evaluated prefill-intensive single-node profiles, AInfer-PD reduces rollout makespan by 7. p. 11–22.5% versus the same engine with multiplexing disabled and by 24.8–32.9% versus SGLang [1]. p. 7 The two-node profiles yield reductions of 18.0–35.3% and 18.3–31.8%, respectively [7]. Fine-grained segment boundaries add 8.6–19.8% over whole-epoch asynchronous enqueue in the measured crossed topology [7]. AInfer-PD therefore shortens the rollout critical path without a second model instance or KV transfer [10]. p. 7
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AInfer-PD can coordinate P/D collective order across ranks to remove observed progress cycles in ADP/ATP paths by ordering intersecting P collectives after D on every rank.
-
The system extends DeepEP with phase-owned buffers, counters, workspaces, events, and queue-pair ranges so normal-P and low-latency-D communication can coexist within one runtime while retaining shared model weights and KV state.
-
AInfer-PD enables concurrent execution on crossed ADP/ATP and DeepEP paths by coordinating P/D collective order across ranks while giving the two DeepEP paths independent communication state.
-
The design maintains shared model weights and KV storage while coordinating P and D on the same devices, reducing fixed-workload rollout completion time by 7.1–22.5% relative to the same AInfer engine with P/D multiplexing disabled and by 24.8–32.9% relative to SGLang on two-node deployments.
-
Fine-grained boundaries reduce completion time by a further 8.6–19.8% over whole-epoch asynchronous enqueue in a same-engine ablation, allowing for more precise control over when D can enter P execution segments.
-
The system utilizes a rank-aligned segment turnstile to coordinate when D may enter across ranks, ensuring that
every rank reaches the P boundary before D enqueue and the post-D phase before the next P release.
Sources
- DeepSeek-V3 Technical Report
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
- DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
- Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
- ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends