AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts
summary
The gist
Rollout generation often dominates the wall-clock time of large-scale reinforcement learning (RL) [1] and AInfer-PD reduces fixed-workload rollout completion time by 7.1–22.5% relative to the same…
In short
AInfer-PD solves performance bottlenecks in large-scale reinforcement learning rollouts where prefill (P) and decode (D) phases interfere on shared hardware. It introduces in-place P/D multiplexing for distributed Mixture of Experts (MoE) models by coordinating collective communication order across ranks and isolating phase-specific communication states, significantly reducing rollout time.
Key concepts
- Prefill-Decode Interference
- In agentic RL rollouts, prefill work (P) and decode work (D) often happen simultaneously on shared accelerators. This coexistence causes interference because P can disrupt the latency-sensitive D phase by issuing conflicting communication requests or sharing mutable protocol states, leading to longer total rollout completion times.
- In-Place P/D Multiplexing
- This technique allows both prefill and decode phases to share the same model weights and KV cache on the same devices. AInfer-PD manages this by coordinating communication order across ranks and ensuring that P and D paths maintain independent communication states, enabling safe concurrent execution.
- Phase-Owned Communication Domains
- DeepEP faces state ownership issues when P and D run concurrently. AInfer-PD addresses this by creating separate communication domains within the runtime for each phase. This ensures that the prefill path uses one set of communication rules while the decode path uses another, preventing conflicts between them.
- Ordered Phase-Specific Collectives
- The system establishes a specific cross-rank order for intersecting P/D collectives. By using a distributed turnstile and selective stream routing, AInfer-PD ensures that conflicting communication requests are ordered correctly across ranks, allowing the decode phase to proceed while prefill segments execute in controlled segments.
Terminology used across episodes
This episode discusses
- AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts · Paper Radio
- DeepSeek-V3 Technical Report
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
- DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
- Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
- ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
The paper
AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts · Read on arXiv
Ant Group, China
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts".
Paper summary: Tom: So we're looking at this paper called AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts.
Jane: It looks like the main problem they're tackling is that when you have agentic reinforcement learning rollouts, you have prefill work and decode work happening at the same time, and they share the same accelerators.
Lu: Right, so those asynchronous trajectories introduce this persistent mix of prefill work and decode work because other trajectories are still decoding while new ones are being prefetched zero <ref:2609.00993#pg2>. And that co-location on shared hardware means that prefill can mess with the latency-sensitive decode work and just slow down the whole rollout >
Meng: But typically, when we look at these systems in practice, we separate prefill and decode to avoid that interference, which means you need separate device pools and you have to transfer the KV cache between them zero <ref:2609.00993#pg2>. So if you share devices for speed, how do you handle this persistent mix of prefill and decode?
Tom: That's what AInfer-PD claims it solves by extending in-place multiplexing to these distributed Mixture of Experts, or MoE, rollouts one <ref:2609.00993#pg1>. Basically, they want to keep sharing the model weights and the KV cache on the same devices while coordinating how prefill and decode happen together >
Jane: And their core idea is that they coordinate the order of those prefill and decode collectives across all the different ranks in a way that keeps everything safe during concurrent execution two <ref:2609.00993#pg2>. They give each of those two DeepEP paths, which handle prefill and decode, independent communication states >
Lu: The design coordinates the collective order across ranks and separates the mutable state of those two paths so they can run concurrently without stepping on each other's toes three <ref:2609.00993#pg2>. They retain shared model weights and KV storage while managing prefill and decode on the same devices >
Meng: But how do they manage that coordination when things are happening in different orders? I mean, if you have intersecting collectives, how do you make sure they don't clash? What specific mechanism does AInfer-PD use to handle those conflicting communication paths?
Paper summary: Tom: Well, the paper talks about establishing one cross-rank order for each pair of intersecting prefill and decode collectives >
Jane: It seems like they use a distributed turnstile to establish the same host enqueue order on every rank for that specific pair, and then a selective stream route to realize that order for the conflicting device kernels four. That sounds like a pretty clever way to enforce discipline across different devices >
Lu: And it’s not just about ordering those collectives; they also add phase-owned communication domains within one process level runtime five. This is key because it isolates the mutable protocol state, which keeps the normal prefill path and the low latency decode path from interfering with each other's specific state management >
Meng: So if we think about deployment, this sounds like it solves a real headache for large MoE systems that combine attention operations with distributed expert execution two <ref:2609.00993#pg2>. But what’s the actual performance gain they are seeing when they test this against the baseline? What did they measure?
Tom: They showed reductions in rollout makespan of seven point one to twenty-two point five percent versus the same engine with multiplexing disabled >
Jane: That's a pretty big chunk, especially when you compare it to SGLang, where they see reductions of up to thirty-two point nine percent across single node prefill-intensive profiles one. The two node profiles also show reductions between eighteen point zero and thirty-five point three percent >
Lu: They also found that adding fine grained segment boundaries adds another eight point six to nineteen point eight percent over whole epoch asynchronous enqueue in the measured crossed topology seven. That suggests you can get even more speedup by breaking up the prefill work into smaller pieces >
Meng: If I'm shipping this to production, I need to know what these segment boundaries mean for my hardware utilization and latency spikes. Does this method create a predictable pattern of communication overhead? What are the practical costs of managing these turnstiles across many ranks?
Tom: The paper suggests that AInfer-PD shortens the critical path without needing a second model instance or any KV transfer, which is significant because it keeps everything on one set of devices >
Paper summary: Jane: So for someone who only listens to this show, the main point is that you can run prefill and decode at the same time on shared hardware with MoE models and get much faster rollout times by carefully ordering how those two phases communicate >
Lu: The limitation they mention is that this in-place approach still requires separate device pools if you need to totally disaggregate your resources, or if you can't tolerate any KV transfer at all nine. It's not a perfect solution for every single hardware setup >
Meng: I see. So it’s very focused on the in-place scenario where sharing devices and state is the main goal, rather than a full separation of concerns nine. What about the deepEP state isolation? Is that really just about protocol state or something more substantial?
Lu: The isolation specifically targets mutable protocol state, not things like NIC bandwidth, caches, or unreserved memory on the small processing units >
Tom: It’s important to remember they are coordinating P and D on the same devices while keeping the model weights and KV cache shared across that same pool >
Jane: So AInfer-PD is essentially a way to manage the communication plumbing between prefill and decode when you're running large MoE models on a single set of GPUs, without having to move your state around ten.
Lu: Exactly. They order the registered collectives and give DeepEP phase owned state while keeping one model and KV replica, which is a key design choice for their in-place system >
Meng: Okay, so the big picture is that this could mean much faster agentic RL rollouts on existing hardware without having to buy extra GPUs or manage complex offloading schemes ten.
Tom: That's the core claim. It shortens the rollout critical path significantly across those single node prefill-intensive profiles where you have persistent P and D coexistence >
Jane: We'll leave you there. We’ve been talking about AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts one <ref:2609.00993#pg1,AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE>.
Conclusion: Tom: So we’ve been talking about AInfer-PD, and now we're coming to the end of this discussion about it.
Jane: It’s really interesting how they tackle that persistent problem of prefill and decode phases interfering when they share hardware in distributed Mixture of Experts systems.
Tom: Yeah, so they are basically saying you can keep using one set of devices for both parts of the rollout without having to move your data around constantly.
Lu: Exactly, it’s about coordinating the communication between those two different execution paths across all the ranks in a way that keeps them safe from clashing.
Meng: So what does that actually mean for me when I'm trying to deploy this at scale? Does it just mean less waiting time on my end?
Jane: It means they’ve measured real results, and the rollout time shrinks by up to twenty point five percent compared to systems where you turn off this multiplexing feature.
Tom: And they did even better when comparing it against SGLang, showing a reduction of nearly thirty point three percent on two-node setups.
Lu: The core mechanism is ordering the intersecting collectives and giving those two different execution paths their own independent communication states so they can run at the same time.
Jane: But there are still trade-offs, and the authors mention that this in-place approach still needs separate device pools if you want to totally separate your resources for other reasons.
Meng: So it’s not a magic fix for every single deployment scenario, just an optimization for when sharing devices is the main constraint.
Tom: It shortens the critical path significantly without needing a second model instance or any extra data transfer between devices.
Jane: That’s what they’re claiming, and it really changes how we think about running large MoE models where prefill and decode are constantly happening together.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game