BatchGen: An Architecture for Scalable and Efficient Batch Inference
summary
The gist
Batch inference has become one of the fastest-growing and most resource-intensive modes of AI computation Batch inference introduces fundamentally new system requirements The sequence coroutine…
In short
BatchGen introduces a sequence coroutine compute model to overcome inefficiencies in batch inference. It treats sequences as fine-grained, event-driven coroutines that can be dynamically paused, combined, or migrated across GPUs. This architecture allows for flexible scheduling, mitigating stragglers and improving utilization on resource-constrained hardware.
Key concepts
- Sequence Coroutine Compute Model
- This model represents each sequence as a small, independent event-driven coroutine. Unlike traditional batching where all sequences run simultaneously, this allows the system to pause a sequence when needed and resume it later, enabling dynamic reorganization of work across devices.
- yield Primitive
- The 'yield' primitive is used by the scheduler to pause a sequence coroutine. This suspension allows the runtime to immediately switch context and resume another inactive sequence on a different GPU. This event-driven scheduling maximizes GPU utilization by ensuring there is always useful work available.
- COMBINE Primitive
- The 'combine' primitive enables dynamic batch size adjustment for sparse models, particularly in Mixture-of-Experts (MoE) layers. It allows the system to merge yielded sequences into larger batches at runtime when necessary, overcoming the limitation of small batch sizes that hinder MoE performance.
- Dynamic Scheduling Phases
- The scheduler manages sequence flow through four phases at each memory page boundary: Sync, Eviction, Extension, and Refill. These phases dynamically adjust memory usage and manage the continuous flow of sequences to maintain high throughput and handle long-tail decoding stragglers effectively.
Terminology used across episodes
This episode discusses
- BatchGen: An Architecture for Scalable and Efficient Batch Inference · Paper Radio
- Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
- MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
- A Survey on Efficient Inference for Large Language Models
The paper
BatchGen: An Architecture for Scalable and Efficient Batch Inference · Read on arXiv
Tairan Xu, Leyang Xue, Zhan Lu, Jinfu Deng, Hongyang Xiao, Yinsicheng Jiang, Congjie He, Matej Sandor, Le Xu, Luo Mai
University of Edinburgh Tencent
Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-sequence load variation that emerges only at runtime, and the ability to sustain high utilization across large fleets of GPUs. Existing systems fail to meet these requirements, losing substantial fractions of achievable throughput. We introduce a new architectural foundation for batch inference: the sequence coroutine compute model, which represents each sequence as a fine-grained, event-driven coroutine. This model exposes expressive primitives that allow the runtime to reorganize work dynamically, enabling larger expert-level batches, mitigating stragglers, reallocating work across devices, and maintaining utilization even on cost-effective or memory-constrained GPUs. Building on this abstraction, we implement BatchGen, a production-ready system that uses the coroutine model at cluster scale. On a 128-GPU cluster, BatchGen reduces batch completion time by up to 2.3 times, and on memory-constrained accelerators it outperforms the strongest offloading baseline by up to 9.6 times. We will open-source BatchGen at https://github.com/batchgen-project/batchgen
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "BatchGen: An Architecture for Scalable and Efficient Batch Inference".
Title and authors: Tom: So we’re looking at this paper called "BatchGen: An Architecture for Scalable and Efficient Batch Inference," and it sounds like they're tackling a really big problem in how we run large AI models. It suggests that current batch inference systems aren't built right for the real demands of scaling up to millions of sequences.
Jane: That makes sense, Tom, because when you try to serve a model to millions of users at once, you run into huge problems with load variation and keeping all your GPUs busy. What they propose is this new sequence coroutine compute model where every single sequence is treated like an event-driven coroutine.
Lu: Exactly. This abstraction lets the system reorganize work dynamically using four specific operations: yield, combine, partition, and migrate. These primitives are what allow the runtime to pause a sequence or combine sequences together to form much larger batches for those expert layers that are currently struggling with small batch sizes.
Meng: I’m curious about how they handle those stragglers you mentioned earlier. If one part of the sequence is slow, does this model help keep the whole GPU from sitting idle waiting for it?
Tom: It does; the paper explains that these primitives allow them to partition stragglers across different idle GPUs or migrate that coroutine state to a device with more available resources, which keeps utilization high even when things get uneven.
Jane: And this dynamic reorganization is key because it lets them build bigger expert-level batches on the fly, which is something existing architectures just can't naturally express. It seems like they’re building the scheduling directly into the computation itself.
Lu: Right, and I also liked how they describe the runtime implements event-driven scheduling at a cluster scale using that yield primitive to suspend one sequence and immediately resume another inactive one on a different GPU, which really maximizes utilization by making sure GPUs always have something useful to process.
Meng: But running that dynamic scheduling across a whole cluster sounds complicated. How do you keep the memory management clean when you're moving and combining these coroutines around? That’s where I get nervous about deployment costs at scale.
Tom: They addressed that with a unified memory layout, which handles both host and device memory management, along with paged KV-cache management. This means they use host memory as a unified store for shared model parameters and the KV cache, and they can checkpoint the KV cache on yield and restore it on combine without needing distributed synchronization.
Jane: So it minimizes that peak GPU memory usage significantly because you aren't constantly duplicating state across devices when you're moving things around. It sounds like a very smart way to manage resources during the prefill phase, too.
Title and authors: Lu: The paper also talks about aggressive offloading during the prefill phase to hide those data transfer costs, which is important when you’re trying to keep latency low for users. They even use dynamic programming and lightweight profiling and simulation to build a directed acyclic graph of a layer's forward pass so they can select the configuration with the shortest critical path.
Meng: I wonder about that dynamic selection process. If you’re constantly profiling and simulating to pick the best schedule, what’s the overhead on top of just running inference? That sounds like it could eat into your total cost of ownership if it's too heavy.
Tom: The paper shows they achieved up to a nine point six times speedup under limited resources because they are smart about enumerating candidate configurations and selecting the one with the shortest critical path using that dynamic programming approach. It’s not just guessing; it’s optimizing based on what the specific layer needs at that moment.
Jane: That is impressive when you look at their results; they showed an increase in batching efficiency for expert layers and a reduction in long-tail decoding stragglers by up to two point three times on a twelve eighty gpu cluster. It seems to address those specific bottlenecks we see in real traffic.
Lu: And the comparison against leading inference engines was interesting because BatchGen outperformed them even on memory-constrained accelerators, achieving up to a nine point six times speedup under those settings. That shows the architecture is robust beyond just having enough raw hardware.
Meng: If I’m deploying this, I need to know how scalable it really is when we move from a few machines to a hundred machines. The paper mentioned that scaling to twelve eight GPUs showed sub-linear speedup because of communication overhead in MoE layers; does that mean the performance gain flattens out quickly?
Tom: Yes, the authors noted that scaling to twelve eight GPUs resulted in sub-linear speedup due to communication overhead specifically within MoE layers. However, they still managed to outperform even the best parallel decoding configuration by two point two times without needing manual tuning.
Jane: So while it doesn't scale perfectly linearly with instance count in those specific MoE scenarios, it still offers a significant improvement over what we’d get otherwise when you look at the overall throughput numbers. It really shows how much better the scheduling is compared to traditional fixed batching.
Lu: Overall, the core idea they are pushing is that sequences themselves give them the right granularity for dynamic scheduling, which is a departure from static batching schemes. This sequence coroutine model is general enough to apply to any generation workload whose stages have distinct compute and memory requirements.
Title and authors: Meng: So, what does this mean practically for someone who just wants to deploy a model without becoming a systems architect? Is this something that requires deep knowledge of distributed computing infrastructure to use effectively?
Tom: Well, it is deployed as an industry partner’s production cluster managed by a Ray orchestrator, running as independent instances. It provides a compatible and expressive interface for batch inference across multi-node and multi-GPU setups, so the goal is to make it usable at scale without requiring you to manually tune every single scheduling parameter.
Jane: And I think that’s the big implication—it moves the complexity from your deployment setup into the architecture itself. It's designed to be expressive enough that it adapts to what a model actually needs at runtime, rather than forcing us into a rigid structure upfront.
Lu: The paper introduces this event-driven sequence coroutine architecture as a new computation model for large-scale batch inference, which is really opening up a rich systems research agenda for designing the next generation of AI engines because it’s not tied to just one type of workload anymore.
Tom: So, to wrap up, BatchGen proposes this event-driven sequence coroutine architecture with primitives like yield and combine that lets you dynamically reorganize work across devices and handle memory efficiently. It’s a powerful way to manage the load variation we see in batch inference today.
Jane: It sounds like a really solid piece of research because it takes the inherent variability of sequence computation and gives the system tools to manage it intelligently at cluster scale, which is exactly what we need for production AI serving.
Lu: Yeah, I think the biggest thing here is that this architecture is general; it applies to any generation workload whose stages have distinct compute characteristics and memory requirements, which makes it much more versatile than previous solutions.
Meng: For me, the main thing is that if you can get a nine point six times speedup on constrained hardware without needing manual tuning for every layer, that’s valuable because those constraints are often where we run into real deployment issues.
Tom: It's certainly a massive result when you look at the performance numbers they cited, showing consistent improvements across various workloads and even outperforming existing engines in memory-constrained settings.
Jane: So it seems like this paper, "BatchGen: An Architecture for Scalable and Efficient Batch Inference," provides a new foundation for how we think about batch inference by using sequence coroutines to enable dynamic work reorganization. It’s definitely something worth keeping an eye on as we move into larger and more complex models.
The paper's summary: Tom: So, we’ve been talking about this new architecture called BatchGen that uses sequence coroutines to handle batch inference better than what we have now at scale.
Jane: Right, so you’re telling me the core idea is treating each sequence like an event-driven program that can be paused and moved around dynamically instead of just shoving them all into one big static block?
Tom: Exactly. It introduces these four operations—yield, combine, partition, and migrate—which let the system reorganize work on the fly to mitigate stragglers or reallocate tasks across different GPUs.
Jane: That sounds like a lot of moving parts under the hood. How does this actually translate into speed for someone running a model?
Tom: Well, when they ran it on one hundred GPUs, they saw performance gains that were pretty significant—they increased batching efficiency for those expert layers and reduced those long-tail decoding stragglers by about two point three times.
Jane: Two point three times on what kind of workload are we talking about? And how does this help someone who is just trying to deploy a model without becoming a distributed computing expert?
Tom: It means they hit up to nine point six times speedup when the hardware is limited, which is huge because most of the problems at scale aren't just having enough raw power, it's managing that load variation.
Jane: And this memory management part you mentioned earlier—the unified store for host and device memory—does that mean less peak GPU usage when things are moving around?
Tom: It does. They use the host memory as a shared space for model parameters and the KV cache, so they can checkpoint that cache on a yield and restore it on a combine without needing complex distributed synchronization.
Jane: That’s smart resource management. So, if you take this idea—these coroutines—and apply it to other types of AI tasks, what else could we be doing with this model?
Tom: The paper says the sequence coroutine abstraction is really general. It’s not just for one type of generation task; it applies to any workload where the different stages have different compute needs and memory requirements.
Jane: So, it’s a flexible foundation that could potentially apply to many things, not just text generation?
Tom: Right. The authors are looking at extending BatchGen to other models and tasks as future work, but the architecture itself is designed to be broadly applicable across various AI engine designs.
The paper's improvements: Tom: We’re looking at what BatchGen suggests are these major improvements to the system's operation, moving beyond just the core concept of sequence coroutines.
Jane: So, what’s new here? Beyond reorganizing work dynamically with those four primitives, does it get smarter about *how* it decides what to do at any given moment?
Tom: It gets much better at configuration selection. Instead of just guessing a batch size or a schedule, the system uses lightweight profiling and simulation to build a directed acyclic graph for each layer’s forward pass.
Jane: So they're essentially running little simulations on the layer itself to pick the best way to run it, which sounds like it could save so much time in setup.
Tom: It does. They use dynamic programming on that graph to select the configuration with the shortest critical path, which directly optimizes for batch completion time.
Jane: That connects back to those stragglers we talked about earlier; if you can dynamically pick a faster path for a specific layer, it helps keep everything moving.
Tom: Exactly. And they’ve got this memory management unified, where the host and device memory share a space for parameters and the KV cache.
Jane: So that means when the system pauses one sequence or moves another to a different GPU, it doesn't have to constantly copy massive amounts of data across devices?
Tom: Right. They can checkpoint that cache on a yield and restore it on a combine without needing complex distributed synchronization, which really cuts down on peak GPU memory usage.
Jane: That’s huge for deployment costs because you’re not paying for massive amounts of extra VRAM just to handle the movement.
Tom: And they showed this setup can achieve up to nine point six times speedup even when you are running under very constrained hardware settings, which is a big win for real-world deployment.
Jane: It seems like they’ve made the system robust enough that it doesn't just work well on ideal hardware but actually gets better when things get tight.
Tom: That’s the point. They’ve built this architecture to be adaptable, and they showed it handles load variation and resource scarcity much more effectively than previous systems.
Jane: So, if you take this ability to dynamically adapt the execution plan based on real-time layer characteristics, what does that mean for the future of large AI models?
Conclusion: Tom: So to wrap up, BatchGen introduces this event-driven sequence coroutine architecture as a new way to handle batch inference at scale by allowing dynamic work reorganization across devices and optimizing configuration selection based on layer needs.
Jane: It sounds like they’ve built something that is much more adaptive than the fixed batch sizes we’ve been using for a long time.
Tom: Exactly. The whole point is that the sequence itself provides the right granularity for dynamic scheduling, which is a big shift away from static batching schemes.
Jane: So what does this mean practically for those of us who just want to deploy models without becoming a distributed computing expert?
Tom: It means the complexity of scheduling and optimization is moved into the architecture itself so you don't have to manually tune every single parameter for every layer.
Jane: That’s a big deal for production use because it makes the system more self-managing under changing conditions.
Lu: I think what this paper really establishes is a general computational model, one that applies to any generation workload where the different stages have distinct compute and memory requirements.
Meng: From my side, if we can get consistent performance gains across various workloads and even on constrained hardware without manual tuning, that’s valuable because those constraints are often where we run into real deployment issues.
Lalam: I’m confident in this result; the model is absolutely sure that this architecture provides a reliable path toward more efficient inference across the board.
Tom: So, the paper does show consistent improvements across various workloads and even outperforms leading inference engines on memory-constrained accelerators by up to nine point six times.
Jane: That’s a strong number, especially since it tackles those long-tail decoding stragglers and boosts efficiency for expert layers in a twelve eighty gpu cluster.
Lu: The implication is that this sequence coroutine model opens up a rich research agenda for designing the next generation of AI engines because it’s not tied to just one type of workload anymore.
Meng: But I also see the caveat they mentioned: scaling to twelve eight GPUs showed sub-linear speedup due to communication overhead in MoE layers, so we still have those bottlenecks if we go massive.
Tom: That’s true; it shows the limits of even this powerful system when you hit high communication demands in a model like a Mixture of Experts.
Jane: So, while it’s incredibly flexible, the performance gain isn't always perfectly linear when you scale up to huge MoE models.
Tom: Right. But even with those limitations, BatchGen still manages to outperform the best parallel decoding configuration by two point two times without needing any manual tuning on our end.
Lu: This architecture is general enough that it’s not just for text generation; it applies to many different AI tasks where the stages have distinct compute characteristics.
Jane: It really feels like this paper, BatchGen: An Architecture for Scalable and Efficient Batch Inference, provides a solid new foundation for how we think about batch inference by using sequence coroutines to enable dynamic work reorganization.
Tom: It’s a powerful way to manage the load variation we see in batch inference today, so I’m really excited about seeing what people do next with this model.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game