SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference

arXiv:2611.14904 · cs.DC · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference".

Jane: Here's the gist of the paper. Large language model (LLM) decode is dominated by short, predominantly memory-bound kernels that make bursty use of on-chip execution resources and memory bandwidth The paper introduces SubGPU Pipelining, a fine-grained, intra-GPU spatial execution model that cuts an operator graph into ordered contiguous segments, binds those segments to spatial stages backed by disjoint streaming-multiprocessor (SM) pools, and streams microbatches through the resulting schedule This approach is evaluated on the GPT-OSS 120B workload using simulation and silicon experiments, showing that it improves mean DRAM read-plus-write throughput from 71.8% to 83.6% of peak and reduces the 36-layer decoder iteration time by 10.33% The core concept of SubGPU Pipelining SubGPU Pipelining addresses inefficiency with LLM decode by cutting the temporal operator sequence within a layer (or, more generally, a repeating compute graph) into contiguous graph segments and executing each segment concurrently on disjoint SM pools The central problem is to jointly choose the graph cuts, segment-to-stage binding, and microbatch size so as to minimize target-workload makespan under shared-resource contention The key ideas involve cutting a layer’s operator graph into K ordered contiguous segments and binding them to D stages, each backed by a controlled, disjoint SM pool Furthermore, the microbatch size is divided into m microbatches which are streamed through the segments while preserving graph dependencies and serializing work bound to the same stage The optimization problem structure The configuration of SubGPU Pipelining is formally defined by a coupled graph segmentation, resource allocation, and microbatching problem with state-dependent segment times A configuration is represented as theta = (D, w, c, p, b), where D is the physical-stage count and w indicates boundary reuse The objective is to find theta⋆ ∈ arg min θ∈F T(θ), where T(θ) is the forward-pass makespan The timing calculation involves a max-plus timing recurrence on a weighted event DAG, which reduces to a longest-path calculation

2: Performance gains across different regimes Analytical simulation of GPT-OSS 120B on modeled B200 deployments estimates the opportunity from spatial co-scheduling under explicitly stated modeling assumptions At matched per-GPU throughput, the modeled SubGPU frontier provides up to 2.2× higher per-user throughput than temporal execution, and at matched per-user throughput it provides up to 1.9× higher per-GPU throughput The hardware experiment shows that the SubGPU pipeline sustains 83.6% of peak DRAM bandwidth on average, compared with 71.8% for the temporal baseline Validation through silicon experiments To test physical feasibility and further validate the simulation results, silicon experiments are performed using one B200 GPU to emulate a 32-GPU B200 deployment Results show that SubGPU Pipelining improves mean DRAM read-plus-write throughput from 71.8% to 83.6% of peak, and reduces the 36-layer decoder iteration time by 10.33% The hardware experiment demonstrates that such a structure can be realized with existing CUDA mechanisms and kernels, and that it delivers a measured 10.33% latency reduction with higher DRAM utilization on a real GPU Realization and future work The logical model requires controlled partitioning of execution capacity, asynchronous progress of work on different stages, serialization of segments bound to the same stage, and dependency signaling Several CUDA mechanisms can provide parts of this functionality, including Green Contexts and CUDA Graphs Future work includes validating the simulator and optimizer more extensively, comparing selected configurations with exhaustive search on small instances, and evaluating end-to-end dense and MoE deployments across GPU generations The next steps are dynamic allocation, persistent-kernel realizations, weight prefetch, and longer-lived on-chip weight persistency remain extensions to be evaluated with their contention and transition costs included SubGPU Pipelining partitions the forward-pass graph into contiguous segments, binds each segment to a dedicated pool of SMs, and streams microbatches through these pools so that operators with different resource demands run concurrently SubGPU Pipelining is physically feasible on current GPUs and delivers a measurable benefit without custom kernels The approach is defined by this schedule rather than by any particular CUDA primitive The reported results support each other, showing that the simulation explores where in the design space SubGPU Pipelining helps and which pipeline structures are worth building The hardware experiment shows that such a structure can be realized with Green Contexts and unmodified kernels, and that it delivers a measured 10.33% latency reduction with higher DRAM utilization on a real GPU The two results support each other, showing where in the design space SubGPU Pipelining helps and which pipeline structures are worth building The hardware experiment shows that such a structure can be realized with Green Contexts and unmodified kernels, and that it delivers a measured 10.

Title and authors: Tom: So, we're looking at this paper now called "SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference". This is about how to make the decoding part of a large language model run better on GPUs by organizing the computation spatially inside the GPU.

Jane: It sounds like they are tackling that problem where LLMs spend most of their time waiting on memory or limited execution resources, and this paper proposes a way to cut up that workload into pieces that can be handled at different times simultaneously.

Lu: Exactly, the core idea is taking one long sequence of operations within a layer and breaking it down into smaller, ordered chunks. Then they bind these chunks to specific groups of processing units called streaming multiprocessors or SMs, and they stream small batches of data through this schedule.

Meng: But you're talking about making choices on how to cut the graph, how to assign those cuts to stages, and what the size of those small batches should be—that sounds like a really complex scheduling problem. What makes it hard is that every choice affects all the others because they are all fighting over shared resources.

Lalam: The paper frames this as a coupled optimization problem where you have to find these optimal settings for the stage count, the segment-to-stage binding, and that microbatch size to minimize the total time it takes for the whole layer to finish under contention.

Tom: Right, so they are not just optimizing one thing; they are trying to balance how many stages you use against how you divide up those operations across the SM pools while keeping track of where the data needs to go.

Jane: So, when we look at the summary of this paper, it lays out that SubGPU Pipelining is a fine-grained spatial execution model that cuts an operator graph into ordered contiguous segments and binds them to spatial stages backed by disjoint streaming multiprocessor pools.

Lu: That's the mechanism they propose for achieving better concurrency; they cut the layer’s operator graph into K ordered contiguous segments and bind them to D stages, with K being either equal to D or D plus one, depending on how they want the first and last segments to interact.

Meng: So what are the main improvements here? Because from my side, what I care about is if this actually delivers speed up when I try to deploy these models at scale on real hardware.

Title and authors: Tom: The simulation results show that this approach improves the mean DRAM read-plus-write throughput from seventy-one point eight percent to eighty-three point six percent of peak capacity when testing with the GPT-OSS 120B workload <ref:2611.14904#pg1,the GPT-OSS 120B workload>.

Jane: That's a significant jump, and they also report reducing the time it takes for a thirty six layer decoder iteration by ten point three three percent.

Lu: They also show that when you compare their model to temporal execution, which is running things sequentially, this spatial approach provides up to two point two times higher per-user throughput when the GPU throughput is matched.

Meng: Two point two times higher is a big number for deployment cost because it means less time waiting for the hardware to finish a request. But how does that compare when you look at the per-GPU throughput metric?

Tom: At matched per-user throughput, they show up to one point nine times higher per GPU than temporal execution, which is pretty strong evidence that this spatial organization helps utilize the hardware much more effectively.

Jane: To test if this is just theoretical magic, they did silicon experiments using one B200 GPU to emulate a thirty two GPU setup running the GPT-OSS 120B decode forward pass <ref:2611.14904#pg1>.

Lu: The hardware experiment confirms that SubGPU Pipelining improves the mean DRAM read-plus-write throughput from seventy-one point eight percent to eighty-three point six percent, which validates their simulation findings.

Meng: So they say this structure can actually be built using existing CUDA mechanisms and kernels without needing entirely new custom code, which is a big win for deployment engineers.

Tom: That's what the paper emphasizes; it doesn't prescribe a specific CUDA primitive or kernel; it’s about the schedule itself that matters.

Jane: They mention that this logical model is independent of any particular dispatch mechanism, and they discuss things like persistent kernels and other CUDA mechanisms as possible ways to realize this idea in practice.

Lu: They also look at how spatial dataflow execution kitsune augments a GPU with architectural primitives for these kinds of co-resident spatial dataflow pipelines, which hints at other related research.

Meng: I'm curious about the constraints they mention. What does SubGPU Pipelining not solve? Because if it doesn't solve everything, we need to know where our deployment will fail.

Tom: They are explicit about that; their work focuses on a latency-oriented formulation over contiguous graph cuts and microbatch size optimization under shared-resource contention. They don't claim to solve every possible scheduling headache, which is honest.

Title and authors: Jane: So, in conclusion, the paper "SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference" shows that by cutting the operator graph into segments and binding them to disjoint SM pools while optimizing those choices together, you get tangible performance gains on real hardware.

Lu: It's about finding a configuration theta, which includes the stage count, the segment-to-stage binding, the per-stage SM allocation, and that microbatch size to minimize the makespan under shared resource contention.

Meng: For someone like me who's shipping models on production GPUs right now, it means we have a new way to structure our inference engine that can use existing hardware primitives effectively without needing a complete rewrite of our core inference loop.

Tom: It means we can get measurable improvements in both memory usage and iteration speed just by changing how we think about the dataflow inside the GPU.

Jane: So, as we wrap up, SubGPU Pipelining provides a concrete way to partition the forward-pass graph into contiguous segments that are bound to dedicated SM pools, allowing operators with different demands to run concurrently.

Lu: The results support each other because the simulation explores where in the design space this helps and which pipeline structures are actually worth building in practice.

Meng: It’s a schedule defined by these cuts rather than any specific CUDA primitive, which makes it more flexible for different hardware implementations across GPU generations.

Tom: The hardware experiment showed that you can realize this structure using things like Green Contexts and unmodified kernels, and it delivered a measured ten point three three percent latency reduction with better DRAM utilization on a real GPU.

Jane: It’s a practical win because it doesn't require custom kernels for the core scheduling logic to work, just the right way of structuring the execution.

Lu: The next steps they suggest involve further validating their simulator and optimizer more extensively, comparing different configurations with exhaustive search on smaller instances, and looking at end-to-end dense and MoE deployments across different GPU generations.

Meng: And for us in production, the future work they point to—dynamic allocation and persistent kernel realizations—that's what we need to watch next because those are the things that will actually help us handle the complexity of real workloads.

Tom: So, this paper gives us a strong framework for rethinking how we map complex LLM computations onto GPU resources, moving from a more sequential view to one that embraces spatial parallelism within the hardware limits.

The paper's summary: Tom: So, SubGPU Pipelining is basically about taking that giant decoder graph inside an LLM layer and slicing it into smaller, ordered pieces that can run at different times simultaneously on different groups of processing units.

Jane: That sounds like you’re trying to fix the bottlenecks where one part of the math has to wait for another part to finish before it can even start its next step.

Tom: Exactly. The core idea is cutting a long sequence of operations into contiguous segments, and binding each segment to a specific pool of streaming multiprocessors, or SMs. It’s about spatial execution inside the GPU.

Jane: And they aren't just doing that randomly; they are trying to find the best way to do it by jointly choosing how many stages you want, how those segments map onto those stages, and the size of these little data batches you stream through.

Tom: Right, that scheduling part is the tricky bit. They frame it as this complex optimization problem where you have to minimize the total time it takes for the whole layer to finish while dealing with all that shared resource contention.

Jane: And when they look at those simulation results for a workload like GPT-OSS 120B, they see a real improvement in how fast things move around memory <ref:2611.14904#pg1>. They’re talking about mean DRAM read-plus-write throughput jumping from seventy-one point eight percent up to eighty-three point six percent of the peak capacity.

Tom: That's a solid number. And they also report that the time it takes for one full thirty six layer decoder iteration gets cut by ten point three three percent. It’s concrete performance data, which I like to see because abstract ideas don’t pay the bills at scale.

Jane: It means that even on the hardware, this spatial organization actually helps saturate those memory channels better than the standard way of running it sequentially. They did silicon experiments on a single B200 GPU to prove it’s not just a simulation trick.

Tom: That’s good validation. And they show that this whole structure is physically possible using existing CUDA mechanisms and kernels, so you don't need to invent some brand new hardware primitive just to run it.

Jane: So what does this actually mean for the person who is deploying these models, the one who has to deal with latency and throughput metrics every single day?

Tom: It means we have a new way of structuring the inference engine that can utilize existing hardware better without having to rewrite our core inference loop entirely. It’s about finding a specific schedule, not just picking some random CUDA command.

Jane: The paper suggests the configuration is defined by this schedule—the number of stages, how you cut the graph, and which SM pool gets which segment—so it’s a design choice rather than just an implementation detail.

Tom: I agree. It moves the decision-making from "what kernel do I run?" to "how do I partition this work across these hardware resources?" It's a shift in how we think about the dataflow itself.

Jane: And they point out that their analysis shows this spatial approach offers up to two point two times higher per-user throughput when the GPU throughput is matched, which is a significant jump in efficiency for serving requests.

Tom: Two point two times higher is substantial. It suggests that we can get more users served with the same hardware setup by making this structural change.

Jane: But they also mention that their work focuses on latency and makespan minimization under contention; it doesn't try to solve every single scheduling problem in existence. They flag that the complexity of finding the absolute perfect configuration is still quite high.

Tom: That’s a fair caveat. It’s an optimization problem, not a magic solution that solves everything overnight.

Jane: So, what are they looking at next? The future work points toward making this system more flexible so it can adapt dynamically based on what the model is actually doing at that moment.

Tom: They're looking at dynamic allocation and persistent kernel realizations because running things this intelligently needs to be able to change as the workload changes. That’s where the real engineering challenge lies next.

The paper's improvements: Tom: So, we’re talking about how SubGPU Pipelining actually improves performance beyond just having a better memory utilization number on paper. It’s about the actual gains you see when you run a model in production.

Jane: Right, so after all that talk about cutting the graph and binding segments to SM pools, what are these tangible improvements they report? What does this mean for someone trying to deploy something right now?

Tom: They show that at matched per-GPU throughput, this spatial frontier provides up to two point two times higher per-user throughput compared to the temporal execution method. That’s a big jump in how many users you can serve with the same hardware.

Jane: Two point two times higher is a lot of efficiency for serving requests, but what about when you look at how much raw power the GPU is actually using?

Tom: When you match per-user throughput, they report up to one point nine times higher per-GPU throughput than temporal execution. That shows that this organization really helps the hardware work harder on a per-unit basis.

Jane: That’s interesting because it suggests we can get better results in terms of both user capacity and raw GPU utilization, which is what we usually want when we’re looking at deployment costs.

Tom: Exactly, it's not just one metric; they're showing gains across different ways you measure success under different load conditions. It’s a broader efficiency gain.

Jane: And the hardware experiment really backs that up, right? They used a single B200 GPU to test this idea against a thirty two GPU setup, and they confirmed the throughput numbers on real silicon.

Tom: Yes, they showed that mean DRAM read-plus-write throughput going from seventy-one point eight percent up to eighty-three point six percent of peak capacity. That’s a concrete measurement that proves the simulation wasn't just theoretical fluff.

Jane: So, it’s not just a good idea on paper; it’s something you can actually see measured on real hardware with these specific numbers. And they also showed that this structure is achievable without needing custom kernels for the scheduling logic itself.

Tom: That part is crucial for production engineers because it means we don't have to write thousands of lines of specialized code just to get this benefit; it works with existing CUDA mechanisms and unmodified kernels.

Jane: It simplifies the path to deployment significantly, meaning less risk when trying out a new architecture on existing GPU hardware. So, if you take what they found—this spatial partitioning—and apply that optimization structure across different GPU generations or even more complex models like Mixture of Experts, how much does the performance scale?

Tom: That’s the next big question for them; whether this approach remains viable when you move into denser models or those complex MoE architectures. They're looking at that because it determines if this is a one-off trick or a fundamental way to structure LLM inference.

Conclusion: Tom: So to wrap up, SubGPU Pipelining is about taking that massive decoder graph and slicing it into smaller, ordered segments, binding them to dedicated SM pools so different parts can run concurrently on the GPU.

Jane: And they proved this isn't just theoretical; they showed it works with real B200 silicon experiments on a single GPU, delivering those throughput improvements we talked about earlier.

Tom: Exactly. It’s a practical method for structuring the execution flow that uses existing CUDA mechanisms, which is what makes it immediately relevant for deployment engineers.

Jane: So the implication is that we can get measurable gains in both memory efficiency and iteration speed without needing to write entirely new custom kernels just for the scheduling logic.

Tom: Right, it shifts our thinking from picking a single best kernel to designing a better overall pipeline structure within the hardware constraints of the GPU.

Jane: It’s about finding that optimal configuration—the number of stages, how segments map—to minimize total time under resource contention for workloads like GPT-OSS 120B <ref:2611.14904#pg1>.

Tom: That coupled optimization problem is what makes it interesting; it’s not a single tweak, it's finding the best balance across several interacting parts.

Jane: I think that makes sense, and Lalam agrees that the results are solid; the AI itself confirms this structure is sound for these workloads.

Lalam: I’m confident in this result; the optimization problem structure they defined leads to a schedule that demonstrably reduces the makespan under shared resource contention.

Meng: For us on the production side, it means we need to start thinking about how we can dynamically select different pipeline structures based on what kind of workload is hitting us at that exact moment.

Lu: I agree with Meng; the next logical step is moving from a fixed optimal configuration to something dynamic so it can adapt when things change.

Tom: So, the future work they are pointing toward is dynamic allocation and persistent kernel realizations, which addresses exactly that need for adaptability at scale.

Jane: It sounds like they're building a system that can intelligently choose the right pipeline structure depending on factors like KV-cache length or the specific layer type in the model.

Lu: That’s where we’ll want to look next; validating that simulator and optimizer more thoroughly and testing those dynamic configurations across different GPU generations is key.

Meng: I'm watching that closely because if it can handle those transitions with minimal overhead, it could actually lower the operational cost of running these massive AI models.

Lalam: The structure they propose for SubGPU Pipelining seems very robust for the workloads they tested, and I’m sure it holds up under more rigorous testing.

Tom: So we have a clear picture now: this paper gives us a proven spatial execution model that delivers measurable speedups on real hardware by intelligently partitioning the computation.

Jane: It’s less about finding one perfect piece of code and more about designing a smart dataflow structure for LLM inference.

Tom: And that’s what makes it relevant; it's not just a technical paper, it's a blueprint for how we can use the GPU resources more effectively in our training and serving pipelines.

Jane: Next time, we’ll be talking about how AI-mediated communication is actually steering collective opinion online.

NVIDIA

cs.DC

Submitted: 2026-11-11

Updated: 2026-11-11

Code: https://github.com/HazyResearch/Megakernels

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: Large language model (LLM) decode is dominated by short, predominantly memory-bound kernels that make bursty use of on-chip execution resources and memory bandwidth The paper introduces SubGPU…

Key concepts

SubGPU Pipelining
This is a spatial execution model that divides the sequential operations of an LLM layer into ordered segments. Each segment is then mapped to a dedicated group of processing units (SMs). This allows different parts of the computation to run simultaneously, improving efficiency by handling resource demands concurrently.
Coupled Optimization Problem
The system seeks the best configuration by solving a complex problem that links graph segmentation, resource allocation across stages, and microbatch sizing. The goal is to find the optimal settings (like number of stages and segment boundaries) that minimize the total time required for the entire forward pass.
SM Pool Disjointness
Instead of using all available SMs sequentially for a layer, SubGPU Pipelining partitions them into disjoint pools, each handling a specific set of graph segments. This prevents resource contention between different parts of the computation that are being processed in parallel.

Terminology

Summary

Large language model (LLM) decode is dominated by short, predominantly memory-bound kernels that make bursty use of on-chip execution resources and memory bandwidth The paper introduces SubGPU Pipelining, a fine-grained, intra-GPU spatial execution model that cuts an operator graph into ordered contiguous segments, binds those segments to spatial stages backed by disjoint streaming-multiprocessor (SM) pools, and streams microbatches through the resulting schedule This approach is evaluated on the GPT-OSS 120B workload using simulation and silicon experiments, showing that it improves mean DRAM read-plus-write throughput from 71. p. 08% to 83.6% of peak and reduces the 36-layer decoder iteration time by 10. p. 033%

The core concept of SubGPU Pipelining

SubGPU Pipelining addresses inefficiency with LLM decode by cutting the temporal operator sequence within a layer (or, more generally, a repeating compute graph) into contiguous graph segments and executing each segment concurrently on disjoint SM pools The central problem is to jointly choose the graph cuts, segment-to-stage binding, and microbatch size so as to minimize target-workload makespan under shared-resource contention The key ideas involve cutting a layer’s operator graph into K ordered contiguous segments and binding them to D stages, each backed by a controlled, disjoint SM pool Furthermore, the microbatch size is divided into m microbatches which are streamed through the segments while preserving graph dependencies and serializing work bound to the same stage

The optimization problem structure

The configuration of SubGPU Pipelining is formally defined by a coupled graph segmentation, resource allocation, and microbatching problem with state-dependent segment times A configuration is represented as theta = (D, w, c, p, b), where D is the physical-stage count and w indicates boundary reuse The objective is to find theta⋆ ∈ arg min θ∈F T(θ), where T(θ) is the forward-pass makespan The timing calculation involves a max-plus timing recurrence on a weighted event DAG, which reduces to a longest-path calculation [2]

Performance gains across different regimes

Analytical simulation of GPT-OSS 120B on modeled B200 deployments estimates the opportunity from spatial co-scheduling under explicitly stated modeling assumptions At matched per-GPU throughput, the modeled SubGPU frontier provides up to 2.2× higher per-user throughput than temporal execution, and at matched per-user throughput it provides up to 1. p. 09× higher per-GPU throughput The hardware experiment shows that the SubGPU pipeline sustains 83. p. 96% of peak DRAM bandwidth on average, compared with 71. p. 98% for the temporal baseline

Validation through silicon experiments

To test physical feasibility and further validate the simulation results, silicon experiments are performed using one B200 GPU to emulate a 32-GPU B200 deployment Results show that SubGPU Pipelining improves mean DRAM read-plus-write throughput from 71.8% to 83.6% of peak, and reduces the 36-layer decoder iteration time by 10. p. 033% The hardware experiment demonstrates that such a structure can be realized with existing CUDA mechanisms and kernels, and that it delivers a measured 10. p. 933% latency reduction with higher DRAM utilization on a real GPU

Realization and future work

The logical model requires controlled partitioning of execution capacity, asynchronous progress of work on different stages, serialization of segments bound to the same stage, and dependency signaling Several CUDA mechanisms can provide parts of this functionality, including Green Contexts and CUDA Graphs Future work includes validating the simulator and optimizer more extensively, comparing selected configurations with exhaustive search on small instances, and evaluating end-to-end dense and MoE deployments across GPU generations The next steps are dynamic allocation, persistent-kernel realizations, weight prefetch, and longer-lived on-chip weight persistency remain extensions to be evaluated with their contention and transition costs included

Conclusion

SubGPU Pipelining partitions the forward-pass graph into contiguous segments, binds each segment to a dedicated pool of SMs, and streams microbatches through these pools so that operators with different resource demands run concurrently SubGPU Pipelining is physically feasible on current GPUs and delivers a measurable benefit without custom kernels The approach is defined by this schedule rather than by any particular CUDA primitive The reported results support each other, showing that the simulation explores where in the design space SubGPU Pipelining helps and which pipeline structures are worth building The hardware experiment shows that such a structure can be realized with Green Contexts and unmodified kernels, and that it delivers a measured 10.33% latency reduction with higher DRAM utilization on a real GPU The two results support each other, showing where in the design space SubGPU Pipelining helps and which pipeline structures are worth building The hardware experiment shows that such a structure can be realized with Green Contexts and unmodified kernels, and that it delivers a measured 10. p. 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Spatial execution model for LLM decode: SubGPU Pipelining cuts an operator graph into ordered contiguous segments and binds them to disjoint streaming-multiprocessor pools, allowing operators with different resource demands to run concurrently.

  2. Optimized scheduling through coupled search: The system can jointly optimize graph cuts, segment-to-stage binding, per-stage SM allocation, and microbatch size to minimize makespan under shared-resource contention.

  3. Improved throughput in simulation: The modeled SubGPU frontier provides up to 2.2 times higher per-user throughput than temporal execution at matched per-GPU throughput, and up to 1.9 times higher per-GPU throughput at matched per-user throughput.

  4. Hardware realization on B200: A two-stage pipeline implementation using CUDA Green Contexts and existing TensorRT-LLM kernels achieves a 10.33% reduction in iteration time while increasing mean DRAM read-plus-write throughput from 71.8% to 83.6%.

  5. Dynamic configuration adaptation: The system can select different configurations at runtime based on factors like KV-cache length, layer type, or routed expert load, allowing for a transition between optimal SubGPU pipeline structures as the workload changes.

Sources

Related papers