Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures

arXiv:2610.15899 · cs.DC, cs.LG · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures".

Jane: The gist: Lightweight scheduling policies can improve latency-throughput tradeoffs in continuously batched LLM serving without changing model architectures, kernels, or learned schedulers.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So we're diving into the specifics of "Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures." What exactly is this paper proposing as a solution to this batching problem?

Tom: Well, they set up a testbench using real Hugging Face execution and they evaluate eight different scheduling policies across one thousand one hundred fifty-two GPU model-scheduler-workload runs <ref:2610.15899#pg1,real Hugging Face execution and>. They used traffic from ShareGPT derived workloads and tested them on six Qwen, Gemma, and Mistral models running on Modal L40S, A100, and H100 GPUs.

Lu: The paper shows that the scheduler choice definitely changes four things: throughput, time to first token TTFT, completion latency, and tail behavior across all those different setups.

Meng: I wonder how much of that performance gain actually comes from the scheduling versus just the batch size itself? Because I'm always looking at how much overhead this introduces in a real production environment.

Lalam: The results show that aging shortest-prompt-first is a strong practical baseline, but they also find that oracle remaining-length scheduling isn't enough on its own because hardware and batching effects matter too.

The paper's summary: Tom: Let's unpack what the paper summarizes. They start by defining the problem in autoregressive LLM inference, which is that a request goes through a prefill stage and then repeatedly through a decode stage where we choose which requests get served at each token step.

Jane: So, the central idea they summarize is that this scheduling decision at each decode step is where the system has repeated opportunities to choose which requests to include in the next GPU step.

Lu: They mention classical scheduling theory suggests shortest-job-first and shortest-remaining-processing-time can reduce mean completion time if you know or can estimate job sizes, but LLM serving makes those things complicated because the request cost depends on both prompt length and output length.

Meng: So, the complexity comes from that output length uncertainty. If we don't know how long a response will be, estimating the true remaining work is hard.

Lalam: They address this by using a trace-driven testbench where the target output length is known from the ShareGPT response length to determine when a request completes, which lets them compare heuristics against an oracle style policy that uses that known remaining output length.

The paper's improvements: Jane: Now, let's look at the specific improvements they suggest in "Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures." What are these actionable pieces of advice for us?

Tom: They show that the scheduler choice is a meaningful systems lever. For instance, aging shortest-prompt-first is presented as a strong practical baseline, especially when you look at L40S and mixed workloads.

Lu: But they also found that simply knowing the job size isn't enough because batching and hardware effects shape performance in ways that make a single scheduler universally optimal impossible to find.

Meng: The paper highlights that the best scheduler actually changes depending on the GPU, the model, and even the workload you're running. That means we can't just pick one setting and assume it will work everywhere.

Lalam: They also point out that better length estimates only help if they preserve batching stability and fairness; oracle remaining-length scheduling isn't enough on its own by itself.

Conclusion: Jane: So to wrap up, what's the big picture here? What does this whole investigation into "Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures" actually mean for serving AI models in practice?

Tom: It means that we don't need massive, complex new architectures or totally different inference kernels to get better performance. We can make improvements by tuning the scheduling policy based on deployment-time constraints.

Lu: The key finding is that the best scheduler changes depending on the GPU, model, and workload you're running. So serving systems should expose scheduling as a control you can manage when you deploy something in production.

Meng: For us engineers at startup level, this suggests we need to be very careful about what we expose to the user; a deployment-dependent scheduler is much more robust than a fixed one.

Lalam: I'll just add that better length estimates only help when they preserve batching stability and fairness; oracle remaining-length scheduling is not enough by itself, which was a key finding in this paper.

Tom: That's the gist of it. We can get better latency and throughput by focusing on these lightweight changes to how we manage those active requests during decode steps.

Jane: It’s a lot of data, but it shows that understanding the interaction between prefill, decode, and batching policy is a first-order performance issue that needs careful tuning.

Ashman Mehra, Akshan Agrawal, Aryeh Carmi

Carnegie Mellon University

cs.DC, cs.LG

Submitted: 2026-10-11

Updated: 2026-10-11

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist: Lightweight scheduling policies can improve latency-throughput tradeoffs in continuously batched LLM serving without changing model architectures, kernels, or learned schedulers.

Key concepts

Continuous Batching: This technique keeps a GPU busy by dynamically adding new requests to a batch as older ones finish. It's used in LLM serving because generation happens step-by-step (token by token), allowing the system to efficiently fill the GPU with multiple ongoing generations.
Head-of-Line Blocking: This occurs when a very long generation request stays at the front of a batch for many steps. This can delay shorter, quicker requests waiting behind it, negatively impacting overall system responsiveness and fairness.
Scheduling Policies: These are the rules used by the serving system to decide which active requests get processed at each decode step. Examples include FIFO (first-in, first-out) or Shortest Prompt First (SPF), and their choice directly impacts how fast users see responses and how efficiently the GPU is used.

Summary

The gist: Lightweight scheduling policies can improve latency-throughput tradeoffs in continuously batched LLM serving without changing model architectures, kernels, or learned schedulers.

Problem Definition and Research Questions

Serving large language models is fundamentally different from serving traditional neural networks because generation is autoregressiveA request is not completed by a single forward pass; instead, the system repeatedly executes decode steps, each producing one new token while carrying forward a growing key-value cache.In an online setting, many requests with different prompt lengths, output lengths, and arrival times compete for the same GPU.This makes scheduling a first-order systems problem: the runtime must decide which requests receive service at each decode step while keeping the GPU sufficiently full to maintain high throughput.Continuous batching has become a standard technique for improving LLM serving efficiency.However, continuous batching does not eliminate head-of-line blocking.If long generations remain in the active batch for many decode iterations, they can delay shorter jobs that would otherwise finish quickly.This project investigates a practical question: how much can continuously batched LLM serving improve using only lightweight scheduling changes?Rather than proposing a new model architecture, a new inference kernel, or a full production serving stack, we isolate the scheduling policy as the variable of interest.The goal is not to find a universally optimal scheduler, but to understand how scheduler choice affects latency, throughput, and tail behavior across workloads, models, and GPUs.This research makes three contributions.We implement a pluggable continuous-batching testbench for comparing LLM serving schedulers under fixed runtime constraints.We construct a ShareGPT-derived workload suite with bursty, low-load, steady, uniform, and mixed short/long request patterns.We evaluate eight scheduling policies across six openweight models and three GPU classes.Scheduler choice changes throughput, TTFT, completion latency, and tail behavior under fixed model, GPU, workload, batch size, and prefill budget.Aging shortest-prompt-first is the strongest practical baseline in many L40S and mixed-workload settingswhile oracle remaining-length scheduling shows that job-size knowledge alone is insufficient because batching and hardware effects also shape performance.RQ3 is supported by the cross-GPU results.The best scheduler changes with GPU, model, and workloadso serving systems should expose scheduling as a deployment-time control.Key takeaways. RQ3: the best scheduler changes with GPU, model, and workloadso serving systems should expose scheduling as a deployment-time control.RQ2: better length estimates help only when they preserve batching stability and fairness; oracle remaining length is not enough by itself.RQ3 is supported by the cross-GPU results.The best scheduler changes with GPU, model, and workloadso serving systems should expose scheduling as a deployment-time control.Key takeaways. RQ3: the best scheduler changes with GPU, model, and workloadso serving systems should expose scheduling as a deployment-time control.RQ2: better length estimates help only when they preserve batching stability and fairness; oracle remaining length is not enough by itself.RQ3 is supported by the cross-GPU results.The best scheduler changes with GPU, model, and workloadso serving systems should expose scheduling as a deployment-time control.During prefill, the model processes the prompt and initializes the request’s KV cache.During decode, the model repeatedly generates one token conditioned on the previous tokens.Since decode proceeds incrementally, the serving runtime has repeated opportunities to choose which requests to include in the next GPU step.At each decode step, the scheduler ranks active requests and selects at most max batch size requests for service.The scheduler affects at least four outcomes.It affects throughput because larger or more stable batches improve GPU utilization.It affects time-to-first-token because admission and first service determine how quickly a user sees an initial response.It affects completion latency because scheduling determines how long each request takes from arrival to final token.It also affects tail latency and fairness because aggressive short-job policies can improve average latency while delaying long or low-priority requests.The core challenge is that a deployable scheduler usually does not know the true remaining work of a request.Prompt length is visible at arrival time, but output length is often unknown.In our trace-driven testbench, target output length is known from the ShareGPT response length and is used to determine when a request completes.This lets us compare both practical heuristics and an oracle-style shortest-remaining-tokens policy that uses the trace-derived remaining output length.FIFO is attractive because it is simple and predictable, but it can allow long requests to occupy decode capacity while shorter requests wait.Policies inspired by shortest-job-first or shortest-remaining-processing-time may reduce mean and tail latency by finishing short requests earlier.RQ2 asks how much estimation quality mattersSome policies use visible prompt length as a proxy for job size, while others estimate remaining decode length from prompt length, request class, and online observations.We evaluate whether these estimates improve performance and whether inaccurate estimates can hurt compared to simpler heuristics.RQ3 asks if the best scheduling policy depends on hardware, model, and workloadGPU architectures differ in memory bandwidth, compute throughput, and batching behavior.Models differ in size, context handling, and decode cost.Workloads differ in arrival pattern and length distribution.We therefore expect no single scheduler to dominate universallyFirst-in, first-out (FIFO) Arrival order.Service-tier priority (Tier) Service tier, then arrival order.Shortest prompt first (SPF) Visible prompt length.Longest prompt first (LPF) Negative prompt length.Shortest remaining tokens first (SRTF) GPT-3 output length.

Improvements for AI systems

Continuous batching testbench implementation allows for rigorous comparison of lightweight scheduling policies under fixed model, GPU, workload, batch size, and prefill budget, enabling precise tuning of latency-throughput tradeoffs without requiring changes to model architectures or kernels.
Deployment-dependent scheduler tuning enables the system to select the optimal policy based on hardware constraints; for instance, the best choice is deployment-dependent as evidenced by findings that L40S favors aging shortest-prompt-first on p95, A100 is less friendly.
Incorporating aging and fairness mechanisms into short-job policies can mitigate detrimental effects of long requests; this addresses the finding that aging shortest-prompt-first is a strong practical baseline while simultaneously reducing starvation risk.
Adaptive fair LCB policies can be used to balance conflicting objectives by incorporating multiple signals, as they are noted for combining estimated remaining work, uncertainty, starvation prevention, service-tier fairness, and KV-cache pressure.
The system can dynamically adjust scheduling based on observed online data; this is supported by the policy Estimated shortest tokens first (ESTF) updates per-class and global output-length averages from completed requests, providing a mechanism for real-time adaptation.

Abstract

Large language model serving is increasingly limited not only by model size and GPU speed, but also by scheduling during autoregressive decoding, where continuous batching interleaves requests at token granularity as sequences arrive and finish. We study whether lightweight scheduling policies can improve latency-throughput tradeoffs without changing model architectures, kernels, or deploying learned schedulers. We implement a Python/PyTorch continuous-batching testbench with real Hugging Face execution and evaluate eight policies across 1,152 GPU-model-scheduler-workload runs using ShareGPT-derived traffic, six Qwen, Gemma, and Mistral models, and Modal L40S, A100, and H100 GPUs. Scheduler choice changes throughput, TTFT, completion latency, and tail behavior under fixed model, GPU, workload, batch size, and prefill budget. Aging shortest-prompt-first is the strongest practical baseline in many L40S and mixed-workload settings, while oracle remaining-length scheduling shows that job-size knowledge alone is insufficient because batching and hardware effects also shape performance. The custom dataset has been uploaded to Hugging Face.