Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
The research investigated how simple scheduling policies can improve latency and throughput in continuously batched LLM serving without changing the model itself. By testing eight different scheduling policies against various models, GPUs, and workloads, the study found that no single scheduler is universally best; the optimal choice depends on hardware, model type, and workload characteristics.