Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
The work investigates how batch size affects performance when grouping requests that share common initial text prefixes. Findings show that smaller batches with high prefix homogeneity yield much higher throughput because they maximize spatial and temporal locality in the KV cache accesses. The proposed Feather scheduler uses reinforcement learning to dynamically balance batch size and prefix sharing for optimal efficiency.