HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving

Name: HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving
Uploaded: 2026-10-17T09:11:19.073260+00:00
Duration: 343 s
Description: HAServe co-designs request scheduling and KV-cache management to optimize fast online LLM serving for segmented and skewed workloads. It uses a workload abstraction called HorizonSchema to decide which requests to admit and which KV-cache entries to keep, aiming to reduce tail latency.

HAServe co-designs request scheduling and KV-cache management to optimize fast online LLM serving for segmented and skewed workloads. It uses a workload abstraction called HorizonSchema to decide which requests to admit and which KV-cache entries to keep, aiming to reduce tail latency.

Open the full episode