SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Researchers from Shanghai Jiao Tong University and Alibaba propose SMetric to optimize LLM scheduling for agentic workloads. Unlike human chats, AI agents rely heavily on KV cache reuse, often exceeding 80%. SMetric balances session-centric scheduling by prioritizing load balance for initial requests and memory reuse for subsequent ones, preventing cluster imbalance.