Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
cs.DC, cs.AI, cs.LG
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: 12 pages, 8 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM agents drive inference engines with closed-loop sessions whose context grows every turn, and serving them economically depends on prefix caching…
Abstract
LLM agents drive inference engines with closed-loop sessions whose context grows every turn, and serving them economically depends on prefix caching: a hit costs an incremental prefill of about a thousand tokens, a miss recomputes a 10--40K-token context. We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime--a 10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes--while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction). The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers. Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation; a first-order workload-balance condition, calibrated independently, brackets when the regime can drain. Getting out is the harder problem: throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes. A CPU offload tier delays the collapse but does not prevent it. A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested.
Sources
- Formal Analysis of Metastable Failures in Software Systems
- Service-Induced Congestion in Memory-Constrained LLM Serving
- CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
- Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees
- Characterizing Metastable Faults and Failures
- Scheduling Mixed RL Rollouts Beyond Prefix Locality
- Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
- When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
- When Cache Pressure Becomes a Trap: Metastability in Agent Serving
- Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving
- EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
- KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
- TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends