Inside the Trap: A Renewal Law for Prefix-Cache Collapse in LLM Agent Serving
cs.DC, cs.PF
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/AkshatValse/surviving-llmstampede
Terminology
Sources
- Formal Analysis of Metastable Failures in Software Systems
- Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
- Service-Induced Congestion in Memory-Constrained LLM Serving
- TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
- Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
- Locality-aware Fair Scheduling in LLM Serving
- CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
- Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
- LLM Query Scheduling with Prefix Reuse and Latency Constraints
- A versatile and accurate approximation for LRU cache performance
- Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
- The Llama 3 Herd of Models
- Scheduling Mixed RL Rollouts Beyond Prefix Locality
- ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
- Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
- TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
- Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends