When Cache Pressure Becomes a Trap: Metastability in Agent Serving
cs.DC
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/llm-d/llm-d-router
Terminology
Sources
- Service-Induced Congestion in Memory-Constrained LLM Serving
- Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
- CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
- DeepSeek-V3 Technical Report
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- MSF-Model: Queuing-Based Analysis and Prediction of Metastable Failures in Replicated Storage Systems
- MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
- ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
- Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
- TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
- Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
- Jamba: A Hybrid Transformer-Mamba Language Model
- Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
- When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
- Autellix: An Efficient Serving Engine for LLM Agents as General Programs
- Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving
- MiniMax-01: Scaling Foundation Models with Lightning Attention
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends