SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving
cs.DC
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/ulab-uiuc/LLMRouter
Terminology
Sources
- Accelerating Large Language Model Decoding with Speculative Sampling
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- Latent Space Communication via K-V Cache Alignment
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
- Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
- Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
- Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
- DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- RouteLLM: Learning to Route LLMs with Preference Data
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
- SWE-smith: Scaling Data for Software Engineering Agents
- From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialogue
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends