GoodServe: Globally-coordinated Routing of Agentic LLM Inferences for High Goodput
cs.DC
Submitted: 2026-05-16
Updated: 2026-11-02
Code: https://github.com/BerriAI/litellm
Terminology
Sources
- Efficient and Scalable Agentic AI with Heterogeneous Systems
- SLICE: SLO-Driven Scheduling for LLM Inference on Edge Computing Devices
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- The Llama 3 Herd of Models
- M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
- SEW: Self-Evolving Agentic Workflows for Automated Code Generation
- HEXGEN-FLOW: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL
- AskDB: An LLM Agent for Natural Language Interaction with Relational Databases
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Preble: Efficient Distributed Prompt Scheduling for LLM Serving
- AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure
- STAR: Decode-Phase Rescheduling for LLM Inference
- Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
- SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips
- Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends