LLM Serving Optimization with Variable Prefill and Decode Lengths
math.OC, cs.AI, cs.LG
Submitted: 2025-08-08
Updated: 2026-09-29
Code: https://github.com/armancohan/long-summarization
Terminology
Sources
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
- Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
- A Universal Load Balancing Principle and Its Application to Large Language Model Serving
- Adaptively Robust LLM Inference Optimization under Prediction Uncertainty
- A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
- Online Scheduling for LLM Inference with KV Cache Constraints
- OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
- GPT-4 Technical Report
- Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
- Don't Stop Me Now: Embedding Based Scheduling for LLMs
- BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
- Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Second-Order Optimization with Asynchronous Curvature
- Online Control via Counterfactual Tracking
- Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability