From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment
cs.CL, cs.DC
Submitted: 2026-10-04
Updated: 2026-10-04
Code: https://github.com/predibase/lorax
Sources
- Yi: Open Foundation Models by 01.AI
- SLOs-Serve: Optimized Serving of Multi-SLO LLMs
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- The Llama 3 Herd of Models
- Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
- CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference
- AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
- Artificial Intelligence Index Report 2025
- A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services
- S-LoRA: Serving Thousands of Concurrent LoRA Adapters
- Qwen2 Technical Report
- LLaMA: Open and Efficient Foundation Language Models
- Fast Distributed Inference Serving for Large Language Models
- FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
- Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
- AlpaCare:Instruction-tuned Large Language Models for Medical Application
- SGLang: Efficient Execution of Structured Language Model Programs
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering