Layer-wise CPU-GPU Scheduling for LLM Inference on Memory-Limited Consumer GPUs
cs.DC
Submitted: 2026-11-01
Updated: 2026-11-01
Code: https://github.com/huggingface/accelerate
Terminology
Sources
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- On-Device Language Models: A Comprehensive Review
- LLM4Drive: A Survey of Large Language Models for Autonomous Driving
- Large Language Models for Robotics: A Survey
- OPT: Open Pre-trained Transformer Language Models
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends