Placing LLM Lookup Tables by Per-Layer Prefetch Deadlines
cs.DC, cs.AI, cs.LG
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: 15 pages, 5 figures, 10 tables
Code: https://github.com/vllm-project/vllm
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Memory Layers at Scale
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Low-Bit N-Gram Embeddings for Language Models
- N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
- ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories
- DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- Memory Offloading for Large Language Model Inference with Latency SLO Guarantees
- Pooling Engram Conditional Memory in Large Language Models using CXL
- TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
- N-Grammer: Augmenting Transformers with latent n-grams
- STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
- WildChat: 1M ChatGPT Interaction Logs in the Wild
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends