InferScale: GPU-Native KV Injection for Personalized LLM Serving

Name: InferScale: GPU-Native KV Injection for Personalized LLM Serving
Uploaded: 2026-10-04T00:01:50.750803+00:00
Duration: 348 s
Description: InferScale is a GPU-native system that injects personalized LLM memory directly into vLLM's cache to eliminate repeated prompt prefilling. It achieves near-constant time-to-first-token regardless of context size by precomputing and reusing KV states for retrieved memory facts, maintaining high retrieval accuracy.

InferScale is a GPU-native system that injects personalized LLM memory directly into vLLM's cache to eliminate repeated prompt prefilling. It achieves near-constant time-to-first-token regardless of context size by precomputing and reusing KV states for retrieved memory facts, maintaining high retrieval accuracy.

Open the full episode