Linear KV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs
The episode discusses 'Linear KV,' a method for position-independent caching in hybrid LLMs. The hosts explain that instead of mathematically composing all cached states, using just the last chunk's state as a starting point is sufficient. This approach improves quality on some models and significantly reduces latency across all tested configurations.