A JoLT for the KV cache: Near-lossless KV cache compression via joint Lagrangian allocation of Tucker ranks and a rotated residual for llms
The episode discusses a paper introducing JoLT, a method for near-lossless compression of LLM KV caches. It utilizes joint Lagrangian allocation of Tucker ranks and residuals to efficiently manage memory under a single byte budget. This approach achieves significant efficiency gains while maintaining high fidelity, enabling reliable long-context AI applications.