Continuous Semantic Caching for Low-Cost LLM Serving
This work creates a theoretical framework for caching LLM responses across an infinite query space where costs and arrival probabilities are unknown. It uses an epsilon-net to discretize the continuous space, allowing algorithms to make decisions by comparing expected serving costs against mismatch costs. The research provides rigorous guarantees for both offline learning and online adaptation.