LipCache: A Local Inference Proxy with Certified Caching for Edge Image Classification Service

LipCache: A Local Inference Proxy with Certified Caching for Edge Image Classification Service

Abstract Summary:

As edge-side vision services continue to expand toward low-latency, high-throughput scenarios, reducing the inference cost of vision models without sacrificing reliability has become a central concern. Existing semantic caching methods largely rely on empirical similarity thresholds; while such thresholds improve hit rates, they tend to introduce silent misclassifications near decision boundaries. To address this, we propose LipCache, a certified semantic caching framework for image classification. Without modifying the existing deployed main model, MainNet, the framework introduces a lightweight network, GuardNet, that maps inputs into a low-dimensional feature space subject to a Lipschitz constraint. It then computes a per-sample certified reuse radius from the local classification margin and the spectral norm of the classification head. At runtime, a cached result is reused only when the query feature falls inside the certified reuse ball; otherwise, the query falls back to MainNet. Thus, cache hits are transformed from empirical threshold tests into geometric certification decisions with explicit theoretical boundaries. Across standard image classification tasks like CIFAR, Tiny-ImageNet, and SVHN, LipCache achieves a measured speedup of up to 1.65× with limited end-to-end accuracy degradation, while all accepted cache hits satisfy the GuardNet-side certified-consistency condition. Furthermore, an enhanced GuardNet training recipe substantially improves cache hit rates in the Tiny-ImageNet multi-class extension while maintaining a certified-consistency rate of 100%. These results demonstrate that per-sample certified reuse can reduce main-model fallback while preserving theoretical consistency, providing a feasible approach to reliable cache-assisted inference at the edge.

Introduction Summary:

Deep-neural-network-based vision services have been widely deployed across diverse domains such as autonomous-driving perception, intelligent manufacturing inspection, real-time surveillance, and mobile augmented reality. The dual demand of these services for inference accuracy and response speed has made high-accuracy neural models a de facto standard component. However, the parameter count and computational complexity of such models keep growing, so that a single inference already consumes considerable compute and memory resources. This tension is further amplified in industrial edge scenarios. Unlike occasional photo recognition on a phone, industrial edge devices—such as production-line cameras and roadside perception units—typically operate continuously at high sampling rates, injecting a large volume of concurrent queries into the inference service within very short intervals. When these queries converge at the cloud, they readily trigger spikes in queries-per-second, network congestion, and server overload, severely threatening the overall service availability and tail latency. In other words, the root cause of the problem lies not only in the high cost of a single inference, but more critically in the fact that a large number of semantically redundant queries are repeatedly executed without discrimination.

The rise of the edge computing paradigm offers a structural opportunity to mitigate this risk. By pushing part of the computation or storage capability to edge nodes close to the data source, the system can potentially absorb a fraction of the query load locally, thereby reducing its dependence on the remote cloud link. New challenges arise, however: the latency and memory budgets of edge hardware are extremely tight and far from sufficient to host a full-sized model comparable to its cloud counterpart. Directly deploying a high-accuracy classifier on an edge device almost inevitably faces a sharp conflict between accuracy and resources.

To address this tension, existing work proceeds along roughly two lines. The first line aims to reduce the execution cost of the model at the edge, including compact architecture design, model compression, and early-exit strategies. These methods substantially improve edge deployability, yet their unit of optimization remains how to run the model more cheaply; they do not touch upon a more fundamental question: for the many semantically repetitive queries, is it really necessary to run the model every time? The cache-assisted inference starts precisely from this gap. It exploits the pervasive redundancy of visual patterns in edge workloads and reuses previously computed results to skip part of the model invocations directly. However, current reuse strategies suffer from a structural weakness in reliability: exact matching is extremely sensitive to compression, cropping, and illumination changes, and can hardly enable effective reuse in real deployments; semantic caching, although it enlarges the reuse region, mostly relies on heuristic similarity thresholds and provides no formal guarantee on the correctness of the reused labels. Consequently, cache hits near decision boundaries may introduce silent misclassifications and uncontrollable accuracy loss. A caching strategy that combines an explainable safety boundary with practical reuse efficiency is still missing.

This paper proposes LipCache, a certified semantic caching framework that comprehensively exploits the storage and computation capability of edge devices. Its core design principle is modularity: rather than modifying the deployed high-accuracy main model (MainNet), it introduces a lightweight guard network, GuardNet, as a new branch to construct the semantic cache. Through operator-norm-controlled convolutions and spectral normalization, GuardNet maps inputs to a smooth low-dimensional feature space, and, starting from the local classification margin and the Lipschitz bound of the classification head, computes a per-sample certified reuse radius offline for each cached sample. Online, only when the query feature falls inside the certified ball of some cached sample, namely when their distance is less than the certified reuse radius, does the system reuse the stored label; otherwise it falls back to MainNet to preserve accuracy. Thereby, a cache hit is converted from an empirical threshold decision into a geometric certification decision with an explicit theoretical boundary.

We validate the core certification mechanism of LipCache on benchmarks with markedly different statistical structure: CIFAR (natural images), Tiny-ImageNet (higher-resolution fine-grained inter-class differences), and SVHN (domain-shifted digits and characters). This choice is designed to isolate the stability of the certified radius under different task geometries, rather than to exhaust all visual task scales; larger-scale settings (e.g., full ImageNet) and non-i.i.d. streaming scenarios are left as open directions discussed in Section 7. On these three tasks, the average hit rates under the tight certified radius reach 0.342, 0.472, and 0.502, respectively, with end-to-end accuracies of 0.921, 0.867, and 0.960, and corresponding measured speedups of 1.32×, 1.31×, and 1.65×; moreover, the certified-consistency rate of all results is 100%. Comparisons with empirical thresholds and the global nearest-neighbor baseline show that the certified radius is the only reuse boundary that can stably preserve certified consistency across the three tasks. Furthermore, in the Tiny-ImageNet multi-class extension, an enhanced GuardNet training recipe improves the hit rates of the 20/30/50-class tasks from 0.056/0.005/0.0004 to 0.423/0.254/0.124, while both the end-to-end accuracy and the certified consistency are preserved.

The main contributions are as follows:

1. We propose a dual-model cache-assisted inference framework that comprehensively exploits the storage and computation capability of edge devices and, without modifying MainNet, makes certified reuse decisions through a lightweight GuardNet.

2. Starting from the local classification margin and the Lipschitz bound of the GuardNet classification head, we derive a per-sample certified reuse radius in the feature space, providing a computable and interpretable rule for safe semantic reuse.

3. We systematically evaluate LipCache across multiple benchmark tasks, multi-class extensions, and edge-deployment boundaries, verifying the effectiveness of the certified radius as the only cross-task stable certified boundary and revealing the potential of the enhanced training recipe in enlarging the certified reuse region.

Related Work Summary:

Edge image classification faces a structural tension: high-accuracy neural models keep growing in their computational and memory demands, whereas edge devices must satisfy tight latency, energy, storage, and connectivity constraints. A practical edge system therefore requires more than just a smaller classifier. It must reduce computation, exploit redundancy among correlated inputs, and decide when a low-cost shortcut is safe rather than merely fast. The literature addresses these needs along three main lines and one important bridge. Efficient edge inference reduces the cost of running a model through compact architectures, adaptive execution, hardware–software co-design, or edge–cloud partitioning. Cache-assisted inference and semantic reuse avoid part of the execution by reusing results from temporally or semantically correlated inputs. Certified nearest-neighbor retrieval forms a bridge, studying when a nearest-neighbor decision in an embedding space remains invariant under perturbation. Lipschitz control and certification provide the tools that turn margins and continuity bounds into local stability guarantees. LipCache connects these lines: it uses a lightweight GuardNet as a cache-oriented front end to the high-accuracy MainNet, and accepts reuse only inside a per-sample certified reuse radius in the GuardNet feature space.

Problem Description Summary:

Consider an image classification system whose input space is X ⊆ Rn and whose label set is Y = 1,..., C. The system may invoke a high-accuracy classifier M: X → Y, but a single inference of M is costly—in edge scenarios this may amount to a combination of GPU time, cloud communication latency, and other overhead. For any input x ∈ X, the system ultimately needs to output a prediction ŷ ∈ Y.

A key observation is that edge vision workloads are not composed of mutually independent random samples. The input–label pairs (x, y) are drawn from a joint distribution D that often exhibits pronounced structural redundancy: influenced by factors such as a fixed camera viewpoint, periodic sampling, and slow scene changes, consecutively arriving queries are highly clustered in pixel space or semantic space, and adjacent inputs frequently share the same label. Under these conditions, executing M in full for every query means that a large amount of computation is wasted on samples whose answers could have been inferred from existing results. This observation points to a natural optimization path: cache a subset of historical predictions, and when a new query is sufficiently similar to some cached sample, directly reuse its label, thereby skipping the invocation of M.

However, the above path faces a fundamental tension. If the reuse condition is too loose, queries near a decision boundary may be incorrectly matched to an inappropriate cached label, leading to silent accuracy loss; if the reuse condition is too strict, cache hits will be so rare that the savings in M invocations become negligible. The core challenge is therefore not whether to introduce a cache, but how to define a reuse rule that simultaneously achieves a sufficient hit rate to yield meaningful system gains and constrains the resulting prediction errors to a controllable range.

Cache-assisted inference can thus be formalized as the following constrained optimization problem: min InvokeRateM (π) subject to Accsys (π) ≥ AccM − ∆acc, where ∆acc ≥ 0 is a user-specified accuracy tolerance. A smaller ∆acc enforces a more conservative reuse strategy, tending to depress the hit rate to reduce accuracy risk; a larger ∆acc allows more aggressive cache reuse, trading potential accuracy for a lower M invocation frequency.

LipCache Framework Summary:

LipCache adopts a dual-model design, comprising a high-accuracy main model M and a lightweight guard model G. The main model provides the reference prediction: ŷM (x) = arg maxc∈Y Mc(x). The guard model maps each input to a low-dimensional feature space, G: X → Rd. Let z = G(x); an affine classifier operates in this space: s(z) = W z + b, where W ∈ RC×d, b ∈ RC, yielding the GuardNet prediction ŷG (x) = arg maxc sc(G(x)).

Each cache entry is a triple (zi, ỹi, ri), containing a GuardNet feature, a payload label, and a certified reuse radius. The training objective is designed as a layered framework: Ltotal = (1 − α)LCE + αLdistill + λmLmargin, where α ∈ [0, 1] balances direct supervision against an optional distillation signal from M, and λm controls the margin-regularization strength. When α = 0, the objective reduces to the pure cross-entropy training used in the 10-class main experiments of this paper.

Lipschitz control requires the feature map to satisfy Lip(G) ≤ 1. To this end, GuardNet imposes an operator-norm constraint on all parameterized layers. For a convolutional layer whose kernel is K ∈ Rcout×cin×k1×k2, the Lipschitz constant of the convolution operator is given by its frequency-domain representation: Lip(K) = maxω σmax(K̂(ω)), where K̂(ω) is the frequency-domain matrix obtained by zero-padding the kernel and applying a two-dimensional real FFT. Dividing the kernel by max(Lip(K), 1) before each forward pass enforces the 1-Lipschitz constraint. Linear layers use standard spectral normalization; ReLU, non-overlapping average pooling, and the final tanh projection are all non-expansive. Since the Lipschitz constant of each individual layer does not exceed 1, by submultiplicativity of operator norms under composition, the overall guard network satisfies Lip(G) ≤ 1.

Margin regularization directly serves cache efficiency: the certified reuse radius ri is proportional to the GuardNet classification margin, so explicitly encouraging large margins is a key lever for improving the hit rate. Let c1 and c2 denote the indices of the top-1 and top-2 logits: m(x) = [FG (x)]c1 − [FG (x)]c2. To avoid blindly enlarging the margin on already misclassified samples, the regularizer is applied only to the correctly classified subset of a batch B: Scorr = (x, y) ∈ B arg maxc∈Y[FG (x)]c = y, Lmargin = −(1/max(1, Scorr)) Σ(x,y)∈Scorr m(x).

For each candidate sample xi, we first compute zi = G(xi), the GuardNet label yiG = arg maxc [FG (xi)]c, and the payload ỹi selected by the construction strategy. This paper considers three payload semantics. The first is the proxy label, which directly stores ỹi = yiG; in this case the cached payload is naturally aligned with the certificate, and it is the default setting adopted in the main-text experiments. The second is the MainNet prediction scheme, which stores ỹi = ŷM (xi); the third is the ground-truth scheme, which stores ỹi = yi.

At runtime, each input xnew is encoded as znew = G(xnew), and the hit set is H(znew) = i ∥znew − zi∥2 < ri. If H ≠ ∅, it takes the payload from the nearest hit entry: i∗(xnew) = arg mini∈H(znew) ∥znew − zi∥2. The system predicts ŷsys(xnew) = ỹi∗(xnew) if H(znew) ≠ ∅, otherwise ŷM(xnew) if H(znew) = ∅. On a miss, the system falls back to M; if write-back is enabled, a missed sample may be added to the cache under the same construction strategy.

The computational overhead of the above pipeline directly determines the practical gain of the framework. Let CG and CM denote the per-query cost of GuardNet and MainNet, respectively. When maintaining N cache entries in a d-dimensional feature space, the expected cost per query is E[Cost] = O(CG + N d + InvokeRateM CM). When CG + N d is much smaller than CM, the system can reduce the overall overhead in expectation even if the hit rate is not high.

Theoretical Analysis Summary:

The structure of the …

Based on this paper, here are specific improvements to AI systems and what the improved systems can do:

1. Certified Semantic Caching for Inference Acceleration

• Improvement: Implement a dual-model architecture (MainNet + GuardNet) with Lipschitz-constrained feature extraction and per-sample certified reuse radii, replacing heuristic similarity thresholds with geometrically provable cache-hit decisions.

• Improved System Capability: An edge inference system that accelerates image classification by 1.3–1.65× while guaranteeing that every cache hit is theoretically consistent with the guard model's prediction, eliminating silent misclassifications near decision boundaries.

2. Formal Safety Guarantees for Reuse Decisions

• Improvement: Derive and enforce a certified reuse radius r i = m i / (sigma max(W) sqrt 2) from local classification margins and spectral norm bounds, with optional tighter per-class radii r i,j = (f y i G(z i) - f j(z i)) / W y i G - W j 2.

• Improved System Capability: An AI system that can provably guarantee label invariance within a defined input neighborhood, enabling safe reuse of prior computations without risking incorrect outputs—critical for autonomous driving, medical imaging, or industrial inspection where silent errors are unacceptable.

3. Margin-Aware Training for Cache Efficiency

• Improvement: Incorporate margin regularization L margin = -1 over S corr sum(x,y) in S corr m(x) into the guard network's training objective, applied only to correctly classified samples to avoid amplifying errors.

• Improved System Capability: A classifier that learns feature spaces with larger decision-boundary margins, increasing cache hit rates by up to 7.5× (e.g., from 0.056 to 0.423 in 20-class tasks) without sacrificing certified consistency, making caching viable for multi-class and fine-grained recognition.

4. Modular Integration Without Retraining Main Model

• Improvement: Deploy GuardNet as an external, lightweight branch that leaves the existing high-accuracy MainNet untouched, using distillation (L distill) to align GuardNet's predictions with MainNet's.

• Improved System Capability: Organizations can retrofit existing production classifiers with certified caching by adding a small guard network (e.g., 355 KB), achieving immediate latency reductions without retraining or modifying their core models—enabling seamless adoption in legacy systems.

5. Adaptive Cache Management with Certified Entry Selection

• Improvement: Use radius-prioritized de-duplication for cache population, prioritizing entries with larger certified radii, and support multiple payload semantics (proxy label, MainNet label, ground truth) with explicit trade-offs.

• Improved System Capability: An AI system that automatically maintains a cache of high-value, certifiably safe samples, maximizing hit rates under memory constraints (e.g., increasing cache from 100 to 400 samples per class boosts hits from 0.288 to 0.392 on CIFAR-10) while allowing operators to choose between strict safety (proxy labels) or higher accuracy (MainNet labels).

6. Robustness to Input Perturbations

• Improvement: Leverage the certified radius to provide inherent robustness guarantees against additive noise, JPEG compression, and mild geometric transforms, as demonstrated by 100% certified-consistency rates under stress tests.

• Improved System Capability: An image classification service that maintains correctness guarantees even when inputs are degraded by real-world conditions (compression artifacts, sensor noise), with graceful performance degradation (e.g., hit rate drops from 0.343 to 0.310 under strong JPEG) rather than silent failures.

7. Edge-Cloud Load Balancing with Latency-Aware Decisions

• Improvement: Implement a cost model E[Cost] = O(C G + Nd + InvokeRate M C M) to dynamically decide between edge-local cache lookup and cloud fallback based on measured RTT, encoding cost, and search overhead.

• Improved System Capability: A distributed inference system that optimally balances edge and cloud resources, achieving speedups of 1.22–1.42× across LAN/WiFi/cellular connections and maintaining acceptable p99 latency (e.g., 177 ms at 1,000 queries/s) even under heavy load, by adaptively routing queries based on network conditions.

8. Scalable Multi-Class Certification

• Improvement: Apply the enhanced training recipe (combining cross-entropy, distillation, and margin regularization) to scale certified caching from 10 to 50 classes, maintaining 100% certified consistency while improving hit rates from near-zero to 0.124–0.423.

• Improved System Capability: A general-purpose classification system that can be extended to larger label spaces (e.g., product recognition, scene understanding) while preserving theoretical safety guarantees, making certified caching practical beyond toy benchmarks.

9. Energy-Aware Future Architecture (SNN-Based GuardNet)

• Improvement: Explore spiking neural network (SNN) variants of GuardNet for event-driven, energy-efficient operation, with teacher-student distillation from ANN guards (currently 74.26% accuracy vs. 76.84% ANN, with potential for neuromorphic hardware deployment).

• Improved System Capability: A future ultra-low-power edge system that performs certified caching using spiking neurons, reducing energy consumption by orders of magnitude compared to ANN-based guards, suitable for battery-powered IoT devices, while maintaining safety guarantees once SNN training and Lipschitz control are refined.

10. Interpretable Confidence for Cache Hits

• Improvement: Provide per-sample certified radii as interpretable confidence scores, allowing operators to visualize the safety margin of each cached decision (e.g., correct hits have mean distance ratios of 0.75–0.80 vs. 0.89–0.91 for incorrect hits).

• Improved System Capability: An AI system that outputs not just predictions but also provable safety bounds for each reused result, enabling human operators to audit decisions, set custom risk thresholds, and trust the system in high-stakes applications where explainability is mandatory.

As edge-side vision services continue to expand toward low-latency, high-throughput scenarios, reducing the inference cost of vision models without sacrificing reliability has become a central concern. Existing semantic caching methods largely rely on empirical similarity thresholds; while such thresholds improve hit rates, they tend to introduce silent misclassifications near decision boundaries. To address this, we propose LipCache, a certified semantic caching framework for image classification. Without modifying the existing deployed main model, MainNet, the framework introduces a lightweight network, GuardNet, that maps inputs into a low-dimensional feature space subject to a Lipschitz constraint. It then computes a per-sample certified reuse radius from the local classification margin and the spectral norm of the classification head. At runtime, a cached result is reused only when the query feature falls inside the certified reuse ball; otherwise, the query falls back to MainNet. Thus, cache hits are transformed from empirical threshold tests into geometric certification decisions with explicit theoretical boundaries. Across standard image classification tasks like CIFAR, Tiny-ImageNet, and SVHN, LipCache achieves a measured speedup of up to 1.65 times with limited end-to-end accuracy degradation, while all accepted cache hits satisfy the GuardNet-side certified-consistency condition. Furthermore, an enhanced GuardNet training recipe substantially improves cache hit rates in the Tiny-ImageNet multi-class extension while maintaining a certified-consistency rate of 100%. These results demonstrate that per-sample certified reuse can reduce main-model fallback while preserving theoretical consistency, providing a feasible approach to reliable cache-assisted inference at the edge.