Efficient Test-Time Adaptation through Latent Subspace Coefficients Search

The gist: ELaTTA is a gradient-free framework for single-instance Test-Time Adaptation that adapts each test sample by optimizing a low-dimensional coefficient vector in a source-induced principal latent subspace, achieving stateof-the art accuracy while reducing compute by up to 63× and peak memory by up to 11×

How it works

ELaTTA is designed as a gradient-free TTA method for edge deployment that avoids weight updates by searching for a few latent coefficients in a sourcederived principal subspace. This approach decouples adaptation complexity from backbone size, constrains adaptation drift, and limits catastrophic forgetting in continual settings The framework freezes model weights and adapts each test sample by optimizing a compact k-D coefficient vector with a forward-only objective while keeping all network weights fixed.

Latent Subspace Construction

The offline preparation step involves building a compact latent subspace from source-domain features in a one-off offline stage, after which all source data is discarded and never accessed during deployment. Given a randomly selected set of N source samples, the procedure computes the source mean µs and forms the centered feature matrix Zes before computing a rank-k truncated SVD Zes ≈ UkΣ(z)k V⊤k, where Vk ∈ R D×k has orthonormal columns. This subspace basis Vk is then permanently discarded after computation, retaining only the subspace basis Vk.

Entropy-Based Test-Time Objective

At test time, groundtruth labels are unavailable, so ELaTTA adopts Shannon entropy minimization as a standard self-supervised objective to encourage confident predictions. The adaptation does not update network parameters but searches for a low-dimensional latent coefficients p ∈ R k within the subspace Vk to obtain an adapted latent zadapted = zt + Vkp. A straightforward test-time objective is to minimize the pointwise entropy as min p∈Rk L(p) where L(p) ≜ H(Cls(zt + Vkp)).

Distributional Search via CMA-ES

The optimization of the smoothed objective J(m, Σ) is performed using CMA-ES, a gradient-free optimizer for continuous black-box objectives. The search distribution p ∼ N (m, Σ) is optimized by minimizing J(m, Σ) ≜ Ep∼N(m,Σ) h L(p). CMA-ES maintains a Gaussian search distribution and performs covariance-adaptive updates closely related to natural-gradient steps on the Gaussiansmoothed entropy objective. The algorithm returns the p∗ with the smallest fitness value and corresponding output yˆ∗.

Theoretical Guarantees and Efficiency

Theoretical analysis shows that adaptation is confined to a controlled k-dimensional subspace because adaptation in span(Vk) cannot introduce any update in span(Vk)⊥. Proposition III.2 makes explicit that optimizing (m, Σ) corresponds to descending a Gaussian-smoothed loss in coefficient space, while the induced smoothing in latent space is confined to span(Vk). The expected pointwise loss of the deployed sample p∗ is upperbounded by the smoothed objective E[L(p∗)] ≤ min t E[J(mt, Σt)]. This framework achieves a GFLOPs value of 16.95 and requires only 696 MB in memory, making it highly efficient compared to other methods.

Empirical Results and Deployment

ELaTTA achieves state-of-the-art accuracy under both strict and continual singleinstance protocols across six benchmarks. It reduces compute by up to 63× and memory by up to 11× compared to standard TTA baselines. The framework is demonstrated on a ZYNQ-7020 platform, confirming its practical on-device deployability. Furthermore, the results show that ELaTTA remains highly competitive in continual batch settings and exhibits superior generalization capabilities across severe distribution shifts like DomainNet-126. The analysis of QELaTTA variants shows that QELaTTA-V1 minimizes resource overhead by quantizing p to 1-bit, achieving an average accuracy of 57.63% on the ImageNet series.

Conclusion

ELaTTA presents an efficient single-instance TTA framework with a low-dimensional latent coefficient search in a source-derived principal subspace, effectively optimizing a Gaussian-smoothed entropy objective while keeping all network weights frozen. Instantiated with forward-only CMA-ES, ELaTTA achieves negligible on-device state, making it practical under tight memory and latency constraints. Across diverse datasets, architectures, and distribution shifts, ELaTTA achieves strong robustness in both strict and continual singleinstance settings while substantially reducing compute and peak memory. It is also noted that the fundamental bottleneck lies in the absolute loss of representation semantics rather than the latent adaptation mechanism itself. The next step involves algorithm-hardware co-design to further improve end-to-end efficiency on deployed systems.

--- Page 1 ---

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 1

Efficient Test-Time Adaptation through Latent Subspace Coefficients Search Xinyu Luo, Jie Liu, Member, IEEE,Kecheng Chen, Junyi Yang,Graduate Student Member, IEEE,Bo Ding,Arindam Basu,Fellow, IEEE,and Haoliang Li,Member, IEEE Abstract—Real-world deployment often exposes models to distribution shifts, making test-time adaptation (TTA) critical for robustness. Yet most TTA methods are unfriendly to edge deployment, as they rely on backpropagation, activation buffering, or test-time mini-batches, leading to high latency and memory overhead. We propose ELaTTA (Efficient Latent TestTime Adaptation), a gradient-free framework for single-instance TTA under strict on-device constraints. ELaTTA freezes model weights and adapts each test sample by optimizing a lowdimensional coefficient vector in a source-induced principal latent subspace, pre-computed offline via truncated SVD and stored with negligible overhead. At inference, ELaTTA encourages prediction confidence by optimizing the k-D coefficients with CMA-ES, effectively optimizing a Gaussian-smoothed objective and improving stability near decision boundaries. Across six benchmarks and multiple architectures, ELaTTA achieves stateof-the art accuracy under both strict and continual singleinstance protocols, while reducing compute by up to 63× and peak memory by up to 11×. We further demonstrate on-device deployment on a ZYNQ-7020 platform. Index Terms—Test-time Adaptation, Latent Subspace, Efficient Deployment, Image Classification, Keyword Spotting.

--- Page 2 ---

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 2

GFLOPs Accuracy (%) MEMO Methods (Memory Size) Fig. 1. Accuracy, computation, and peak memory of various TTA methods under the single-instance setting on ImageNet-C with ViT-Base model. No Adapt 55.03% SAR 54.82% Elas. Pix. JPEG Acc. 66.77% No Adapt 55.03% CoTTA 66.66% MEMO 49.21%.

--- Page 3 ---

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 3

I I INTRODUCTION THE heterogeneity of data in real-world applications poses a significant challenge for modern machine learning systems. During deployment, the data encountered (a.k.a. target domain) often deviates from the training data (a.k.a. source domain), resulting in out-of-distribution (OOD) data [1]–[3]. This distribution shift undermines the assumption of identical training and test distributions, causing models to struggle in generalizing effectively. Testtime adaptation (TTA) has emerged as a promising solution, allowing models to adapt dynamically to OOD data during inference, which is critical for ensuring robust and reliable AI systems in real-world settings [4]–[7]. TTA is particularly relevant for on-device and edge deployment, where models are embedded in safety and privacy critical pipelines such as mobile authentication, in-vehicle perception, and voice assistant wake-up [8], [9]. In such scenarios, adaptation often has to be performed locally. A large body of TTA methods performs gradient-based optimization to adjust parameters during inference.

1. Bold header Latent Subspace Coefficient Adaptation for Edge Deployment: The system can perform single-instance Test-Time Adaptation (TTA) by optimizing a lowdimensional coefficient vector in a source-induced principal latent subspace, which allows adaptation without backpropagation, activation buffering, or test-time mini-batches.

2. Smoothed Entropy Optimization via Distributional Search: By optimizing the objective minm, Σ≻0 J(m, Σ) ≜ Ep∼N(m,Σ) h H(Cls(zt + Vkp)), the system stabilizes adaptation by minimizing pointwise entropy and reducing decision flips and confirmation bias.

3. Hardware-Agnostic Deployment: The framework is designed for strict on-device constraints, achieving significant resource reduction, specifically reducing compute by up to 63× and peak memory by up to 11×, and demonstrating practical deployability on platforms like the ZYNQ-7020.

4. Continual Adaptation Efficiency: For streaming scenarios, the system supports continual single-instance adaptation by using a continual warm-start with the previous solution, which allows it to maintain high accuracy while avoiding error accumulation or catastrophic forgetting over long streams due to repeated parameter updates.

5. Robustness to Domain Shifts: The latent subspace approach ensures that adaptation in span(Vk) cannot introduce any update in span(Vk)⊥, providing a mechanism for adaptation that is bounded by the degrees of freedom of per-sample updates and performs well across severe shifts like DomainNet-126.

6. Quantized Inference Efficiency: The introduction of QELaTTA variants allows the system to operate under extreme numerical constraints, with QELaTTA-V1 achieving performance close to full-precision ELaTTA while reducing the optimization to controlling k binary switches, substantially lowering compute and memory costs.