EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems

arXiv:2505.00986 · cs.LG, cs.CV · Submitted 2025-05-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems".

Jane: The gist Deep neural networks (DNNs) are widely used for perception in embodied visual systems such as drones and AR glasses,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We’ve just covered how EmbodiTTA uses domain shift detection to decide when to trigger adaptation, and now we want to look at what this framework actually claims about its overall approach >

Jane: Right. Essentially, the paper is focused on moving away from continuous test-time adaptation because it’s too costly for real-world deployment on embodied systems like drones or AR glasses >

Lu: They state that existing methods, like continual TTA, which adapt the model on every incoming batch before inference are impractical because they cause substantial memory overhead and energy consumption >

Meng: So they are proposing this on-demand TTA as a practical alternative that only activates adaptation when there's a significant domain shift detected >

Tom: That’s right. They claim that continuous adaptation during stable periods is wasteful because performance stays stable for long durations before dropping noticeably after about three thousand frames, suggesting we don't need to update every batch > <ref:2505.00986#pg2>

Jane: And they introduce EmbodiTTA as the solution, which is an efficient on-demand TTA framework for accurate and efficient TTA on embodied devices >

Lu: The paper sets out the main problem clearly by pointing out that C-TTA relies on backpropagation during inference, which creates memory overhead and this cost gets worse as batch size increases >

Meng: So they are tackling that specific issue with a focus on making adaptation manageable within those constraints rather than just trying to optimize for perfect accuracy at any cost >

Tom: Exactly. They claim their framework avoids unnecessary updates during stable periods by monitoring each incoming batch for potential domain shifts >

Jane: And the implication is that this makes TTA viable for real-world use because it’s not constantly running and incurring huge costs on every single data point >

Lu: The authors are arguing that you can achieve a practical and efficient model adaptation during inference without needing to adapt the model on every batch before inference >

Meng: So the main takeaway is that efficiency trumps continuous updating when dealing with streaming data streams under strict memory and energy budgets >

Tom: Right, so we’ve established *why* this approach is necessary, which brings us to the practical details of how they actually build this system >

Jane: And now we need to talk about the specific mechanisms they use to make on-demand TTA actually work >

Lu: Let's look at those three techniques again, because that’s where the real engineering work happens >

Meng: I'm ready for it. Lay out the three innovative parts of EmbodiTTA for us >

Tom: Alright, let’s break down those components so we can see how they fit together in practice >

Conclusion: Tom: So we’ve covered the details on detection, selection, and decoupling of BN updates for the framework called EmbodiTTA today >

Jane: It really boils down to this idea that you only adapt when a significant domain shift is detected, which is the central concept behind on-demand TTA >

Lu: The authors are proving that you can build a system where adaptation only happens when it's truly necessary for performance improvement >

Meng: And they showed that this saves energy and computation by avoiding unnecessary updates during stable periods >

Tom: And the results, like the forty-seven point two percent reduction in energy consumption and the latency of up to six point seven times faster compared to baselines really prove it’s a practical thing >

Jane: It shows that TTA can be made a reality for embodied systems when you manage it smartly rather than just blindly applying every single batch update >

Lu: The paper’s title, EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems is really descriptive of what the work accomplishes >

Meng: It’s not just about the technical trick; it's about making adaptation fit into real deployment scenarios where you have limits on memory and energy >

Tom: Exactly. It fits that need perfectly. So, this paper shows a viable way forward for handling model adaptation in resource-constrained environments >

Singapore Management University · Samsung AI Center–Cambridge

cs.LG, cs.CV

Submitted: 2025-05-02

Updated: 2026-10-09

Code: https://github.com/rwightman/pytorch-image-models

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist Deep neural networks (DNNs) are widely used for perception in embodied visual systems such as drones and AR glasses, where models must process streaming data under strict memory and energy

Key concepts

On-demand TTA: This paradigm triggers model adaptation only when the system detects a significant change in the data distribution (domain shift). Instead of adapting constantly, it waits for a clear need before performing computationally expensive adaptation steps, making it much more efficient for edge devices.
Lightweight Domain Shift Detection: The framework uses an exponential moving average (EMA) strategy to track sample-wise entropy over time. By comparing the current sample's entropy against a historical baseline, it determines if the data distribution has shifted enough to warrant adaptation, thus activating TTA only when necessary.
Source Domain Selection: This module intelligently chooses the best pre-trained model from a pool of candidates for adaptation. It clusters samples and measures domain similarity using L2 distance to select the source domain that is most closely aligned with the current input data, ensuring high adaptation accuracy.
Decoupled Batch Normalization (BN) Update: This scheme manages memory constraints by separating the updating of BN statistics from parameter fine-tuning. It updates statistics asynchronously using a larger batch size and fine-tunes parameters using a small batch size on reliable samples, allowing adaptation within strict memory limits.

Terminology

Adaptation — 132× in this paper · explained in 1 episode
Domain Shift — 63× in this paper · explained in 6 episodes
Accuracy — 51× in this paper · explained in 1 episode
Distribution Shift — 9× in this paper · explained in 9 episodes
Robustness — 8× in this paper · explained in 5 episodes
Exponential Moving Average — 7× in this paper
Exponential Moving Average (EMA) — 7× in this paper · explained in 2 episodes
Batch Normalization — 6× in this paper
Computational overhead — 6× in this paper · explained in 2 episodes
Distribution Shifts — 6× in this paper · explained in 2 episodes
Domain Adaptation — 3× in this paper · explained in 5 episodes
Edge Devices — 3× in this paper · explained in 2 episodes
Energy Efficiency — 3× in this paper · explained in 3 episodes
Semantic Segmentation — 3× in this paper · explained in 2 episodes
BN with Group Normalization — 2× in this paper
Benchmark — 2× in this paper · explained in 2 episodes
Core — 2× in this paper · explained in 2 episodes
Embodied AI — 2× in this paper · explained in 2 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Generalizability — 2× in this paper · explained in 6 episodes
Loss Function — 2× in this paper · explained in 2 episodes
Test-time Adaptation (TTA) — 2× in this paper · explained in 3 episodes
Activation Sparsity — 1× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Calibration — 1× in this paper · explained in 7 episodes
Corruption Robustness — 1× in this paper · explained in 2 episodes
Decoupling — 1× in this paper · explained in 2 episodes
Exchangeability — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Sparsity — 1× in this paper · explained in 4 episodes
Unsupervised Learning — 1× in this paper · explained in 2 episodes

Summary

The gist Deep neural networks (DNNs) are widely used for perception in embodied visual systems such as drones and AR glasses, where models must process streaming data under strict memory and energy budgets. EmbodiTTA introduces a novel paradigm called on-demand TTA, which triggers adaptation only when a significant domain shift is detected, presenting an efficient solution for resource-constrained devices.

How it works

EmbodiTTA comprises three techniques to achieve resource-efficient test-time adaptation (EmbodiTTA: Resource-Efficient Test-Time Adaptation for Embodied Visual Systems; Page 1). These techniques are:

a lightweight domain shift detection mechanism to activate on-demand TTA only when needed, drastically reducing the overall computational overhead (EmbodiTTA comprises three techniques: 1) a lightweight domain shift detection mechanism to activate on-demand TTA only when needed, drastically reducing the overall computational overhead; Page 1).
a source domain selection module to choose the appropriate starting point for adaptation (Instead of always adapting from the immediately preceding domain as in C-TTA, we propose a similar-domain selection pipeline that identifies and selects the most relevant source domain for adaptation; Page 3).
a decoupled Batch Normalization (BN) update scheme to reduce the memory overhead and achieve stable adaptation (we designed a decoupled BN update scheme that adapts the BN statistics and BN parameters asynchronously with different batch sizes, enabling effective model adaptation within a constrained memory budget; Page 2).

Domain Shift Detection

The framework utilizes an exponential moving average (EMA) strategy to smooth sample-wise entropy by incorporating historical entropy and reducing the influence of a single sample (we introduce an exponential moving average (EMA) strategy to smooth the sample-wise entropy by incorporating historical entropy and reducing the influence of a single sample; Page 5). The formula for calculating the EMA entropy is Et = (1 − m) · Et−1 + m · xt, where Et represents the EMA entropy at time t, m denotes the momentum factor (with a value between 0 and 1), and xt is the entropy value of the current input sample at time t that can be directly obtained from the inference process; Page 5). After each adaptation, EmbodiTTA records the EMA entropy over the next few samples as an EMAbase, which reflects the current model’s capability on the adapted domain data (After each adaptation, EmbodiTTA records the EMA entropy over the next few samples (e.g., 100) as the entropy baseline (EMAbase), which reflects the current model’s capability on the adapted domain data; Page 5). Adaptation is triggered when EMAsample − EMAbase exceeds a threshold (EMAthr), where EMAthr is a user-defined parameter that trades off adaptation frequency against energy efficiency; Page 5).

Source Domain Selection

The source domain selection module identifies and selects the most relevant source domain for adaptation, leading to higher accuracy (Instead of always adapting from the immediately preceding domain as in C-TTA, we propose a similar-domain selection pipeline that identifies and selects the most relevant source domain for adaptation; Page 3). This process involves two essential steps:

constructing a pool with enough candidates to ensure effectiveness from the start (we propose to split the training dataset into multiple subsets and adapt the pre-trained model on each subset; Page 5). Domain features are extracted using the output BN mean from the second BN layer as domain features, and samples are clustered into m subsets using K-Means algorithm (we feed each sample through the pre-trained model and utilize the output BN mean from the second BN layer as the domain features; Page 6).
measuring the domain similarity to identify the candidate most closely aligned with the new domain (we compute the L2 distance between µdomain and each candidate’s stored BN mean µc, and select the closest one: c∗ = arg min c∥µdomain − µc∥2; Page 6). This selected candidate serves as the initialization point for subsequent adaptation (This selected candidate serves as the initialization point for the subsequent adaptation process; Page 6).

Decoupled BN Update

The decoupled BN update strategy separates the updating of BN statistics and BN parameters to manage memory constraints (we propose a decoupled BN update scheme that adapts the BN statistics and BN parameters asynchronously with different batch sizes, enabling effective model adaptation within a constrained memory budget; Page 2).

The first step updates the BN statistics with a larger batch size during a forward pass using an Exponential Moving Average (EMA) approach (we employ an Exponential Moving Average (EMA) approach to integrate the BN statistics of the current batch with historical BN statistics, defined as Sk = (1 − m) · Sk−1 + m · Bk; Page 6).
The second step fine-tunes the BN parameters with a small batch size during backpropagation using an entropy-minimization loss function that only considers reliable samples where entropy falls below a threshold τ (we compute the entropy-minimization loss only over the reliable subset S, defined as samples whose entropy falls below a threshold τ; Page 6).

Evaluation Results

EmbodiTTA achieves superior performance across various settings, showing higher TTA accuracy while reducing energy consumption by 47.2% and latency by up to 6.7× compared to baselines (Our results show that EmbodiTTA achieves higher TTA accuracy while reducing energy consumption by 47.2%, with comparable memory usage; Page 1). For the CORe50 dataset, EmbodiTTA reaches an average mIoU of 83.7% at batch size 1 and substantially outperforms the strongest baseline, SAR (Our results show that EmbodiTTA achieves the best performance across all batch-size settings, reaching 83.7% at batch size 1 and substantially outperforming the strongest baseline, SAR (76.8%); Page 2). Furthermore, EmbodiTTA remains stable across all batch sizes in latency evaluation, processing 10,000 samples in 239s on CIFAR10-C (Our results show that EmbodiTTA processes 10,000 samples in 239s, 6.7× faster than MECTA (latency baseline); Page 7).

Generalizability Across Model Architectures

EmbodiTTA consistently outperforms all baselines on MobileNetV2 and MobileViT (EmbodiTTA consistently outperforms all baselines on MobileNetV2 and MobileViT (Table VI); Page 8). It is also the only method effective at batch size 1 across both architectures, confirming that its gains generalize beyond ResNet-50 (EmbodiTTA consistently outperforms all baselines on MobileNetV2 and MobileViT (Table VI), and remains the only method effective at batch size 1 across both architectures, confirming that its gains generalize beyond ResNet-50; Page 8).

Conclusion

EmbodiTTA proposes a novel concept called on-demand TTA for resource-constrained embodied systems, which triggers adaptation only when a domain shift is detected (We propose a novel concept called on-demand TTA for resource-constrained embodied systems, which triggers adaptation only when a domain shift is detected; Page 8). EmbodiTTA comprises three key components: domain shift detection to monitor distribution shifts on the fly, source domain selection to optimize the efficacy of the source model for adaptation, and decoupled BN adaptation to update the model efficiently under limited memory constraints (EmbodiTTA comprises three key components: domain shift detection to monitor distribution shifts on the fly, source domain selection to optimize the efficacy of the source model for adaptation, and decoupled BN adaptation to update the model efficiently under limited memory constraints; Page 8). The experimental results show that EmbodiTTA outperforms baselines across nearly all evaluation settings in accuracy, latency, and energy consumption while maintaining comparable memory overhead (In summary, EmbodiTTA outperforms nearly all baselines across metrics and offers the strongest trade-off among accuracy, memory footprint, and energy consumption; Page 8).

The gist The paper introduces EmbodiTTA, a novel on-demand TTA framework designed for resource-constrained embodied devices that triggers adaptation only when a significant domain shift is detected.

How it works

EmbodiTTA comprises three techniques to achieve resource-efficient test-time adaptation (EmbodiTTA comprises three techniques: 1) a lightweight domain shift detection mechanism to activate on-demand TTA only when needed, drastically reducing the overall computational overhead; Page 1). These techniques are:

a lightweight domain shift detection mechanism to activate on-demand TTA only when needed, drastically reducing the overall computational overhead (EmbodiTTA comprises three techniques: 1) a lightweight domain shift detection mechanism to activate on-demand TTA only when needed, drastically reducing the overall computational overhead; Page 1).
a source domain selection module to choose the appropriate starting point for adaptation (Instead of always adapting from the immediately preceding domain as in C-TTA, we propose a similar-domain selection pipeline that identifies and selects the most relevant source domain for adaptation; Page 3).
a decoupled Batch Normalization (BN) update scheme to reduce the memory overhead and achieve stable adaptation (we designed a decoupled BN update scheme that adapts the BN statistics and BN parameters asynchronously with different batch sizes, enabling effective model adaptation within a constrained memory budget; Page 2).

Domain Shift Detection

Improvements for AI systems

Bold domain shift detection will be based on EMA entropy calculated using Et = (1 − m) · Et−1 + m · xt, allowing for a lightweight mechanism to monitor distribution shifts without needing ground truth labels, which is essential for real-time edge deployment.
Source domain selection will utilize a process where the model selects the candidate domain most similar to the new domain from a candidate pool by computing the L2 distance between the current batch's BN mean and each stored BN mean, leading to higher accuracy and faster convergence.
Decoupled Batch Normalization (BN) updates will be implemented by updating BN statistics with a larger batch size during a forward pass and then performing parameter fine-tuning with a small batch size using an entropy-minimization loss function only over reliable samples, as defined by "Lent = 1/S Σ X i − X c c=1 pi,c log pi,c, S = [i H(pi) < τ]."
The resulting EmbodiTTA system can perform object recognition and semantic segmentation on resource-constrained devices like the Jetson Orin Nano with a latency reduction of up to 6.7× and energy consumption reduction of 47.2% compared to baselines, maintaining high TTA accuracy across various batch sizes, including batch size 1.
The improved AI system will be deployable in real-world embodied scenarios where models must process streaming data under strict memory and energy budgets by only triggering adaptation when a significant domain shift is detected, thereby reducing adaptation frequency by up to 90% over C-TTA methods.

Abstract

Continual Test-time adaptation (CTTA) continuously adapts the deployed model on every incoming batch of data. While achieving optimal accuracy, existing CTTA approaches present poor real-world applicability on resource-constrained edge devices, due to the substantial memory overhead and energy consumption. In this work, we first introduce a novel paradigm -- on-demand TTA -- which triggers adaptation only when a significant domain shift is detected. Then, we present OD-TTA, an on-demand TTA framework for accurate and efficient adaptation on edge devices. OD-TTA comprises three innovative techniques: 1) a lightweight domain shift detection mechanism to activate TTA only when it is needed, drastically reducing the overall computation overhead, 2) a source domain selection module that chooses an appropriate source model for adaptation, ensuring high and robust accuracy, 3) a decoupled Batch Normalization (BN) update scheme to enable memory-efficient adaptation with small batch sizes. Extensive experiments show that OD-TTA achieves comparable and even better performance while reducing the energy and computation overhead remarkably, making TTA a practical reality.