Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning

arXiv:2605.26473 · eess.SY, cs.SY · Submitted 2026-05-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning".

Dev: Here's the gist of the paper. Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications <ref:2605.26473#pg2> This paper proposes Orion, a holistic framework designed to co-optimize training latency, plasticity, and stability of state-of-the-art OCL models under strict memory constraints <ref:2605.26473#pg2>.

Rosa: The Core Problem Addressed Online continual learning (OCL) is essential for systems that must adapt in real time in dynamic environments such as transportation, agriculture, and defense <ref:2605.26473#pg2> However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability <ref:2605.26473#pg2>.

Dev: Unlike offline parameter tuning, an online and self-adaptive approach is essential for robust on-device deployment <ref:2605.26473#pg2>.

Rosa: The inherent trade-off space of on-device OCL is inherently conflicted, as batch scaling reduces latency but inflates memory, while larger buffers help stability but risk OOM <ref:2605.26473#pg2>.

Paper summary: Rosa: So we're looking at this paper called "Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning." It’s about making online continual learning, which is learning in real time from new data, work on devices that have very tight memory limits.

Dev: Right. The core idea is that online continual learning is super important for things like self-driving cars or robots that need to adapt right away to new situations as they go. But doing it on small devices means you can't just keep adding more memory, which messes up the trade-offs between how fast you train, how flexible the model stays, and how stable it is.

Taro: So if a system gets overloaded with data or forgets old stuff because of memory limits, that’s a real problem when you are deploying this on an embedded system. I wonder what happens when the world throws something totally unexpected at the robot, and its memory management struggles to keep up?

Rosa: Exactly. This paper proposes Orion as this holistic framework that tries to balance training speed, plasticity, and stability while sticking to those strict memory constraints for state-of-the-art online continual learning models.

Dev: It does this by using something called URGE, which they describe as a unified runtime indicator based on the buckets effect principle. This indicator is supposed to tell the system which resource is actually the scarcest one right now.

Taro: So if we think about it practically, this means Orion dynamically reallocates memory across all those different parts of an online learning system, like how much data you process in a batch and how big your replay buffer is.

Rosa: That’s right. They coordinate that memory reallocation across both the operating system and the actual application level to keep things balanced according to what URGE is measuring.

Dev: The paper shows this relationship clearly in a visualization, showing that you can't just pick one setting because training latency, plasticity, stability, and memory usage all compete against each other.

Taro: So the naive question someone would ask is how does this actually work when things go wrong? If the system hits a memory wall during an unexpected event in a real-world scenario?

Rosa: Orion claims it adapts to different hardware platforms, from high-performance edge servers down to memory-constrained embedded devices, and it consistently meets those memory constraints.

Dev: They report that Orion achieved an average speedup of twelve point one three times compared to online baselines when training latency was the main thing they were targeting <ref:2605.26473#pg3>. But crucially, they say the reduction in plasticity and stability was less than ten percent compared to a brute-force offline method <ref:2605.26473#pg2>.

Taro: A ten percent hit on stability sounds manageable if it means we can deploy this system without running out of memory during a critical moment, right?

Rosa: That’s what they found. They also have this robotic case study where they validated the practical usability of Orion, showing it auto-balances those performance metrics without running into any out of memory errors during deployment.

Paper summary: Dev: It's interesting how low the overhead is reported too. They say the execution overhead is around zero point forty-seven percent to two point ten percent, and memory overhead is almost negligible between zero point zero one six percent to zero point zero four two percent across a lot of scenarios.

Taro: That low overhead makes it sound like it could actually be deployable on hardware that isn't just a fancy server, which is what we need for real-time applications.

Rosa: It seems the main idea behind this paper, "Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning," is providing a system that can handle the complex trade-offs between training speed and model quality when memory is extremely limited on devices.

Dev: The authors focus on co-optimizing those three things—latency, plasticity, and stability—by using that unified runtime indicator, URGE, to manage memory automatically across all components of the online learning process.

Taro: So for someone who just listens to the show who doesn't know deep learning jargon, what does this actually mean for their daily life? Does it mean their phone gets smarter faster in a way that doesn't crash?

Rosa: It means that for applications requiring real-time adaptation, like autonomous systems, Orion provides a way to ensure the system keeps learning and adapting even when resources are severely limited on the device.

Dev: The implication is that we move away from having to manually tune batch sizes or buffer sizes based on guesswork; instead, a runtime mechanism handles that complexity by monitoring the system's scarcest resource.

Taro: It suggests a level of robustness for these online learning systems that we haven't fully seen yet when we push them into truly unpredictable environments where memory pressure spikes randomly.

Rosa: And this framework is built on established tools, it integrates with things like the Avalanche pipeline, which means it’s designed to work within existing training infrastructures for both traditional and resource-constrained settings.

Dev: The paper does mention that Orion supports a lot of model level optimizations too, like automated porting or quantization, and system level things like gradient checkpointing or micro-batch execution.

Taro: So it’s not just about the memory management; it’s a comprehensive way to make online continual learning more practical for real-world deployment on edge devices.

Rosa: That seems to be the main point of this work, connecting the theoretical trade-off space of OCL with practical, resource-constrained on-device realities.

Dev: Overall, "Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning" presents a novel method for managing training latency, plasticity, and stability in ondevice OCL systems under strict memory constraints.

Taro: It shows that you can get significant speedups without making a huge sacrifice in the quality of what the model actually learns over time.

Rosa: The real takeaway is that this framework can be adapted to other continual learning systems, which lays a foundation for future on-device learning in intelligent systems generally.

Conclusion: Rosa: So we’re wrapping up our look at Orion, this paper titled "Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning."

Dev: It’s about how they managed to balance training speed and model stability when memory is really tight on a device.

Taro: They used something called URGE, which is like a runtime indicator that tells the system what resource is running out first.

Rosa: That's right. It dynamically reallocates memory across the learning components—the batch processing, the replay buffers—based on what it measures in real time.

Dev: And they showed that this system can handle different hardware, from big edge servers down to small embedded systems with tight memory limits.

Taro: The results showed significant speedups in training, but they kept the drop in stability and plasticity under ten percent compared to a standard offline method.

Rosa: That's a key finding because it means you can get faster learning without completely ruining the quality of what the model actually remembers.

Dev: It also has very low execution overhead, only about one percent, which makes it look like it could actually run on real hardware without needing a huge power draw.

Taro: What this means for us is that we might finally be able to deploy truly adaptive online learning systems in unpredictable environments where memory pressure spikes unexpectedly.

Rosa: It suggests that the way we manage resources in these systems needs to be totally self-adaptive instead of relying on fixed settings we guess beforehand.

Dev: The authors are showing us a way to handle those trade-offs—latency versus stability—without constantly having to manually tune every single parameter on the fly.

Taro: This framework could be really useful when we think about future autonomous systems that have to learn continuously while operating in places where resources are constantly fluctuating.

Rosa: It lays a foundation for how other online continual learning methods can be made more practical and deployable on actual devices, not just in simulations.

Zexin Li, Nikil Dutt, Cong Liu

University of California, Riverside · University of California, Irvine

eess.SY, cs.SY

Submitted: 2026-05-26

Updated: 2026-11-02

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 82/100

The gist: Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications <ref:2605.26473#pg2> This paper proposes Orion…

Key concepts

Online Continual Learning (OCL)
OCL allows machine learning models, like those in autonomous systems, to learn continuously from new data streams in real-time. This is crucial for dynamic environments where the system must adapt immediately without needing a full retraining cycle.
URGE
URGE is a unified runtime indicator that determines the system's current performance based on its scarcest resource. It uses this indicator to dynamically reallocate memory across different parts of the OCL training process, ensuring optimal performance under memory pressure.
Trade-off Space
This refers to the inherent conflicts in OCL systems where improving one aspect, such as reducing training latency by using smaller batches, often negatively impacts another aspect like model stability or increasing memory usage. Orion manages this space by automatically finding the best balance.
Buckets Effect Principle
This principle states that a system's overall performance is limited by its scarcest resource. Orion uses this to guide its memory management decisions, focusing on optimizing the component that is currently causing the most significant performance bottleneck.

Terminology

Summary

Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications <ref:2605. p. 0

Where the paper says it

Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning Zexin Li*1 , Nikil Dutt†2, and Cong Liu‡1 1University of California, Riverside 2University of California, Irvine Abstract Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications. However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability.

26473#pg2> This paper proposes Orion, a holistic framework designed to co-optimize training latency, plasticity, and stability of state-of-the-art OCL models under strict memory constraints <ref:2605. p. 0
Where the paper says it

Unlike offline parameter tuning, which cannot account for the dynamic shift in memory pressure and workload complexity as OCL progresses, an online and self-adaptive approach is essential for robust on-device deployment. This paper proposes Orion, a holistic framework designed to co-optimize training latency, plasticity, and stability of state-of-the-art OCL models under strict memory constraints, enabling feasible on-device deployment.

26473#pg2>.

The Core Problem Addressed

Online continual learning (OCL) is essential for systems that must adapt in real time in dynamic environments such as transportation, agriculture, and defense <ref:2605.26473#pg2> However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability <ref:2605. p. 0

Where the paper says it

Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning Zexin Li*1 , Nikil Dutt†2, and Cong Liu‡1 1University of California, Riverside 2University of California, Irvine Abstract Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications. However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability.

26473#pg2>. Unlike offline parameter tuning, an online and self-adaptive approach is essential for robust on-device deployment <ref:2605. p. 0
Where the paper says it

However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability. Unlike offline parameter tuning, which cannot account for the dynamic shift in memory pressure and workload complexity as OCL progresses, an online and self-adaptive approach is essential for robust on-device deployment.

26473#pg2>. The inherent trade-off space of on-device OCL is inherently conflicted, as batch scaling reduces latency but inflates memory, while larger buffers help stability but risk OOM <ref:2605. p. 1
Where the paper says it

This gap is critical, as autonomous robots and edge devices increasingly require real-time adaptation under strict hardware budgets in the field, far from data centers and without opportunity for offline retraining. More challenging, the trade-off space of on-device OCL is inherently conflicted: batch scaling reduces latency but inflates memory, larger buffers help stability, but risk OOM, and stronger optimization improves plasticity while increasing compute and memory traffic.

26473#pg2>.

The Self-Adaptive Mechanism

At its core, Orion leverages URGE, a unified runtime indicator grounded in the “Buckets effect” principle that system performance is bounded by its scarcest resource <ref:2605.26473#pg2>. This indicator dynamically reallocates memory across OCL components by jointly coordinating batch processing, replay buffers, and optimization strategies at both the OS and application level <ref:2605. p. 0

Where the paper says it

This paper proposes Orion, a holistic framework designed to co-optimize training latency, plasticity, and stability of state-of-the-art OCL models under strict memory constraints, enabling feasible on-device deployment. At its core, Orion leverages URGE, a unified runtime indicator grounded in the “Buckets effect” principle that system performance is bounded by its scarcest resource, to dynamically reallocate memory across OCL components by jointly coordinating batch processing, replay buffers, and optimization strategies at both the OS and application level.

26473#pg2>. The URGE equation reflects the system’s state by integrating performance metrics such as plasticity, stability, training latency, and memory pressure <ref:2605. p. 1
Where the paper says it

While URGE’s formulation is intentionally pragmatic and interpretable, inspired by Liebig’s Law of the Minimum [20], we demonstrate it is robust across diverse benchmarks, hardware platforms, and OCL algorithms. By integrating performance metrics (plasticity, stability, latency), balancing memory usage with system constraints to prevent OOM errors, and adapting to workload changes using time-dependent thresholds, URGE enables autonomous memory reallocation across OCL components.

26473#pg2>.

System-Level Optimizations

Orion introduces practical engineering enhancements to address OCL deployment bottlenecks <ref:2605.26473#pg2>. These include a unified, multi-threaded data prefetching module that leverages idle CPU resources to proactively load streaming and replay data, reducing training bottlenecks <ref:2605. p. 1

Where the paper says it

In addition, we prototype a practical system for real-world deployment by extending the Avalanche-lib library [21, 22] with system-level optimizations tailored for embedded and edge platforms. A key module is our unified, multi-threaded data prefetcher that leverages idle CPU resources to proactively load streaming and replay data, reducing training bottlenecks.

26473#pg2>. All system-level optimizations are implemented as transparent modules within the prototype, ensuring compatibility with the standard Avalanche pipeline and minimal engineering overhead for end users <ref:2605. p. 5
Where the paper says it

Notably, we prototype a unified, multi-threaded data prefetching module that leverages idle CPU resources to mitigate data loading latency and improve end-to-end training performance. All system-level optimizations are implemented as transparent, drop-in modules within our prototype, ensuring compatibility with the standard Avalanche pipeline and minimal engineering overhead for end users.

26473#pg2>.

Performance and Practical Utility

Orion was evaluated on one edge server and two GPU-enabled autonomous embedded systems, demonstrating its ability to balance training latency, plasticity, and stability in OCL training <ref:2605.26473#pg2>. The results show that Orion achieves significant training speedups while maintaining balanced performance and effectively adapting to various scenarios <ref:2605. p. 0

Where the paper says it

To further demonstrate its practical utility, Orion is integrated into a realistic autonomous navigational robot powered by OCL. The results show that Orion achieves significant training speedups while maintaining balanced performance and effectively adapting to various scenarios, all with minimal runtime, memory, and energy overhead, making Orion a practical solution for on-device continual learning.

26473#pg2>. Furthermore, a robotic case study validated the practicality and efficacy of Orion in real-world OCL scenarios, successfully auto-balancing training latency, plasticity, and stability without encountering any OOM errors during deployment <ref:2605. p. 2
Where the paper says it

• Practical Usability: A robotic case study validated the practicality and efficacy of Orion in real-world OCL scenarios. It successfully auto-balances training latency, plasticity, and stability, without encountering any OOM errors during deployment (Sec.

26473#pg2>.

Overhead Analysis

Orion exhibits low execution overhead ranging from 0.47% to 2.10%, almost negligible memory overhead between 0. p. 2

Where the paper says it

• Low Overhead: Orion exhibits low execution overhead ranging from 0. 10%, almost negligible memory overhead between 0.

016% to 0.042%, and negligible energy overhead of less than 0. p. 2
Where the paper says it

10%, almost negligible memory overhead between 0. 042%, and negligible energy overhead of less than 0.

1% across a rich set of OCL scenarios <ref:2605. p. 2
Where the paper says it

042%, and negligible energy overhead of less than 0. 1% across a rich set of OCL scenarios (Sec.

26473#pg2>. This low overhead establishes Orion as a lightweight, deployable solution for sustainable on-device OCL <ref:2605. p. 0
Where the paper says it

To further demonstrate its practical utility, Orion is integrated into a realistic autonomous navigational robot powered by OCL. The results show that Orion achieves significant training speedups while maintaining balanced performance and effectively adapting to various scenarios, all with minimal runtime, memory, and energy overhead, making Orion a practical solution for on-device continual learning.

26473#pg2>. The memory management for optimization is managed by dynamically adjusting batch processing memory, replay buffer memory, and optimization strategy consumption based on URGE <ref:2605. p. 7
Where the paper says it

Memory allocation for optimization strategies is managed as follows: MOt = ( MOadvanced, if URGEt ≥ Thrt MOdefault, if URGEt < Thrt (5) where MOadvanced represents the memory consumption of more advanced optimization strategies that can improve the model’s ability to learn new knowledge, and MOdefault represents the memory consumption of less memory-intensive optimization strategies. Note that memory management for optimization is particularly challenging because it is correlated to batch size and replay buffer size in the above and could be hard to estimate precisely.

26473#pg2>.

Conclusion

This paper presents Orion, a holistic solution for managing training latency, plasticity, and stability in ondevice OCL for memory-limited autonomous embedded systems <ref:2605.26473#pg2>. The study highlights the trade-offs required in OCL systems on resource-constrained platforms <ref:2605. p. 0

Where the paper says it

Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning Zexin Li*1 , Nikil Dutt†2, and Cong Liu‡1 1University of California, Riverside 2University of California, Irvine Abstract Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications. However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability.

p. 0
Where the paper says it

Orion: Enabling Self-adaptive Memory Management for On-device Online Continual Learning Zexin Li*1 , Nikil Dutt†2, and Cong Liu‡1 1University of California, Riverside 2University of California, Irvine Abstract Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications. However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability.

26473#pg2>. Orion effectively auto-balances these trade-offs while adhering to memory constraints <ref:2605. p. 3
Where the paper says it

3 Motivation This section thoroughly examines the critical trade-offs in on-device OCL under stringent memory constraints, highlighting specific challenges through a series of realistic case studies. Understanding and quantifying these trade-offs provides valuable insights into how to effectively co-optimize plasticity, stability, and resource usage, guiding the development of more robust and efficient deployable solutions.

p. 3
Where the paper says it

3 Motivation This section thoroughly examines the critical trade-offs in on-device OCL under stringent memory constraints, highlighting specific challenges through a series of realistic case studies. Understanding and quantifying these trade-offs provides valuable insights into how to effectively co-optimize plasticity, stability, and resource usage, guiding the development of more robust and efficient deployable solutions.

26473#pg2>. We evaluate Orion across state-of-the-art replay-based OCL algorithms, diverse benchmarks, and three heterogeneous hardware platforms, demonstrating its effectiveness and versatility <ref:2605. p. 2
Where the paper says it

Key highlights of Orion include: • Overall Effectiveness and Versatility: Orion was evaluated on one edge server and two GPU-enabled autonomous embedded systems, demonstrating its ability to balance training latency, plasticity, and stability in OCL training. It adapts to different platforms, benchmarks, OCL algorithms, and user preferences while consistently meeting memory constraints.

p. 2
Where the paper says it

Key highlights of Orion include: • Overall Effectiveness and Versatility: Orion was evaluated on one edge server and two GPU-enabled autonomous embedded systems, demonstrating its ability to balance training latency, plasticity, and stability in OCL training. It adapts to different platforms, benchmarks, OCL algorithms, and user preferences while consistently meeting memory constraints.

26473#pg2>. The design principles of Orion can extend to other continual learning systems, laying a foundation for future on-device OCL systems in intelligent systems <ref:2605. p. 1
Where the paper says it

Beyond algorithms, system-level efforts remain limited: LifeLearner targets hardware-level optimization for meta-continual learning [17], Ekya optimizes offline CL pipelines on multi-GPU servers [18], and Latent Replay focuses on algorithmic changes without systems co-design [19]. Yet none of these efforts jointly address runtime memory constraints, algorithmic efficacy, and system-level co-design for on-device OCL.

26473#pg2>.

Related Work

Online Continual Learning (OCL) addresses the challenge of sequential learning from streaming data <ref:2605.26473#pg2> Replay-based methods, such as ER [1], mitigate catastrophic forgetting by revisiting past experiences during new task learning <ref:2605.26473#pg2>. GEM [3] and AGEM [4] preserve prior knowledge using gradient-based constraints, while GSS [5] enhances memory efficiency by storing informative samples <ref:2605. p. 1

Where the paper says it

Constraintand regularization-based methods, e. , GEM [3], AGEM [4], EWC [6], and LwF [7], preserve past knowledge via gradient projections or parameter consolidation.

26473#pg2>. Orion is the first comprehensive framework to dynamically co-optimize latency, energy efficiency, and learning efficacy for online continual learning on embedded platforms <ref:2605. p. 5
Where the paper says it

1 Design Overview Our proposed system, named Orion, is a holistic framework designed to optimize OCL systems through self-adaptive runtime memory management, balancing system performance and resource constraints. Importantly, Orion is transparent to the online continual learning process and can be seamlessly integrated into existing OCL frameworks.

p. 5
Where the paper says it

1 Design Overview Our proposed system, named Orion, is a holistic framework designed to optimize OCL systems through self-adaptive runtime memory management, balancing system performance and resource constraints. Importantly, Orion is transparent to the online continual learning process and can be seamlessly integrated into existing OCL frameworks.

26473#pg2>.

Orion is highly adaptable across diverse datasets, models, and hardware <ref:2605.26473#pg2>. Built atop Avalanche-lib [21, 22], it seamlessly integrates with scalable DNN training solutions for both traditional and resource-constrained environments <ref:2605.26473#pg2>. Beyond navigational robotics, Orion’s modular and transparent architecture readily extends to other on-device scenarios <ref:2605. p. 0

Where the paper says it

To further demonstrate its practical utility, Orion is integrated into a realistic autonomous navigational robot powered by OCL. The results show that Orion achieves significant training speedups while maintaining balanced performance and effectively adapting to various scenarios, all with minimal runtime, memory, and energy overhead, making Orion a practical solution for on-device continual learning.

26473#pg2>. At the model level, it supports automated porting [55], model merging [56], compression [57–63], and quantization [64–66]. p. 0
Where the paper says it

This capability is critical for robotics and other embodied AI systems that must adapt in real time in dynamic environments such as transportation, agriculture, and defense [9–13]. Unlike offline retraining on fixed datasets, OCL updates the model incrementally in a single pass over new experiences, making both algorithmic behavior and systems design central to practical deployment [14, 15].

At the system level, it can integrate memory-saving optimizations like gradient checkpointing [67, 68], micro-batch execution [69, 70], recomputation [70, 71], and selective layer updates [72–74]. Orion is the first comprehensive framework to dynamically co-optimize latency, energy efficiency, and learning efficacy for online continual learning on embedded platforms <ref:2605.26473#pg2>. The study highlights the trade-offs required in OCL systems on resource-constrained platforms <ref:2605.26473#pg2>. Orion effectively auto-balances these trade-offs while adhering to memory constraints <ref:2605.26473#pg2>. We evaluate Orion across state-of-the-art replay-based OCL algorithms, diverse benchmarks, and three heterogeneous hardware platforms, demonstrating its effectiveness and versatility <ref:2605.26473#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Self-adaptive memory management for online continual learning systems can dynamically reallocate memory across components by jointly coordinating batch processing, replay buffers, and optimization strategies at both OS and application levels to maintain plasticity, stability, and training latency under strict memory constraints.

  2. The improved AI system can achieve significant training speedups while maintaining balanced performance and effectively adapting to various scenarios on resource-constrained platforms with minimal runtime memory, energy overhead (less than 0.1%), and execution overhead (below 2.1%).

  3. The system can autonomously manage the trade-offs between plasticity, stability, and latency by calculating a unified runtime indicator called URGE based on the Buckets effect principle to guide autonomous memory reallocation across training components.

  4. The system can implement a time-dependent threshold that decreases over time to encourage more aggressive optimization policies as training advances, ensuring the system adapts to increasing computational complexity during OCL progression.

  5. The improved AI system can utilize a unified, multi-threaded data prefetching module that leverages idle CPU resources to proactively load streaming and replay data, reducing training bottlenecks and maximizing system resource utilization.

  6. The framework can automatically configure hyperparameters based on user-specified importance orders for metrics like memory, plasticity, stability, and training latency by normalizing weights to ensure the URGE calculation reflects these preferences accurately.

Abstract

Online continual learning (OCL) enables real-time adaptation to new data, making it crucial for dynamic robotic applications. However, its practical deployment is hindered by memory constraints in resource-limited systems, which affect key trade-offs in training latency, plasticity, and stability. Unlike offline parameter tuning, which cannot account for the dynamic shift in memory pressure and workload complexity as OCL progresses, an online and self-adaptive approach is essential for robust on-device deployment. This paper proposes Orion, a holistic framework designed to co-optimize training latency, plasticity, and stability of state-of-the-art OCL models under strict memory constraints, enabling feasible on-device deployment. At its core, Orion leverages URGE, a unified runtime indicator grounded in the ``Buckets effect'' principle that system performance is bounded by its scarcest resource, to dynamically reallocate memory across OCL components by jointly coordinating batch processing, replay buffers, and optimization strategies at both the OS and application level. Furthermore, Orion introduces system-level data prefetching techniques to maximize efficiency. A system prototype of Orion has been implemented using the widely adopted Avalanche-lib and thoroughly evaluated across a diverse range of OCL algorithms, benchmarks, and hardware platforms commonly used in autonomous robotic applications. To further demonstrate its practical utility, Orion is integrated into a realistic autonomous navigational robot powered by OCL. The results show that Orion achieves significant training speedups while maintaining balanced performance and effectively adapting to various scenarios, all with minimal runtime, memory, and energy overhead, making Orion a practical solution for on-device continual learning.

Sources

Related papers