CubeDAgger: Interactive Imitation Learning for Dynamic Systems with Efficient yet Low-risk Interaction

arXiv:2505.04897 · cs.RO, cs.LG · Submitted 2025-05-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "CubeDAgger: Interactive Imitation Learning for Dynamic Systems with Efficient yet Low-risk Interaction".

Dev: Here's the gist of the paper. Interactive imitation learning makes an agent’s control policy robust by stepwise supervisions from an expert The proposed method is designed on a baseline, EnsembleDAgger, with three improvements These improvements are adding a regularization to explicitly activate the threshold for deciding the supervision timing The second improvement transforms the expert-agent switching system to an optimal consensus system of multiple action candidates Third, autoregressive colored noise is injected to the agent’s actions for time-consistent exploration The proposed CubeDAgger is tested in three simulation environments with different dynamic characteristics The results show that CubeDAgger improves the robustness of the trained policies over the conventional EnsembleDAgger without sacrificing dynamic stability during data collection Additionally, a real-world scooping task (see Fig.

Rosa: 1) is demonstrated Although baselines make human supervision errouneous and imitation unsatisfactory, CubeDAgger can acquire robust policies in just 30 minutes of interaction while maintaining comfortable supervision.

Dev: The paper introduces CubeDAgger as a novel Dagger variant to enable Interactive Imitation Learning for dynamic tasks The method is designed on EnsembleDAgger with three ‘C’ improvements.

Rosa: These modifications are: 1) Controlled: To reduce the errors of the hand-tuned safety set, the ensemble uncertainty is explicitly controlled to/under the specified thresholds This is formulated by minimizing a loss function that includes an inequality constraint on ensemble uncertainty.

Paper summary: Rosa: So, looking at CubeDAgger: Interactive Imitation Learning for Dynamic Systems with Efficient yet Low-risk Interaction by Taisuke Kobayashi et al., the main idea is that standard expert-agent switching systems fail on dynamic tasks because they cause abrupt control changes that violate stability.

Dev: The authors propose CubeDAgger to fix this by adding three specific improvements: controlling supervision timing with regularization, creating an optimal consensus system for multiple action candidates, and injecting autoregressive colored noise for time-consistent exploration.

Taro: What this means in practice is they are focusing on making the agent's learning process smoother, ensuring that when it switches its actions based on expert feedback, the robot doesn't suddenly jerk or become unstable.

Rosa: It’s a practical demonstration that by structuring the interaction with controlled timing and using consensus systems for those action candidates, you can get policies that are more robust while still being safe during training in dynamic settings.

Dev: They showed this works in simulations across three different environments and even on a real-world scooping task where it took only thirty minutes of interaction to acquire robust policies.

Taro: But they also have to be honest about the limitations, which is that human experts can still select noisy actions, so the collected data isn't perfect for everything.

Rosa: So, in simple terms, CubeDAgger is an attempt to build a safer and more structured way for robots to learn complex behaviors interactively by managing the timing and exploration explicitly rather than relying on simpler switching methods.

Conclusion: Rosa: So, CubeDAgger is about taking interactive imitation learning for dynamic systems and making it safer by controlling how much supervision happens during training.

Dev: Right, they're building on something called EnsembleDAgger but adding three specific things to make the interaction more controlled and less risky.

Taro: I mean it's about reducing those sudden jumps in control commands that can mess up a robot when it’s moving around dynamically.

Rosa: Exactly, they introduced this idea of controlling the supervision timing with a regularization term so the system decides when to learn based on uncertainty.

Dev: And they also made a consensus system for those multiple action candidates, which stops those abrupt switching behaviors we see in other methods.

Taro: That means instead of just flipping between two choices instantly, the agent finds a smoother path between them while learning.

Rosa: Plus, they injected this autoregressive colored noise into the actions during exploration to keep things consistent over time instead of just random noise.

Dev: So what does this mean for real-world application? It’s about getting robust policies faster in dynamic scenarios without needing a huge amount of perfect expert data.

Taro: They showed it works well in simulations with different movement patterns and even on a real scooping task where they got good results in just thirty minutes of interaction.

Rosa: The main point is that you can get a policy that's tough enough to handle the world's surprises without breaking when you're actively collecting data.

Dev: But they did admit that human experts can still pick tricky actions, so the collected data isn't always perfect, which means we might need more noise-robust learning later on.

Taro: True, the system still needs to be careful when the world throws something unexpected at it during those thirty minutes.

Rosa: So CubeDAgger gives us a way to learn robust control policies interactively by making the process itself more structured and less unstable.

Dev: It’s a step toward building safer learning loops for robots that have to operate in real-time environments where stability is key.

National Institute of Informatics · The Graduate University for Advanced Studies

cs.RO, cs.LG

Submitted: 2025-05-08

Updated: 2026-11-05

Comments: accepted for publication in IROS2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: Interactive imitation learning makes an agent’s control policy robust by stepwise supervisions from an expert The proposed method is designed on a baseline, EnsembleDAgger…

Key concepts

Controlled
This improvement manages the ensemble's uncertainty by explicitly setting thresholds. It minimizes a loss function that includes an inequality constraint on this uncertainty, ensuring the agent learns within safe boundaries defined by human supervision.
Consensus
Instead of abrupt action switching, this system uses an optimal consensus design based on multiple action candidates from the ensemble. This involves solving an Lp norm minimization problem to find a continuous decision between actions, preventing jerky control commands.
Colored Noise
Autoregressive colored noise is injected into the agent's actions instead of simple white noise. This specific noise generation method ensures time-consistent exploration, allowing the agent to explore more effectively without disrupting the expert's established behaviors.

Terminology

Summary

Interactive imitation learning makes an agent’s control policy robust by stepwise supervisions from an expert The proposed method is designed on a baseline, EnsembleDAgger, with three improvements These improvements are adding a regularization to explicitly activate the threshold for deciding the supervision timing The second improvement transforms the expert-agent switching system to an optimal consensus system of multiple action candidates Third, autoregressive colored noise is injected to the agent’s actions for time-consistent exploration The proposed CubeDAgger is tested in three simulation environments with different dynamic characteristics The results show that CubeDAgger improves the robustness of the trained policies over the conventional EnsembleDAgger without sacrificing dynamic stability during data collection Additionally, a real-world scooping task (see Fig. p. 0 1) is demonstrated Although baselines make human supervision errouneous and imitation unsatisfactory, CubeDAgger can acquire robust policies in just 30 minutes of interaction while maintaining comfortable supervision. p. 0

Methodology Overview

The paper introduces CubeDAgger as a novel Dagger variant to enable Interactive Imitation Learning for dynamic tasks The method is designed on EnsembleDAgger with three ‘C’ improvements. These modifications are:

  1. Controlled: To reduce the errors of the hand-tuned safety set, the ensemble uncertainty is explicitly controlled to/under the specified thresholds This is formulated by minimizing a loss function that includes an inequality constraint on ensemble uncertainty.

  2. Consensus: To effectively utilize the multiple action candidates of the ensemble model while eliminating abrupt changes in action due to the switching system, a consensus system is optimally designed This involves solving an Lp norm minimization problem for the i-th action dimension.

  3. Colored: To facilitate exploration while not interfering with the expert behaviors, autoregressive colored noise is introduced for time-consistent exploration The red noise is generated using an autoregressive model as follows.

Control Mechanism Design

The consensus system is formulated as a central tendency of multiple action candidates, i.e., the agent and the expert's actions The formulation involves an Lp norm minimization problem for the i-th action dimension. Different properties of pi are derived according to pi and wk, where pi is defined as a function of ρi. p. 3 The weight wk is designed using the safety definition in EnsembleDAgger. p. 4 This design allows for continuous a c decisions rather than simple switching systems. p. 4

Exploration Strategy

The exploration strategy utilizes autoregressive colored noise instead of white noise Red noise is selected since it can be generated with an autoregressive model as follows. The agent's action is then given by a πk = µπk + 2√K/3σπkϵ red t,k. This design ensures that the gain, 2√K/3, increases the expert’s confidence level relatively only to K ∼ 2K. p. 4

Experimental Validation

Simulations verify that CubeDAgger improves robustness over EnsembleDAgger without sacrificing dynamic stability during data collection Real-world scooping experiments demonstrate that CubeDAgger can learn robust policies from scratch based on just 30 minutes of interaction The proposed method successfully executed all episodes safely while promoting diversity in situations through moderate perturbations.

Conclusion and Limitations

CubeDAgger successfully improved the robustness of trained policies while maintaining dynamic stability during data collection compared to EnsembleDAgger However, as found in the experiments, human experts in dynamic systems cannot always select optimal actions, leading to noisy data. To mitigate this adverse effect, it should be desirable to integrate noise-robust learning algorithms and/or data selection methods. p. 7

References

[1] C. Celemin, R. p. 7 Perez-Dattari, E. p. 7 Chisari, G. p. 7 Franzese, ´ L. p. 7 de Souza Rosa, R. p. 7 Prakash, Z. p. 7 Ajanovic, M. p. 7 Ferraz, A. Valada, ´ J. p. 7 Kober, et al. p. 7, “Interactive imitation learning in robotics: A survey,” Foundations and Trends® in Robotics

[2] S. Ross, G. Gordon, and D. p. 7 Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in International Conference on Artificial Intelligence and Statistics. p. 7 JMLR Workshop and Conference Proceedings

[3] M. Kelly, C. Sidrane, K. p. 7 Driggs-Campbell, and M. p. 7 p. 7 J. Kochenderfer, “Hg-dagger: Interactive imitation learning with human experts,” in International Conference on Robotics and Automation. p. 7 IEEE

[4] K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer, “Ensembledagger: A bayesian approach to safe imitation learning,” in IEEE/RSJ International Conference on Intelligent Robots and Systems. p. 7 IEEE

[5] Y. Cui, D. Isele, S. Niekum, and K. p. 7 Fujimura, “Uncertainty-aware data aggregation for deep imitation learning,” in International Conference on Robotics and Automation. p. 7 IEEE

[6] H. Oh and T. Matsubara, “Leveraging demonstrator-perceived precision for safe interactive imitation learning of clearance-limited tasks,” IEEE Robotics and Automation Letters, vol. p. 7 9, no. 4, pp. 3387–3394

[7] D. G. Black, D. Andjelic, and S. p. 7 E. Salcudean, “Evaluation of communication and human response latency for (human) teleoperation,” IEEE Transactions on Medical Robotics and Bionics, vol. p. 7 6, no. 1, pp. 53–63

[8] D. Liberzon and A. p. 7 S. Morse, “Basic problems in stability and design of switched systems,” IEEE control systems magazine, vol. p. 7 19, no. 5, pp. 59–70

[9] R. Hoque, A. Balakrishna, C. p. 7 Putterman, M. p. 7 Luo, D. S. Brown, D. Seita, B. Thananjeyan, E Novoseller, and K Goldberg, “Lazydagger: Reducing context switching in interactive imitation learning,” in IEEE international conference on automation science and engineering

[10] R. Hoque, A. Balakrishna, E Novoseller, A Wilcox, D. p. 7 S. Brown, and K Goldberg, “Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning,” in Conference on Robot Learning PMLR

[11] R. Hoque, L. Y. Chen, S. Sharma, K. Dharmarajan, B Thananjeyan, P Abbeel, and K Goldberg, “Fleet-dagger: Interactive robot fleet learning with scalable human supervision,” in Conference on Robot Learning PMLR

[12] M. Laskey, J. Lee, R. Fox, A. Dragan, and K Goldberg, “Dart: Noise injection for robust imitation learning,” in Conference on robot learning PMLR

[13] H. Oh, H. Sasaki, B Michael, and T Matsubara, “Bayesian disturbance injection: Robust imitation learning of flexible policies for robot manipulation,” Neural Networks, vol. p. 7 158, pp. 42–58

[14] D. S. Brown, W. Goo, and S. p. 7 Niekum, “Better-than-demonstrator imitation learning via automatically-ranked demonstrations,” in Conference on robot learning PMLR

[15] R. S. Sutton and A. p. 7 G. Barto, Reinforcement learning: An introduction. p. 7 MIT press, 2018

[16] R. C. Wilson, E Bonawitz, V D Costa, and R B Ebitz, “Balancing exploration and exploitation with information and randomization,” Current opinion in behavioral sciences, vol. p. 7 38, pp. 49–56

[17] S. Adams, T Cody, and P A Beling, “A survey of inverse reinforcement learning,” Artificial Intelligence Review, vol. p. 7 55, no. 6, pp. 4307–4346

[18] J. Eßer, N Bach, C Jestel, O Urbann, and S Kernerer “Guided reinforcement learning: A review and evaluation for efficient and effective real-world robotics [survey],” IEEE Robotics & Automation Magazine, vol. p. 7 30, no. 2, pp. 67–85

[19] J. Luo, P Dong, Y Zhai, Y Ma, and S Levine, “Rlif: Interactive imitation learning as reinforcement learning,” in International Conference on Learning Representations, 2024

[20] M. Bain and C Sammut, “A framework for behavioural cloning. p. 7” in Machine Intelligence 15, 1995, pp. p. 7 103–129

[21] W. E. L Ilboudo, T Kobayashi, and T Matsubara, “Adaterm: Adaptive t-distribution estimated robust moments for noise-robust stochastic gradient optimization,” Neurocomputing, vol. p. 7 557, p. 126692, 2023

[22] K. Brantley, W Sun, and M Henaff, “Disagreement-regularized imitation learning,” in International Conference on Learning Representations, 2020

[23] T. Kobayashi, “Lira: Light-robust adversary for model-based reinforcement learning in real world,” Robotics and Autonomous Systems, p. p. 7 105057, 2025

[24] M. Chavent and J Saracco, “On central tendency and dispersion measures for intervals and hypercubes,” Communications in Statistics—Theory and Methods, vol. p. 7 37, no. 9, pp. 1471–1482

[25] I. F Oliveira and R H Takahashi, “An enhancement of the bisection method average performance preserving minmax optimality,” ACM Transactions on Mathematical Software, vol. p. 7 47, no. 1, pp. 1–24

[26] O Eberhard, J Hollenstein, C Pinneri, and G Martius, “Pink noise is all you need: Colored noise exploration in deep reinforcement learning,” in The Eleventh International Conference on Learning Representations, 2023

[27] T. Kobayashi, “Revisiting experience replayable conditions,” Applied Intelligence, vol. p. 7 54, no. 19, pp. 9381–9394

[28] I. Osband, C Blundell, A Pritzel, and B Van Roy, “Deep exploration via bootstrapped dqn,” Advances in neural information processing systems, vol. p. 7 29, 2016

[29] M. Oquab, T Darcet, T Moutakanni, H V Vo, M Szafraniec, V Khalidov, P Fernandez, D HAZIZA, F Massa, A El-Nouby et al. p. 7, “Dinov2: Learning robust visual features without supervision,” Transactions on Machine Learning Research

[30] H Jaeger, “Adaptive nonlinear system identification with echo state networks,” Advances in neural information processing systems, vol. 15, 2002

Pruning redundant reasoning tokens

The paper addresses the problem that expert-agent switching systems cause discrete and abrupt changes in control commands to the robot, which could easily violate the robot’s dynamic stability. This approach is unsuitable for IIL on dynamic systems because of this issue. p. 1

How the training signal is built

The agent's policy is optimized towards π∗ by minimizing the loss function L(θ) This problem is solved by, for example, stochastic gradient descent (e.g. AdaTerm [21] employed in this paper). p. 1 The loss function is L(θ) = E(sn,an)∼D [- ln π(an sn; θ)] / z. p. 1

Where the gains come from

The proposed CubeDAgger successfully improved the robustness of trained policies while maintaining dynamic stability during data collection compared to EnsembleDAgger In real-world scooping demonstration, CubeDAgger successfully acquired robust policies while safely collecting data without excessively interfering with the human operator. Furthermore, the proposed method could be conducted three trials comfortably. p. 6 The test results of the agent policies conducted every five episodes are depicted in Fig. p. 6 6. Consistent with subjective ease of operation, the average a − a c every 5 minutes was smallest with the proposed method. p. 6 The success rate of scooping the three balls was 8/15 among successful episodes. p. 6 The proposed CubeDAgger could more efficiently facilitate the exploration to improve the robustness without violating the dynamic stability. p. 6

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Controlled ensemble uncertainty: The proposed modification optimizes ensemble models not only according to equation (1) but also with an inequality constraint on ensemble uncertainty, preventing all action candidates from perfectly matching the expert's action while limiting deviation to a threshold. This reduces misjudgments of safety while ensuring the effectiveness of the ensemble models.

  2. Consensus system among ensemble actions: The switching decision is transformed into a consensus-making problem using an Lp norm minimization problem to derive an optimal control action c from multiple candidates, which ensures a central tendency is derived from multiple action candidates with safety taken into account. This allows for continuous control decisions rather than simple discrete switching systems.

  3. Colored noise for time-consistent exploration: Autoregressive colored noise is injected into the agent's actions to facilitate exploration while not interfering with expert behaviors, preventing the white noise from causing oscillatory behaviors that might make human operation difficult and cause motion sickness. This design compensates for the difference in the number of candidates from the agent and expert by increasing the expert's confidence level.

Related papers