RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes

Object interaction tasks have been a focus of advances in imitation learning <ref:2606.24403#pg2> End-to-end methods, dominated by diffusion and flow-based variants have shown leaps in performance while sacrificing interpretability <ref:2606.24403#pg2> In this paper, we revisit a few modern imitation learning benchmarks for object interactions, with the aim of composing a framework that repurposes principled theories of manipulation, preserving both performance and interpretability <ref:2606.24403#pg2> The current work shows promise in using simple interpretable building blocks to learn manipulation skills <ref:2606.24403#pg2>.

RE4 Framework Overview

The RE4 framework composes four lightweight steps—Retrieve, Reframe, Replan, and Replay—inside a receding horizon loop, with every step conditioned on the current manipulation mode <ref:2606.24403#pg2> The design rests on two principles: (i) Relative reframing expressing a demonstration’s actions relative to the manipulated object <ref:2606.24403#pg2> and (ii) Manipulation mode conditioning every step on whether the object is grasped keeps RE4 correct across the discontinuity at contact <ref:2606.24403#pg2>.

Key Components of RE4

The four key steps are defined as follows:

1. Retrieve: RETRIEVE chooses which demonstration to follow and where to follow and returns a single source frame j⋆ through a two-stage lookup, with the manipulation mode as a hard filter ensuring that only frames with matching modes are admissible <ref:2606.24403#pg2>.

2. Reframe: REFRAME transports that frame’s action chunk into the current object frame and emits an end-effector set-point xnear <ref:2606.24403#pg2>.

3. Replan: REPLAN drives the end-effector to xnear <ref:2606.24403#pg2>.

4. Replay: REPLAY executes the reframed chunk <ref:2606.24403#pg2>.

Pose Estimation Strategy

A key component identified is the estimate of the pose of the target object, and while general-purpose 6DoF object pose estimation is non-trivial, we explore lightweight model-free self-supervised training only from the demonstration data to estimate object pose <ref:2606.24403#pg2>. This strategy is surprisingly effective for the studied benchmarks, while remaining lightweight in training <ref:2606.24403#pg2>. For prehensile cases, GETPOSE can be learned in a self-supervised fashion from the demonstration data alone <ref:2606.24403#pg2>.

Performance and Robustness

RE4 matches or exceeds baseline policies across every task and modality, attaining task performance on par with or above the strongest generative/parametric baseline <ref:2606.24403#pg2>. The framework demonstrates robustness in sparse data regimes, observing smaller performance drops compared to Diffusion Policy <ref:2606.24403#pg2>. Specifically, RE4 reaches strong scores at an earlier Tmax when evaluating Push-T on adversarially-sparse initial conditions <ref:2606.24403#pg2>.

Limitations and Future Work

A limitation is that pose estimation error propagates since it conditions the retrieval metric dq and the REFRAME transform <ref:2606.24403#pg2>. Furthermore, we have not evaluated the estimator on a diverse enough set of objects to claim general applicability, particularly for non-prehensile interactions <ref:2606.24403#pg2>. Future work will explore more integrated combinations of learned modules, topological properties of demonstrations, and planning to compose hybrid extensible architectures <ref:2606.24403#pg2>.

Experiments and Benchmarks

We benchmark four tasks across two suites: Push-T [29], Lift [30], Can [30], and Square [30] <ref:2606.24403#pg2>. We evaluate both state-based and image-based variants for each task <ref:2606.24403#pg2>. The results show RE4 is competitive on Lift [30] and Can [30] and strongest on the harder Square [30] task <ref:2606.24403#pg2>. Sparse observation experiments confirm RE4 outperforms Diffusion Policy in sparse data domain <ref:2606.24403#pg2>. The results indicate promising performance in imitation learning benchmarks that outperform baselines, while additionally expressing interpretability (from our retrieval), lightweight training (only image-based pose estimation from demonstration), and robustness to sparse data regimes <ref:2606.24403#pg2>.

References

[1] M. T. Mason. Toward robotic manipulation <ref:2606.24403#pg2>

[2] A. Correia and L. A. Alexandre. A survey of demonstration learning <ref:2606.24403#pg2>

[3] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R Tedrake, and S Song Diffusion policy: Visuomotor policy learning via action diffusion <ref:2606.24403#pg2>

[4] D Wang et al Equivariant diffusion policy <ref:2606.24403#pg2>

[5] S Jiang et al Streaming flow policy: Simplifying diffusion/flow policies by treating robot trajectories as flow trajectories <ref:2606.24403#pg2>

[6] T Bonnaire et al Why diffusion models don’t memorize: The role of implicit dynamical regularization in training <ref:2606.24403#pg2>

[7] C He et al Demystifying diffusion policies: Action memorization and simple lookup table alternatives <ref:2606.24403#pg2>

[8] P Vitiello et al One-shot imitation learning: A pose estimation perspective <ref:2606.24403#pg2>

[9] Y Li et al Geometry-aware policy imitation <ref:2606.24403#pg2>

[10] J Kuffner and J Xiao Motion for manipulation tasks <ref:2606.24403#pg2>

[11] K Hauser and J Latombe Multi-modal motion planning in non-expansive spaces <ref:2606.24403#pg2>

[12] Z Kingston et al Decoupling constraints from sampling-based planners <ref:2606.24403#pg2>

[13] M Dogar and S Srinivasa A framework for push-grasping in clutter <ref:2606.24403#pg2>

[14] E R Vieira et al Persistent homology for effective non-prehensile manipulation <ref:2606.24403#pg2>

[15] Z Kingston et al Using experience to improve constrained planning on foliations for multi-modal problems <ref:2606.24403#pg2>

[16] A J Ijspeert et al Dynamical movement primitives: learning attractor models for motor behaviors <ref:2606.24403#pg2>

[17] S Sobti et al A sampling-based motion planning framework for complex motor actions <ref:2606.24403#pg2>

[18] B Wen et al Foundationpose: Unified 6d pose estimation and tracking of novel objects <ref:2606.24403#pg2>

[19] H Shen et al Efficient diffusion models: A survey <ref:2606.24403#pg2>

[20] A Prasad et al Consistency policy: Accelerated visuomotor policies via consistency distillation <ref:2606.24403#pg2>

[21] J Pari et al The surprising effectiveness of representation learning for visual imitation <ref:2606.24403#pg2>

[22] N Di Palo and E Johns On the effectiveness of retrieval, alignment, and replay in manipulation <ref:2606.24403#pg2>

[23] Y Huang et al Generalized task-parameterized skill learning <ref:2606.24403#pg2>

[24] G Franzese et al Generalizable motion policies through keypoint parameterization and transportation maps <ref:2606.24403#pg2>

[25] J Sosa and D Hogg Self-supervised 3d human pose estimation from a single image <ref:2606.24403#pg2>

[26] T Pan et al Task and motion planning for execution in the real <ref:2606.24403#pg2>

[27] R Shome et al Towards robust product packing with a minimalistic end-effector <ref:2606.24403#pg2>

[28] S M LaValle and J J Kuffner Jr Randomized kinodynamic planning <ref:2606.24403#pg2>

[29] P Florence et al Implicit behavioral cloning <ref:2606.24403#pg2>

[30] A Mandlekar et al What matters in learning from offline human demonstrations for robot manipulation <ref:2606.24403#pg2>

[31] G Wolberg and S Zokai Robust image registration using log-polar transform <ref:2606.24403#pg2>

[32] K He et al Identity mappings in deep residual networks <ref:2606.24403#pg2>

[33] Y Zhou et al On the continuity of rotation representations in neural networks <ref:2606.24403#pg2>

[34] S Ross G Gordon and D Bagnell A reduction of imitation learning and structured prediction to no-regret online learning <ref:2606.24403#pg2>

[35] K Sridhar et al Memory-consistent neural networks for imitation learning <ref:2606.

1. Retrieval-based imitation learning can be improved by using a mode-aware retrieval that is conditioned on manipulation modes to ensure only relevant demonstration frames are considered, as stated in Retrieval For Push-T. This prevents the system from being bound to demonstration coverage by enforcing only frames with matching modes are admissible.

2. Pose estimation for object interaction can be improved by training a lightweight model-free self-supervised estimator directly from demonstration data, which is effective for prehensile cases and avoids the overhead of general-purpose 6DoF pose estimators. This allows the system to recover the manipulated object’s pose using only demonstration data—no object geometry or mesh, no supervision.

3. Reframe can be enhanced by explicitly modeling mode transitions during transformation, as shown in Algorithm 2 REFRAME where the transport operation is defined differently for Mtransit and Mtransfer modes. This ensures that the system correctly handles the discontinuity at contact by either transporting the demo into query’s object frame or expressing current agent in demo’s object frame.

4. Replan can be made more robust by integrating a failure detection mechanism specifically tied to manipulation mode constraints, such as defining FAILURE as a specific condition in Mtransfer mode. This allows the system to detect an unsuccessful grasp and bypass retrieval and reset to a fallback state, here chosen trivially to be initial state.