RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback".
Dev: Here's the gist of the paper. RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning Interface Design and Hardware RUMI is designed as a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85 A scissorlike input drives the coupled fingers while a rotary encoder measures the resulting grasp coordinate Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations The hardware features a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers A Teensy 4.0 microcontroller coordinates on-device sensing and actuation while synchronizing measurements Augmented Reality Visualization The method integrates the tracking system into a virtual reality headset to provide real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform During data collection, RUMI poses are stored in the registered robot base frame established by the AR system IRIS renders a MuJoCo-based digital shadow of the target robot during recording to allow the operator to observe the robot configuration associated with the demonstrated motion in real time This visualization allows users to recognize infeasible poses while recording rather than filtering them only after collection Force-Aware Teleoperation Mechanism The haptic feedback mechanism supports force-aware teleoperation for simulated and real robots During teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load This mechanism allows the same handheld device to support both in-the-wild collection and teleoperation Data Quality and Policy Training The AR condition results in fewer number of joint limit- and velocity violations compared to the w/o AR condition The average manipulability is 19.6% higher than the w/o AR condition, although the standard deviation is also greater For the CoinFT sensor, force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors The DIGIT variant outperforms the raw variant in the box sorting task because it lacks information about different boxes’ textures Simulation Results and Force Control Demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy that persists in policy rollouts The learned policies preserve this separation during rollout, producing mean forces of 5.42 N and 19.86 N for the bulb and block respectively This indicates that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning by providing a consistent physical interface across these modes Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot REFERENCES
1: B.
Rosa: D.
Dev: Argall, S.
Rosa: Chernova, M.
Title and authors: Rosa: Alright, moving into the specific details of this paper called "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," we’re going to break down what exactly this system is doing.
Dev: We've talked about the hardware matching, but let’s focus on the interface design itself. They describe RUMI as a handheld platform that is specifically matched to the Robotiq 2F-eighty-five robot’s nonlinear kinematics and finger attachments <ref:2611.15802#pg1,matched to the Robotiq 2F-85>.
Taro: That matching part is huge because generic interfaces often fail when you have those specific, complicated movements of a real gripper; this system uses a scissor like input drive to couple the motion of the operator’s thumb and index finger directly to the gripper fingers.
Rosa: And they emphasize that these interchangeable fingers support both force torque sensing and vision based tactile sensing for contact rich demonstrations. That means you get both force information and visual feedback during interaction.
Dev: They also detail how they integrated the tracking system into a virtual reality headset to provide real time augmented reality feedback about the kinematic feasibility of motion on the target robot platform.
Taro: During data collection, they store these RUMI poses in the registered robot base frame established by that AR system, and then IRIS renders a MuJoCo based digital shadow of the target robot during recording.
Rosa: That allows the operator to observe the robot configuration associated with the demonstrated motion in real time so they can see if it’s physically possible before they commit to recording it.
Dev: The paper also introduces this haptic feedback mechanism that supports force aware te operation for simulated and real robots, giving the operator feedback such as vibration indicating remote contact and the approximate magnitude of the commanded load.
Taro: So, so it’s not just a capture device; it’s an active training tool where you can actually feel the forces involved in manipulating that robot remotely.
Rosa: It really is a powerful combination because RUMI provides all these elements together: matching hardware, AR visualization, and force aware teleoperation capability.
Dev: This setup aims to give us data that isn't just visually correct but physically grounded and force aware, which is what we really need for robust robot learning.
The paper's summary: Rosa: So, summarizing the core contributions of "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," it boils down to three main things.
Dev: First, they present RUMI itself as a Robotiq 2F-eighty-five matched handheld interface featuring nonlinear actuation and interchangeable fingertips <ref:2611.15802#pg1>.
Taro: Second, they developed an AR pipeline for visualizing and validating robot configurations during robot free data collection to help operators recognize infeasible poses while recording rather than filtering them only after collection.
Rosa: And third, they introduced a haptic feedback mechanism that allows for gripper force control during simulated and real teleoperation, giving the operator feedback on remote contact and load magnitude.
Dev: So, in essence, they are releasing the device design and software framework along with a dataset featuring force torque and vision based tactile sensor data.
Taro: That dataset is what makes it practical for others to actually train their own policies without having to build all this complicated hardware from scratch.
Rosa: It’s about providing a consistent way to get embodiment matched, tactile aware demonstrations that are ready for robot learning right now.
Dev: The implication is that we have a new way to capture demonstrations where the physical interaction is directly linked to the robot's specific capabilities and constraints.
The paper's improvements: Rosa: Now let’s look at the improvements they suggest in this work, because they don’t just stop at presenting RUMI; they outline how this system can be enhanced.
Dev: They point out that by using an AR pipeline, we get enhanced data quality for robot learning because it allows operators to visualize and validate robot configurations during robot free collection.
Taro: This is important because it lets people recognize infeasible poses while recording, which is a big improvement over just filtering them later when you review the data.
Rosa: They also improved the policies themselves by developing imitation policies capable of handling force aware demonstrations using different fingertip modalities like CoinFT and DIGIT sensors.
Dev: This means they can train models that preserve the target gripper geometry while enabling natural input before contact and force measurement after contact, which is a big step in modeling real physical behavior.
Taro: And they showed that these learned policies preserve an object dependent force strategy during rollout, producing mean forces of five point four two newtons and nineteen point eight six newtons for the bulb and block respectively.
Rosa: That result is significant because it confirms that the blocker provides a useful contact reference for demonstrating force sensitive behavior in simulation.
Dev: It also showed a clear correlation between task success rates and the availability of their fingertip tactile sensors, proving those sensors are essential for successful demonstrations.
Conclusion: Rosa: So, to wrap up on "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," it’s really about providing a unified platform for data collection.
Dev: In summary, the main implication is that we now have a method to collect high quality demonstrations where the physical interaction is directly tied to the robot's specific capabilities through matching hardware and rich feedback loops.
Taro: It means we can get datasets that are not just visually correct but physically grounded and force aware, which is what we need for robust robot learning.
Rosa: For future work, they suggest using generative AI to overcome the single view limitation as well as providing more informative visual cues about external forces acting upon the simulated or teleoperated robot.
Dev: And that sounds like the next step—using generative AI to move beyond just seeing what’s there to understanding what’s happening physically around the robot.
Taro: I think those future directions are smart because they focus on making the system even more aware of those subtle external forces, not just basic contact.
Rosa: Yeah, it really shows that this work is laying a foundation for learning systems that can handle real world complexity. RUMI is a really solid piece of hardware and software integration for this goal.
Siemens Germany University of Technology Nuremberg Germany Technical University of Munich Germany
cs.RO, cs.AI
Submitted: 2026-11-11
Updated: 2026-11-11
Project page: https://utn-air.github.io/RUMI
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning Interface…
Key concepts
- RUMI
- A handheld universal manipulation interface designed specifically to match the kinematics, finger attachments, and viewpoint of a particular robot gripper (like the Robotiq 2F-85). It uses a scissor-like mechanism where operator finger motion directly drives the coupled gripper fingers, providing direct physical correspondence between human input and robot action.
- Augmented Reality Visualization
- The system integrates tracking into a VR headset to provide real-time AR feedback. This allows operators to see a digital shadow (MuJoCo rendering) of the target robot during recording, visualizing the kinematic feasibility of their movements before physical contact occurs, which improves data quality.
- Force-Aware Teleoperation
- A mechanism that provides haptic feedback during remote operation. It uses a blocker to control low-resistance aperture before contact and converts subsequent squeezing into a measured force command. This allows operators to feel remote contact and the approximate load magnitude, supporting both in-the-wild collection and teleoperation.
- Tactile Sensing Modalities (CoinFT/DIGIT)
- The system utilizes different fingertip sensors, such as CoinFT or DIGIT, which support force/torque sensing. These sensors allow the robot to reliably push objects into alignment by providing force-adjusted trajectories, enabling the training of policies that are sensitive to contact forces.
Terminology
Summary
RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning
Interface Design and Hardware
RUMI is designed as a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85 A scissorlike input drives the coupled fingers while a rotary encoder measures the resulting grasp coordinate Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations The hardware features a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers A Teensy 4.0 microcontroller coordinates on-device sensing and actuation while synchronizing measurements
Augmented Reality Visualization
The method integrates the tracking system into a virtual reality headset to provide real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform During data collection, RUMI poses are stored in the registered robot base frame established by the AR system IRIS renders a MuJoCo-based digital shadow of the target robot during recording to allow the operator to observe the robot configuration associated with the demonstrated motion in real time This visualization allows users to recognize infeasible poses while recording rather than filtering them only after collection
Force-Aware Teleoperation Mechanism
The haptic feedback mechanism supports force-aware teleoperation for simulated and real robots During teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load This mechanism allows the same handheld device to support both in-the-wild collection and teleoperation
Data Quality and Policy Training
The AR condition results in fewer number of joint limit- and velocity violations compared to the w/o AR condition The average manipulability is 19.6% higher than the w/o AR condition, although the standard deviation is also greater For the CoinFT sensor, force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors The DIGIT variant outperforms the raw variant in the box sorting task because it lacks information about different boxes’ textures
Simulation Results and Force Control
Demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy that persists in policy rollouts The learned policies preserve this separation during rollout, producing mean forces of 5.42 N and 19.86 N for the bulb and block respectively This indicates that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors
Conclusion
RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning by providing a consistent physical interface across these modes Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot
REFERENCES
[1] B. D. Argall, S. Chernova, M. p. 6 REFERENCES
[1] B. Chernova, M. Chernova, M. Veloso, and B. Veloso, and B. Browning, “A survey
of robot learning from demonstration,” Robot.Where the paper says it
Where the paper says it
Where the paper says it
[2] G. Liu, W. Sun, and P. p. 6 469–483, 2009. Sun, and P. Sun, and P. Li, “Motion capture and AR based programming by demonstration for industrial robots using handheld teaching
device,” Sci.Where the paper says it
Where the paper says it
[3] M. J. Kim et al., “OpenVLA: An open-source vision-language-action model,” in Proc. p. 6 Li, “Motion capture and AR based programming by demonstration for industrial robots using handheld teaching
device,” Sci. , “OpenVLA: An open-source vision-language-action
model,” in Proc. , “OpenVLA: An open-source vision-language-action
model,” in Proc. Robot Learn. , “OpenVLA: An open-source vision-language-action
model,” in Proc. Robot Learn. , “OpenVLA: An open-source vision-language-action
model,” in Proc. Robot Learn. , “OpenVLA: An open-source vision-language-action
model,” in Proc. Robot Learn. , “OpenVLA: An open-source vision-language-action
model,” in Proc. Robot Learn.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[4] Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic learning datasets and RT-X models,” in Proc. IEEE Int. Conf. Robot Autom. p. 0 RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot
Learning with Augmented Reality and Haptic Feedback
Kevin Avdiu∗1,3
, Seongjin Bien∗2
, Yannik Blei∗2
, Philine Meister1
,
Johannes Hechtl1,2
, Philipp Schmitt1
, Kai Wurm1
, Georg von Wichert1
, Wolfram Burgard∗2
Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale,
force-aware datasets remain scarce. Handheld interfaces can
help by transmitting interaction forces directly through the
operator’s arm, but most existing designs are tailored to simple
linear grippers and do not match the nonlinear mechanics of
widely used robotic grippers. RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot
Learning with Augmented Reality and Haptic Feedback
Kevin Avdiu∗1,3
, Seongjin Bien∗2
, Yannik Blei∗2
, Philine Meister1
,
Johannes Hechtl1,2
, Philipp Schmitt1
, Kai Wurm1
, Georg von Wichert1
, Wolfram Burgard∗2
Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale,
force-aware datasets remain scarce. Handheld interfaces can
help by transmitting interaction forces directly through the
operator’s arm, but most existing designs are tailored to simple
linear grippers and do not match the nonlinear mechanics of
widely used robotic grippers. RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot
Learning with Augmented Reality and Haptic Feedback
Kevin Avdiu∗1,3
, Seongjin Bien∗2
, Yannik Blei∗2
, Philine Meister1
,
Johannes Hechtl1,2
, Philipp Schmitt1
, Kai Wurm1
, Georg von Wichert1
, Wolfram Burgard∗2
Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale,
force-aware datasets remain scarce. Handheld interfaces can
help by transmitting interaction forces directly through the
operator’s arm, but most existing designs are tailored to simple
linear grippers and do not match the nonlinear mechanics of
widely used robotic grippers. RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot
Learning with Augmented Reality and Haptic Feedback
Kevin Avdiu∗1,3
, Seongjin Bien∗2
, Yannik Blei∗2
, Philine Meister1
,
Johannes Hechtl1,2
, Philipp Schmitt1
, Kai Wurm1
, Georg von Wichert1
, Wolfram Burgard∗2
Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale,
force-aware datasets remain scarce. Handheld interfaces can
help by transmitting interaction forces directly through the
operator’s arm, but most existing designs are tailored to simple
linear grippers and do not match the nonlinear mechanics of
widely used robotic grippers. RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot
Learning with Augmented Reality and Haptic Feedback
Kevin Avdiu∗1,3
, Seongjin Bien∗2
, Yannik Blei∗2
, Philine Meister1
,
Johannes Hechtl1,2
, Philipp Schmitt1
, Kai Wurm1
, Georg von Wichert1
, Wolfram Burgard∗2
Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale,
force-aware datasets remain scarce. Handheld interfaces can
help by transmitting interaction forces directly through the
operator’s arm, but most existing designs are tailored to simple
linear grippers and do not match the nonlinear mechanics of
widely used robotic grippers.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[5] C. Chi et al., “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” 2024
[6] Zhaxizhuoma et al., “FastUMI: A scalable and hardware-independent universal manipulation interface with dataset,” in Proc. p. 6 [6] Zhaxizhuoma et al. , “FastUMI: A scalable and hardware-independent
universal manipulation interface with dataset,” in Proc.Where the paper says it
[7] K. Liu et al., “FastUMI-100K: Advancing data-driven robotic manipulation with a large-scale UMI-style dataset,” in arXiv:2510. p. 6 (CoRL), 2025. , “FastUMI-100K: Advancing data-driven robotic manipulation with a large-scale UMI-style dataset,” in arXiv:2510.Where the paper says it
[8] H. Choi et al. p. 6 08022v1,
2025. Choi et al. 08022v1,
2025. Choi et al. Choi et al. , “In-the-wild compliant manipulation with UMI-FT,” in
arXiv:2601.Where the paper says it
Where the paper says it
Where the paper says it
[9] E. Helmut, N. Funk, T. Schneider, C. p. 6 01IS24083). We
further thank Volker Schneider and Devadas Sheela for their
assistance with hardware development and setup. Schneider, C. de Farias, and J. de Farias, and J. Peters, “Tactileconditioned diffusion policy for force-aware robotic manipulation,” in
Proc.Where the paper says it
Where the paper says it
Where the paper says it
[10] L. Wu et al., “FreeTacMan: Robot-free visuo-tactile data collection system for contact-rich manipulation,” in arXiv:2506. p. 6 (ICRA), 2026. , “FreeTacMan: Robot-free visuo-tactile data collection
system for contact-rich manipulation,” in arXiv:2506.Where the paper says it
[11] W. Liu, J. Wang, Y. Wang, W. Wang, and C. p. 6 01941v4, 2026. Wang, and C. Wang, and C. Lu, “ForceMimic:
Force-centric imitation learning with force-motion capture system for
contact-rich manipulation,” in Proc.Where the paper says it
Where the paper says it
[12] G. Lee et al., “ManipForce: Force-guided policy learning with frequency-aware representation for contact-rich manipulation,” in arXiv:2509. p. 7 , “ManipForce: Force-guided policy learning with
frequency-aware representation for contact-rich manipulation,” in
arXiv:2509. 19047v1, 2025.Where the paper says it
[13] Q. Zeng et al. p. 7 19047v1, 2025. Zeng et al. Zeng et al. , “ActiveUMI: Robotic manipulation with active perception from robot-free human demonstrations,” in arXiv:2510.Where the paper says it
Where the paper says it
[14] Y. Huang, S. Li, X. Li, and W. Ding, “UMIGen: A unified framework for egocentric point cloud generation and cross-embodiment robotic imitation learning,” in arXiv:2511. p. 7 01607v1,
2025. Ding, “UMIGen: A unified framework
for egocentric point cloud generation and cross-embodiment robotic
imitation learning,” in arXiv:2511.Where the paper says it
[15] J. Wang et al. p. 6 01941v4, 2026. Wang, and C. Wang et al. , “XRZero-G0: Pushing the frontier of dexterous robotic manipulation with interfaces, quality and ratios,” in
arXiv:2604.Where the paper says it
Where the paper says it
[16] Z. Wang, “UMI-3D: Extending universal manipulation interface from vision-limited to 3D spatial perception,” in arXiv:2604. p. 7 13001v2, 2026. Wang, “UMI-3D: Extending universal manipulation interface from
vision-limited to 3D spatial perception,” in arXiv:2604.Where the paper says it
[17] T. Ohkawa et al. p. 7 14089v1, 2026. Ohkawa et al.Where the paper says it
[18] Y. Xu et al., “EgoGuide: Egocentric guidance for efficient robot-free demonstration collection and learning,” in arXiv:2606. p. 7 , “YUBI: Yielding universal bidigital interface for
bimanual dexterous manipulation at scale,” in 1st ICRA Workshop on
”Beyond Teloperation: Learning from Diverse Human and Simulation
Data”, 2026. , “EgoGuide: Egocentric guidance for efficient robot-free
demonstration collection and learning,” in arXiv:2606.Where the paper says it
[19] Simple AI et al., “HiFi-UMI: Learning deployable manipulation policies from high-fidelity UMI data alone,” in arXiv:2607. p. 7 [19] Simple AI et al. , “HiFi-UMI: Learning deployable manipulation
policies from high-fidelity UMI data alone,” in arXiv:2607.Where the paper says it
[20] F. Liu, C. Li, Y. Qin, J. Xu, P. Abbeel, and R. p. 7 25895v1,
2026. Abbeel, and R. Abbeel, and R. Chen, “ViTaMIn:
Learning contact-rich tasks through robot-free visuo-tactile manipulation interface,” in arXiv:2504.Where the paper says it
Where the paper says it
[21] X. Zhu, B. Huang, and Y. p. 7 06156v2, 2025. Huang, and Y. Huang, and Y. Li, “Touch in the wild: Learning fine-grained
manipulation with a portable visuo-tactile gripper,” in Adv. Li, “Touch in the wild: Learning fine-grained
manipulation with a portable visuo-tactile gripper,” in Adv. Neural Inf.Where the paper says it
Where the paper says it
Where the paper says it
[22] Y. Xu, L. Wei, P. An, Q. Zhang, and Y. p. 7 (NeurIPS), 2025. Zhang, and Y. Zhang, and Y. Li, “exUMI: Extensible
robot teaching system with action-aware task-agnostic tactile representation,” in Proc.Where the paper says it
Where the paper says it
[23] T. Wu et al., “MagiClaw: A dual-use, vision-based soft gripper for bridging the human demonstration to robotic deployment gap,” in arXiv:2509. p. 7 (CoRL), 2025. , “MagiClaw: A dual-use, vision-based soft gripper for
bridging the human demonstration to robotic deployment gap,” in
arXiv:2509.Where the paper says it
[24] C. Li et al., “ViTaMIn-B: A reliable and efficient visuo-tactile bimanual manipulation interface,” in arXiv:2511. p. 7 19169v1, 2025. , “ViTaMIn-B: A reliable and efficient visuo-tactile bimanual manipulation interface,” in arXiv:2511.Where the paper says it
[25] T. Cheng et al. p. 7 05858v2, 2025. Cheng et al. Cheng et al. , “TacUMI: A multi-modal universal manipulation
interface for contact-rich tasks,” in arXiv:2601.Where the paper says it
Where the paper says it
[26] S. Luo et al., “OmniUMI: Towards physically grounded robot learning via human-aligned multimodal interaction,” in arXiv:2604. p. 7 14550v1, 2026. , “OmniUMI: Towards physically grounded robot learning
via human-aligned multimodal interaction,” in arXiv:2604.Where the paper says it
[27] M. Xu et al., “DexUMI: Using human hand as the universal manipulation interface for dexterous manipulation,” in Proc. p. 7 10647v3,
2026. , “DexUMI: Using human hand as the universal manipulation interface for dexterous manipulation,” in Proc.Where the paper says it
[28] O. Rayyan, J. Abanes, M. Hafez, A. Tzes, and F. p. 7 (CoRL), 2025. Tzes, and F. Tzes, and F. Abu-Dakka, “Mvumi: A scalable multi-view interface for cross-embodiment learning,”
IEEE Access, vol.Where the paper says it
Where the paper says it
[29] C. Xu et al., “RealDexUMI: A wearable universal manipulation interface for dexterous robot learning,” in arXiv:2606. p. 7 104 545–104 555, 2026. , “RealDexUMI: A wearable universal manipulation
interface for dexterous robot learning,” in arXiv:2606.Where the paper says it
[30] A. Hajj-Ahmad et al. p. 7 06033v2, 2026. Hajj-Ahmad et al. Hajj-Ahmad et al. , “Koala gripper: Co-designing robotic grippers
and data-capture devices for scaling dexterous manipulation learning,”
in arXiv:2608.Where the paper says it
Where the paper says it
[31] Robotiq, “Adaptive grippers: 2F-85 and 2F-140,” Product documentation, accessed: Sep. 15, 2026 [Online]. p. 1 RUMI follows these directions by matching the nonlinear
finger kinematics and mounting geometry of the Robotiq 2F85, enabling straightforward deployment on existing robot
platforms without requiring a custom robot-side gripper or
additional compatibility solution. It also uses live robot
projection during collection for online kinematic checking. [31] Robotiq, “Adaptive grippers: 2F-85 and 2F-140,” Product
documentation, accessed: Sep. Available:
https://robotiq.Where the paper says it
Where the paper says it
[32] H. Choi et al., “Coinft: A coin-sized, capacitive 6-axis force torque sensor for robotic applications,” arXiv preprint arXiv:2503. p. 7 Choi et al. , “Coinft: A coin-sized, capacitive 6-axis force torque
sensor for robotic applications,” arXiv preprint arXiv:2503.Where the paper says it
[33] M. Lambeta et al. p. 7 19225,
2025. Lambeta et al. Lambeta et al. , “Digit: A novel design for a low-cost compact highresolution tactile sensor with application to in-hand manipulation,”
IEEE Robot.Where the paper says it
Where the paper says it
[34] X. Jiang et al. p. 7 3838–3845, 2020. Jiang et al. Jiang et al. , “IRIS: An immersive robot interaction system,” in
Proc.Where the paper says it
Where the paper says it
[35] E. Todorov, T. p. 7 (CoRL), 2025. Todorov, T. Todorov, T. Erez, and Y. Erez, and Y. Tassa, “Mujoco: A physics engine for
model-based control,” in Proc. Tassa, “Mujoco: A physics engine for
model-based control,” in Proc. IEEE/RSJ Int. IEEE/RSJ Int. Robots Syst.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[36] T. Julg ¨ et al. p. 7 (IROS), 2012. Julg ¨ et al. Julg ¨ et al. , “Robot Control Stack: A lean ecosystem for robot
learning at scale,” in Proc.Where the paper says it
Where the paper says it
[37] T. Z. Zhao, V. Kumar, S. Levine, and C. p. 7 (ICRA),
2026. Levine, and C. Levine, and C. Finn, “Learning fine-grained
bimanual manipulation with low-cost hardware,” in Proc.Where the paper says it
Where the paper says it
[38] Y. Chen et al. p. 7 Abbeel, and R. Chen, “ViTaMIn:
Learning contact-rich tasks through robot-free visuo-tactile manipulation interface,” in arXiv:2504. Chen et al. , “Manipulationnet: An infrastructure for benchmarking
real-world robot manipulation with physical skill challenges and
embodied multimodal reasoning,” in arXiv:2603.Where the paper says it
Where the paper says it
[39] A. George, S. Gano, P. Katragadda, and A. p. 7 04363, 2026. Katragadda, and A. Katragadda, and A. Farimani, “Vital
pretraining: Visuo-tactile pretraining for tactile and non-tactile manipulation policies,” in Proc.Where the paper says it
Where the paper says it
[40] K. He, X. Zhang, S. Ren, and J. p. 7 (ICRA),
2025. Ren, and J. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. Sun, “Deep residual learning for image
recognition,” in Proc. Pattern Recognit.Where the paper says it
Where the paper says it
Where the paper says it
[41] Dileepkumar Khajana, “Spiral light bulb,” GrabCAD
--- Page 5 ---
TABLE I
REPORTED CAPABILITIES OF HANDHELD DEMONSTRATION INTERFACES.: REPORTED; –: NOT REPORTED. p. 2 TABLE I
REPORTED CAPABILITIES OF HANDHELD DEMONSTRATION INTERFACES. ✓: REPORTED; –: NOT REPORTED.Where the paper says it
--- Page 6 ---
Fig. 2. Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. p. 3 Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The
hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T)
sensing. Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The
hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T)
sensing. Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The
hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T)
sensing.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 7 ---
Fig. 2. Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (c) The nonlinear transmission couples finger motion, while a selectively engaged blocker constrains further closure
--- Page 8 ---
Fig. 4a.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Enhanced data quality for robot learning by using an AR pipeline to visualize and validate robot configurations during robot-free data collection. This allows operators to
recognize infeasible poses while recording rather than filtering them only after collection.
-
Development of imitation policies capable of handling force-aware demonstrations using different fingertip modalities like CoinFT and DIGIT sensors. The system can train models that
preserve the target gripper geometry while enabling natural input before contact and force measurement after contact.
-
Integration of a haptic feedback mechanism supporting force-aware teleoperation for simulated and real robots. This mechanism allows the operator to receive feedback such as
vibration feedback indicating remote contact and the approximate magnitude of the commanded load.
-
Creation of a universal manipulation interface matched to nonlinear kinematics, finger attachments, and viewpoint of specific robotic grippers like the Robotiq 2F-85. This enables direct correspondence between operator input and gripper actuation by utilizing a
scissorlike input drives the coupled fingers
mechanism.
Abstract
Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator's arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface's capability for collecting force-aware teleoperation datasets in simulation.
Sources
- FastUMI-100K: Advancing Data-driven Robotic Manipulation with a Large-scale UMI-style Dataset
- In-the-Wild Compliant Manipulation with UMI-FT
- FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
- ManipForce: Force-Guided Policy Learning with Frequency-Aware Representation for Contact-Rich Manipulation
- ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrations
- UMIGen: A Unified Framework for Egocentric Point Cloud Generation and Cross-Embodiment Robotic Imitation Learning
- XRZero-G0: Pushing the Frontier of Dexterous Robotic Manipulation with Interfaces, Quality and Ratios
- UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception
- EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning
- HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
- ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface
- MagiClaw: A Dual-Use, Vision-Based Soft Gripper for Bridging the Human Demonstration to Robotic Deployment Gap
- ViTaMIn-B: A Reliable and Efficient Visuo-Tactile Bimanual Manipulation Interface
- TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks
- OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction
- RealDexUMI: A Wearable Universal Manipulation Interface for Dexterous Robot Learning
- CoinFT: A Coin-Sized, Capacitive 6-Axis Force Torque Sensor for Robotic Applications
- ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation