RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback

arXiv:2611.15802 · cs.RO, cs.AI · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback".

Dev: Here's the gist of the paper. RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning Interface Design and Hardware RUMI is designed as a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85 A scissorlike input drives the coupled fingers while a rotary encoder measures the resulting grasp coordinate Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations The hardware features a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers A Teensy 4.0 microcontroller coordinates on-device sensing and actuation while synchronizing measurements Augmented Reality Visualization The method integrates the tracking system into a virtual reality headset to provide real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform During data collection, RUMI poses are stored in the registered robot base frame established by the AR system IRIS renders a MuJoCo-based digital shadow of the target robot during recording to allow the operator to observe the robot configuration associated with the demonstrated motion in real time This visualization allows users to recognize infeasible poses while recording rather than filtering them only after collection Force-Aware Teleoperation Mechanism The haptic feedback mechanism supports force-aware teleoperation for simulated and real robots During teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load This mechanism allows the same handheld device to support both in-the-wild collection and teleoperation Data Quality and Policy Training The AR condition results in fewer number of joint limit- and velocity violations compared to the w/o AR condition The average manipulability is 19.6% higher than the w/o AR condition, although the standard deviation is also greater For the CoinFT sensor, force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors The DIGIT variant outperforms the raw variant in the box sorting task because it lacks information about different boxes’ textures Simulation Results and Force Control Demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy that persists in policy rollouts The learned policies preserve this separation during rollout, producing mean forces of 5.42 N and 19.86 N for the bulb and block respectively This indicates that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning by providing a consistent physical interface across these modes Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot REFERENCES

1: B.

Rosa: D.

Dev: Argall, S.

Rosa: Chernova, M.

Title and authors: Rosa: Alright, moving into the specific details of this paper called "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," we’re going to break down what exactly this system is doing.

Dev: We've talked about the hardware matching, but let’s focus on the interface design itself. They describe RUMI as a handheld platform that is specifically matched to the Robotiq 2F-eighty-five robot’s nonlinear kinematics and finger attachments <ref:2611.15802#pg1,matched to the Robotiq 2F-85>.

Taro: That matching part is huge because generic interfaces often fail when you have those specific, complicated movements of a real gripper; this system uses a scissor like input drive to couple the motion of the operator’s thumb and index finger directly to the gripper fingers.

Rosa: And they emphasize that these interchangeable fingers support both force torque sensing and vision based tactile sensing for contact rich demonstrations. That means you get both force information and visual feedback during interaction.

Dev: They also detail how they integrated the tracking system into a virtual reality headset to provide real time augmented reality feedback about the kinematic feasibility of motion on the target robot platform.

Taro: During data collection, they store these RUMI poses in the registered robot base frame established by that AR system, and then IRIS renders a MuJoCo based digital shadow of the target robot during recording.

Rosa: That allows the operator to observe the robot configuration associated with the demonstrated motion in real time so they can see if it’s physically possible before they commit to recording it.

Dev: The paper also introduces this haptic feedback mechanism that supports force aware te operation for simulated and real robots, giving the operator feedback such as vibration indicating remote contact and the approximate magnitude of the commanded load.

Taro: So, so it’s not just a capture device; it’s an active training tool where you can actually feel the forces involved in manipulating that robot remotely.

Rosa: It really is a powerful combination because RUMI provides all these elements together: matching hardware, AR visualization, and force aware teleoperation capability.

Dev: This setup aims to give us data that isn't just visually correct but physically grounded and force aware, which is what we really need for robust robot learning.

The paper's summary: Rosa: So, summarizing the core contributions of "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," it boils down to three main things.

Dev: First, they present RUMI itself as a Robotiq 2F-eighty-five matched handheld interface featuring nonlinear actuation and interchangeable fingertips <ref:2611.15802#pg1>.

Taro: Second, they developed an AR pipeline for visualizing and validating robot configurations during robot free data collection to help operators recognize infeasible poses while recording rather than filtering them only after collection.

Rosa: And third, they introduced a haptic feedback mechanism that allows for gripper force control during simulated and real teleoperation, giving the operator feedback on remote contact and load magnitude.

Dev: So, in essence, they are releasing the device design and software framework along with a dataset featuring force torque and vision based tactile sensor data.

Taro: That dataset is what makes it practical for others to actually train their own policies without having to build all this complicated hardware from scratch.

Rosa: It’s about providing a consistent way to get embodiment matched, tactile aware demonstrations that are ready for robot learning right now.

Dev: The implication is that we have a new way to capture demonstrations where the physical interaction is directly linked to the robot's specific capabilities and constraints.

The paper's improvements: Rosa: Now let’s look at the improvements they suggest in this work, because they don’t just stop at presenting RUMI; they outline how this system can be enhanced.

Dev: They point out that by using an AR pipeline, we get enhanced data quality for robot learning because it allows operators to visualize and validate robot configurations during robot free collection.

Taro: This is important because it lets people recognize infeasible poses while recording, which is a big improvement over just filtering them later when you review the data.

Rosa: They also improved the policies themselves by developing imitation policies capable of handling force aware demonstrations using different fingertip modalities like CoinFT and DIGIT sensors.

Dev: This means they can train models that preserve the target gripper geometry while enabling natural input before contact and force measurement after contact, which is a big step in modeling real physical behavior.

Taro: And they showed that these learned policies preserve an object dependent force strategy during rollout, producing mean forces of five point four two newtons and nineteen point eight six newtons for the bulb and block respectively.

Rosa: That result is significant because it confirms that the blocker provides a useful contact reference for demonstrating force sensitive behavior in simulation.

Dev: It also showed a clear correlation between task success rates and the availability of their fingertip tactile sensors, proving those sensors are essential for successful demonstrations.

Conclusion: Rosa: So, to wrap up on "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," it’s really about providing a unified platform for data collection.

Dev: In summary, the main implication is that we now have a method to collect high quality demonstrations where the physical interaction is directly tied to the robot's specific capabilities through matching hardware and rich feedback loops.

Taro: It means we can get datasets that are not just visually correct but physically grounded and force aware, which is what we need for robust robot learning.

Rosa: For future work, they suggest using generative AI to overcome the single view limitation as well as providing more informative visual cues about external forces acting upon the simulated or teleoperated robot.

Dev: And that sounds like the next step—using generative AI to move beyond just seeing what’s there to understanding what’s happening physically around the robot.

Taro: I think those future directions are smart because they focus on making the system even more aware of those subtle external forces, not just basic contact.

Rosa: Yeah, it really shows that this work is laying a foundation for learning systems that can handle real world complexity. RUMI is a really solid piece of hardware and software integration for this goal.

Siemens Germany University of Technology Nuremberg Germany Technical University of Munich Germany

cs.RO, cs.AI

Submitted: 2026-11-11

Updated: 2026-11-11

Project page: https://utn-air.github.io/RUMI

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning Interface…

Key concepts

RUMI
A handheld universal manipulation interface designed specifically to match the kinematics, finger attachments, and viewpoint of a particular robot gripper (like the Robotiq 2F-85). It uses a scissor-like mechanism where operator finger motion directly drives the coupled gripper fingers, providing direct physical correspondence between human input and robot action.
Augmented Reality Visualization
The system integrates tracking into a VR headset to provide real-time AR feedback. This allows operators to see a digital shadow (MuJoCo rendering) of the target robot during recording, visualizing the kinematic feasibility of their movements before physical contact occurs, which improves data quality.
Force-Aware Teleoperation
A mechanism that provides haptic feedback during remote operation. It uses a blocker to control low-resistance aperture before contact and converts subsequent squeezing into a measured force command. This allows operators to feel remote contact and the approximate load magnitude, supporting both in-the-wild collection and teleoperation.
Tactile Sensing Modalities (CoinFT/DIGIT)
The system utilizes different fingertip sensors, such as CoinFT or DIGIT, which support force/torque sensing. These sensors allow the robot to reliably push objects into alignment by providing force-adjusted trajectories, enabling the training of policies that are sensitive to contact forces.

Terminology

Summary

RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning

Interface Design and Hardware

RUMI is designed as a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85 A scissorlike input drives the coupled fingers while a rotary encoder measures the resulting grasp coordinate Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations The hardware features a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers A Teensy 4.0 microcontroller coordinates on-device sensing and actuation while synchronizing measurements

Augmented Reality Visualization

The method integrates the tracking system into a virtual reality headset to provide real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform During data collection, RUMI poses are stored in the registered robot base frame established by the AR system IRIS renders a MuJoCo-based digital shadow of the target robot during recording to allow the operator to observe the robot configuration associated with the demonstrated motion in real time This visualization allows users to recognize infeasible poses while recording rather than filtering them only after collection

Force-Aware Teleoperation Mechanism

The haptic feedback mechanism supports force-aware teleoperation for simulated and real robots During teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load This mechanism allows the same handheld device to support both in-the-wild collection and teleoperation

Data Quality and Policy Training

The AR condition results in fewer number of joint limit- and velocity violations compared to the w/o AR condition The average manipulability is 19.6% higher than the w/o AR condition, although the standard deviation is also greater For the CoinFT sensor, force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors The DIGIT variant outperforms the raw variant in the box sorting task because it lacks information about different boxes’ textures

Simulation Results and Force Control

Demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy that persists in policy rollouts The learned policies preserve this separation during rollout, producing mean forces of 5.42 N and 19.86 N for the bulb and block respectively This indicates that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors

Conclusion

RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning by providing a consistent physical interface across these modes Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot

REFERENCES

[1] B. D. Argall, S. Chernova, M. p. 6

Where the paper says it

REFERENCES [1] B. Chernova, M.

Veloso, and B. p. 6
Where the paper says it

Chernova, M. Veloso, and B.

Browning, “A survey of robot learning from demonstration,” Robot. p. 6
Where the paper says it

Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robot.

Auton. Syst., vol. 57, no. 5, pp. 469–483, 2009

[2] G. Liu, W. Sun, and P. p. 6

Where the paper says it

469–483, 2009. Sun, and P.

Li, “Motion capture and AR based programming by demonstration for industrial robots using handheld teaching device,” Sci. p. 6
Where the paper says it

Sun, and P. Li, “Motion capture and AR based programming by demonstration for industrial robots using handheld teaching device,” Sci.

Rep., vol. 14, 2024

[3] M. J. Kim et al., “OpenVLA: An open-source vision-language-action model,” in Proc. p. 6

Where the paper says it

Li, “Motion capture and AR based programming by demonstration for industrial robots using handheld teaching device,” Sci. , “OpenVLA: An open-source vision-language-action model,” in Proc.

Conf. Robot Learn. p. 6
Where the paper says it

, “OpenVLA: An open-source vision-language-action model,” in Proc. Robot Learn.

p. 6
Where the paper says it

, “OpenVLA: An open-source vision-language-action model,” in Proc. Robot Learn.

p. 6
Where the paper says it

, “OpenVLA: An open-source vision-language-action model,” in Proc. Robot Learn.

p. 6
Where the paper says it

, “OpenVLA: An open-source vision-language-action model,” in Proc. Robot Learn.

p. 6
Where the paper says it

, “OpenVLA: An open-source vision-language-action model,” in Proc. Robot Learn.

(CoRL), 2024

[4] Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic learning datasets and RT-X models,” in Proc. IEEE Int. Conf. Robot Autom. p. 0

Where the paper says it

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback Kevin Avdiu∗1,3 , Seongjin Bien∗2 , Yannik Blei∗2 , Philine Meister1 , Johannes Hechtl1,2 , Philipp Schmitt1 , Kai Wurm1 , Georg von Wichert1 , Wolfram Burgard∗2 Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers.

p. 0
Where the paper says it

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback Kevin Avdiu∗1,3 , Seongjin Bien∗2 , Yannik Blei∗2 , Philine Meister1 , Johannes Hechtl1,2 , Philipp Schmitt1 , Kai Wurm1 , Georg von Wichert1 , Wolfram Burgard∗2 Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers.

p. 0
Where the paper says it

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback Kevin Avdiu∗1,3 , Seongjin Bien∗2 , Yannik Blei∗2 , Philine Meister1 , Johannes Hechtl1,2 , Philipp Schmitt1 , Kai Wurm1 , Georg von Wichert1 , Wolfram Burgard∗2 Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers.

p. 0
Where the paper says it

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback Kevin Avdiu∗1,3 , Seongjin Bien∗2 , Yannik Blei∗2 , Philine Meister1 , Johannes Hechtl1,2 , Philipp Schmitt1 , Kai Wurm1 , Georg von Wichert1 , Wolfram Burgard∗2 Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers.

p. 0
Where the paper says it

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback Kevin Avdiu∗1,3 , Seongjin Bien∗2 , Yannik Blei∗2 , Philine Meister1 , Johannes Hechtl1,2 , Philipp Schmitt1 , Kai Wurm1 , Georg von Wichert1 , Wolfram Burgard∗2 Abstract— Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator’s arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers.

, 2024

[5] C. Chi et al., “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” 2024

[6] Zhaxizhuoma et al., “FastUMI: A scalable and hardware-independent universal manipulation interface with dataset,” in Proc. p. 6

Where the paper says it

[6] Zhaxizhuoma et al. , “FastUMI: A scalable and hardware-independent universal manipulation interface with dataset,” in Proc.

Conf. Robot Learn. (CoRL), 2025

[7] K. Liu et al., “FastUMI-100K: Advancing data-driven robotic manipulation with a large-scale UMI-style dataset,” in arXiv:2510. p. 6

Where the paper says it

(CoRL), 2025. , “FastUMI-100K: Advancing data-driven robotic manipulation with a large-scale UMI-style dataset,” in arXiv:2510.

08022v1, 2025

[8] H. Choi et al. p. 6

Where the paper says it

08022v1, 2025. Choi et al.

p. 6
Where the paper says it

08022v1, 2025. Choi et al.

, “In-the-wild compliant manipulation with UMI-FT,” in arXiv:2601. p. 6
Where the paper says it

Choi et al. , “In-the-wild compliant manipulation with UMI-FT,” in arXiv:2601.

09988v1, 2026

[9] E. Helmut, N. Funk, T. Schneider, C. p. 6

Where the paper says it

01IS24083). We further thank Volker Schneider and Devadas Sheela for their assistance with hardware development and setup.

de Farias, and J. p. 6
Where the paper says it

Schneider, C. de Farias, and J.

Peters, “Tactileconditioned diffusion policy for force-aware robotic manipulation,” in Proc. p. 6
Where the paper says it

de Farias, and J. Peters, “Tactileconditioned diffusion policy for force-aware robotic manipulation,” in Proc.

IEEE Int. Conf. Robot Autom., 2026

[10] L. Wu et al., “FreeTacMan: Robot-free visuo-tactile data collection system for contact-rich manipulation,” in arXiv:2506. p. 6

Where the paper says it

(ICRA), 2026. , “FreeTacMan: Robot-free visuo-tactile data collection system for contact-rich manipulation,” in arXiv:2506.

01941v4, 2026

[11] W. Liu, J. Wang, Y. Wang, W. Wang, and C. p. 6

Where the paper says it

01941v4, 2026. Wang, and C.

Lu, “ForceMimic: Force-centric imitation learning with force-motion capture system for contact-rich manipulation,” in Proc. p. 6
Where the paper says it

Wang, and C. Lu, “ForceMimic: Force-centric imitation learning with force-motion capture system for contact-rich manipulation,” in Proc.

IEEE Int. Conf. Robot Autom., 2025

[12] G. Lee et al., “ManipForce: Force-guided policy learning with frequency-aware representation for contact-rich manipulation,” in arXiv:2509. p. 7

Where the paper says it

, “ManipForce: Force-guided policy learning with frequency-aware representation for contact-rich manipulation,” in arXiv:2509. 19047v1, 2025.

19047v1, 2025

[13] Q. Zeng et al. p. 7

Where the paper says it

19047v1, 2025. Zeng et al.

, “ActiveUMI: Robotic manipulation with active perception from robot-free human demonstrations,” in arXiv:2510. p. 7
Where the paper says it

Zeng et al. , “ActiveUMI: Robotic manipulation with active perception from robot-free human demonstrations,” in arXiv:2510.

01607v1, 2025

[14] Y. Huang, S. Li, X. Li, and W. Ding, “UMIGen: A unified framework for egocentric point cloud generation and cross-embodiment robotic imitation learning,” in arXiv:2511. p. 7

Where the paper says it

01607v1, 2025. Ding, “UMIGen: A unified framework for egocentric point cloud generation and cross-embodiment robotic imitation learning,” in arXiv:2511.

09302v1, 2025

[15] J. Wang et al. p. 6

Where the paper says it

01941v4, 2026. Wang, and C.

, “XRZero-G0: Pushing the frontier of dexterous robotic manipulation with interfaces, quality and ratios,” in arXiv:2604. p. 7
Where the paper says it

Wang et al. , “XRZero-G0: Pushing the frontier of dexterous robotic manipulation with interfaces, quality and ratios,” in arXiv:2604.

13001v2, 2026

[16] Z. Wang, “UMI-3D: Extending universal manipulation interface from vision-limited to 3D spatial perception,” in arXiv:2604. p. 7

Where the paper says it

13001v2, 2026. Wang, “UMI-3D: Extending universal manipulation interface from vision-limited to 3D spatial perception,” in arXiv:2604.

14089v1, 2026

[17] T. Ohkawa et al. p. 7

Where the paper says it

14089v1, 2026. Ohkawa et al.

, “YUBI: Yielding universal bidigital interface for bimanual dexterous manipulation at scale,” in 1st ICRA Workshop on ”Beyond Teloperation: Learning from Diverse Human and Simulation Data”, 2026

[18] Y. Xu et al., “EgoGuide: Egocentric guidance for efficient robot-free demonstration collection and learning,” in arXiv:2606. p. 7

Where the paper says it

, “YUBI: Yielding universal bidigital interface for bimanual dexterous manipulation at scale,” in 1st ICRA Workshop on ”Beyond Teloperation: Learning from Diverse Human and Simulation Data”, 2026. , “EgoGuide: Egocentric guidance for efficient robot-free demonstration collection and learning,” in arXiv:2606.

14665v1, 2026

[19] Simple AI et al., “HiFi-UMI: Learning deployable manipulation policies from high-fidelity UMI data alone,” in arXiv:2607. p. 7

Where the paper says it

[19] Simple AI et al. , “HiFi-UMI: Learning deployable manipulation policies from high-fidelity UMI data alone,” in arXiv:2607.

25895v1, 2026

[20] F. Liu, C. Li, Y. Qin, J. Xu, P. Abbeel, and R. p. 7

Where the paper says it

25895v1, 2026. Abbeel, and R.

Chen, “ViTaMIn: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface,” in arXiv:2504. p. 7
Where the paper says it

Abbeel, and R. Chen, “ViTaMIn: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface,” in arXiv:2504.

06156v2, 2025

[21] X. Zhu, B. Huang, and Y. p. 7

Where the paper says it

06156v2, 2025. Huang, and Y.

Li, “Touch in the wild: Learning fine-grained manipulation with a portable visuo-tactile gripper,” in Adv. p. 7
Where the paper says it

Huang, and Y. Li, “Touch in the wild: Learning fine-grained manipulation with a portable visuo-tactile gripper,” in Adv.

Neural Inf. p. 7
Where the paper says it

Li, “Touch in the wild: Learning fine-grained manipulation with a portable visuo-tactile gripper,” in Adv. Neural Inf.

Process. Syst., 2025

[22] Y. Xu, L. Wei, P. An, Q. Zhang, and Y. p. 7

Where the paper says it

(NeurIPS), 2025. Zhang, and Y.

-L. Li, “exUMI: Extensible robot teaching system with action-aware task-agnostic tactile representation,” in Proc. p. 7
Where the paper says it

Zhang, and Y. Li, “exUMI: Extensible robot teaching system with action-aware task-agnostic tactile representation,” in Proc.

Conf. Robot Learn., 2025

[23] T. Wu et al., “MagiClaw: A dual-use, vision-based soft gripper for bridging the human demonstration to robotic deployment gap,” in arXiv:2509. p. 7

Where the paper says it

(CoRL), 2025. , “MagiClaw: A dual-use, vision-based soft gripper for bridging the human demonstration to robotic deployment gap,” in arXiv:2509.

19169v1, 2025

[24] C. Li et al., “ViTaMIn-B: A reliable and efficient visuo-tactile bimanual manipulation interface,” in arXiv:2511. p. 7

Where the paper says it

19169v1, 2025. , “ViTaMIn-B: A reliable and efficient visuo-tactile bimanual manipulation interface,” in arXiv:2511.

05858v2, 2025

[25] T. Cheng et al. p. 7

Where the paper says it

05858v2, 2025. Cheng et al.

, “TacUMI: A multi-modal universal manipulation interface for contact-rich tasks,” in arXiv:2601. p. 7
Where the paper says it

Cheng et al. , “TacUMI: A multi-modal universal manipulation interface for contact-rich tasks,” in arXiv:2601.

14550v1, 2026

[26] S. Luo et al., “OmniUMI: Towards physically grounded robot learning via human-aligned multimodal interaction,” in arXiv:2604. p. 7

Where the paper says it

14550v1, 2026. , “OmniUMI: Towards physically grounded robot learning via human-aligned multimodal interaction,” in arXiv:2604.

10647v3, 2026

[27] M. Xu et al., “DexUMI: Using human hand as the universal manipulation interface for dexterous manipulation,” in Proc. p. 7

Where the paper says it

10647v3, 2026. , “DexUMI: Using human hand as the universal manipulation interface for dexterous manipulation,” in Proc.

Conf. Robot Learn., 2025

[28] O. Rayyan, J. Abanes, M. Hafez, A. Tzes, and F. p. 7

Where the paper says it

(CoRL), 2025. Tzes, and F.

Abu-Dakka, “Mvumi: A scalable multi-view interface for cross-embodiment learning,” IEEE Access, vol. p. 7
Where the paper says it

Tzes, and F. Abu-Dakka, “Mvumi: A scalable multi-view interface for cross-embodiment learning,” IEEE Access, vol.

14, pp. 104 545–104 555, 2026

[29] C. Xu et al., “RealDexUMI: A wearable universal manipulation interface for dexterous robot learning,” in arXiv:2606. p. 7

Where the paper says it

104 545–104 555, 2026. , “RealDexUMI: A wearable universal manipulation interface for dexterous robot learning,” in arXiv:2606.

06033v2, 2026

[30] A. Hajj-Ahmad et al. p. 7

Where the paper says it

06033v2, 2026. Hajj-Ahmad et al.

, “Koala gripper: Co-designing robotic grippers and data-capture devices for scaling dexterous manipulation learning,” in arXiv:2608. p. 7
Where the paper says it

Hajj-Ahmad et al. , “Koala gripper: Co-designing robotic grippers and data-capture devices for scaling dexterous manipulation learning,” in arXiv:2608.

20546v1, 2026

[31] Robotiq, “Adaptive grippers: 2F-85 and 2F-140,” Product documentation, accessed: Sep. 15, 2026 [Online]. p. 1

Where the paper says it

RUMI follows these directions by matching the nonlinear finger kinematics and mounting geometry of the Robotiq 2F85, enabling straightforward deployment on existing robot platforms without requiring a custom robot-side gripper or additional compatibility solution. It also uses live robot projection during collection for online kinematic checking.

Available: https://robotiq. p. 7
Where the paper says it

[31] Robotiq, “Adaptive grippers: 2F-85 and 2F-140,” Product documentation, accessed: Sep. Available: https://robotiq.

com/products/adaptive-grippers

[32] H. Choi et al., “Coinft: A coin-sized, capacitive 6-axis force torque sensor for robotic applications,” arXiv preprint arXiv:2503. p. 7

Where the paper says it

Choi et al. , “Coinft: A coin-sized, capacitive 6-axis force torque sensor for robotic applications,” arXiv preprint arXiv:2503.

19225, 2025

[33] M. Lambeta et al. p. 7

Where the paper says it

19225, 2025. Lambeta et al.

, “Digit: A novel design for a low-cost compact highresolution tactile sensor with application to in-hand manipulation,” IEEE Robot. p. 7
Where the paper says it

Lambeta et al. , “Digit: A novel design for a low-cost compact highresolution tactile sensor with application to in-hand manipulation,” IEEE Robot.

Autom. Lett., vol. 5, no. 3, pp. 3838–3845, 2020

[34] X. Jiang et al. p. 7

Where the paper says it

3838–3845, 2020. Jiang et al.

, “IRIS: An immersive robot interaction system,” in Proc. p. 7
Where the paper says it

Jiang et al. , “IRIS: An immersive robot interaction system,” in Proc.

Conf. Robot Learn., 2025

[35] E. Todorov, T. p. 7

Where the paper says it

(CoRL), 2025. Todorov, T.

Erez, and Y. p. 7
Where the paper says it

Todorov, T. Erez, and Y.

Tassa, “Mujoco: A physics engine for model-based control,” in Proc. p. 7
Where the paper says it

Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in Proc.

IEEE/RSJ Int. p. 7
Where the paper says it

Tassa, “Mujoco: A physics engine for model-based control,” in Proc. IEEE/RSJ Int.

Conf. Intell Robots Syst. p. 7
Where the paper says it

IEEE/RSJ Int. Robots Syst.

, 2012

[36] T. Julg ¨ et al. p. 7

Where the paper says it

(IROS), 2012. Julg ¨ et al.

, “Robot Control Stack: A lean ecosystem for robot learning at scale,” in Proc. p. 7
Where the paper says it

Julg ¨ et al. , “Robot Control Stack: A lean ecosystem for robot learning at scale,” in Proc.

IEEE Int. Conf. Robot Autom., 2026

[37] T. Z. Zhao, V. Kumar, S. Levine, and C. p. 7

Where the paper says it

(ICRA), 2026. Levine, and C.

Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proc. p. 7
Where the paper says it

Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proc.

Robot.: Sci. Syst., 2023

[38] Y. Chen et al. p. 7

Where the paper says it

Abbeel, and R. Chen, “ViTaMIn: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface,” in arXiv:2504.

, “Manipulationnet: An infrastructure for benchmarking real-world robot manipulation with physical skill challenges and embodied multimodal reasoning,” in arXiv:2603. p. 7
Where the paper says it

Chen et al. , “Manipulationnet: An infrastructure for benchmarking real-world robot manipulation with physical skill challenges and embodied multimodal reasoning,” in arXiv:2603.

04363, 2026

[39] A. George, S. Gano, P. Katragadda, and A. p. 7

Where the paper says it

04363, 2026. Katragadda, and A.

B. Farimani, “Vital pretraining: Visuo-tactile pretraining for tactile and non-tactile manipulation policies,” in Proc. p. 7
Where the paper says it

Katragadda, and A. Farimani, “Vital pretraining: Visuo-tactile pretraining for tactile and non-tactile manipulation policies,” in Proc.

IEEE Int. Conf. Robot Autom., 2025

[40] K. He, X. Zhang, S. Ren, and J. p. 7

Where the paper says it

(ICRA), 2025. Ren, and J.

Sun, “Deep residual learning for image recognition,” in Proc. p. 7
Where the paper says it

Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc.

IEEE Conf. Comput. Vis Pattern Recognit. p. 7
Where the paper says it

Sun, “Deep residual learning for image recognition,” in Proc. Pattern Recognit.

, 2016

[41] Dileepkumar Khajana, “Spiral light bulb,” GrabCAD

--- Page 5 ---

TABLE I

REPORTED CAPABILITIES OF HANDHELD DEMONSTRATION INTERFACES.: REPORTED; –: NOT REPORTED. p. 2

Where the paper says it

TABLE I REPORTED CAPABILITIES OF HANDHELD DEMONSTRATION INTERFACES. ✓: REPORTED; –: NOT REPORTED.

GRASP FORCE EXCLUDES WRIST-ONLY LOADS; VISUAL GUIDANCE IS OPERATOR-FACING; ADDED HAPTICS EXCLUDES NATURAL CONTACT

--- Page 6 ---

Fig. 2. Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. p. 3

Where the paper says it

Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T) sensing.

p. 3
Where the paper says it

Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T) sensing.

(a) The hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T) sensing. p. 3
Where the paper says it

Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T) sensing.

(b) The device rests on the back of the hand, while the thumb and index finger operate the rear scissor pads

--- Page 7 ---

Fig. 2. Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (c) The nonlinear transmission couples finger motion, while a selectively engaged blocker constrains further closure

--- Page 8 ---

Fig. 4a.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Enhanced data quality for robot learning by using an AR pipeline to visualize and validate robot configurations during robot-free data collection. This allows operators to recognize infeasible poses while recording rather than filtering them only after collection.

  2. Development of imitation policies capable of handling force-aware demonstrations using different fingertip modalities like CoinFT and DIGIT sensors. The system can train models that preserve the target gripper geometry while enabling natural input before contact and force measurement after contact.

  3. Integration of a haptic feedback mechanism supporting force-aware teleoperation for simulated and real robots. This mechanism allows the operator to receive feedback such as vibration feedback indicating remote contact and the approximate magnitude of the commanded load.

  4. Creation of a universal manipulation interface matched to nonlinear kinematics, finger attachments, and viewpoint of specific robotic grippers like the Robotiq 2F-85. This enables direct correspondence between operator input and gripper actuation by utilizing a scissorlike input drives the coupled fingers mechanism.

Abstract

Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator's arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface's capability for collecting force-aware teleoperation datasets in simulation.

Sources

Related papers