RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback

summary

The gist

RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning Interface…

In short

RUMI is a handheld interface matched to the Robotiq 2F-85 robot, using augmented reality for real-time kinematic feasibility checks during data collection. It incorporates haptic feedback for force-aware teleoperation and uses specialized sensors like CoinFT and DIGIT to capture tactile data. This system enables the collection of high-quality, embodiment-matched demonstrations crucial for training robust robot learning policies.

Key concepts

RUMI
A handheld universal manipulation interface designed specifically to match the kinematics, finger attachments, and viewpoint of a particular robot gripper (like the Robotiq 2F-85). It uses a scissor-like mechanism where operator finger motion directly drives the coupled gripper fingers, providing direct physical correspondence between human input and robot action.
Augmented Reality Visualization
The system integrates tracking into a VR headset to provide real-time AR feedback. This allows operators to see a digital shadow (MuJoCo rendering) of the target robot during recording, visualizing the kinematic feasibility of their movements before physical contact occurs, which improves data quality.
Force-Aware Teleoperation
A mechanism that provides haptic feedback during remote operation. It uses a blocker to control low-resistance aperture before contact and converts subsequent squeezing into a measured force command. This allows operators to feel remote contact and the approximate load magnitude, supporting both in-the-wild collection and teleoperation.
Tactile Sensing Modalities (CoinFT/DIGIT)
The system utilizes different fingertip sensors, such as CoinFT or DIGIT, which support force/torque sensing. These sensors allow the robot to reliably push objects into alignment by providing force-adjusted trajectories, enabling the training of policies that are sensitive to contact forces.

Terminology used across episodes

This episode discusses

The paper

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback · Read on arXiv

Siemens Germany University of Technology Nuremberg Germany Technical University of Munich Germany

Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator's arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface's capability for collecting force-aware teleoperation datasets in simulation.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback".

Dev: Here's the gist of the paper. RUMI presents a handheld universal manipulation interface matched to the Robotiq 2F-85 with augmented reality and haptic feedback to enable force-aware teleoperation and robot learning Interface Design and Hardware RUMI is designed as a handheld platform matched to the nonlinear kinematics, finger attachments, and wrist-camera viewpoint of the Robotiq 2F-85 A scissorlike input drives the coupled fingers while a rotary encoder measures the resulting grasp coordinate Interchangeable fingers support force/torque and vision-based tactile sensing for contact-rich demonstrations The hardware features a crossed, scissor-like linkage that couples the motion of the operator’s thumb and index finger to the gripper fingers A Teensy 4.0 microcontroller coordinates on-device sensing and actuation while synchronizing measurements Augmented Reality Visualization The method integrates the tracking system into a virtual reality headset to provide real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform During data collection, RUMI poses are stored in the registered robot base frame established by the AR system IRIS renders a MuJoCo-based digital shadow of the target robot during recording to allow the operator to observe the robot configuration associated with the demonstrated motion in real time This visualization allows users to recognize infeasible poses while recording rather than filtering them only after collection Force-Aware Teleoperation Mechanism The haptic feedback mechanism supports force-aware teleoperation for simulated and real robots During teleoperation, a contact-triggered blocker provides low-resistance aperture control before contact and converts the subsequent squeeze into a measured force command The Meta Quest controller can additionally provide vibration feedback indicating remote contact and the approximate magnitude of the commanded load This mechanism allows the same handheld device to support both in-the-wild collection and teleoperation Data Quality and Policy Training The AR condition results in fewer number of joint limit- and velocity violations compared to the w/o AR condition The average manipulability is 19.6% higher than the w/o AR condition, although the standard deviation is also greater For the CoinFT sensor, force-adjusted trajectories are able to reliably push the gear into alignment with its neighbors The DIGIT variant outperforms the raw variant in the box sorting task because it lacks information about different boxes’ textures Simulation Results and Force Control Demonstrations collected with haptic feedback exhibit a clear object-dependent force strategy that persists in policy rollouts The learned policies preserve this separation during rollout, producing mean forces of 5.42 N and 19.86 N for the bulb and block respectively This indicates that the blocker provides a useful contact reference for demonstrating force-sensitive behavior and that the learned policy reproduces the resulting object-dependent grasp strategy The results show a clear correlation between task success rates and the availability of their respective fingertip tactile sensors RUMI enables the collection of embodiment-matched, tactile-aware demonstrations for robot learning by providing a consistent physical interface across these modes Future work could include generative AI to overcome the single-view limitation as well as more informative visual cues about external forces acting upon the simulated or teleoperated robot REFERENCES

1: B.

Rosa: D.

Dev: Argall, S.

Rosa: Chernova, M.

Title and authors: Rosa: Alright, moving into the specific details of this paper called "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," we’re going to break down what exactly this system is doing.

Dev: We've talked about the hardware matching, but let’s focus on the interface design itself. They describe RUMI as a handheld platform that is specifically matched to the Robotiq 2F-eighty-five robot’s nonlinear kinematics and finger attachments <ref:2611.15802#pg1,matched to the Robotiq 2F-85>.

Taro: That matching part is huge because generic interfaces often fail when you have those specific, complicated movements of a real gripper; this system uses a scissor like input drive to couple the motion of the operator’s thumb and index finger directly to the gripper fingers.

Rosa: And they emphasize that these interchangeable fingers support both force torque sensing and vision based tactile sensing for contact rich demonstrations. That means you get both force information and visual feedback during interaction.

Dev: They also detail how they integrated the tracking system into a virtual reality headset to provide real time augmented reality feedback about the kinematic feasibility of motion on the target robot platform.

Taro: During data collection, they store these RUMI poses in the registered robot base frame established by that AR system, and then IRIS renders a MuJoCo based digital shadow of the target robot during recording.

Rosa: That allows the operator to observe the robot configuration associated with the demonstrated motion in real time so they can see if it’s physically possible before they commit to recording it.

Dev: The paper also introduces this haptic feedback mechanism that supports force aware te operation for simulated and real robots, giving the operator feedback such as vibration indicating remote contact and the approximate magnitude of the commanded load.

Taro: So, so it’s not just a capture device; it’s an active training tool where you can actually feel the forces involved in manipulating that robot remotely.

Rosa: It really is a powerful combination because RUMI provides all these elements together: matching hardware, AR visualization, and force aware teleoperation capability.

Dev: This setup aims to give us data that isn't just visually correct but physically grounded and force aware, which is what we really need for robust robot learning.

The paper's summary: Rosa: So, summarizing the core contributions of "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," it boils down to three main things.

Dev: First, they present RUMI itself as a Robotiq 2F-eighty-five matched handheld interface featuring nonlinear actuation and interchangeable fingertips <ref:2611.15802#pg1>.

Taro: Second, they developed an AR pipeline for visualizing and validating robot configurations during robot free data collection to help operators recognize infeasible poses while recording rather than filtering them only after collection.

Rosa: And third, they introduced a haptic feedback mechanism that allows for gripper force control during simulated and real teleoperation, giving the operator feedback on remote contact and load magnitude.

Dev: So, in essence, they are releasing the device design and software framework along with a dataset featuring force torque and vision based tactile sensor data.

Taro: That dataset is what makes it practical for others to actually train their own policies without having to build all this complicated hardware from scratch.

Rosa: It’s about providing a consistent way to get embodiment matched, tactile aware demonstrations that are ready for robot learning right now.

Dev: The implication is that we have a new way to capture demonstrations where the physical interaction is directly linked to the robot's specific capabilities and constraints.

The paper's improvements: Rosa: Now let’s look at the improvements they suggest in this work, because they don’t just stop at presenting RUMI; they outline how this system can be enhanced.

Dev: They point out that by using an AR pipeline, we get enhanced data quality for robot learning because it allows operators to visualize and validate robot configurations during robot free collection.

Taro: This is important because it lets people recognize infeasible poses while recording, which is a big improvement over just filtering them later when you review the data.

Rosa: They also improved the policies themselves by developing imitation policies capable of handling force aware demonstrations using different fingertip modalities like CoinFT and DIGIT sensors.

Dev: This means they can train models that preserve the target gripper geometry while enabling natural input before contact and force measurement after contact, which is a big step in modeling real physical behavior.

Taro: And they showed that these learned policies preserve an object dependent force strategy during rollout, producing mean forces of five point four two newtons and nineteen point eight six newtons for the bulb and block respectively.

Rosa: That result is significant because it confirms that the blocker provides a useful contact reference for demonstrating force sensitive behavior in simulation.

Dev: It also showed a clear correlation between task success rates and the availability of their fingertip tactile sensors, proving those sensors are essential for successful demonstrations.

Conclusion: Rosa: So, to wrap up on "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," it’s really about providing a unified platform for data collection.

Dev: In summary, the main implication is that we now have a method to collect high quality demonstrations where the physical interaction is directly tied to the robot's specific capabilities through matching hardware and rich feedback loops.

Taro: It means we can get datasets that are not just visually correct but physically grounded and force aware, which is what we need for robust robot learning.

Rosa: For future work, they suggest using generative AI to overcome the single view limitation as well as providing more informative visual cues about external forces acting upon the simulated or teleoperated robot.

Dev: And that sounds like the next step—using generative AI to move beyond just seeing what’s there to understanding what’s happening physically around the robot.

Taro: I think those future directions are smart because they focus on making the system even more aware of those subtle external forces, not just basic contact.

Rosa: Yeah, it really shows that this work is laying a foundation for learning systems that can handle real world complexity. RUMI is a really solid piece of hardware and software integration for this goal.

More episodes

← Home