MRNaB: Mixed Reality-based Robot Navigation Interface using Optical-see-through MR-beacons
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "MRNaB: Mixed Reality-based Robot Navigation Interface using Optical-see-through MR-beacons".
Paper summary: Rosa: So to summarize this paper on MRNaB, their main thesis is that traditional two dimensional displays limit how much information you can see at once when you're trying to navigate a robot >
Dev: They argue that Mixed Reality technology fixes this by enhancing the dimensionality of visualization, letting users perceive multiple pieces of information concurrently using these MR-beacons >
Rosa: The specific approach they propose is this MRNaB interface which uses optical see through MR-beacons created with an air tap in the real world >
Dev: And these beacons are designed to be persistent, giving you multi destination visualization and acting as a signal transmitter for the robot navigation, which is what eliminates needing to repeatedly input where the robot is >
Rosa: They break down how it works into four main functions: Add, Move, Delete and Select which lets you manage these beacons in your space >
Dev: And they emphasize that this database system means you don't have to specify the robot's location every single time you run the program which is a big deal for repetitive tasks like delivery >
Taro: It sounds like it’s designed to streamline the interaction, moving away from purely command and control inputs on a flat screen >
Rosa: Exactly. They show how this approach improves user experience and operational efficiency in robot navigation when compared to traditional methods >
Conclusion: Dev: Looking at this paper, MRNaB by Iglesius, Kobayashi, Uranishi, and Takemura is really about making robot navigation more intuitive for the person using it >
Rosa: It’s about moving beyond the limitations of flat screens and giving users a richer way to see their robot's path in the real world >
Dev: The implication is that if we can get these systems into regular use, we could see a significant reduction in the time required for people to direct robots to specific locations >
Taro: So, it means less wasted time getting setup and more actual movement toward the goal when using this interface >
Rosa: That’s right. It’s about using augmented reality to build a better feedback loop between the user and the robot's navigation needs >
Dev: And while they noted issues like occlusion or far distance, it suggests that for tasks requiring repeated navigation, this mixed reality approach has real promise because it significantly cuts down on the necessary beacon settings frequency >
Eduardo Iglesius, Masato Kobayashi, Yuki Uranishi, Haruo Takemura
Graduate School of Information Science and Technology, The University of Osaka
cs.RO
Submitted: 2024-03-28
Updated: 2025-05-15
Project page: https://mertcookimg.github.io/mrnab
Importance score: 72/100
The gist: Recent advancements in robotics have led to the development of numerous interfaces to enhance the intuitiveness of robot navigation.
Key concepts
- MR-beacons
- These are virtual markers created using optical-see-through mixed reality technology. Users create them in the real world using an 'air tap' gesture, which allows the robot system to perceive and use these persistent markers for navigation.
- Air Tap Gesture
- This is a specific physical action users perform in the real world to create or interact with MR-beacons. It serves as the primary input method for adding new beacons, moving existing ones, and selecting them for robot guidance.
- ROS 2 Integration
- The system uses ROS 2 to connect the robot (Kachaka) with the user interface. Co-localization is achieved by placing a QR image target on the floor that matches the map on the robot, allowing accurate transformation of MR-beacon coordinates into usable robotic data.
- Persistent Beacons
- MRNaB creates beacons that remain visible and locatable throughout a session. This feature enables multi-destination visualization and acts as a signal transmitter, reducing the need for users to repeatedly specify the robot's location.
Terminology
Summary
Recent advancements in robotics have led to the development of numerous interfaces to enhance the intuitiveness of robot navigation. p. 0 This paper proposes the Mixed Reality-based Robot Navigation Interface using an Optical-see-through MR-beacons, a novel approach that uses MR-beacons created with an “air tap”, situated in the real world. p. 0 The results show significant improvements in user performance, both objectively and subjectively, confirming that the MRNaB enhances navigation efficiency and user experience. p. 0
System Overview
The MRNaB system is constructed into four primary functions: “Add”, “Move”, “Delete”, and “Select”. These functions allow for the addition of MR-beacons, location movement, its deletion, and the selection of MR-beacons for navigation purposes respectively. p. 0 The system employs a database that eliminates the need for users to repeatedly specify the robot’s location. p. 1
Key Features and Functionality
The MRNaB provides persistent MR-beacons, enabling multi-destination visualization and functioning as a signal transmitter for robot navigation. The system supports the “air tap” gesture to create beacons, move their location, select them for navigation, and delete them from the floor. p. 7 Specifically, the “Add” function generates an MR-beacon from the floor by performing an “air tap” movement. p. 3 The “Move” function allows users to adjust the pose of an existing MR-beacon. p. 3
Technical Implementation Details
The system utilizes ROS 2 for interfacing the robot (Kachaka) with the user. Co-localization is achieved by using vuforia, putting a QR image target on the floor level which resembles the map topic on the robot. p. 2 The MR-beacon design mirrors those of the real robot, and its coordinates are measured relative to this image target for easier transformation to ROS-type data. p. 2
Experimental Validation
Experiments compared MRNaB with a traditional 2D system using a computer display and mouse. Objective indices showed that participants required less beacon actions before navigation, such as an average of 1. p. 659 tries for the MRNaB system compared to 2. p. 695 tries for the 2D system overall. p. 2 Navigation number measurement results per task indicated that participants required less navigation attempts using MRNaB compared to the traditional 2D system. p. 6
Conclusion and Limitations
The study concludes that despite users taking longer time to place the beacon accurately, MRNaB required less beacon settings frequency which significantly decreased the amount of navigation needed. Subjective evaluation showed that viewing detailed environmental information increased confidence and preference for users. p. 7 Limitations include occlusion leading to optical illusions and difficulty establishing beacons at very far distances. p. 7 The authors plan to address these issues in future work by implementing solutions for occlusion and incorporating a minimap for far-distance establishment. p. 7
REFERENCES
[1] Quigley M, Conley K, Gerkey B, et al. ROS: an open-source Robot Operating System. p. 8 Vol. 3; 2009
[2] Ngo AT, Tran NH, Ton TP, et al. Simulation of hybrid autonomous underwater vehicle based on ros and gazebo. p. 8 In: 2021 International Conference on Advanced Technologies for Communications (ATC); 2021
[3] Cruz AB, Sousa A, Reis LP. Controller for real and simulated wheelchair with a multimodal interface using gazebo and ros. p. 8 In: 2020 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC); 2020
[4] Beraldo G, Castaman N, Bortoletto R, et al. Ros-health: An open-source framework for neurorobotics. p. 8 In: 2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR); 2018
[5] Velamala SS, Patil D, Ming X. Development of ros-based gui for control of an autonomous surface vehicle. p. 8 In: 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO); 2017
[6] Hoang KC, Chan WP, Lay S, et al. ARviz: An Augmented RealityEnabled Visualization Platform for ROS Applications. p. 8 IEEE Robotics & Automation Magazine. p. 8 2022;29(1):58–67
[7] Salvato M, Heravi N, Okamura AM, et al. Predicting hand-object interaction for improved haptic feedback in mixed reality. p. 8 IEEE Robotics and Automation Letters. p. 8 p. 8 p. 8 p. 8 p. 8 2022;7(2):3851–3857
[8] Devo A, Mao J, Costante G, et al. Autonomous single-image drone exploration with deep reinforcement learning and mixed reality. p. 8 IEEE Robotics and Automation Letters. 2022;7(2):5031–5038
[9] Makhataeva Z, Varol HA. Augmented reality for robotics: A review. p. 8 Robotics. 2020;9(2)
[10] Fan W, Guo X, Feng E, et al. Digital twin-driven mixed reality framework for immersive teleoperation with haptic rendering. p. 8 IEEE Robotics and Automation Letters. 2023;8(12):8494–8501
[11] Penco L, Momose K, McCrory S, et al. Mixed reality teleoperation assistance for direct control of humanoids. p. 8 IEEE Robotics and Automation Letters. 2024;9(2):1937–1944
[12] Zhang C, Lin C, Leng Y, et al. An effective head-based hri for 6d robotic grasping using mixed reality. p. 8 IEEE Robotics and Automation Letters. 2023;8(5):2796–2803
[13] Lee J, Lim T, Kim W. Investigating the Usability of Collaborative Robot Control Through Hands-Free Operation Using Eye Gaze and Augmented Reality. p. 8 In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems; 2023
[14] Chen J, Sun B, Pollefeys M, et al. A 3D Mixed Reality Interface for Human-Robot Teaming. p. 8 arXiv preprint arXiv:231002392. p. 8 2023;Available: arXiv:2310. p. 802392 [cs.RO]
[15] Zhang G, Zhang D, Duan L, et al. Accessible Robot Control in Mixed Reality. p. 8 arXiv preprint arXiv:230602393. p. 8 2023;Available: arXiv:2306. p. 802393 [cs.RO]
[16] Koenig N, Howard A. Design and use paradigms for Gazebo, an opensource multi-robot simulator. p. 8 In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems; Vol. p. 8 3; 2004
[17] Cepeda JS, Chaimowicz L, Soto R. Exploring Microsoft Robotics Studio as a Mechanism for Service-Oriented Robotics. p. 8 In: Proc. of the Latin American Robotics Symposium and Intelligent Robotics Meeting; 2010
[18] Michel O. WebotsTM: Professional Mobile Robot Simulation. p. 8 International Journal of Advanced Robotic Systems. p. 8 2004;1
[19] Rajapaksha DD, Mohamed Nuhuman MN, Gunawardhana SD, et al. Web Based User-Friendly Graphical Interface to Control Robots with ROS Environment. p. 8 In: Proc. of the International Conference on Information Technology Research; 2021
[20] Fernando WAM, Jayawardena C, Rajapaksha UUS. Developing A UserFriendly Interface from Robotic Applications Development. p. 8 In: Proc. of the International Research Conference on Smart Computing and Systems Engineering; Vol. p. 8 5; 2022
[21] Aarizou M. ROS-based web application for an optimized multi-robots multi-users manipulation. p. 8 In: Proc. of the National Conference in Computer Science Research and its Applications; 2023
[22] Kam H, Lee SH, Park T, et al. RViz: a toolkit for real domain data visualization. p. 8 Telecommunication Systems. p. 8 2015;60:1–9
[23] Tiddi I, Bastianelli E, Bardaro G, et al. A User-friendly Interface to Control ROS Robotic Platforms. p. 8 In: Proc. of the International Workshop on the Semantic Web; 2018
[24] Walker ME, Hedayati H, Szafir D. Robot teleoperation with augmented reality virtual surrogates. p. 8 In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI); 2019
[25] Wu M, Dai SL, Yang C. Mixed Reality Enhanced User Interactive Path Planning for Omnidirectional Mobile Robot. p. 8 Applied Sciences. p. 8 2020;10(3):1135
[26] Gu M, Cosgun A, Chan WP, et al. Seeing Thru Walls: Visualizing Mobile Robots in Augmented Reality. p. 8 In: Proc. of the IEEE International Conference on Robot & Human Interactive Communication. p. 8 IEEE; 2021
[27] Walker M, Hedayati H, Lee J, et al. Communicating Robot Motion Intent with Augmented Reality. p. 8 In: Proc. of the ACM/IEEE International Conference on Human-Robot Interaction; 2018
[28] Owaki K, Techasarntikul N, Shimonishi H. Human Behavior Analysis in Human-Robot Cooperation with AR Glasses. p. 8 In: Proc. of the IEEE International Symposium on Mixed and Augmented Reality; Los Alamitos, CA, USA. p. 8 IEEE Computer Society; 2023
[29] Zolotas M, Demiris Y. Towards Explainable Shared Control using Augmented Reality. p. 8 In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems; 2019
[30] Zolotas M, Elsdon J, Demiris Y. Head-Mounted Augmented Reality for Explainable Robotic Wheelchair Assistance. p. 8 In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems; 2018
[31] Regal F, Petlowany C, Pehlivanturk C, et al. AugRE: Augmented Robot Environment to Facilitate Human-Robot Teaming and Communication. p. 8 In: Proc. of the IEEE International Conference on Robot and Human Interactive Communication; 2022
[32] Kot T, Nov´ak P, Bajak J. Using HoloLens to create a virtual operator station for mobile robots. p. 8 In: Proc. of the International Carpathian Control Conference; 2018
[33] Angelopoulos A, Hale A, Shaik H, et al. Drone Brush: Mixed Reality Drone Path Planning. p. 8 In: Proc. of the ACM/IEEE International Conference on Human-Robot Interaction; 2022
[34] Quesada RC, Demiris Y. Holo-SpoK: Affordance-Aware Augmented Reality Control of Legged Manipulators. p. 8 In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems; 2022
[35] Cruz C, Cerro J, Barrientos A. Mixed-reality for quadruped-robotic guidance in SAR tasks. p. 8 Journal of Computational Design and Engineeringy. p. 8 2023;10
[36] Brooke J. SUS: A quick and dirty usability scale. p. 8 Usability Eval Ind. p. 8 1995;189
[37] Shapiro SS, Wilk MB. An Analysis of Variance Test for Normality (Complete Samples). p. 8 Biometrika. p. 8 1965;52(3/4):591–611
[38] Rey D, Neuh¨auser M. Wilcoxon-Signed-Rank Test. p. 6 International Encyclopedia of Statistical Science. p. 8 2011;:1658–1659
B is the token BARE, with no backticks around it and no quotation marks around them.
Pruning redundant reasoning tokens
The paper proposes MRNaB, a mixed reality navigation interface with an MR-beacon shaped like the robot, which can be used for multiple destinations by utilizing the “air tap” gesture. This approach improves destination visualization and enhances navigation efficiency. p. 1
How the training signal is built
The system supports persistent goal poses, allowing users to set frequently used destinations, beneficial for repetitive navigation tasks in various settings. The database system ensures that previously set destinations are retrieved upon rerunning the program. p. 2
Where the gains come from
The experiment demonstrated that participants required less beacon actions to navigate the robot accurately using our system compared to a traditional 2D system. Furthermore, participants required less navigation for the robot to reach its destination using our system compared to the traditional 2D system. p. 6 The MRNaB was also proven to increase user’s confidence in order to navigate the robot to the right direction. p. 7
The system is integrated with a database system, eliminating the need for users to create a new MR-beacon each time the project is restarted. This ensures multi-destination visualization and maintains data persistence across sessions. p. 7 The experiment showed that participants required less action time to complete the task using traditional 2D system compared to our system. p. 6
The ability to see the beacon throughout the entire process increased confidence and preference of users. However, learning the “air tap” gesture introduced a subjective element to the responses which contributed further to no statistical significance observed in these cases. p. 7 The effectiveness of our method can be seen through comprehensive real-world experiments. p. 7
In contrast, participants could see the targeted location in the real world leading them to take more time to place it accurately. The traditional system did not provide a visible target location causing participants to place the beacon more quickly. p. 7 However, users performed fewer beaconrelated actions and navigation attempts suggesting improved efficiency in directing the robot to its destination. p. 7
The ability to see the beacon throughout the entire process increased confidence and preference of users. Additionally, viewing detailed environmental information led to higher scores for these indices. p. 7 However, as most participants lacked experience with MR head-mounted displays particularly the “air tap” gesture a learning curve was evident. p. 7 The unfamiliarity with the device necessitated a period of adaptation before users could comfortably utilize the system resulting in higher scores for professional assistance and practice necessity. p. 7
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve robot navigation efficiency by providing a unified view of virtual and real-world elements, which minimizes
the need for attention shifts
between the map and the real environment. The system can navigate directly in the real world compared to 2D systems, as seen in the result whereparticipants were able to navigate the robot directly in the real world.
-
Enhance multi-destination visualization by using a persistent MR-beacon that retains its position even after restarting, eliminating
the need for users to repeatedly specify the robot’s pose for frequently visited destinations.
This allows users to visualize multiple destinations concurrently. -
Increase user confidence and preference by providing visual feedback on the robot's destination in the real world, as noted by the finding that
the ability to see the beacon throughout the entire process increased confidence and preference of users.
-
Reduce redundant interactions by eliminating the need for users to repeatedly specify locations through a database system, which
eliminates the need for users to repeatedly specify the robot’s location
after a project is restarted. -
Improve real-world spatial understanding by using MR-beacons shaped like the robot, which makes it
easier for user to visualize where the robot wants to go even for multiple destinations.
Sources
- A 3D Mixed Reality Interface for Human-Robot Teaming
- EgoSpot:Egocentric Multimodal Control for Hands-Free Mobile Manipulation
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation