MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation".
Paper summary: Rosa: We just touched on the core idea of MRReP, which is using Mixed Reality to let users draw paths physically instead of just clicking on a screen.
Dev: The authors state that autonomous robots in shared indoor environments often need paths that respect human movement, like avoiding pedestrians or maintaining a certain clearance.
Rosa: Conventional path planners only optimize geometric costs and they don't offer strong support for human operators to explicitly define the route itself.
Dev: So this paper introduces MRReP as an interface that enables users to draw a Hand-drawn Reference Path directly on the physical floor using hand gestures.
Rosa: The core claim is that this drawn path is integrated into the robot navigation stack through a custom Hand-drawn Reference Path Planner, which turns those point sequences into a global path for autonomous navigation.
Dev: They also developed three fundamental functions for manipulating this path: ADD to create a new HRP, CLEAR to delete all existing ones, and SEND to transmit the data via ROS two to trigger robot navigation along that specified route <ref:2604.00059#pg1>.
Rosa: The main contribution they highlight is that in their within-subject experiment against a conventional 2D baseline interface, MRReP enhanced path specification accuracy and usability while also enabling more stable path specification in the physical environment <ref:2604.00059#pg1,within-subject experiment against a conventional 2D baseline interface>.
Dev: They found that MRReP enabled more faithful and intuitive path specification directly in the physical environment than the 2D system, which improved both accuracy and subjective usability while reducing perceived workload <ref:2604.00059#pg1>.
Rosa: And they showed that this method is better for incorporating human spatial intention into robot navigation than relying on traditional methods like cost tuning or waypoint input.
Dev: So, to summarize, MRReP is an MR-based interface that allows users to draw robot navigation paths in the physical environment using hand gestures and it shows this improves path specification accuracy and usability compared to a conventional 2D interface <ref:2604.00059#pg1>.
Taro: It’s interesting because they are showing how direct physical interaction can lead to better path fidelity than purely digital inputs.
Rosa: Right, and that brings us nicely into the conclusion of what this whole paper is trying to tell us about this new way of working with robots.
Conclusion: Dev: So looking at the paper "MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation" and the authors Takumi Taki, Masato Kobayashi, Yuki Uranishi, and others, what’s the bigger picture here?
Rosa: This paper presents MRReP as a way to let users specify robot navigation paths by drawing them physically in a Mixed Reality setting using hand gestures.
Dev: The key implication is that this method supports more faithful path specification than conventional 2D interfaces and it improves both accuracy and usability, which is significant for how humans interact with autonomous systems <ref:2604.00059#pg1>.
Rosa: It suggests that directly specifying the route in the physical environment using MR is an effective approach for incorporating human spatial intention into mobile robot navigation.
Dev: So, instead of relying on indirect methods or discrete inputs, we can have a system where a person can just draw the path they want to see it instantly translated into a global path by the robot.
Rosa: That way, we’re giving humans more direct control over the shape and placement of the path, which is much better for tasks in complex indoor spaces.
Dev: The work suggests that this kind of interface could be really useful for making robot navigation feel more intuitive for people working alongside them in shared spaces.
Rosa: It moves us toward a future where specifying paths isn't just about inputting coordinates, but about intuitively sketching the desired path in the actual space around the robot.
Takumi Taki, Masato Kobayashi, Yuki Uranishi
Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka · Graduate School of Maritime Sciences, Kobe University
cs.RO
Submitted: 2026-03-31
Updated: 2026-03-31
Journal ref: IEEE 35th International Conference on Robot and Human Interactive Communication (RO-MAN), Kitakyushu, Japan, 2026, pp. 191-198
DOI: 10.1109/RO-MAN72286.2026.11714993
Code: https://github.com/pf-robotics/kachaka-api
Project page: https://mertcookimg.github.io/mrrep
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 82/100
The gist: Autonomous mobile robots operating in humanshared indoor environments often require paths that reflect human spatial intentions…
Key concepts
- Handdrawn Reference Path (HRP)
- This is a path that users define physically on the floor using hand gestures while wearing a HoloLens 2. It represents the desired route or trajectory for an autonomous mobile robot, allowing humans to specify complex paths intuitively rather than through abstract digital inputs.
- MRReP Interface
- MRReP is a Mixed Reality interface that lets users draw and edit paths directly in the physical world. It uses hand gestures to create, clear, and send path data to the robot system, bridging the gap between human spatial intent and robot navigation commands.
- Path Specification Accuracy
- This refers to how closely the path specified by a user matches their intended route. The MRReP system showed higher accuracy than conventional 2D methods because users could visually define paths in their actual environment, leading to more faithful path representation.
- Send Function
- The SEND function transmits the stored HRP data from the Mixed Reality interface to ROS 2. This action triggers the robot navigation system to follow the specific, hand-drawn path that was previously defined by the user.
Terminology
Summary
Autonomous mobile robots operating in humanshared indoor environments often require paths that reflect human spatial intentions, such as avoiding interference with pedestrian flow or maintaining comfortable clearance <ref:2604. p. 000059#pg2> MRReP, a Mixed Reality-based interface that enables users to draw a Handdrawn Reference Path (HRP) directly on the physical floor using hand gestures <ref:2604. p. 000059#pg2>. The results demonstrated that MRReP enhanced path specification accuracy, usability, and perceived workload, while enabling more stable path specification in the physical environment <ref:2604. p. 000059#pg2>.
Problem Statement
Conventional path planners primarily optimize geometric costs and provide limited support for explicit route specification by human operators <ref:2604.00059#pg2>. In practical operation, there are many situations in which an operator has a preferred path in mind, concerning the shape and placement of the path itself rather than only the final destination <ref:2604. p. 000059#pg2>. Operators often have to rely on indirect methods such as cost tuning, repeated replanning, or discrete goal and waypoint input, which may not faithfully capture the desired path <ref:2604. p. 000059#pg2>.
System Architecture and Functionality
The proposed system allows users wearing a HoloLens 2 to draw a Hand-drawn Reference Path (HRP) directly in the physical environment through hand gestures <ref:2604.00059#pg2>. The interface provides three fundamental functions for HRP manipulation: ADD, CLEAR, and SEND <ref:2604. p. 200059#pg2>.
-
ADD Function is designed to create a new HRP by generating a piecewise linear path through hand gestures <ref:2604. p. 200059#pg2>.
-
CLEAR Function deletes all existing HRPs when selected, removing corresponding coordinate records from the database <ref:2604. p. 200059#pg2>.
-
SEND Function transmits the stored HRP data to ROS 2 and triggers robot navigation along the specified path <ref:2604. p. 200059#pg2>.
Implementation Details
The system uses two communication links: Unity–ROS 2 and ROS 2–Kachaka <ref:2604.00059#pg2>. Coordinate systems of Unity and ROS 2 are aligned through HoloLens 2-based QR code recognition using the Vuforia Engine <ref:2604. p. 300059#pg2>. Path data management stores the coordinate sequence in a JSON file managed by a Unity script to restore the previous state after reboot <ref:2604. p. 300059#pg2>. Path processing involves implementing a custom node that converts HRP point sequences into a global path and using a custom global planner to provide this path to Navigation2 <ref:2604. p. 300059#pg2>.
Experimental Validation
A within-subject experiment compared MRReP against a conventional 2D baseline interface <ref:2604.00059#pg2>. Evaluation metrics included Hand-drawn Path Accuracy, Number of Drawing Attempts, Task Completion Time, Path Stability, and Usability and Cognitive Load <ref:2604. p. 400059#pg2>. The MR system achieved higher scores on primary fidelity-related metrics like Precision and Recall compared to the 2D system <ref:2604. p. 500059#pg2>. However, some MR-specific challenges included physical discomfort such as arm fatigue during gesture-based interaction <ref:2604. p. 600059#pg2>.
Conclusion
This paper presented MRReP, a Mixed Reality-based interface that enables users to specify robot navigation paths through hand gestures in the physical environment <ref:2604.00059#pg2>. The proposed system supports more faithful path specification than conventional 2D interfaces <ref:2604. p. 600059#pg2>. A within-subject experiment showed that MRReP improved path specification accuracy, usability, and perceived workload relative to the 2D baseline <ref:2604. p. 600059#pg2>. These results suggest that direct path specification in the physical environment using MR is an effective approach for incorporating human spatial intention into mobile robot navigation <ref:2604. p. 600059#pg2>.
REFERENCES
[1] N. Abdul Khaleq and A. p. 7 Al-Araji, “A one decade survey of autonomous mobile robot systems,” International Journal of Electrical and Computer Engineering, vol. p. 7 11, pp. 4891–4906, 2021 <ref:2604. p. 700059#pg2>
[2] K. T. Eriksen and L. p. 7 Bodenhagen, “Understanding human-robot teamwork in the wild: The difference between success and failure for mobile robots in hospitals,” in 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (ROMAN), 2023, pp. p. 7 277–284
[3] D. C. Herath, L. Martin, S. Doolan, and J. p. 7 B. Grant, “Robots and aged care: A case study assessing implementation of service robots in an aged care home,” in 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2023, pp. p. 7 1641–1647
[4] A. Sidaoui, I. p. 7 H. Elhajj, and D. p. 7 Asmar, “Human-in-the-loop augmented mapping,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). p. 7 IEEE, 2018, pp. p. 7 3190–3195 <ref:2604. p. 700059#pg2>
[5] M. Skubic, D. Anderson, S. p. 7 Blisard, D. p. 7 Perzanowski, and A. p. 7 Schultz, “Using a hand-drawn sketch to control a team of robots,” Autonomous Robots, vol. p. 7 22, pp. 399–410, 2007
[6] D. Sakamoto, K. p. 7 Honda, M. Inami, and T. p. 7 Igarashi, “Sketch and run: A stroke-based interface for home robots,” in CHI 2009: Digital Life New World - Proceedings of the 27th International Conference on Human Factors in Computing Systems, ser. p. 7 Conference on Human Factors in Computing Systems - Proceedings, ACM. p. 7 ACM, 2009, pp. p. 7 197–200
[7] T. Taki, M. Kobayashi, E. p. 0 Iglesius, N. p. 7 Chiba, S. Shirai, and Y. p. 7 Uranishi, “MRHaD: Mixed reality-based hand-drawn map editing interface for mobile robot navigation,” in 2025 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). p. 7 IEEE, 2025, pp. p. 7 p. 7 1888–1895
[8] H. R. Kam, S.-H. Lee, T. Park, and C. p. 7-H. Kim, “RViz: a toolkit for real domain data visualization,” Telecommunication Systems, vol. p. 7 60, pp. 337–345, 2015
[9] D. Shah, B. Osinski, S. p. 7 Levine ´ et al. p. 7, “LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,” in Conference on robot learning. p. 7 pmlr, 2023, pp. p. 7 492–504
[10] B. Yu, Q. Yuan, K. Li, H. Kasaei, and M. p. 7 Cao, “Co-NavGPT: Multi r obot cooperative visual semantic navigation using vision language models,” IEEE Robotics and Automation Letters, vol. p. 7 11, no. 2, pp. 2122–2129, 2025
[11] J. Bao and R. Yonetani, “Path planning using instruction-guided probabilistic roadmaps,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). p. 7 IEEE, 2025, pp. 505–511
[12] K. Omer, E. Torta, and A. p. 7 Monteriu, “Semantic-enhanced path planning for safety-centric indoor robots navigation,” in 2024 10th International Conference on Automation, Robotics and Applications (ICARA). p. 7 IEEE, 2024, pp. p. 7 p. 7 185–190
[13] A. Sridhar, D. p. 7 Shah, C. Glossop, and S. p. 7 Levine, “NoMaD: Goal masked diffusion policies for navigation and exploration,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). p. 7 IEEE, 2024, pp. 63–70
[14] M. Wu, S.-L. Dai, and C. p. 7 Yang, “Mixed reality enhanced user interactive path planning for omnidirectional mobile robot,” Applied Sciences, vol. p. 7 10, no. 3, p. 1135, 2020
[15] P. Yu, A. Arora, Y. Song, G. Soh, N. M. L. Sharon, and A. p. 7 Kumar, “A mixed reality platform for online motion planning of mobile robots via hololens 2,” in Proc. p. 7 of the 9th Intl. p. 7 Conf. of Asian Society for Precision Engg. p. 7 and Nanotechnology (ASPEN 2022). p. 7 Singapore, 2022, pp. p. 7 817–828
[16] K. C. Hoang, W. P. Chan, S. Lay, A. Cosgun, and E. p. 7 A. Croft, “ARviz: An augmented reality-enabled visualization platform for ROS applications,” IEEE Robotics & Automation Magazine, vol. p. 7 29, no. 1, pp. 58–67, Mar. p. 0 2022
[17] E. Iglesius, M. p. 7 p. 7 Kobayashi, Y. p. 0 Uranishi, and H. p. 7 Takemura, “MRNaB: mixed reality-based robot navigation interface using optical-seethrough mr-beacons,” Advanced Robotics, vol. p. 7 39, no. 11, pp. 663–677, 2025
[18] A. Fang, X. Xu, Y. Lin, A. Gilbert, and D. p. 7 Panagou, “Mixed reality meets robotic systems: A hololens2-enabled waypoint navigation interface,” in HRI 2025 Workshop VAM-HRI
[19] M. Gu, E. Croft, and A. p. 7 Cosgun, “AR Point &Click: An interface for setting robot navigation goals,” pp. p. 7 38–49, 2022
[20] E. Iglesius, M. Kobayashi, and Y. p. 0 Uranishi, “MRPoS: Mixed reality-based robot navigation interface using spatial pointing and speech with large language model,” 2026
[21] G. Baker, T. Bridgwater, P. p. 7 Bremner, and M. p. 7 Giuliani, “Towards an immersive user interface for waypoint navigation of a mobile robot,” arXiv preprint arXiv:2003. p. 712772, 2020
[23] F. Boniardi, A. p. 7 Valada, W. Burgard, and G. p. 7 D. Tipaldi, “Autonomous indoor robot navigation using a sketch interface for drawing maps and routes,” in 2016 IEEE International Conference on Robotics and Automation, ICRA 2016, Stockholm, Sweden, May 16–21, 2016
[24] A. H. Tan, A. Fung, H. Wang, and G. p. 7 Nejat, “Mobile robot navigation using hand-drawn maps: A vision language model approach,” IEEE Robotics and Automation Letters, 2025
[25] J. Hwang, R. C. Arkin, and D. p. 7 Kwon, “Mobile robots at your fingertip: Bezier curve on-line trajectory generation for supervisory control,” in 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, Nevada, USA, October 27–November 1, 2003
[26] X. Hou, W. Li, Y. Han, A. Wang, Y. Yang, L. Liu, and Y. p. 7 Tian, “A novel mobile robot navigation method based on hand-drawn paths,” IEEE Sensors Journal, vol. p. 7 20, no. 19, pp. 11 660–11 673, 2020
[27] J. A. Frank and V. p. 7 Kapila, “Path bending: Interactive human-robot interfaces with collision-free correction of user-drawn paths,” in Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI 2015, Atlanta, GA, USA, March 29–April 01, 2015
[30] M. Skubic, C. Bailey, and G. p. 7 Chronis, “A sketch interface for mobile robots,” in Proceedings of the IEEE International Conference on Systems, Man & Cybernetics: Washington, D. p. 7C., USA, 5–8 October 2003
[31] Preferred Robotics, Inc., “kachaka-api,” https://github. p. 7com/pf-robotics/kachaka-api/blob/main/docs/ROS2. p. 7md
[32] S. Macenski, S. p. 7 Singh, F. Mart´ın, and J. p. 7 Gines, “Regulated pure pursuit ´ for robot path tracking,” Autonomous Robots, vol. p. 7 47, no. 6, pp. 685–694, 2023
[33] D. Rey and M. Neuhauser, “Wilcoxon-signed-rank test,” ¨ International encyclopedia of statistical science, pp. p. 7 1658–1659, 2011
[34] J. Brooke et al. p. 7, “SUS-A quick and dirty usability scale,” Usability evaluation in industry, vol. p. 7 189, no. 194, pp. 4–7, 1996
[35] S. Hart, “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research,” Human mental workload/Elsevier, 1988
[2604.00059v1] arXiv:2604. p. 000059v1 [cs. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved path specification accuracy by enabling users to
draw a Handdrawn Reference Path (HRP) directly on the physical floor using hand gestures.
This reduces mental translation between a 2D map and the 3D physical environment, leading tomore faithful and intuitive path specification directly in the physical environment than a conventional 2D interface.
-
Enhanced path stability by allowing users to
specify how the robot should move
directly in the physical space. This is achieved by converting the user-specified point sequence into a global path within the robot navigation stack via a custom Hand-drawn Reference Path Planner, whichreduces the gap between screen-based editing and real-world spatial intention.
-
Reduced perceived workload for human operators by providing an interface that allows direct physical specification. The system shows results indicating that MRReP achieves
lower NASA-TLX scores than the 2D system, indicating better usability and lower workload,
despite challenges likephysical discomfort such as arm fatigue during gesture-based interaction.
-
Increased path following accuracy by allowing the robot to navigate along a path defined by a
continuous HRP on the physical floor in an MR environment.
This direct specification is shown to allow participants tocover the target trajectory more completely, reducing omissions and unintended shortcuts,
resulting in higher Recall metrics. -
Provided robust state restoration capability for navigation sessions through persistent data storage. The system stores the coordinate sequence of HRPs in a JSON file managed by a Unity script, enabling
the previous state to be restored after reboot
by loading this data at startup.
Sources
- ARviz -- An Augmented Reality-enabled Visualization Platform for ROS Applications
- MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model
- Towards an immersive user interface for waypoint navigation of a mobile robot
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation