Human-Machine Interfaces for Subsea Telerobotics: From Soda-straw to Natural Language Interactions
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Human-Machine Interfaces for Subsea Telerobotics".
Dev: Here's the gist of the paper. This review systematically charts the evolution of human-machine interfaces (HMIs) in subsea telerobotics, tracing a progression from early first-person "soda-straw" consoles characterized by narrow field-of-view camera feeds to modern interfaces incorporating gesture recognition, virtual reality (VR), and natural language processing (NLP).
Rosa: The analysis is structured around three core perspectives: operator experience—encompassing immersive feedback fidelity, cognitive workload assessment, and ergonomic design; robotic autonomy—focusing on contextual understanding and task execution capabilities; and the quality of bidirectional communication between the human operator and the machine.
Dev: A key focus is placed on persistent limitations in current systems, specifically regarding immersive feedback fidelity, intuitive control mechanisms, and the lack of cross-platform standardization.
Rosa: The paper introduces a structured analysis framework based on "technology clusters," which delineate how human-machine interaction occurs across different system architectures.
Title and authors: Rosa: Okay, so looking at the paper’s suggestions for improvement, they focus on making autonomy more adaptive based on the operator's skill level.
Dev: They suggest that adaptive autonomy systems can analyze the operator’s skill level and adjust autonomously by figuring out which parts of a mission can be handled independently and when the human operator needs to step in.
Taro: That means if the water is calm, it can engage autopilot mode, but if things get turbulent or confined spaces appear, it prompts the operator to take control.
Rosa: Right. And then they look at learning from demonstration systems, which could learn knowledge bases from demonstrations provided by an operator to perform similar tasks autonomously in the future.
Dev: They suggest enhancing that by accounting for how varied the quality of those demonstrations is and combining Bayesian learning with probabilistic path planning to fix any gaps between what was demonstrated and what the robot can actually operate in.
Taro: That sounds like they’re trying to make sure a learned task doesn't fail when the real world throws a little curve at it.
Rosa: And then there's the idea of natural language processing interfaces using large language models, which could enable robots to reason logically and suggest alternative strategies based on a long conversation.
Dev: It moves beyond just executing simple commands toward having real-time natural conversations where the robot has a cognitive understanding of the goal and actively participates in making decisions.
Taro: That’s moving from simple instruction following to actual collaborative reasoning, which is pretty significant for complex underwater missions.
Rosa: Another suggestion is latency mitigation for long distance operations, which would use predictive control algorithms and intent aware robotic systems to deal with communication loss.
Dev: These systems could anticipate future states or switch over to autonomous mode when communication breakdown is about to happen, which helps cut down on the impact of those delays on the operator experience.
Taro: That addresses the immediate danger of lag in a high-stakes environment.
Rosa: Then there’s modeling haptic sensations from remote underwater settings using physics driven learning and generative models to get higher fidelity haptic feedback.
Dev: That tackles the problem of translating that high fidelity hydrodynamic data into feedback that actually engages the operator's sense of touch while still being ergonomic for them to use.
The paper's summary: Rosa: So, wrapping up this discussion on Human-Machine Interfaces for Subsea Telerobotics: From Soda-straw to Natural Language Interactions, the paper shows a clear path from very basic controls to much more intelligent interactions.
Dev: It summarizes that the key is figuring out which command and feedback modalities—language, gesture, haptics, or camera vision—should be prioritized based on how complex the task is and what the environmental constraints are.
Taro: They also show that digital replicas like simulators and twins can really help improve operator skills by allowing mission rehearsals and replays, which helps analyze cognitive load.
Rosa: Overall, they emphasize that understanding shared autonomy is essential, especially adapting it to the operator’s skill level and how much mental work they’re already doing.
Dev: The paper points out that a major limitation of current systems is still the fidelity of immersive feedback and getting those control mechanisms to be truly intuitive across different platforms.
Taro: And they also flag that there's still a lack of cross platform standardization, which makes deploying these advanced interfaces much harder in practice right now.
Rosa: So, the future research directions they suggest—like learning from demonstration and cognitively adaptive systems—are really focused on making these interfaces more intelligent and less brittle when things go wrong.
Dev: It’s clear that moving toward natural language interaction and predictive control for latency is where the real push is going in this area.
Taro: I think the biggest implication is that we need to build systems that can gracefully handle the unexpected, not just execute a pre-planned path perfectly.
The paper's improvements: Rosa: So, we've seen how they evolved the interfaces from those old soda straws to these new stuff, and now they're talking about some specific ways to make them better in the real world.
Dev: Yeah, this paper isn't just showing what exists; it’s pointing out exactly where things still fall apart. They focus on three main areas: how the operator feels using the interface, how smart the robot actually is when doing its job, and how well that two-way conversation works.
Taro: And they flag some big issues right away—specifically with immersive feedback fidelity, making controls feel intuitive, and just not having standard ways to connect different systems.
Rosa: They propose three major improvements to tackle those problems. First is this idea for adaptive autonomy systems where the robot learns the operator’s skill level and adjusts its own help accordingly.
Dev: That means if the water gets rough or it gets too tight, the robot can switch into a more autonomous mode, but only when it knows you can handle it. It's about figuring out what parts of a mission are safe for AI to take over independently.
Taro: I like that idea because in real conditions, things change fast. If the system gets smart enough to know when to let go and when to prompt you, that makes the whole operation much more manageable for a human.
Rosa: Then they talk about learning from demonstration systems where the robot learns tasks by watching a human do them, but they also stress fixing problems with those demonstrations being inconsistent.
Dev: They suggest using Bayesian learning combined with probabilistic path planning to bridge the gap between what you showed the robot and what it can actually do when it's in the water. That handles those discrepancies in operational space.
Taro: So it’s not just about copying a move; it’s about understanding why that move worked and how to adapt if things are slightly different.
Rosa: And then there’s this big idea about natural language processing using large language models, which they see as a way for the robot to actually reason with you.
Dev: They're not just talking about simple commands anymore; they want the robot to have a real conversation, understand your goal deeply, and actively suggest alternative strategies based on that long back-and-forth.
Taro: That moves it from simple execution to genuine collaboration where the machine is thinking alongside you about how to solve the problem.
Rosa: Another crucial area is latency mitigation for those long distance operations where communication can drop out.
Dev: They suggest using predictive control algorithms and systems that are aware of intent so the robot can guess what's going to happen or switch to autonomous mode before there’s a total communication breakdown. It manages that delay impact on your experience.
Taro: That’s about building in a safety net for when the connection gets shaky, so you don't lose control completely just because of a few milliseconds of lag.
Rosa: Finally, they look at modeling haptic sensations using physics-driven learning and generative models to make remote feedback feel more real.
Dev: That means taking all that high-fidelity data about the water and translating it into a tactile sensation for you that actually feels like you’re there, while still being comfortable for you to use.
Taro: If they can nail those three things—adaptive autonomy, better learning from demonstrations, and low latency—then we move much closer to interfaces that feel like extensions of our own bodies underwater.
Conclusion: Rosa: So we've spent time looking at how they're trying to upgrade those old soda straws in subsea telerobotics, and now they're showing us what these improvements actually mean for the future of remote operation.
Dev: Basically, this paper is about moving beyond simple visual feeds toward interfaces that can adapt to the operator’s skill level and handle communication failures better.
Taro: It’s about making the robot smarter in a way that anticipates problems instead of just reacting to them when things go wrong out there.
Rosa: I think the big implication is that we're finally getting closer to systems where the human isn't just a remote pilot, but a true co-pilot who can manage complex, unpredictable environments.
Dev: It’s about building safety cases for these more intelligent systems, which means designing controls that can handle those unexpected failures gracefully rather than crashing when things get messy.
Taro: I just hope they keep pushing on the real-world failure modes because that's where most of the actual learning happens when you’re dealing with something like deep-sea robotics.
Rosa: The paper, "Human-Machine Interfaces for Subsea Telerobotics: From Soda-straw to Natural Language Interactions," really sets a good baseline for what these next steps should look like.
Dev: It shows that we need to focus on those three things—adaptive autonomy, robust learning from demonstrations, and better communication resilience—to make it viable outside of just a controlled lab.
Taro: I agree with the latency part; if you can predict a drop in connection before it happens, the whole experience becomes much less stressful for the person on the other end.
Rosa: Yeah, that’s right. It shows that the direction for this research is moving toward true cognitive understanding from both sides of that connection.
Dev: We'll be looking at how these new interaction methods actually scale up when you move from a small test site to a deep-sea deployment with massive data streams.
Taro: Next time, we’re going to look at some papers focusing more on those autonomous planning systems and how they handle the sheer complexity of multi-modal inputs.
Adnan Abdullah, David Blow, Ruo Chen, Thanakon Uthai, Eric Jing Du, Md Jahidul Islam
RoboPI Laboratory and ICIC Laboratory, University of Florida
cs.RO
Submitted: 2024-12-02
Updated: 2026-11-08
Comments: 28 pages. First pre-print released in 2024; revised as technical report in November 2026
Code: https://github.com/DroidPlanner/Tower
Project page: https://robopi.ece.ufl.edu/subsea_tr.html
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: This review systematically charts the evolution of human-machine interfaces (HMIs) in subsea telerobotics…
Key concepts
- Cognitive Workload Assessment
- This measures how mentally taxed an operator is while controlling a robot underwater. It focuses on evaluating the difficulty of understanding feedback, making decisions quickly, and managing the overall complexity of the task. High workload indicates poor interface design that needs simplification.
- Bidirectional Communication Quality
- This examines how effectively information flows both from the human operator to the robot (commands) and from the robot back to the human (feedback). Good quality means operators receive timely, relevant data, and robots accurately convey their status and intent.
- Technology Clusters
- This is a framework used to categorize different ways humans interact with machines in subsea robotics. It groups interfaces based on system architecture, helping researchers compare current solutions against guidelines for sensory input, platform compatibility, and user comfort.
Terminology
Summary
This review systematically charts the evolution of human-machine interfaces (HMIs) in subsea telerobotics, tracing a progression from early first-person We systematically analyze the state-of-the-art literature through three interrelated perspectives:
operator experience (including immersive feedback, cognitive workload, and ergonomic design), robotic
autonomy (contextual understanding and task execution), and the quality of bidirectional communication
between human and machine. Emphasis is placed on interface features to highlight persistent limitations in
current systems, notably in immersive feedback fidelity, intuitive control mechanisms, and the lack of crossplatform standardization. We systematically analyze the state-of-the-art literature through three interrelated perspectives:
operator experience (including immersive feedback, cognitive workload, and ergonomic design), robotic
autonomy (contextual understanding and task execution), and the quality of bidirectional communication
between human and machine. Emphasis is placed on interface features to highlight persistent limitations in
current systems, notably in immersive feedback fidelity, intuitive control mechanisms, and the lack of crossplatform standardization.soda-straw
consoles characterized by narrow field-of-view camera feeds to modern interfaces incorporating gesture recognition, virtual reality (VR), and natural language processing (NLP). The analysis is structured around three core perspectives: operator experience—encompassing immersive feedback fidelity, cognitive workload assessment, and ergonomic design; robotic autonomy—focusing on contextual understanding and task execution capabilities; and the quality of bidirectional communication between the human operator and the machine. p. 0Where the paper says it
Where the paper says it
The paper introduces a structured analysis framework based on To facilitate
comparative evaluation, we introduce the concept of “technology clusters" that delineate how human-machine
interaction happens across different system architectures. First, we categorize and evaluate the state-of-theart (SOTA) subsea telerobotics interfaces and simulators based on a set of guidelines such as their choice
of sensory modalities, platform compatibility, user comfort, and optimal use cases (Section 3-4). First, we categorize and evaluate the state-of-theart (SOTA) subsea telerobotics interfaces and simulators based on a set of guidelines such as their choice
of sensory modalities, platform compatibility, user comfort, and optimal use cases (Section 3-4). Then, we
present a detailed review of human-robot teaming in subsea telerobotics, highlighting various aspects of shared
autonomy, bidirectional interaction, and intelligence (Section 5-6). Then, we
present a detailed review of human-robot teaming in subsea telerobotics, highlighting various aspects of shared
autonomy, bidirectional interaction, and intelligence (Section 5-6). Finally, we analyze the SOTA approaches
and trends to address the challenges in subsea HMIs, highlighting open problems and use cases for future
research (Section 7).technology clusters,
which delineate how human-machine interaction occurs across different system architectures. It first categorizes and evaluates the state-of-the-art (SOTA) subsea telerobotics interfaces and simulators against guidelines concerning sensory modalities, platform compatibility, user comfort, and optimal use cases. p. 2Where the paper says it
Where the paper says it
Where the paper says it
The review spans two primary categories: commercially available interfaces with public documentation and peer-reviewed academic interfaces. p. 2 For
industry leaders, the comparative insights into current systems and emerging trends will inform the design
of futuristic HMI solutions. Review Scope And Criteria
This review spans two primary HMI categories: commercially available interfaces with publicly accessible
documentation and peer-reviewed interfaces within the academic literature. 2 organizes key milestones in the evolution of HMI technologies into three distinct eras:
(i) foundational space and ocean exploration during the 20th century that laid the base for HMI development,
(ii) the emergence of marine robotic companies, novel software and hardware solutions in the early 21st century,
and (iii) the recent shift toward immersive VR-integrated and language-driven interfaces. The motivation behind
classifying into these eras is not only due to their chronological relevance but also to highlight how technical,
commercial, and human factors have co-evolved in shaping the capabilities and expectations of subsea HMI
systems. 4 Multimodalities Interface
Besides vision, human sensorimotor control relies on multimodal sensory feedback such as auditory and
somatosensory (tactile and proprioceptive) cues to interpret the consequences of initiated actions [65]. Haptic
feedback is the most recognized one in enhancing XR systems; researchers have explored innovative methods for
simulating haptotactile stimulation (e. Emphasis is placed on interface features to highlight persistent limitations in
current systems, notably in immersive feedback fidelity, intuitive control mechanisms, and the lack of crossplatform standardization. Additionally, we assess the role of simulators and digital twins as scalable tools
for operator training and system prototyping. Then, we
present a detailed review of human-robot teaming in subsea telerobotics, highlighting various aspects of shared
autonomy, bidirectional interaction, and intelligence (Section 5-6). Finally, we analyze the SOTA approaches
and trends to address the challenges in subsea HMIs, highlighting open problems and use cases for future
research (Section 7). It
aims to provide foundational knowledge to students, researchers, scientists, and engineers engaged in HMI
development. The classification and open research questions will offer a roadmap to young researchers for
systematic investigation into interface design, autonomy integration, and human-robot collaboration.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive Autonomy Systems can analyze an operator’s skill level to adjust autonomy intelligently by determining
which parts of a mission can be handled independently and when operator intervention is necessary.
This allows the robot to engage autopilot mode in calm waters but prompt the operator for control during turbulent conditions or confined spaces. -
Learning from Demonstration (LfD) systems can learn knowledge bases from operator-provided demonstrations to autonomously perform analogous tasks in the future. This is enhanced by accounting for variability in demonstration quality and combining Bayesian learning with probabilistic path planning to resolve discrepancies between the demonstration space and the robot’s operational space.
-
Natural Language Processing (NLP) interfaces, specifically utilizing Large Language Models (LLMs), can enable robots to reason logically and propose alternative strategies based on a long conversation. This moves beyond simple command execution toward
real-time natural conversations, where the robot would possess a cognitive understanding of the goal and actively engage in decision-making.
-
Latency Mitigation for Long Distance Operation can utilize predictive control algorithms and intent-aware robotic systems to account for communication loss. These systems can anticipate future states or switch to autonomous mode when communication breakdown is imminent, mitigating the impact of delays on operator experience.
-
Hydrodynamic Sensing & Haptic Feedback Modeling can employ physics-driven learning and generative models to model haptic sensations from remote underwater settings with higher fidelity. This addresses the challenge of translating high-fidelity hydrodynamic data into intuitive feedback that engages operators' sensory perception while maintaining ergonomics.
Abstract
This review explores the evolution of human-machine interfaces (HMIs) in subsea telerobotics, charting the progression from traditional first-person "soda-straw" consoles-- characterized by narrow field-of-view camera feeds-- to contemporary interfaces leveraging gesture recognition, virtual reality, and natural language processing. The review makes three key contributions. First, we introduce a structured framework that analyzes subsea HMIs through three interrelated aspects: operator experience, robotic autonomy, and bidirectional human-machine communication. Second, using these aspects, we compare visual, haptic, extended-reality, gesture, and natural-language interfaces; examine the progression from direct teleoperation to shared autonomy; and assess the distinct roles of simulators and digital twins in training, prototyping, and mission support. Third, we synthesize the reported evidence to distinguish established industrial tools, selectively deployed technologies, and research-stage capabilities, while identifying limitations in interoperability, communication reliability, immersive feedback, and field validation. From this synthesis, we outline standardization needs and research directions for hydrodynamic feedback, latency-aware shared autonomy, grounded language interaction, and real-to-simulation transfer. The report concludes by identifying open research questions and proposing a roadmap toward more intelligent and intuitive subsea HMI systems.
Sources
- OceanPlan: Hierarchical Planning and Replanning for Natural Language AUV Piloting in Large-scale Unexplored Ocean Environments
- Haptic-enabled Mixed Reality System for Mixed-initiative Remote Robot Control
- OceanChat: Piloting Autonomous Underwater Vehicles in Natural Language
- OceanGPT: A Large Language Model for Ocean Science Tasks
- Word2Wave: Language Driven Mission Programming for Efficient Subsea Deployments of Marine Robots
- Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination
- Digital Twins Below the Surface: Enhancing Underwater Teleoperation
- BlueME: Robust Underwater Robot-to-Robot Communication Using Compact Magnetoelectric Antennas
- Underwater Human-Robot and Human-Swarm Interaction: A Review and Perspective
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- AquaFuse: Waterbody Fusion for Physics Guided View Synthesis of Underwater Scenes
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation