Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators

arXiv:2608.06219 · cs.RO, cs.HC · Submitted 2026-08-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators".

Dev: Here's the gist of the paper. Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers Participants completed teleoperation tasks more efficiently and accurately with the touchscreen interface, achieving a 53.5% reduction in completion time (median: 2.50 vs.

Rosa: 5.38 min), higher in-area coverage on the sinusoidal path (90.7% vs.

Dev: 84.1%), and lower overshoot on both path geometries compared with the joystick Cognitive load, quantified via NASA-TLX (0–100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17.3%) and was lowest under the autonomous one-click mode (31; -21 points vs.

Rosa: joystick, -40.4%; -12 vs.

Paper summary: Rosa: So we're looking at this paper called Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators. Basically it tackles the problem that right now operating robotic arms in tricky places requires too much skill and attention with the old joystick controllers.

Dev: Right, so what's the main idea here? What is this paper trying to show us about making these interfaces better?

Rosa: It designs a new touchscreen interface that maps your finger movements directly to how the robot moves. The claim is that this gives you much finer control over speed and lets you visualize things on screen, making surface interaction more natural and precise than what's available now Dev That sounds like it could really simplify tasks like swabbing or sampling in places where you can’t be there physically.

Taro: I wonder if that direct mapping is actually practical when you need to deal with real-world constraints, like latency and bandwidth? Does this interface actually work reliably outside of a perfectly controlled lab setting?

Dev: That's a good question, Taro. The authors designed the control architecture using a hybrid impedance-admittance scheme to handle those exact issues. They use Cartesian impedance for tracking the tangential motion—that's the movement along the surface—and an admittance loop for the normal direction, which is how they regulate that contact force Rosa That means while you touch to move along a path, the system automatically manages how hard it pushes against whatever surface it's touching.

Taro: So if you are swabbing and you accidentally push too hard, this system should compensate by regulating the force rather than just letting your input dictate a sudden impact? What happens when the world misbehaves and that force regulation needs to happen really fast?

Dev: The paper mentions they put a distal six degrees of freedom force torque sensor right at the robot's wrist, which closes that force loop locally. This way, the crucial normal-force regulation is decoupled from how slow your network connection is Rosa It also has this surface-lock behavior where if the measured normal force exceeds a certain threshold, it switches from free mapping to sticking the tool onto the surface Taro That sounds like a safety feature for when things get messy.

Rosa: And they quantified how much better it is using this setup. They compared three conditions: a conventional Xbox joystick, this new touchscreen interface, and an autonomous single-click mode Dev The results show that using the touchscreen cut down the task completion time by fifty-three point five percent, going from a median of five point three eight minutes with the joystick to two point five zero minutes with the new interface Taro That's a big jump in efficiency for repetitive tasks.

Dev: And it wasn't just speed, though. They also looked at how much mental effort the operator had to put in, which they measured using something called NASA-TLX scores Rosa The cognitive load dropped from an average score of fifty-two with the joystick down to forty-three with the touchscreen interface, that’s a nine point drop Taro

Paper summary: Taro: A nine point drop in mental load is significant for sustained remote operation. What about when the system is just running on its own, that single-click mode? Does it feel less stressful than even the new touchscreen setup?

Dev: It does. The autonomous one-click mode had the lowest cognitive load score of thirty-one Rosa That was a twenty point drop compared to the joystick and a twenty-seven point drop compared to this touchscreen interface Taro So, for someone just watching or supervising, that single-click approach seems much less taxing on their brain.

Rosa: It really shows how intuitive these mappings can be when they target surface contact specifically, unlike many previous touch interfaces that focused more on just moving the end effector in free space Dev The study suggests this particular design targets reducing operator burden by letting you emulate direct physical interaction with the environment Taro It’s about making the control feel more like you're actually touching things.

Dev: That's the core of it. They coupled that touch mapping with a control scheme that enforces a desired normal-force profile while keeping the tangential tracking compliant Rosa The whole goal was to reduce manual force modulation during tasks like swabbing because the robot handles that part autonomously Taro But they did flag a limitation: this interface primarily targets contact-rich surface manipulation, and it might not be as robust for purely free-space positioning or things where you need incredibly stable contact under really poor communication conditions Dev

Rosa: So, to summarize this paper Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators, the authors are showing that by mapping finger movements directly to robot motion and using a hybrid control scheme with force regulation, you can achieve more precise surface interaction with significantly reduced operator workload Taro The implication is that for industries like nuclear or complex surface inspection where precision and sustained attention matter, this could make remote operation much more viable than current setups Dev

Taro: It shifts the focus from just getting a point to making the entire remote interaction process smoother for the human operator Rosa It’s less about fighting the machine and more about smoothly guiding it through a physical task. That seems like a big shift in how we design these systems, right?

Dev: Exactly. The paper highlights that this system reduces completion time by fifty-three point five percent on tasks like the sinusoidal path, which is hard to get with joysticks Taro It also shows that when you factor in the cognitive load reduction, especially with the autonomous mode, it becomes a much more practical tool for real-time application Rosa

Rosa: So yeah. This paper suggests that intuitive mapping combined with intelligent force control is a way forward for making robotic manipulators work better on surfaces where things are tricky Dev We'll keep an eye on how they tackle those real operational constraints in future work, because that’s where the next challenge will be Taro

Conclusion: Rosa: So we've been looking at this paper, "Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators." Basically, they’re showing how mapping finger movements directly to robot motions can make operating these arms much easier on the human side.

Dev: Right. They built a touchscreen interface that maps continuous finger movements straight to the manipulator's motion, and it gives you better velocity control and integrates visualization. It aims for more natural surface interaction.

Taro: So, what's the big picture here? Is this just a fancy way to move the arm around in a lab setting, or does it actually matter when we take these systems out into rough environments?

Rosa: Well, it’s about reducing operator workload. The study shows that participants using this touchscreen interface were much faster at tasks. They saw a fifty-three point five percent reduction in completion time compared to the old joystick controllers.

Dev: That speed increase is significant, but the paper also looked at how much mental effort it took to do the job, quantified by NASA TLX scores. The load dropped by nine points when switching from the joystick to this touchscreen setup.

Taro: I'm interested in what happens when things go wrong out there. The authors mentioned they designed a hybrid impedance-admittance control architecture to handle latency and force regulation simultaneously. How does that actually behave when the connection between you and the robot is shaky?

Rosa: They addressed that by putting a force torque sensor right at the robot's wrist so they could regulate contact forces locally, even if the network link was slow. They also have this surface-lock behavior where if it touches too hard, it switches to sticking to the surface instead of just pushing harder.

Dev: So they decoupled the normal force regulation from network latency by putting that sensor close to the action. That’s a clever way to build a safety case for real-time control, even in noisy environments.

Taro: What does this mean for someone who only listens? It means we are moving away from needing perfect, continuous joystick input and toward something more tactile and automated during surface tasks.

Rosa: It shifts the focus from just controlling every single joint manually to guiding the interaction with the surface itself through touch. This suggests that for complex, contact-rich manipulation, intuitive interfaces might be a key way to make remote work practical instead of just theoretical.

Juan Jose Garc ía Cardenas, Alperen Kenan, Hamidreza Raei, Paul Bremner, Manuel Giuliani, Arash Ajoudani, Ádiana Tapus

ENSTA Institut Polytechnique de Paris · University of the West of England

cs.RO, cs.HC

Submitted: 2026-08-06

Updated: 2026-08-06

Comments: 9 pages, 7 figures, accepted for presentation at the IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 2026), Kitakyushu, Japan, 24-28 August 2026

DOI: 10.1109/RO-MAN72286.2026.11714783

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments This study designs and evaluates a novel touchscreen…

Key concepts

Touchscreen Teleoperation Interface
A novel control system where an operator interacts with a touchscreen by moving their finger. This finger movement is directly translated into the robot's motion, aiming to make remote operation feel more intuitive and direct, similar to touching the object itself.
Hybrid Impedance-Admittance Architecture
A control design that combines two methods: impedance for tracking desired paths (like tangential movement) and admittance for regulating contact forces (like normal force). This hybrid approach ensures precise motion while maintaining stable, repeatable contact during surface tasks.
Normal-Force Regulation via Admittance Loop
The system uses an admittance loop to manage the force perpendicular to the surface. Instead of manually adjusting pressure, this loop converts errors in measured contact force into specific motion commands for the robot, ensuring consistent and safe interaction with surfaces.
Cognitive Load (NASA-TLX)
A measure used to quantify how mentally demanding a task is for the operator. The study found that switching from a joystick to the touchscreen reduced this mental effort significantly, making the operation less taxing for the user.

Terminology

Summary

Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers Participants completed teleoperation tasks more efficiently and accurately with the touchscreen interface, achieving a 53. p. 0

Where the paper says it

In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator attention, which conventional joystick interfaces struggle to support effectively. This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers.

5% reduction in completion time (median: 2. p. 0
Where the paper says it

Participants completed teleoperation tasks more efficiently and accurately with the touchscreen interface, achieving a 53. 5% reduction in completion time (median: 2.

50 vs. 5.38 min), higher in-area coverage on the sinusoidal path (90. p. 0
Where the paper says it

5% reduction in completion time (median: 2. 38 min), higher in-area coverage on the sinusoidal path (90.

7% vs. 84.1%), and lower overshoot on both path geometries compared with the joystick Cognitive load, quantified via NASA-TLX (0–100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17. p. 0
Where the paper says it

1%), and lower overshoot on both path geometries compared with the joystick. Cognitive load, quantified via NASA-TLX (0–100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17.

3%) and was lowest under the autonomous one-click mode (31; -21 points vs. p. 0
Where the paper says it

Cognitive load, quantified via NASA-TLX (0–100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17. 3%) and was lowest under the autonomous one-click mode (31; -21 points vs.

joystick, -40. p. 0
Where the paper says it

Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators Juan Jose Garc ´ ´ıa Cardenas ´ 1 ⋆ , Alperen Kenan2 ⋆ , Hamidreza Raei3 ⋆ , Paul Bremner2 , Manuel Giuliani4 , Arash Ajoudani3 and Adriana Tapus1 Abstract— Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments. In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator attention, which conventional joystick interfaces struggle to support effectively.

4%; -12 vs. touchscreen, -27. p. 0
Where the paper says it

Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators Juan Jose Garc ´ ´ıa Cardenas ´ 1 ⋆ , Alperen Kenan2 ⋆ , Hamidreza Raei3 ⋆ , Paul Bremner2 , Manuel Giuliani4 , Arash Ajoudani3 and Adriana Tapus1 Abstract— Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments. In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator attention, which conventional joystick interfaces struggle to support effectively.

9%)

Interface Design and Mapping

The novel interface is designed to target contact-rich surface manipulation by coupling collocated touch mapping with a control scheme that enforces a desired normal-force profile while maintaining compliant tangential tracking along the surface The interface aims to (i) reduce operator workload through intuitive control mappings that emulate direct interaction with the environment; (ii) improve task performance and consistency by enabling continuous tangential path execution while autonomously tracking a target normal-force profile during surface manipulation; and (iii) enhance operator safety by enabling remote operation in hazardous environments Figure 1 illustrates the working principle of the touchscreen teleoperation interface, where the planar position of the operator’s finger on the screen is mapped to the robot end-effector coordinates to interact with the surface, while a camera stream of the surface is displayed on the screen to enhance intuitiveness Critically, while the operator specifies tangential motion through touch, the robot autonomously regulates the normal contact force, reducing manual force modulation during swabbing tasks

Control Architecture and Implementation

The control design adopts a hybrid impedance-admittance architecture to address three constraints: (i) non-negligible end-to-end latency in the teleoperation link, (ii) strict regulation of the normal contact force to guarantee repeatable, verifiable swabbing, and (iii) mapping 2D touchscreen input to the motion of a 6-DoF end effector over planar, inclined, or curved surfaces The tangential directions (x–y) are governed by Cartesian impedance for compliant position tracking, while the normal direction (z) is regulated by an admittance loop that converts force error into a motion reference in the task/contact frame A distal 6-DoF force/torque sensor closes the z-axis force loop locally at the robot (PC4), so stability-critical normal-force regulation is decoupled from network latency Autonomy is integrated via a surface-lock behaviour: when the measured normal force in the end-effector frame exceeds a small threshold Fth, the controller switches from free 3-D mapping to contact mode that sticks the tool to the surface The total joint torque is decomposed as τ = τ imp xy + τ adm z + τnull + g(q), where τ imp xy enforces planar tracking, and τ adm z enforces normal force regulation

Performance Metrics and Cognitive Load

The comparative user study involved 20 participants executing tasks under three conditions: (C1) using a conventional Xbox joystick, (C2) using the custom touchscreen-based interface, and (C3) a supervised single-click autonomous playback mode In terms of general performance, the touchscreen condition (C2) had a median of 2.50 min compared to 5. p. 0

Where the paper says it

38 min), higher in-area coverage on the sinusoidal path (90. 1%), and lower overshoot on both path geometries compared with the joystick.

38 min with the joystick condition (C1), representing a 53. p. 5
Where the paper says it

50 min compared to 5. 38 min with the joystick condition (C1), representing a 53.

5% reduction The average coverage on the sinusoidal path improved from 84. p. 6
Where the paper says it

9; Wilcoxon, Z = -1. In the more demanding sinusoidal path, average coverage improved from 84.

1% (SD = 4.8) with C1 to 90.7% (SD = 3.9) with C2, an absolute improvement of 6. p. 6
Where the paper says it

7% (SD = 3. 9) with C2, an absolute improvement of 6.

6% Cognitive load estimation comparison shows a stepwise decrease in (z)-CLI across interfaces: C2 < C1 and C3 < C2, with the omnibus test confirming a robust Interface effect (RM–ANOVA: (F(2, 38) = 27. p. 6
Where the paper says it

Cognitive load estimation comparison contrasts (Figure 6) show a stepwise decrease in (z)-CLI across interfaces: C2 < C1 and C3 < C2. The omnibus test confirmed a robust Interface effect (RM–ANOVA: (F(2, 38) = 27.

1, p < 0.01, η2p = 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved task performance through direct mapping of finger movements to robotic motions, enabling finer velocity control than conventional controllers. This allows for continuous tangential path execution while autonomously tracking a target normal-force profile during surface manipulation.

  2. Reduced cognitive load by implementing a hybrid impedance force controller that decouples the control scheme: "the tangential directions (x–y) are governed by Cartesian impedance for compliant position tracking, while the normal direction (z) is regulated by an admittance loop that converts force error into a motion reference in the task/contact frame."

  3. Enhanced safety and precision through a surface-lock behavior: when the measured normal force in the end-effector frame exceeds a small threshold Fth, the controller switches from free 3-D mapping to contact mode that sticks the tool to the surface.

  4. Improved operator trust and reduced workload by integrating multimodal physiological sensing: The system can use blink rate, facial temperature, and GSR to derive a Cognitive-Load Index (CLI) and a Trust Index (TI), allowing for adaptive shared control where autonomy levels adjust based on real-time operator state.

  5. Optimized control parameter tuning using surface stiffness estimation: The system calculates the local stiffness of the surface using RANSAC regression and aggregates it into a single stiffness value to tune controller parameters, such as Kz p = alpha kest, ensuring tight force regulation around F⋆ z while maintaining well-damped transients.

Abstract

Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments. In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator attention, which conventional joystick interfaces struggle to support effectively. This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers. A comparative user study with 20 participants evaluated task performance and workload using the proposed touchscreen, a conventional joystick, and a single-click autonomous mode. Tasks simulated realistic surface manipulation using a Franka Emika Panda arm, remotely controlled from another country. Kinematic, physiological, and behavioral data were recorded to comprehensively assess task performance, cognitive load, and operator trust across each control condition. Participants completed teleoperation tasks more efficiently and accurately with the touchscreen interface, achieving a 53.5% reduction in completion time (median: 2.50 vs. 5.38 min), higher in-area coverage on the sinusoidal path (90.7% vs. 84.1%), and lower overshoot on both path geometries compared with the joystick. Cognitive load, quantified via NASA-TLX (0-100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17.3%) and was lowest under the autonomous one-click mode (31; -21 points vs. joystick, -40.4%; -12 vs. touchscreen, -27.9%). This research presents an easy-to-implement touchscreen interface that improves performance in teleoperated surface tasks while reducing cognitive load.

Sources

Related papers