Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT

arXiv:2609.07857 · cs.RO, cs.CV · Submitted 2026-09-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT".

Dev: Here's the gist of the paper. Robotic ophthalmic surgery introduces high precision but introduces a ”sensory gap” by decoupling the surgeon from their instrument, resulting in a loss of tactile feedback <ref:2609.07857#pg2> This paper presents a novel haptic feedback system for subretinal injection tasks leveraging Scene Graphs (SG) <ref:2609.07857#pg2> The system bridges the sensory gap by analyzing a physically simulated intraoperative Optical Coherence Tomography (iOCT) feed to construct a real-time surgical SG <ref:2609.07857#pg2> Results demonstrate that the SG-driven haptic feedback improved surgical precision, reducing needle alignment error by 14% (p = 0.044) and improving System Usability Scale (SUS) scores by 8% (p = 0.015) <ref:2609.07857#pg2> System Overview The core of the proposed framework combines a compact imaging setup, a computer vision pipeline, and a semantic rule-based engine to provide context-aware haptic feedback to a robotic input device <ref:2609.07857#pg2> To prove the concept of this work, the proposed framework combines a compact imaging setup, a computer vision pipeline, and a semantic rule-based engine to provide context-aware haptic feedback to a robotic input device <ref:2609.07857#pg2> The system physically simulates iOCT-guided robotic surgery, with an anthropomorphic workspace <ref:2609.07857#pg2> While this optical setup serves as a low-cost proxy for iOCT geometry (fig.

Rosa: 5, left), the Scene Graph framework is designed to be agnostic to the input source, allowing future migration to clinical OCT feeds <ref:2609.07857#pg2> Perception and Scene Graph Construction The perception pipeline processes the video feed to track instruments relative to anatomical features inside the eye <ref:2609.07857#pg2> The system captures the camera feed and sequentially triggers specific detectors for the eye globe (1), retina (4), and needle (6) to generate the geometric primitives required for Scene Graph construction <ref:2609.07857#pg2> The perception pipeline processes the video feed to track instruments relative to anatomical features inside the eye <ref:2609.07857#pg2> To create similarity to OCT, the frame is slightly edited again after it has passed through the pipeline <ref:2609.07857#pg2> The perception pipeline processes the video feed to track instruments relative to anatomical features inside the eye <ref:2609.07857#pg2> Semantic Abstraction and Inference To represent the surgical scene and bridge the gap between low-level camera data and the high-complexity surgical context, we utilized the concept of an SG as our semantic middleware for future decision-making <ref:2609.07857#pg2> For every frame, the image is analyzed, and a corresponding SG is dynamically computed (Figure 5, right) <ref:2609.07857#pg2> The SG serves as the computational foundation of the engine: at each time step, the engine queries the SG topology Gt to infer a discrete surgical state St <ref:2609.07857#pg2> To bridge the gap between continuous sensor data and discrete logic, we utilize the geometric primitives visualized in Figure 4 (Right) <ref:2609.07857#pg2> Haptic Feedback Strategy The system maps these surgical states to a specific haptic grammar <ref:2609.07857#pg2> The goal is to funnel the surgeon from the active feedback states into the silent states <ref:2609.07857#pg2> 1) Nominal Navigation (Silent): When following the Aligned Path, no haptic feedback is applied (Ht = Off) <ref:2609.07857#pg2> 2) Active Guidance (Continuous): States in the Misaligned Path trigger a continuous low-frequency vibration (Ht = Viblow) <ref:2609.07857#pg2> 3) Critical Events (Red/Green): High-priority events trigger distinct patterns <ref:2609.07857#pg2> Study Findings Leveraging haptic feedback did not show any statistically significant change in task completion time <ref:2609.07857#pg2> Safety was quantified by two primary metrics: the frequency of unintended retinal contacts (touching retina without target contact) and the average Alignment Angle θ maintained throughout the study <ref:2609.07857#pg2> The mean alignment error reduced from 7.9 ◦ ± 2.6 ◦ to 6.8 ◦ ± 2.3 ◦ (p = 0.044) in the feedback condition <ref:2609.07857#pg2> Participants successfully evaded the negative Retinal Contact state <ref:2609.07857#pg2> The Visual + Haptic condition successfully promoted an “Align-then-Approach” strategy <ref:2609.07857#pg2> The system received a mean SUS score of 81.9, which corresponds to a rating of “Good”

15: <ref:2609.07857#pg2> We propose that the community should explore SGs not merely for analysis, but also as a semantic middleware for active robotic control <ref:2609.07857#pg2> A significant observation was the user response to the ’Misaligned’ vibration <ref:2609.07857#pg2> This suggests that the system successfully employed negative reinforcement cues to promote safer surgical behavior <ref:2609.07857#pg2> In future work, we will explore continuous guidance and address study limitations, specifically the sample size (N = 16), absence of tissue deformation, and imaging noise <ref:2609.07857#pg2> The ability to implicitly guide movement highlights the system’s potential not only as an intraoperative safety net but also as a tool for skill acquisition in surgical training <ref:2609.07857#pg2> REFERENCES

1: D.

Dev: Tripepi, A.

Rosa: Jalil, N.

Title and authors: Rosa: Let's talk about the title of this work, "Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT". It tells us immediately that they are combining three main things: Scene Graphs, haptic feedback, and simulated iOCT. >

Dev: That sounds like a lot of moving parts. So, what’s the core idea behind using a scene graph to manage the feedback for this robotic surgery? Is it just organizing data better? >

Taro: It seems more than organization. They’re using it as a semantic abstraction layer for the entire surgical scene. This means they are building a model of what is happening—like where the needle is relative to the retina—and then feeding that structure into an inference engine >

Rosa: That inference engine, which they describe as deterministic and rule-based, takes that graph topology and computes a specific surgical state at every time step >

Dev: So instead of having a complex AI try to guess what the surgeon is doing based on raw video pixels, the system uses this pre-defined graph structure to infer the state >

Taro: That’s interesting because it makes the logic more transparent. You can see exactly how the system moves from raw visual data to that discrete surgical state >

Rosa: And they are bridging that gap between continuous sensor data and discrete logic using geometric primitives visualized in their setup, which they call iOCT-fication >

Dev: I like that part about the iOCT simulation. They use a compact imaging setup as a low-cost proxy for real intraoperative iOCT geometry, but the Scene Graph framework is designed to be agnostic, meaning it could eventually take real clinical OCT feeds >

Taro: So they’ve designed it to be flexible enough that if you switch from their simulated setup to actual clinical OCT data later, the core logic based on the scene graph remains intact >

Rosa: Right. The idea is that the semantic structure is the stable part, and the input source—whether it’s a simulation or a real clinical feed—is just plugged into that layer >

Dev: So, if they successfully decouple those layers, it means we don't have to rebuild the entire feedback control architecture every time we switch hardware for different imaging modalities >

The paper's summary: Rosa: To summarize what they did here is that this paper presents a novel haptic feedback system specifically for subretinal injection tasks, addressing the sensory gap inherent in teleoperated robotic surgery. >

Dev: What’s that sensory gap again? I think it’s the loss of tactile feedback when you're operating through a remote console, which leads to unintentional tissue damage because you can't feel the resistance or texture >

Taro: And this system aims to fix that by analyzing a physically simulated iOCT feed to construct a real-time surgical scene graph that informs haptic output on the robotic input device >

Rosa: So it’s essentially taking visual information, structuring it into a scene graph, and using that structure to generate state-dependent haptic feedback based on whether the needle is aligned or misaligned >

Dev: The system maps different spatial positions of the needle within defined zones to specific relations in the graph, which then drives their inference engine to decide which haptic output—silent navigation, continuous vibration, or a critical event pattern—to use >

Taro: This means the system is translating complex visual relationships into simple instructions for a physical device through touch cues >

Rosa: And they demonstrated that this approach improves surgical precision by reducing needle alignment error by fourteen percent and boosting user satisfaction with the System Usability Scale scores by eight percent > <ref:2609.07857#pg1,reducing needle alignment error by 14>

Dev: Those are solid numbers, and they show that using this framework isn't just theoretical; it actually helps improve the actual performance of the task >

Taro: It shows that when you give a surgeon targeted feedback based on where they are in the surgical path, their alignment error goes down from seven point nine degrees to six point eight degrees > <ref:2609.07857#pg3>

The paper's improvements: Rosa: Now let’s look at what they actually suggest as improvements in the system, which is where we move beyond just reporting the results and see how this concept can be applied elsewhere >

Dev: I think one big improvement is that they aren't just relying on force feedback; they are using a layered approach: nominal navigation is silent when following an aligned path >

Taro: So, the system actively funnels the surgeon from those active states into silent states by only applying haptic feedback when there’s a clear need for correction or guidance >

Rosa: That’s the goal, right? They have three levels: nominal navigation is silent when following an aligned path, active guidance triggers a continuous low-frequency vibration if the needle is in the misaligned path, and critical events trigger distinct red or green patterns >

Dev: So that means they are trying to manage cognitive load by letting the surgeon focus on what’s important while filtering out unnecessary noise >

Taro: That idea of using haptic negative reinforcement cues to promote an "align-then-approach" strategy is really interesting because it actively encourages a certain kind of surgical behavior rather than just reacting to errors >

Rosa: It's about the system subtly coaching the surgeon toward safer movement by rewarding them with feedback when they are in the misaligned path >

Dev: The paper also points to how their rule-based inference engine allows for scalability, meaning they can add new entities or relations to the graph schema without having to rewrite the downstream inference engine >

Conclusion: Rosa: So, wrapping up this discussion on "Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT". It sounds like they’ve successfully shown that using a semantic graph structure to bridge the sensory gap is a viable path forward. >

Dev: The main implication I see is that future robotic systems shouldn't just focus on high precision in isolation; they need to integrate contextual awareness into their control architecture >

Taro: And for autonomy research, it opens up possibilities where the system can be used not just as a safety net but also as a tool for skill acquisition in surgical training because of that guidance mechanism >

Rosa: I think the key thing here is that they successfully employed negative reinforcement cues to promote safer surgical behavior, which is something we should keep exploring >

Dev: And remember, this system was evaluated using sixteen participants on an anthropomorphic head phantom and a custom-built surgical robot >

Taro: It’s good to see the potential for implicit guidance in movement, because that suggests systems could be used to teach surgeons how to move safely even when they aren't directly feeling the tissue >

School of Medicine and Health, Department of Ophthalmology, Technical University of Munich (TUM) · Munich Institute of Robotics and Machine Intelligence (MIRMI) · Department of Biomedical Engineering, University of Alberta

cs.RO, cs.CV

Submitted: 2026-09-07

Updated: 2026-09-07

Comments: 7 pages, 9 figures, 2 tables. Accepted at the 2026 IEEE RAS/EMBS 11th International Conference on Biomedical Robotics and Biomechatronicsnics (BioRob 2026)

Journal ref: 2026 11th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Edmonton, AB, Canada, 2026, pp. 1082-1087

DOI: 10.1109/BioRob66782.2026.11681904

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 98/100

The gist: Robotic ophthalmic surgery introduces high precision but introduces a ”sensory gap” by decoupling the surgeon from their instrument, resulting in a loss of tactile feedback <ref:2609.07857#pg2>

Key concepts

Scene Graph (SG): A semantic middleware that represents the surgical scene. It analyzes video feeds to identify and track objects like instruments, the eye globe, and retina. This abstraction bridges raw visual data with high-level surgical context, allowing for intelligent decision-making based on relationships between elements.
iOCT Simulation: A low-cost proxy setup that physically simulates intraoperative Optical Coherence Tomography (iOCT) geometry. This simulation provides the necessary geometric primitives for the Scene Graph to build a realistic understanding of the surgical workspace, making the system agnostic to future clinical OCT feeds.
Haptic Feedback Mapping: The strategy that translates discrete surgical states derived from the SG into specific tactile cues for a robotic device. It defines feedback levels: silent navigation when aligned, continuous low-frequency vibration when misaligned (Active Guidance), and distinct patterns for critical events.
Semantic Middleware: The role of the Scene Graph in serving as a foundational layer for control logic. Instead of hardcoding every surgical rule, the engine queries the SG's topology to infer discrete surgical states, enabling flexible and scalable decision-making across different robotic tasks.

Terminology

Abstraction — 5× in this paper · explained in 1 episode
Scene Graphs — 5× in this paper
Cognitive Load — 4× in this paper · explained in 2 episodes
System Usability Scale — 4× in this paper
Core — 3× in this paper · explained in 2 episodes
Graph Construction — 3× in this paper · explained in 3 episodes
Technical University of Munich — 3× in this paper
Coherence — 2× in this paper · explained in 2 episodes
Decoupling — 2× in this paper · explained in 2 episodes
Attention Mechanism — 1× in this paper · explained in 3 episodes
Contextual Awareness — 1× in this paper · explained in 2 episodes
Dynamic Adaptation — 1× in this paper · explained in 2 episodes
Explainability — 1× in this paper · explained in 4 episodes
Semantic Alignment — 1× in this paper · explained in 6 episodes

Summary

Robotic ophthalmic surgery introduces high precision but introduces a ”sensory gap” by decoupling the surgeon from their instrument, resulting in a loss of tactile feedback <ref:2609.07857#pg2> This paper presents a novel haptic feedback system for subretinal injection tasks leveraging Scene Graphs (SG) <ref:2609.07857#pg2> The system bridges the sensory gap by analyzing a physically simulated intraoperative Optical Coherence Tomography (iOCT) feed to construct a real-time surgical SG <ref:2609.07857#pg2> Results demonstrate that the SG-driven haptic feedback improved surgical precision, reducing needle alignment error by 14% (p = 0.044) and improving System Usability Scale (SUS) scores by 8% (p = 0.015) <ref:2609.07857#pg2>

System Overview

The core of the proposed framework combines a compact imaging setup, a computer vision pipeline, and a semantic rule-based engine to provide context-aware haptic feedback to a robotic input device <ref:2609.07857#pg2> To prove the concept of this work, the proposed framework combines a compact imaging setup, a computer vision pipeline, and a semantic rule-based engine to provide context-aware haptic feedback to a robotic input device <ref:2609.07857#pg2> The system physically simulates iOCT-guided robotic surgery, with an anthropomorphic workspace <ref:2609.07857#pg2> While this optical setup serves as a low-cost proxy for iOCT geometry (fig. 5, left), the Scene Graph framework is designed to be agnostic to the input source, allowing future migration to clinical OCT feeds <ref:2609.07857#pg2>

Perception and Scene Graph Construction

The perception pipeline processes the video feed to track instruments relative to anatomical features inside the eye <ref:2609.07857#pg2> The system captures the camera feed and sequentially triggers specific detectors for the eye globe (1), retina (4), and needle (6) to generate the geometric primitives required for Scene Graph construction <ref:2609.07857#pg2> The perception pipeline processes the video feed to track instruments relative to anatomical features inside the eye <ref:2609.07857#pg2> To create similarity to OCT, the frame is slightly edited again after it has passed through the pipeline <ref:2609.07857#pg2> The perception pipeline processes the video feed to track instruments relative to anatomical features inside the eye <ref:2609.07857#pg2>

Semantic Abstraction and Inference

To represent the surgical scene and bridge the gap between low-level camera data and the high-complexity surgical context, we utilized the concept of an SG as our semantic middleware for future decision-making <ref:2609.07857#pg2> For every frame, the image is analyzed, and a corresponding SG is dynamically computed (Figure 5, right) <ref:2609.07857#pg2> The SG serves as the computational foundation of the engine: at each time step, the engine queries the SG topology Gt to infer a discrete surgical state St <ref:2609.07857#pg2> To bridge the gap between continuous sensor data and discrete logic, we utilize the geometric primitives visualized in Figure 4 (Right) <ref:2609.07857#pg2>

Haptic Feedback Strategy

The system maps these surgical states to a specific haptic grammar <ref:2609.07857#pg2> The goal is to funnel the surgeon from the active feedback states into the silent states <ref:2609.07857#pg2> 1) Nominal Navigation (Silent): When following the Aligned Path, no haptic feedback is applied (Ht = Off) <ref:2609.07857#pg2> 2) Active Guidance (Continuous): States in the Misaligned Path trigger a continuous low-frequency vibration (Ht = Viblow) <ref:2609.07857#pg2> 3) Critical Events (Red/Green): High-priority events trigger distinct patterns <ref:2609.07857#pg2>

Study Findings

Leveraging haptic feedback did not show any statistically significant change in task completion time <ref:2609.07857#pg2> Safety was quantified by two primary metrics: the frequency of unintended retinal contacts (touching retina without target contact) and the average Alignment Angle θ maintained throughout the study <ref:2609.07857#pg2> The mean alignment error reduced from 7.9 ◦ ± 2.6 ◦ to 6.8 ◦ ± 2.3 ◦ (p = 0.044) in the feedback condition <ref:2609.07857#pg2> Participants successfully evaded the negative Retinal Contact state <ref:2609.07857#pg2> The Visual + Haptic condition successfully promoted an “Align-then-Approach” strategy <ref:2609.07857#pg2> The system received a mean SUS score of 81.9, which corresponds to a rating of “Good” [15] <ref:2609.07857#pg2>

Conclusion and Future Work

We propose that the community should explore SGs not merely for analysis, but also as a semantic middleware for active robotic control <ref:2609.07857#pg2> A significant observation was the user response to the ’Misaligned’ vibration <ref:2609.07857#pg2> This suggests that the system successfully employed negative reinforcement cues to promote safer surgical behavior <ref:2609.07857#pg2> In future work, we will explore continuous guidance and address study limitations, specifically the sample size (N = 16), absence of tissue deformation, and imaging noise <ref:2609.07857#pg2> The ability to implicitly guide movement highlights the system’s potential not only as an intraoperative safety net but also as a tool for skill acquisition in surgical training <ref:2609.07857#pg2>

REFERENCES

[1] D. Tripepi, A. Jalil, N. Ally, M. Buzzi, G. Moussa, P.-R. Rothschild, T. Rossi, M. Ferrara, and M. R. Romano [2] G Singh, W W J Jie, M T Sun, R Casson D Selva and W Chan [3] S DiMaio, M Hanuschik and U Kreaden [4] M A Nasseri, M Eder, S Nair, E C Dean M Maier D Zapp C P Lohmann and A Knoll [5] I El Rassi and J-M El Rassi [6] N Enayati, A M Okamura, A Mariani, E Pellegrini M M Coad G Ferrigno and E De Momi [7] M Bergholz, M Ferle and B M Weber [8] A Henriques, K Hoxha, D Zapp P Charbel Issa N Navab and M A Nasseri [9] D L’Abbate, K Prescott B Geraghty V R Kearns and D H W Steel [10] C Irigoyen, A Amenabar Alonso J Sanchez-Molina M Rodr´ıguezHidalgo A Lara-Lopez and J Ruiz-Ederra [11] K Xue, M Groppe, A P Salvetti and R E MacLaren [12] S R Ramamurthy and V P Dave [13] M Islam, L Seenivasan, L C Ming and H Ren [14] C I Nwoye, T Yu, C Gonzalez B Seeliger P Mascagni D Mutter J Marescaux and N Padoy [15] A Bangor, P Kortum and J Miller [16] The paper is supported by the state of Bavaria through Bayerische Forschungsstiftung (BFS) under Grant AZ-1592-23-ForNeRo [17] © 2026 IEEE.

Improvements for AI systems

Context-aware Haptic Feedback System: The improved system can generate state-dependent haptic feedback by using Scene Graphs to infer context-aware vibrotactile cues for the surgeon, bridging visual perception and haptic feedback.
Semantic Middleware for Control: The Scene Graph can serve as a semantic middleware for future decision-making, allowing for the seamless integration of additional surgical contexts like anatomical constraints or workflow phases without re-engineering the control architecture.
Negative Reinforcement Cue Implementation: The system can employ haptic negative reinforcement cues by mapping states in the Misaligned Path to a continuous low-frequency vibration (Ht = Viblow), encouraging users to correct trajectories early, which promotes an Align-then-Approach strategy.
Scalable Feedback Logic: The rule-based inference engine allows for scalability by enabling new entities or relations to be added to the graph schema without altering the downstream inference engine, ensuring that the system remains robust to changes in surgical context.