Real-Time Visual Obstruction Detection in Surgical Augmented Reality
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Real-Time Visual Obstruction Detection in Surgical Augmented Reality".
Jane: Here's the gist of the paper. Surgical augmented reality can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace <ref:2608.00232#pg2> In this paper, we investigate visual obstruction detection for surgical AR and present a latency-aware pipeline that combines vision-language model (VLM)-based surgical-object recognition with segmentation-based obstruction reasoning <ref:2608.00232#pg2> Evaluation results show that the proposed system achieves 87.43% obstruction detection accuracy with an average end-to-end latency of 479 ms, reducing latency by 62.90% compared to a cloud large-model baseline <ref:2608.00232#pg2> System Architecture and Pipeline The proposed system employs a cascaded small-to-large VLM architecture designed for latency efficiency <ref:2608.00232#pg2> 1.
Tom: The edge server first processes the raw surgical image with a VLM to identify task-relevant surgical objects, producing a key-object list <ref:2608.00232#pg2> 2.
Jane: This list is used as text prompts for an open-set segmentation model, which takes the current image and object names as input and generates masks for the specified surgical instruments or tools <ref:2608.00232#pg2> 3.
Tom: The system uses a lightweight model to handle easy frames when its surgical-tool prediction is supported by segmentation consistency, while difficult frames are forwarded to a large VLM with pruned visual tokens <ref:2608.00232#pg2> Pruning redundant reasoning tokens The system incorporates two core modules to improve inference efficiency while preserving the ability to handle challenging surgical frames <ref:2608.00232#pg2> Segmentation-guided early exiting determines whether the system can make an obstruction decision using only the small VLM <ref:2608.00232#pg2> The early-exiting module uses the small VLM to predict task-relevant surgical objects, and if the number of predicted key objects matches the number of clustered object masks, it exits early without invoking the large VLM <ref:2608.00232#pg2> Attention-based visual token pruning reduces the cost of large-model inference when early exiting is rejected <ref:2608.00232#pg2> This module derives token-level importance scores from the small VLM’s attention maps and uses these scores to rank visual-token positions, pruning lower-ranked tokens during large-VLM inference <ref:2608.00232#pg2> Where the gains come from The latency reduction is achieved by combining model-specific prompts and dynamic model invocation strategies <ref:2608.00232#pg2> The small-VLM prompt encourages concise and precise surgical-object names, which facilitate downstream segmentation and early-exit checking <ref:2608.00232#pg2> The large-VLM prompt allows broader object descriptions, enabling more robust recognition when the small VLM prediction may be incomplete or ambiguous <ref:2608.00232#pg2> The deployment choice on the edge server further reduces communication latency compared to remote cloud inference <ref:2608.00232#pg2> Evaluation and Limitations The proposed system achieves an obstruction detection accuracy of 87.43% with an average end-to-end latency of 479 ms <ref:2608.00232#pg2> The cloud large-only baseline has the highest latency among all evaluated configurations, requiring 1119 ms for key-object detection and 1292 ms for endto-end obstruction detection <ref:2608.00232#pg2> The edge small-only baseline yields the lowest latency, with an end-to-end latency of 304 ms, but its detection performance is weaker, achieving 78.86% obstruction detection accuracy <ref:2608.00232#pg2> Failure analysis indicates that remaining errors are mainly caused by key-object recognition failures and segmentation failure <ref:2608.00232#pg2> Future Work Directions Future work will focus on expanding the benchmark to dynamic surgical videos, complex surgical scenes, and more clinically grounded AR guidance content <ref:2608.00232#pg2> The system will jointly model multiple key objects, their corresponding segmentation masks, and their individual obstruction states in realistic environments <ref:2608.00232#pg2> A natural next step is to construct video-based benchmarks for surgical AR obstruction detection using a frame-selection module that uses inexpensive visual cues to filter redundant frames before VLM inference <ref:2608.00232#pg2> This design would reserve VLM inference for informative frames, reducing computation while preserving responsiveness <ref:2608.00232#pg2> In this work, we investigate visual obstruction detection for surgical AR and propose a latency-aware small-to-large VLM pipeline that achieves 87.43% obstruction detection accuracy with an average end-to-end latency of 479 ms <ref:2608.00232#pg2> The results suggest that efficient VLM-based reasoning can support near-real-time obstruction detection in surgical AR, while our failure analysis shows that remaining errors are mainly caused by key-object recognition failures and text-guided segmentation failures <ref:2608.00232#pg2> Building on this study, future work will expand the benchmark to dynamic surgical videos, complex surgical scenes, and more clinically grounded AR guidance content <ref:2608.00232#pg2> Acknowledgments This work was supported in part by NSF grants CSR-2312760, CNS2112562, and IIS-2231975 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency, the Army Research Laboratory, or the U.S.
Paper summary: Tom: The main idea here is that they found a way to detect obstructions by mixing vision language models for object recognition with segmentation-based reasoning for where things are. They claim this approach works well, achieving eighty-seven point four three percent obstruction detection accuracy with an average end-to-end latency of four hundred seventy-nine milliseconds.
Jane: That latency figure is pretty impressive when you consider they reduced the latency by a significant amount, specifically by sixty-two point nine zero percent compared to a cloud large model baseline. It’s about getting high accuracy without making the system too slow for a surgeon to trust it in surgery.
Lu: The system uses an edge server setup where the small vision language model first identifies key surgical objects, and then this list is used as prompts for a segmentation model to generate masks for tools or instruments.
Meng: That sounds like a lot of steps, but the architecture they use involves pruning redundant reasoning tokens to keep the large vision language model inference from getting too costly when it has to handle hard frames.
Lalam: They use two core modules for this efficiency: segmentation-guided early exiting and attention-based visual token pruning when early exiting isn't possible. That’s how they manage the computational load of a large model in a real-time scenario.
Conclusion: Tom: So, looking at "Real-Time Visual Obstruction Detection in Surgical Augmented Reality," the implication is that we can have AR systems that give surgeons warnings about visual interference with near real-time performance. The authors tackled this by combining different model strengths to achieve decent accuracy without crippling the speed.
Jane: What matters for us is seeing how they managed to get that eighty-seven point four three percent accuracy while maintaining an end-to-end latency of four hundred seventy nine milliseconds on the edge server, which is much better than what a cloud large model baseline could do.
Lu: The authors also pointed out their limitations in the paper, specifically noting that remaining errors in their detection mostly come from failures in key-object recognition and segmentation itself. They're not perfect at everything.
Meng: From a deployment standpoint, it’s encouraging that they focused on an edge server setup instead of just relying on remote cloud inference for this kind of critical task. That keeps the communication latency down considerably compared to sending data all the way out and back to the cloud.
Lalam: Ultimately, this study suggests that efficient vision language model reasoning can support near real-time obstruction detection in surgical AR, but it highlights that text-guided segmentation failures are a significant source of error they need to address next.
Department of Computer Science, Duke University · Department of Electrical and Computer Engineering, Duke University
cs.CV
Submitted: 2026-07-31
Updated: 2026-11-02
Comments: ISMAR 2026 Medical XR Workshop
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 79/100
The gist: Surgical augmented reality can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace <ref:2608.00232#pg2> In this paper…
Key concepts
- Visual Obstruction Detection
- This is the core task of identifying when an object in a surgical video view is blocking the surgeon's view or interfering with their work. The system aims to automatically detect these obstructions in real-time so AR tools can provide timely guidance.
- Vision-Language Model (VLM)
- A VLM is an AI model that understands both images and text. In this system, it is used first to identify important surgical objects in the image, and later to generate detailed descriptions needed for accurate obstruction reasoning when necessary.
- Latency-Aware Pipeline
- This refers to a system design focused on speed. The pipeline uses different AI models strategically—switching between small and large versions based on the frame's difficulty—and employs pruning techniques to ensure the overall processing time remains low, crucial for real-time surgical applications.
Terminology
Summary
Surgical augmented reality can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace <ref:2608. p. 000232#pg2> In this paper, we investigate visual obstruction detection for surgical AR and present a latency-aware pipeline that combines vision-language model (VLM)-based surgical-object recognition with segmentation-based obstruction reasoning <ref:2608. p. 000232#pg2> Evaluation results show that the proposed system achieves 87. p. 043% obstruction detection accuracy with an average end-to-end latency of 479 ms, reducing latency by 62. p. 090% compared to a cloud large-model baseline <ref:2608. p. 000232#pg2>
System Architecture and Pipeline
The proposed system employs a cascaded small-to-large VLM architecture designed for latency efficiency <ref:2608.00232#pg2>
-
The edge server first processes the raw surgical image with a VLM to identify task-relevant surgical objects, producing a key-object list <ref:2608. p. 100232#pg2>
-
This list is used as text prompts for an open-set segmentation model, which takes the current image and object names as input and generates masks for the specified surgical instruments or tools <ref:2608. p. 100232#pg2>
-
The system uses a lightweight model to handle easy frames when its surgical-tool prediction is supported by segmentation consistency, while difficult frames are forwarded to a large VLM with pruned visual tokens <ref:2608. p. 000232#pg2>
Pruning redundant reasoning tokens
The system incorporates two core modules to improve inference efficiency while preserving the ability to handle challenging surgical frames <ref:2608.00232#pg2>
** Segmentation-guided early exiting determines whether the system can make an obstruction decision using only the small VLM <ref:2608.00232#pg2>**
The early-exiting module uses the small VLM to predict task-relevant surgical objects, and if the number of predicted key objects matches the number of clustered object masks, it exits early without invoking the large VLM <ref:2608.00232#pg2>
** Attention-based visual token pruning reduces the cost of large-model inference when early exiting is rejected <ref:2608.00232#pg2>**
This module derives token-level importance scores from the small VLM’s attention maps and uses these scores to rank visual-token positions, pruning lower-ranked tokens during large-VLM inference <ref:2608.00232#pg2>
Where the gains come from
The latency reduction is achieved by combining model-specific prompts and dynamic model invocation strategies <ref:2608.00232#pg2>
** The small-VLM prompt encourages concise and precise surgical-object names, which facilitate downstream segmentation and early-exit checking <ref:2608.00232#pg2>**
** The large-VLM prompt allows broader object descriptions, enabling more robust recognition when the small VLM prediction may be incomplete or ambiguous <ref:2608.00232#pg2>**
The deployment choice on the edge server further reduces communication latency compared to remote cloud inference <ref:2608.00232#pg2>
Evaluation and Limitations
The proposed system achieves an obstruction detection accuracy of 87.43% with an average end-to-end latency of 479 ms <ref:2608. p. 000232#pg2>
** The cloud large-only baseline has the highest latency among all evaluated configurations, requiring 1119 ms for key-object detection and 1292 ms for endto-end obstruction detection <ref:2608.00232#pg2>**
The edge small-only baseline yields the lowest latency, with an end-to-end latency of 304 ms, but its detection performance is weaker, achieving 78.86% obstruction detection accuracy <ref:2608. p. 000232#pg2>
Failure analysis indicates that remaining errors are mainly caused by key-object recognition failures and segmentation failure <ref:2608.00232#pg2>
Future Work Directions
Future work will focus on expanding the benchmark to dynamic surgical videos, complex surgical scenes, and more clinically grounded AR guidance content <ref:2608.00232#pg2>
** The system will jointly model multiple key objects, their corresponding segmentation masks, and their individual obstruction states in realistic environments <ref:2608.00232#pg2>**
A natural next step is to construct video-based benchmarks for surgical AR obstruction detection using a frame-selection module that uses inexpensive visual cues to filter redundant frames before VLM inference <ref:2608.00232#pg2>
This design would reserve VLM inference for informative frames, reducing computation while preserving responsiveness <ref:2608.00232#pg2>
Conclusion
In this work, we investigate visual obstruction detection for surgical AR and propose a latency-aware small-to-large VLM pipeline that achieves 87.43% obstruction detection accuracy with an average end-to-end latency of 479 ms <ref:2608. p. 000232#pg2> The results suggest that efficient VLM-based reasoning can support near-real-time obstruction detection in surgical AR, while our failure analysis shows that remaining errors are mainly caused by key-object recognition failures and text-guided segmentation failures <ref:2608. p. 500232#pg2> Building on this study, future work will expand the benchmark to dynamic surgical videos, complex surgical scenes, and more clinically grounded AR guidance content <ref:2608. p. 500232#pg2>
Acknowledgments
This work was supported in part by NSF grants CSR-2312760, CNS2112562, and IIS-2231975 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency, the Army Research Laboratory, or the U.S.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Optimized latency-aware pipeline for surgical AR obstruction detection by combining segmentation-guided early exiting with attention-based visual token pruning to reduce inference overhead while preserving detection reliability. This allows for near real-time surgical AR guidance by significantly reducing latency compared to cloud large model baselines, achieving an average end-to-end latency of 479 ms.
-
Enhanced key object detection accuracy by leveraging a small VLM prompt that encourages
concise and precise surgical-object names,
which facilitates downstream segmentation and early exit checking. This mechanism ensures that the system can make an obstruction decision without invoking the large VLM when predictions aresufficiently reliable.
-
Improved visual token processing cost reduction by using decode-stage attention from the small VLM to derive importance scores, which are then used to rank visual-token positions for pruning in the large VLM. This strategy avoids crossmodel token incompatibility while guiding pruning based on what the small model predicts is most relevant to surgical object recognition.
-
More robust obstruction detection by addressing key-object detection failure cases through improved VLM training or prompt strategies better adapted to
low visual clarity, weak object boundaries, or motion blur.
This directly tackles the primary source of error where incorrect key-object prediction leads to false positive obstructions. -
Increased segmentation reliability for surgical instruments by employing segmentation models like SAM3 that have been trained specifically on fine-grained surgical instruments and low-quality images. This mitigates segmentation failure cases where the model
fails to detect the object and returns no valid mask,
ensuring accurate mask-level obstruction reasoning even with subtle tool structures. -
Future extension to dynamic surgical videos by developing a lightweight frame selection module that uses
inexpensive visual cues, such as feature matching or optical-flow magnitude,
to filter redundant frames before VLM inference. This reserve VLM computation only forinformative frames,
reducing energy cost and latency in real-time video streams.
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs