RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination".
Dev: Here's the gist of the paper. Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence on page 1.
Rosa: To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action on page 1.
Dev: Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets on page 1.
Rosa: Clinical Workflow and Data Preparation The clinical workflow for a carotid artery examination involves a standard procedure where sonographers generally follow established protocols
9: ,
11: on page 2.
Paper summary: Dev: It’s interesting how they structure the entire process into eight sequential stages guided by specific anatomical cues on page three of this paper <ref:2603.01153#pg3>.
Rosa: That’s a lot of specific steps, from "Examine CCA proximal" to "Longitudinal scan completed," which shows they mapped the clinical workflow directly into the system's logic.
Dev: They built a dataset from thirty-two human volumetric scans and created three types of textual annotations for each one: the current scanning stage, an explanation of the key feature for that stage, and what API is next to finish that task on page four.
Rosa: That’s how they got their training data—pairing visual information with detailed text descriptions to teach the model.
Dev: The model itself uses a setup with a Large Language Model like Vicuna v1 point 5 7B and a vision foundation model called PubMedCLIPViT-B/thirty-two which gets frozen on page five.
Rosa: And they use a cross-modality projector, which is just a two-layer MLP, to align the visual and textual features together on page five <ref:2603.01153#pg3>.
Dev: The training involved three steps: first training the retrieval engine's ResMLP using metric learning with triplet loss, then pre-training that cross-modality projector, and finally instruction fine-tuning on page six <ref:2603.01153#pg3>.
Conclusion: Rosa: Looking at this paper RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination by Dianye Huang and colleagues, the main point is how they integrated retrieval augmentation with vision–language reasoning to achieve autonomous scanning.
Dev: It moves beyond just making a decision; it makes the system explain its reasoning, which is important for clinical acceptance because it addresses those safety concerns we talked about earlier.
Rosa: And by using retrieval-augmented in-context learning, they show that you don't need huge training sets to make the system adaptable and generalize better to new situations.
Dev: The results show that the RAG-RUSS@two configuration achieved the highest average accuracy of seventy-nine point one percent, which they claim is strong performance on anatomically well-defined stages on page seven.
Rosa: So, what this means for us is that we’ve got a framework here that can perform full-stage carotid scanning according to clinical protocols autonomously while providing explanations of the current image and the subsequent action.
Dev: It’s a step toward building trustworthy and acceptable robotic ultrasound systems for routine deployment, which is what the authors say on page eight <ref:2603.01153#pg3>.
Rosa: Exactly, so it’s about moving from a black box to something that can articulate its process, giving clinicians confidence in its performance during a complex procedure.
Dianye Huang, Ziping Cong, Nassir Navab, Zhongliang Jiang
Technical University of Munich · Munich Center for Machine Learning
cs.RO
Submitted: 2026-03-01
Updated: 2026-03-01
Comments: Accepted by ICRA
Journal ref: Proc. IEEE Intl. Conf. Robot. Autom. (ICRA), Vienna, Austria, 2026, pp. 10275-10282
DOI: 10.1109/ICRA57385.2026.11697339
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence on page 1.
Key concepts
- RAG-RUSS
- A framework that combines Retrieval-Augmented Generation with robotic ultrasound. It allows the system to perform a complete carotid exam by following clinical steps while clearly explaining what it is doing now and what it plans to do next, making its actions understandable.
- Clinical Workflow Partitioning
- Breaking down the complex carotid examination into eight sequential stages based on anatomical cues. This structure guides the robotic process, ensuring that the ultrasound scan follows established medical protocols, moving systematically from transverse views to longitudinal assessments.
- Retrieval-Augmented Generation (RAG)
- A technique used to improve AI performance when medical data is scarce. Instead of relying only on its training data, the system retrieves relevant context from a database before generating an answer or planning an action, enhancing its accuracy and generalization.
- Cross-modality Projector
- A component that connects visual information (from the ultrasound image) and textual information (from the LLM). It aligns these two different types of data so the AI can effectively reason about the image while understanding its textual instructions.
Terminology
Summary
Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence on page 1. p. 0 RAG-RUSS: A Retrieval-Augmented
Robotic Ultrasound for Autonomous Carotid Examination
Dianye Huang1,2,∗
, Ziping Cong1,∗
, Nassir Navab1,2
, Fellow, IEEE, and Zhongliang Jiang1,2,
†
Abstract— Robotic ultrasound (US) has recently attracted
increasing attention as a means to overcome the limitations
of conventional US examinations, such as the strong operator
dependence. However, the decision-making process of existing
methods is often either rule-based or relies on end-to-end
learning models that operate as black boxes. This has been
seen as a main limit for clinical acceptance and raises safety
concerns for widespread adoption in routine practice. To tackle
this challenge, we introduce the RAG-RUSS, an interpretable
framework capable of performing a full carotid examination
in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action. To tackle
this challenge, we introduce the RAG-RUSS, an interpretable
framework capable of performing a full carotid examination
in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action. Furthermore, given the scarcity of medical data, we incorporate
retrieval-augmented generation to enhance generalization and
reduce dependence on large-scale training datasets.Where the paper says it
Where the paper says it
Where the paper says it
Clinical Workflow and Data Preparation
The clinical workflow for a carotid artery examination involves a standard procedure where sonographers generally follow established protocols [9], [11] on page 2. This workflow typically begins in the transverse (short-axis) plane at the proximal CCA and proceeds distally to visualize the ICA and ECA, followed by a switch to longitudinal views for detailed assessment on page 2. p. 1 The examination is typically
performed first in the transverse (short-axis) plane along the
lumen centerline and then in the longitudinal (long-axis)
plane of the carotid artery [8]. The scan usually begins in
transverse view at the proximal CCA, proceeds distally to
the carotid bulb (carotid sinus), and continues across the
bifurcation to visualize the ICA and ECA. In this study, we partition the carotid scan
process into eight sequential stages (see Fig. 2), each guided
by distinct anatomical cues from the US images.Where the paper says it
Where the paper says it
Examine CCA proximal
and Longitudinal scan completed
on page 3.
Dataset Construction
A high-quality dataset was constructed from 32 human volumetric scans to train the proposed RAG-RUSS on page 4. For each entry, we defined three types of textual annotations: (1) the current scanning stage, (2) an explanation of the key anatomical feature characterizing the stage, and (3) the name of the next API required to complete the given task on page 4. p. 3 For each entry
within the window, in addition to the US images, we defined
three types of textual annotations: (1) the current scanning
stage, such as Stage 1 Examine CCA proximal and Stage
2 Examine CCA distal (see Fig. 2); (2) an explanation of
the key anatomical feature characterizing the stage; and (3)
the name of the next API required to complete the given
task. In this study, three APIs were defined: “tracking
forward,” “tracking backward,” and “rotation clockwise,”
which correspond to advancing the probe, retracting the
probe, and rotating the probe from the transverse plane to the
longitudinal plane, respectively. In total, we collected 15,459
items (N-length sliding window), inlcuding paired visual and
textual information based on 32 reconstructed volumes.Where the paper says it
Where the paper says it
Model Architecture and Training
RAG-RUSS comprises an LLM, a vision foundation model, and a retrieval engine to follow a multi-turn question–answering workflow on page 5. The Large Language Model employed is Vicuna v1. p. 3 )
1) Large Language Model: Vicuna v1. 5 7B [26] is employed as the backbone LLM. It produces three types of
outputs: the current scanning stage stage (see the stage
definitions in Section III-A), an explanation of that stage
explanation, and the prediction of the next API. 2) Ultrasound Image Encoder: A frozen PubMedCLIPViT-B/32 [27] is adopted as the vision encoder for RAGRUSS. the next API to execute. 3) Cross-Modality Projector: Following the LLaVA [26]
design, we employed a two-layer MLP as a cross-modality
projector to align visual and textual features. Training Pipeline
The training of RAG-RUSS can be divided into three
steps, corresponding to the three trainable modules specified
in Fig. We first train the ResMLP of the Retrieval
Engine using metric learning with the triplet loss defined
in Eq.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Performance Evaluation
The performance of the RAG component was evaluated using the Top@k metric, with Top@2 set to 1 if at least one relevant context example appears among the top-k retrieved results on page 7. The RAG module attained an average of 70. p. 5 RAG Retrieval Accuracy
We evaluate the performance of RAG module using the
Top@k metric, defined as follows: for each query, Top@k
is set to 1 if at least one relevant context example appears
among the top-k retrieved results; otherwise, it is 0. The final
score is reported as the average over all queries. During the retrieval, the resulting embedding is compared
against a database of historical scanning contexts to compute
cosine similarity scores. The top-k most similar scanning
contexts (k = 2 in this work) are retrieved and supplied
to the LLM as additional context. 2%
Top@2, with 6. 2% gain from adding one extra context
example. Among all configurations, RAG Only performs the worst with an average
accuracy of 64. In contrast, RAG-RUSS@2 (with two
retrieved examples) achieves the highest average accuracy
(79. In contrast, RAG-RUSS@2 (with two
retrieved examples) achieves the highest average accuracy
(79. 1%), demonstrating strong performance on anatomically
well-defined stages such as CCA proximal (97.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Discussion and Limitations
The experimental results demonstrate that RAG-RUSS@2 achieves the best overall performance, highlighting the effectiveness of augmenting VLMs with retrieval-based context on page 8. However, Return completed emerges as the most challenging stage across all configurations on page 8. p. 6 Discussion
The experimental results demonstrate that RAG-RUSS@2
achieves the best overall performance, highlighting the effectiveness and benefits of augmenting VLMs with retrievalbased context for US scanning. However, a closer examination reveals that Return completed emerges as the most
challenging stage across all configurations. Practical challenges in
real scenarios, such as deformation, will pose additional
challenges in future deployment. Nevertheless, RAG-RUSS
demonstrates that integrating retrieval-augmented generation
with vision–language reasoning can enable accurate, interpretable, and autonomous US scanning.Where the paper says it
Where the paper says it
Conclusion
This work presents RAG-RUSS, an interpretable and autonomous framework for carotid US scanning that follows established clinical workflows and leverages VLM on page 8. The retrieval-augmented in-context learning strategy enhances adaptability and generalization while reducing the amount of training data required on page 8. p. 7 Building on this dataset, we propose RAG-RUSS,
which integrates an LLM, a vision foundation model, and
a retrieval engine. In particular, the retrieval-augmented incontext learning strategy enhances adaptability and generalization while reducing the amount of training data required. In particular, the retrieval-augmented incontext learning strategy enhances adaptability and generalization while reducing the amount of training data required. Experimental results demonstrate that RAG-RUSS
can autonomously perform full-stage carotid scanning in
accordance with clinical protocols. Experimental results demonstrate that RAG-RUSS
can autonomously perform full-stage carotid scanning in
accordance with clinical protocols. Moreover, by providing
explanations of the current image and the subsequent action,
the system takes an important step toward developing trustworthy and acceptable RUSS for routine deployment.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Retrieval-Augmented Generation for Data Scarcity: Incorporate retrieval-augmented generation to
enhance generalization and reduce dependence on large-scale training datasets,
allowing the system to perform accurate stage recognition even with limited specific medical data like the 32 reconstructed volumes used in this work. -
Interpretable Closed-Loop Decision Making: Implement a framework where the system can
explain both the current stage and the next planned action
by leveraging retrieved, similar scan contexts, thereby enablingautonomous, interpretable carotid artery scanning
for clinical acceptance. -
Unified Perception and Action Framework: Develop an intelligent sonography agent that integrates perception, explanation, and action in a unified framework to
predict the next API,
allowing the robotic arm to autonomously plan probe motions across both transverse and longitudinal planes based on real-time image feedback. -
Dynamic Contextual Adaptation: Utilize a VLM with a cross-modality projector trained jointly during pretraining and fine-tuning to align visual embeddings with LLM token embedding space, enabling the model to
reference clinically similar scenarios at query time
for improved stage recognition accuracy. -
Uncertainty-Aware Retrieval Strategy: Employ a Top@k retrieval metric, specifically using k=2 retrieved examples, which the paper demonstrates
improves robustness under uncertainty,
allowing the system to handle ambiguous stages likeReturn completed
more reliably than single-context retrieval.
Abstract
Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence. However, the decision-making process of existing methods is often either rule-based or relies on end-to-end learning models that operate as black boxes. This has been seen as a main limit for clinical acceptance and raises safety concerns for widespread adoption in routine practice. To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action. Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets. The method was trained on data acquired from 28 volunteers, while an additional four volumetric scans recorded from previously unseen volunteers were reserved for testing. The results demonstrate that the method can explain the current scanning stage and autonomously plan probe motions to complete the carotid examination, encompassing both transverse and longitudinal planes.
Sources
- Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication
- From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
- EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopy
- USPilot: An Embodied Robotic Assistant Ultrasound System with Large Language Model Enhanced Graph Planner
- Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation