RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination

arXiv:2603.01153 · cs.RO · Submitted 2026-03-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination".

Dev: Here's the gist of the paper. Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence on page 1.

Rosa: To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action on page 1.

Dev: Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets on page 1.

Rosa: Clinical Workflow and Data Preparation The clinical workflow for a carotid artery examination involves a standard procedure where sonographers generally follow established protocols

9: ,

11: on page 2.

Paper summary: Dev: It’s interesting how they structure the entire process into eight sequential stages guided by specific anatomical cues on page three of this paper <ref:2603.01153#pg3>.

Rosa: That’s a lot of specific steps, from "Examine CCA proximal" to "Longitudinal scan completed," which shows they mapped the clinical workflow directly into the system's logic.

Dev: They built a dataset from thirty-two human volumetric scans and created three types of textual annotations for each one: the current scanning stage, an explanation of the key feature for that stage, and what API is next to finish that task on page four.

Rosa: That’s how they got their training data—pairing visual information with detailed text descriptions to teach the model.

Dev: The model itself uses a setup with a Large Language Model like Vicuna v1 point 5 7B and a vision foundation model called PubMedCLIPViT-B/thirty-two which gets frozen on page five.

Rosa: And they use a cross-modality projector, which is just a two-layer MLP, to align the visual and textual features together on page five <ref:2603.01153#pg3>.

Dev: The training involved three steps: first training the retrieval engine's ResMLP using metric learning with triplet loss, then pre-training that cross-modality projector, and finally instruction fine-tuning on page six <ref:2603.01153#pg3>.

Conclusion: Rosa: Looking at this paper RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination by Dianye Huang and colleagues, the main point is how they integrated retrieval augmentation with vision–language reasoning to achieve autonomous scanning.

Dev: It moves beyond just making a decision; it makes the system explain its reasoning, which is important for clinical acceptance because it addresses those safety concerns we talked about earlier.

Rosa: And by using retrieval-augmented in-context learning, they show that you don't need huge training sets to make the system adaptable and generalize better to new situations.

Dev: The results show that the RAG-RUSS@two configuration achieved the highest average accuracy of seventy-nine point one percent, which they claim is strong performance on anatomically well-defined stages on page seven.

Rosa: So, what this means for us is that we’ve got a framework here that can perform full-stage carotid scanning according to clinical protocols autonomously while providing explanations of the current image and the subsequent action.

Dev: It’s a step toward building trustworthy and acceptable robotic ultrasound systems for routine deployment, which is what the authors say on page eight <ref:2603.01153#pg3>.

Rosa: Exactly, so it’s about moving from a black box to something that can articulate its process, giving clinicians confidence in its performance during a complex procedure.

Dianye Huang, Ziping Cong, Nassir Navab, Zhongliang Jiang

Technical University of Munich · Munich Center for Machine Learning

cs.RO

Submitted: 2026-03-01

Updated: 2026-03-01

Comments: Accepted by ICRA

Journal ref: Proc. IEEE Intl. Conf. Robot. Autom. (ICRA), Vienna, Austria, 2026, pp. 10275-10282

DOI: 10.1109/ICRA57385.2026.11697339

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence on page 1.

Key concepts

RAG-RUSS
A framework that combines Retrieval-Augmented Generation with robotic ultrasound. It allows the system to perform a complete carotid exam by following clinical steps while clearly explaining what it is doing now and what it plans to do next, making its actions understandable.
Clinical Workflow Partitioning
Breaking down the complex carotid examination into eight sequential stages based on anatomical cues. This structure guides the robotic process, ensuring that the ultrasound scan follows established medical protocols, moving systematically from transverse views to longitudinal assessments.
Retrieval-Augmented Generation (RAG)
A technique used to improve AI performance when medical data is scarce. Instead of relying only on its training data, the system retrieves relevant context from a database before generating an answer or planning an action, enhancing its accuracy and generalization.
Cross-modality Projector
A component that connects visual information (from the ultrasound image) and textual information (from the LLM). It aligns these two different types of data so the AI can effectively reason about the image while understanding its textual instructions.

Terminology

Summary

Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence on page 1. p. 0

Where the paper says it

RAG-RUSS: A Retrieval-Augmented Robotic Ultrasound for Autonomous Carotid Examination Dianye Huang1,2,∗ , Ziping Cong1,∗ , Nassir Navab1,2 , Fellow, IEEE, and Zhongliang Jiang1,2, † Abstract— Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence. However, the decision-making process of existing methods is often either rule-based or relies on end-to-end learning models that operate as black boxes.

To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action on page 1. p. 0
Where the paper says it

This has been seen as a main limit for clinical acceptance and raises safety concerns for widespread adoption in routine practice. To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action.

Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets on page 1. p. 0
Where the paper says it

To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action. Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets.

Clinical Workflow and Data Preparation

The clinical workflow for a carotid artery examination involves a standard procedure where sonographers generally follow established protocols [9], [11] on page 2. This workflow typically begins in the transverse (short-axis) plane at the proximal CCA and proceeds distally to visualize the ICA and ECA, followed by a switch to longitudinal views for detailed assessment on page 2. p. 1

Where the paper says it

The examination is typically performed first in the transverse (short-axis) plane along the lumen centerline and then in the longitudinal (long-axis) plane of the carotid artery [8]. The scan usually begins in transverse view at the proximal CCA, proceeds distally to the carotid bulb (carotid sinus), and continues across the bifurcation to visualize the ICA and ECA.

To automate this, the study partitions the scan process into eight sequential stages guided by distinct anatomical cues on page 3. p. 1
Where the paper says it

In this study, we partition the carotid scan process into eight sequential stages (see Fig. 2), each guided by distinct anatomical cues from the US images.

These stages include Examine CCA proximal and Longitudinal scan completed on page 3.

Dataset Construction

A high-quality dataset was constructed from 32 human volumetric scans to train the proposed RAG-RUSS on page 4. For each entry, we defined three types of textual annotations: (1) the current scanning stage, (2) an explanation of the key anatomical feature characterizing the stage, and (3) the name of the next API required to complete the given task on page 4. p. 3

Where the paper says it

For each entry within the window, in addition to the US images, we defined three types of textual annotations: (1) the current scanning stage, such as Stage 1 Examine CCA proximal and Stage 2 Examine CCA distal (see Fig. 2); (2) an explanation of the key anatomical feature characterizing the stage; and (3) the name of the next API required to complete the given task.

In total, we collected 15,459 items (N-length sliding window), including paired visual and textual information based on 32 reconstructed volumes on page 4. p. 3
Where the paper says it

In this study, three APIs were defined: “tracking forward,” “tracking backward,” and “rotation clockwise,” which correspond to advancing the probe, retracting the probe, and rotating the probe from the transverse plane to the longitudinal plane, respectively. In total, we collected 15,459 items (N-length sliding window), inlcuding paired visual and textual information based on 32 reconstructed volumes.

Model Architecture and Training

RAG-RUSS comprises an LLM, a vision foundation model, and a retrieval engine to follow a multi-turn question–answering workflow on page 5. The Large Language Model employed is Vicuna v1. p. 3

Where the paper says it

) 1) Large Language Model: Vicuna v1. 5 7B [26] is employed as the backbone LLM.

5 7B [26] on page 5, and the Ultrasound Image Encoder is a frozen PubMedCLIPViT-B/32 [27] on page 5. p. 3
Where the paper says it

It produces three types of outputs: the current scanning stage stage (see the stage definitions in Section III-A), an explanation of that stage explanation, and the prediction of the next API. 2) Ultrasound Image Encoder: A frozen PubMedCLIPViT-B/32 [27] is adopted as the vision encoder for RAGRUSS.

A cross-modality projector, implemented as a two-layer MLP, aligns visual and textual features on page 5. p. 4
Where the paper says it

the next API to execute. 3) Cross-Modality Projector: Following the LLaVA [26] design, we employed a two-layer MLP as a cross-modality projector to align visual and textual features.

The training pipeline involves three steps: training the ResMLP of the Retrieval Engine using metric learning with triplet loss, pre-training the cross-modality projector, and instruction fine-tuning on page 6. p. 5
Where the paper says it

Training Pipeline The training of RAG-RUSS can be divided into three steps, corresponding to the three trainable modules specified in Fig. We first train the ResMLP of the Retrieval Engine using metric learning with the triplet loss defined in Eq.

Performance Evaluation

The performance of the RAG component was evaluated using the Top@k metric, with Top@2 set to 1 if at least one relevant context example appears among the top-k retrieved results on page 7. The RAG module attained an average of 70. p. 5

Where the paper says it

RAG Retrieval Accuracy We evaluate the performance of RAG module using the Top@k metric, defined as follows: for each query, Top@k is set to 1 if at least one relevant context example appears among the top-k retrieved results; otherwise, it is 0. The final score is reported as the average over all queries.

0% Top@1 and 76. p. 4
Where the paper says it

During the retrieval, the resulting embedding is compared against a database of historical scanning contexts to compute cosine similarity scores. The top-k most similar scanning contexts (k = 2 in this work) are retrieved and supplied to the LLM as additional context.

2% Top@2, with a gain from adding one extra context example on page 7. p. 5
Where the paper says it

2% Top@2, with 6. 2% gain from adding one extra context example.

The RAG-RUSS@2 configuration achieved the highest average accuracy of 79. p. 6
Where the paper says it

Among all configurations, RAG Only performs the worst with an average accuracy of 64. In contrast, RAG-RUSS@2 (with two retrieved examples) achieves the highest average accuracy (79.

1%, demonstrating strong performance on anatomically well-defined stages on page 7. p. 6
Where the paper says it

In contrast, RAG-RUSS@2 (with two retrieved examples) achieves the highest average accuracy (79. 1%), demonstrating strong performance on anatomically well-defined stages such as CCA proximal (97.

Discussion and Limitations

The experimental results demonstrate that RAG-RUSS@2 achieves the best overall performance, highlighting the effectiveness of augmenting VLMs with retrieval-based context on page 8. However, Return completed emerges as the most challenging stage across all configurations on page 8. p. 6

Where the paper says it

Discussion The experimental results demonstrate that RAG-RUSS@2 achieves the best overall performance, highlighting the effectiveness and benefits of augmenting VLMs with retrievalbased context for US scanning. However, a closer examination reveals that Return completed emerges as the most challenging stage across all configurations.

Several limitations remain, including significant computational resources required for longer inputs and the system's reliance on discrete API-based control rather than continuous action execution on page 8. Nevertheless, RAG-RUSS demonstrates that integrating retrieval-augmented generation with vision–language reasoning can enable accurate, interpretable, and autonomous US scanning on page 8. p. 7
Where the paper says it

Practical challenges in real scenarios, such as deformation, will pose additional challenges in future deployment. Nevertheless, RAG-RUSS demonstrates that integrating retrieval-augmented generation with vision–language reasoning can enable accurate, interpretable, and autonomous US scanning.

Conclusion

This work presents RAG-RUSS, an interpretable and autonomous framework for carotid US scanning that follows established clinical workflows and leverages VLM on page 8. The retrieval-augmented in-context learning strategy enhances adaptability and generalization while reducing the amount of training data required on page 8. p. 7

Where the paper says it

Building on this dataset, we propose RAG-RUSS, which integrates an LLM, a vision foundation model, and a retrieval engine. In particular, the retrieval-augmented incontext learning strategy enhances adaptability and generalization while reducing the amount of training data required.

Experimental results demonstrate that RAG-RUSS can autonomously perform full-stage carotid scanning in accordance with clinical protocols on page 8. p. 7
Where the paper says it

In particular, the retrieval-augmented incontext learning strategy enhances adaptability and generalization while reducing the amount of training data required. Experimental results demonstrate that RAG-RUSS can autonomously perform full-stage carotid scanning in accordance with clinical protocols.

Moreover, by providing explanations of the current image and the subsequent action, the system takes an important step toward developing trustworthy and acceptable RUSS for routine deployment on page 8. p. 7
Where the paper says it

Experimental results demonstrate that RAG-RUSS can autonomously perform full-stage carotid scanning in accordance with clinical protocols. Moreover, by providing explanations of the current image and the subsequent action, the system takes an important step toward developing trustworthy and acceptable RUSS for routine deployment.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Retrieval-Augmented Generation for Data Scarcity: Incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets, allowing the system to perform accurate stage recognition even with limited specific medical data like the 32 reconstructed volumes used in this work.

  2. Interpretable Closed-Loop Decision Making: Implement a framework where the system can explain both the current stage and the next planned action by leveraging retrieved, similar scan contexts, thereby enabling autonomous, interpretable carotid artery scanning for clinical acceptance.

  3. Unified Perception and Action Framework: Develop an intelligent sonography agent that integrates perception, explanation, and action in a unified framework to predict the next API, allowing the robotic arm to autonomously plan probe motions across both transverse and longitudinal planes based on real-time image feedback.

  4. Dynamic Contextual Adaptation: Utilize a VLM with a cross-modality projector trained jointly during pretraining and fine-tuning to align visual embeddings with LLM token embedding space, enabling the model to reference clinically similar scenarios at query time for improved stage recognition accuracy.

  5. Uncertainty-Aware Retrieval Strategy: Employ a Top@k retrieval metric, specifically using k=2 retrieved examples, which the paper demonstrates improves robustness under uncertainty, allowing the system to handle ambiguous stages like Return completed more reliably than single-context retrieval.

Abstract

Robotic ultrasound (US) has recently attracted increasing attention as a means to overcome the limitations of conventional US examinations, such as the strong operator dependence. However, the decision-making process of existing methods is often either rule-based or relies on end-to-end learning models that operate as black boxes. This has been seen as a main limit for clinical acceptance and raises safety concerns for widespread adoption in routine practice. To tackle this challenge, we introduce the RAG-RUSS, an interpretable framework capable of performing a full carotid examination in accordance with the clinical workflow while explicitly explaining both the current stage and the next planned action. Furthermore, given the scarcity of medical data, we incorporate retrieval-augmented generation to enhance generalization and reduce dependence on large-scale training datasets. The method was trained on data acquired from 28 volunteers, while an additional four volumetric scans recorded from previously unseen volunteers were reserved for testing. The results demonstrate that the method can explain the current scanning stage and autonomously plan probe motions to complete the carotid examination, encompassing both transverse and longitudinal planes.

Sources

Related papers