The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessment

arXiv:2509.17100 · cs.CV · Submitted 2025-09-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The SAGES Critical View of Safety Challenge".

Jane: Here's the gist of the paper. Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation The SAGES Critical View of Safety (CVS) Challenge presents the first AI competition organized by a surgical society using the CVS in laparoscopic cholecystectomy as an exemplar of surgical quality assessment.

Tom: Challenge Scope and Data Curation The study established the SAGES Critical View of Safety (CVS) Challenge, which involved a global collaboration across 54 institutions in 24 countries engaging hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol.

Jane: The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability.

Tom: Infrastructure for Large-Scale Data Management To manage the large and heterogeneous surgical video and multi-annotator workflows at a global scale, the EndoGlacier framework was developed.

Paper summary: Tom: So, wrapping up this paper "The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessment," it’s about creating this standard testing ground for AI in surgery.

Jane: They showed that by bringing together over a thousand videos from different devices and workflows, they built a way to benchmark how well these AI systems can judge surgical safety.

Lu: It establishes a global standard because of the wide collaboration across fifty-four institutions and the careful curation of one thousand annotated cases following a consensus protocol.

Meng: From my side, it’s interesting that they focused on domain shift robustness, which means testing if the model works when it sees data from devices or techniques it hasn't seen before.

Tom: That robustness gain of seventeen percent in subchallenge C is what really stood out for me, showing the models are doing better than just being good at one specific task.

Jane: The implication for the world is that we’re moving toward a place where AI can provide objective feedback on surgical quality, which helps reduce variability between surgeons.

Lu: It moves us past just having models that are accurate on a test set to having systems that are reliable when facing the messy reality of diverse surgical practice.

Tom: The authors admit their limitation is that the number of videos evaluated is restricted by how many surgeries a surgeon can realistically review per case, which cuts into how much we can test.

Jane: That’s a fair caveat; they acknowledge that testing on thousands of cases per surgeon isn't always feasible in practice.

Meng: So, for deployment right now, the real question is how we take these top models and make them cheap and reliable enough to run consistently in a busy OR.

Tom: Exactly, because even with all this benchmarking data, getting that model onto a production system that actually runs smoothly at scale is the next big hurdle.

Conclusion: Tom: So we’ve been deep in the weeds on this SAGES Critical View of Safety Challenge, and now we’re heading to the conclusion of this paper about how they built this whole testing ground for surgical AI.

Jane: It boils down to them putting together a massive effort involving over a thousand videos from different hospitals and surgeons all over the world, following a really strict set of rules for how those videos get labeled by experts.

Lu: The authors are basically saying they wanted to find out what it’s like when you test an AI system against real-world surgical quality in a way that everyone can agree on.

Meng: I’m interested in the title itself, "The SAGES Critical View of Safety Challenge," because it sounds very serious. It implies there was a specific safety standard they were trying to measure.

Tom: It is serious because they took something subjective—how safe a surgery looks—and turned it into a measurable task with one thousand data points for everyone to look at.

Jane: That’s the core idea, though, moving beyond just building a model that can identify an artery; this is about making sure the AI understands what *safety* actually means in that critical moment before clipping.

Lu: They focused on three main things: how well it identifies safety, how much uncertainty it shows when it’s not sure, and if it keeps working when they use different surgical tools or techniques.

Meng: And the results showed that some models really did get better at handling those changes across all those tests compared to their previous baseline models. That seventeen percent jump in robustness was notable for testing purposes.

Tom: It tells us that just making the AI accurate on one specific thing isn't enough; it has to be tough enough to handle the messy reality of a different operating room.

Jane: So, what does this mean for surgeons or hospitals who want to use this kind of AI in their actual practice? Does it give them a reliable way to check if their tools are performing safely?

Lu: It provides a starting point for creating standardized training recipes and finding those specific architectural choices—like using transformer models—that actually help the system be dependable under pressure.

Tom: We’ve seen how these top performers used those transformer structures, but what we need next is figuring out how to take that high-performing model and make it cheap enough to run on a busy operating room computer every single day.

Jane: That’s the big question now—moving from a brilliant benchmark study to something that actually runs smoothly in a clinical setting without needing super expensive hardware.

University of Strasbourg, CNRS, INSERM, ICube, UMR7357 · IHU Strasbourg · Massachusetts General Hospital, Harvard Medical School · University Hospital Cologne · Global College Shanghai Jiao Tong University · Chang Gung Memorial Hospital Keelung Taiwan · A. Costa Hospital and Maggiore Hospital AUSL Bologna Italy · Institute for Research against Digestive Cancer IRCAD Strasbourg France · Lenox Hill Hospital Northwell Health USA · Albany Medical Center USA

cs.CV

Submitted: 2025-09-21

Updated: 2026-11-11

Comments: 21 pages, 10 figures

Code: https://github.com/SAIIL/CVS_challenge_code

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 94/100

The gist: Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance…

Key concepts

Critical View of Safety (CVS)
This is a specific surgical technique where the surgeon must see all necessary structures clearly before making a critical cut, such as clipping the cystic duct or artery. The AI challenge focused on accurately identifying when this view was achieved during the 90 seconds before that critical moment.
EndoGlacier Framework
This was a large-scale infrastructure developed to manage the massive amount of surgical video data and complex annotation workflows globally. It coordinated three main areas: managing annotators, automating the process, and handling the video files themselves to ensure smooth global collaboration.
Uncertainty Quantification
This refers to how well an AI model can express its confidence in its assessment. The challenge specifically tested this by measuring uncertainty during the CVS task. The findings showed that simply optimizing for high accuracy doesn't guarantee a reliable measure of confidence, highlighting the need for better calibration.
Surgical Pretraining
This involves training AI models on large datasets of surgical videos before they tackle the specific challenge. Using surgical or self-supervised pretraining was found to be a common trait among top performers, suggesting that having prior surgical knowledge helps models perform better in quality assessment tasks.

Terminology

Summary

Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation The SAGES Critical View of Safety (CVS) Challenge presents the first AI competition organized by a surgical society using the CVS in laparoscopic cholecystectomy as an exemplar of surgical quality assessment. p. 0

Where the paper says it

Gemelli IRCCS, Italy zUniversity of Pennsylvania, USA aaDuke University, USA A B S T R A C T Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation. This study presents the SAGES Critical View of Safety (CVS) Challenge, the first AI competition organized by a surgical society, using the CVS in laparoscopic cholecystectomy, a universally recommended yet inconsistently performed safety step, as an exemplar of surgical quality assessment.

Challenge Scope and Data Curation

The study established the SAGES Critical View of Safety (CVS) Challenge, which involved a global collaboration across 54 institutions in 24 countries engaging hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol. The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability. p. 0

Where the paper says it

A global collaboration across 54 institutions in 24 countries engaged hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol. The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability.

Infrastructure for Large-Scale Data Management

To manage the large and heterogeneous surgical video and multi-annotator workflows at a global scale, the EndoGlacier framework was developed. This framework coordinated three streams: (i) annotator management, (ii) orchestration and automation, and (iii) video management. p. 3

Where the paper says it

Coordinated Data Flows for Large-Scale Annotation Managing diverse dataset generation at a global scale requires more than assembling data, it demands infrastructure capable of sustaining continuous, high-quality flows of videos, expert annotations, and coordination across dozens of institutions and contributors. Here, we break down our dataset generation process into three coordinated streams visualized in Figure 2: (1) annotator management, (2) video management, and (3) orchestration and automation.

The annotation management stream involved onboarding 106 potential annotators, resulting in a qualified pool of 20 expert annotators. p. 4
Where the paper says it

As a final gate, we implemented a competency-based exam requiring at least 75% correspondence with expert ratings to ensure conceptual alignment and minimum annotation quality. Through this structured pipeline, we contacted or were contacted by 106 potential annotators, recruited through channels including social media and in-person onboarding campaigns.

Benchmark Task and Dataset Characteristics

The benchmark task focused on assessing the Critical View of Safety (CVS) achievement during the 90-second window immediately preceding clipping of the cystic duct or artery. The dataset comprises 1,000 laparoscopic cholecystectomy cases, with each represented by a 90-second clip annotated at one frame every five seconds. p. 5

Where the paper says it

This design provides both temporal resolution and practical annotation density, aligning the benchmark with the real-world clinical decision context. Dataset Characteristics The SAGES CVS Challenge dataset comprises a total of 1,000 laparoscopic cholecystectomy cases, each represented by a 90-second clip taken from the critical decision window preceding clipping of the cystic duct or artery.

The dataset captures substantial clinical and technical diversity, spanning 23 countries and 8 device vendors. p. 5
Where the paper says it

The dataset captures substantial clinical and technical diversity. 23 countries are represented in the training split and 18 countries in the test split, with an average of 30.

Evaluation Metrics and Performance Trends

Participants were evaluated across three subchallenges: Subchallenge A (CVS Achievement), Subchallenge B (Uncertainty Quantification), and Subchallenge C (Domain Shift Robustness). Top-performing entries showed headline gains over the LG-DG baseline across all three subchallenges, including a 17% relative improvement in robustness in Subchallenge C. p. 1

Where the paper says it

Participants were evaluated across three complementary dimensions essential for clinical deployment—absolute performance requirements, alignment with expert uncertainty, and robustness under real-world domain shifts. Submissions from 13 international teams delivered measurable gains over state-of-the-art in all three areas, offering both methodological advances and insight into the design choices that influence performance.

Common traits of top performers included the use of Transformer or hybrid transformer–CNN models and surgical or self-supervised surgical pretraining. p. 9
Where the paper says it

# Team A ffiliation(s) Architecture Temporal Component Auxiliary Objective Pretraining 1 TUE-VCA Eindhoven University of Technology (Netherlands) PVT-v2 (x5 ensemble) None L2 reconstruction loss Self-supervised (DINO on SurgeNet and GenSurgery datasets) 2 Farm Stanford University (USA) DINOv2 (giant, large), SigLIP, ConvNeXt2-L, InternImage Stacked dilated TCN None Self-supervised (DINO, SigLIP) 3 CVS HUST Huazhong University of Science and Technology (China) ConvNeXt + MLP LSTM ℓ1 loss for confidence prediction Supervised (natural image weights) 4 SDS-HD German Cancer Research Center (DKFZ) (Germany) EVA02-Large Transformer None Segmentation loss on YOLOv8 pseudolabels Endoscapes-pretrained Endoscapes) Self-supervised (MoCov2 on Cholec80, HeiChole, 5 SRV-WEISS University College London (UK) EndoViT + Dense Prediction Transformer, E fficientNet, ConvNeXt None Segmentation + classification fusion Self-Supervised (EndoViT) + Supervised E (EndoViT on Endoscapes segmentation; fficientNet on Endoscapes classes) 6 Caresyntax Caresyntax DenseNet-121 (5× ensemble) None None Supervised (ImageNet weights) 8 HUFT-MedIA Hefei University of Technology (China) Vision Transformer (ViT) 7 FightTumor University Health Network (Canada) ConvNeXt-B None None Supervised (ImageNet weights) + MLP None None Supervised (ImageNet weights) 9 IRCV-URV Universitat Rovira i Virgili (Spain) E fficientNet-B5 + FPN + spatial attention (5x ensemble) None None Supervised (ImageNet weights) 10 mmll NAAMII (Nepal) 1. ResNet-50 + Mask R-CNN + Transformer encoder (temporal 10-frame) 2.

Key Methodological Insights

Analysis revealed that while architectures like Transformer or hybrid transformer–CNN models dominated the very top, leveraging surgical or self-supervised surgical pretraining was common among accuracy/robustness leaders. Ensembling was frequently present among Subchallenge A and C leaders but rare among calibration leaders. The results underscored that optimizing for discriminative performance does not inherently yield confidence alignment, indicating that calibration requires targeted choices. p. 7

Where the paper says it

This design reflected real-world deployment constraints: in surgical AI systems, raw predictive performance, reliability under distribution shifts, and appropriate uncertainty estimation are all critical, yet not inherently aligned. Optimizing for one may compromise another.

The work provides a sharp starting point for focused, controlled ablations and shared training recipes.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved AI system can achieve high performance in predicting majority expert assessment for each of the three CVS criteria by utilizing a mode-of-raters label incorporating annotator confidence: y = 1/3 sum(li - 0.5) · ci, where li is the binary label and ci is the confidence score provided by annotator i.

  2. Improved AI system can reduce Brier error by over 80% compared to the LG-DG baseline in uncertainty quantification tasks through the use of auxiliary objectives or multi-task formulations, as seen in top performers like SDS-HD and Theator.

  3. Improved AI system can enhance domain shift robustness by achieving a 17% relative gain in robustness over state-of-the-art by utilizing methods that explicitly account for the inherent ambiguity in surgical quality assessments under distribution shifts related to device characteristics, case complexity, and regional differences in technique.

Abstract

Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation. This study presents the SAGES Critical View of Safety (CVS) Challenge, the first AI competition organized by a surgical society, using the CVS in laparoscopic cholecystectomy, a universally recommended yet inconsistently performed safety step, as an exemplar of surgical quality assessment. A global collaboration across 54 institutions in 24 countries engaged hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol. The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability. To enable this scale of effort, we developed EndoGlacier, a framework for managing large, heterogeneous surgical video and multi-annotator workflows. Thirteen international teams participated, achieving up to a 17% relative gain in assessment performance, over 62% reduction in calibration error, and a 17% relative improvement in robustness over the state-of-the-art. Analysis of results highlighted methodological trends linked to model performance, providing guidance for future research toward robust, clinically deployable AI for surgical quality assessment.

Sources

Related papers