The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessment
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "The SAGES Critical View of Safety Challenge".
Jane: Here's the gist of the paper. Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation The SAGES Critical View of Safety (CVS) Challenge presents the first AI competition organized by a surgical society using the CVS in laparoscopic cholecystectomy as an exemplar of surgical quality assessment.
Tom: Challenge Scope and Data Curation The study established the SAGES Critical View of Safety (CVS) Challenge, which involved a global collaboration across 54 institutions in 24 countries engaging hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol.
Jane: The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability.
Tom: Infrastructure for Large-Scale Data Management To manage the large and heterogeneous surgical video and multi-annotator workflows at a global scale, the EndoGlacier framework was developed.
Paper summary: Tom: So, wrapping up this paper "The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessment," it’s about creating this standard testing ground for AI in surgery.
Jane: They showed that by bringing together over a thousand videos from different devices and workflows, they built a way to benchmark how well these AI systems can judge surgical safety.
Lu: It establishes a global standard because of the wide collaboration across fifty-four institutions and the careful curation of one thousand annotated cases following a consensus protocol.
Meng: From my side, it’s interesting that they focused on domain shift robustness, which means testing if the model works when it sees data from devices or techniques it hasn't seen before.
Tom: That robustness gain of seventeen percent in subchallenge C is what really stood out for me, showing the models are doing better than just being good at one specific task.
Jane: The implication for the world is that we’re moving toward a place where AI can provide objective feedback on surgical quality, which helps reduce variability between surgeons.
Lu: It moves us past just having models that are accurate on a test set to having systems that are reliable when facing the messy reality of diverse surgical practice.
Tom: The authors admit their limitation is that the number of videos evaluated is restricted by how many surgeries a surgeon can realistically review per case, which cuts into how much we can test.
Jane: That’s a fair caveat; they acknowledge that testing on thousands of cases per surgeon isn't always feasible in practice.
Meng: So, for deployment right now, the real question is how we take these top models and make them cheap and reliable enough to run consistently in a busy OR.
Tom: Exactly, because even with all this benchmarking data, getting that model onto a production system that actually runs smoothly at scale is the next big hurdle.
Conclusion: Tom: So we’ve been deep in the weeds on this SAGES Critical View of Safety Challenge, and now we’re heading to the conclusion of this paper about how they built this whole testing ground for surgical AI.
Jane: It boils down to them putting together a massive effort involving over a thousand videos from different hospitals and surgeons all over the world, following a really strict set of rules for how those videos get labeled by experts.
Lu: The authors are basically saying they wanted to find out what it’s like when you test an AI system against real-world surgical quality in a way that everyone can agree on.
Meng: I’m interested in the title itself, "The SAGES Critical View of Safety Challenge," because it sounds very serious. It implies there was a specific safety standard they were trying to measure.
Tom: It is serious because they took something subjective—how safe a surgery looks—and turned it into a measurable task with one thousand data points for everyone to look at.
Jane: That’s the core idea, though, moving beyond just building a model that can identify an artery; this is about making sure the AI understands what *safety* actually means in that critical moment before clipping.
Lu: They focused on three main things: how well it identifies safety, how much uncertainty it shows when it’s not sure, and if it keeps working when they use different surgical tools or techniques.
Meng: And the results showed that some models really did get better at handling those changes across all those tests compared to their previous baseline models. That seventeen percent jump in robustness was notable for testing purposes.
Tom: It tells us that just making the AI accurate on one specific thing isn't enough; it has to be tough enough to handle the messy reality of a different operating room.
Jane: So, what does this mean for surgeons or hospitals who want to use this kind of AI in their actual practice? Does it give them a reliable way to check if their tools are performing safely?
Lu: It provides a starting point for creating standardized training recipes and finding those specific architectural choices—like using transformer models—that actually help the system be dependable under pressure.
Tom: We’ve seen how these top performers used those transformer structures, but what we need next is figuring out how to take that high-performing model and make it cheap enough to run on a busy operating room computer every single day.
Jane: That’s the big question now—moving from a brilliant benchmark study to something that actually runs smoothly in a clinical setting without needing super expensive hardware.
University of Strasbourg, CNRS, INSERM, ICube, UMR7357 · IHU Strasbourg · Massachusetts General Hospital, Harvard Medical School · University Hospital Cologne · Global College Shanghai Jiao Tong University · Chang Gung Memorial Hospital Keelung Taiwan · A. Costa Hospital and Maggiore Hospital AUSL Bologna Italy · Institute for Research against Digestive Cancer IRCAD Strasbourg France · Lenox Hill Hospital Northwell Health USA · Albany Medical Center USA
cs.CV
Submitted: 2025-09-21
Updated: 2026-11-11
Comments: 21 pages, 10 figures
Code: https://github.com/SAIIL/CVS_challenge_code
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 94/100
The gist: Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance…
Key concepts
- Critical View of Safety (CVS)
- This is a specific surgical technique where the surgeon must see all necessary structures clearly before making a critical cut, such as clipping the cystic duct or artery. The AI challenge focused on accurately identifying when this view was achieved during the 90 seconds before that critical moment.
- EndoGlacier Framework
- This was a large-scale infrastructure developed to manage the massive amount of surgical video data and complex annotation workflows globally. It coordinated three main areas: managing annotators, automating the process, and handling the video files themselves to ensure smooth global collaboration.
- Uncertainty Quantification
- This refers to how well an AI model can express its confidence in its assessment. The challenge specifically tested this by measuring uncertainty during the CVS task. The findings showed that simply optimizing for high accuracy doesn't guarantee a reliable measure of confidence, highlighting the need for better calibration.
- Surgical Pretraining
- This involves training AI models on large datasets of surgical videos before they tackle the specific challenge. Using surgical or self-supervised pretraining was found to be a common trait among top performers, suggesting that having prior surgical knowledge helps models perform better in quality assessment tasks.
Terminology
Summary
Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation The SAGES Critical View of Safety (CVS) Challenge presents the first AI competition organized by a surgical society using the CVS in laparoscopic cholecystectomy as an exemplar of surgical quality assessment. p. 0 Gemelli IRCCS, Italy
zUniversity of Pennsylvania, USA
aaDuke University, USA
A B S T R A C T
Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation. This study presents the SAGES Critical View of Safety (CVS) Challenge, the first
AI competition organized by a surgical society, using the CVS in laparoscopic cholecystectomy, a universally recommended yet
inconsistently performed safety step, as an exemplar of surgical quality assessment.Where the paper says it
Challenge Scope and Data Curation
The study established the SAGES Critical View of Safety (CVS) Challenge, which involved a global collaboration across 54 institutions in 24 countries engaging hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol. The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability. p. 0 A global collaboration across 54 institutions in
24 countries engaged hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a
consensus-validated protocol. The challenge addressed key barriers to real-world deployment in surgery, including achieving high
performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability.Where the paper says it
Infrastructure for Large-Scale Data Management
To manage the large and heterogeneous surgical video and multi-annotator workflows at a global scale, the EndoGlacier framework was developed. This framework coordinated three streams: (i) annotator management, (ii) orchestration and automation, and (iii) video management. p. 3 Coordinated Data Flows for Large-Scale Annotation
Managing diverse dataset generation at a global scale requires more than assembling data, it demands infrastructure capable of sustaining continuous, high-quality flows of videos, expert annotations, and coordination across dozens of institutions
and contributors. Here, we break down our dataset generation
process into three coordinated streams visualized in Figure 2:
(1) annotator management, (2) video management, and (3) orchestration and automation. As a final gate, we
implemented a competency-based exam requiring at least 75%
correspondence with expert ratings to ensure conceptual alignment and minimum annotation quality. Through this structured pipeline, we contacted or were contacted by 106 potential annotators, recruited through channels
including social media and in-person onboarding campaigns.Where the paper says it
Where the paper says it
Benchmark Task and Dataset Characteristics
The benchmark task focused on assessing the Critical View of Safety (CVS) achievement during the 90-second window immediately preceding clipping of the cystic duct or artery. The dataset comprises 1,000 laparoscopic cholecystectomy cases, with each represented by a 90-second clip annotated at one frame every five seconds. p. 5 This design
provides both temporal resolution and practical annotation density, aligning the benchmark with the real-world clinical decision context. Dataset Characteristics
The SAGES CVS Challenge dataset comprises a total of
1,000 laparoscopic cholecystectomy cases, each represented by
a 90-second clip taken from the critical decision window preceding clipping of the cystic duct or artery. The dataset captures substantial clinical and technical diversity. 23 countries are represented in the training split and 18
countries in the test split, with an average of 30.Where the paper says it
Where the paper says it
Evaluation Metrics and Performance Trends
Participants were evaluated across three subchallenges: Subchallenge A (CVS Achievement), Subchallenge B (Uncertainty Quantification), and Subchallenge C (Domain Shift Robustness). Top-performing entries showed headline gains over the LG-DG baseline across all three subchallenges, including a 17% relative improvement in robustness in Subchallenge C. p. 1 Participants were evaluated across three
complementary dimensions essential for clinical deployment—absolute performance requirements, alignment with expert uncertainty, and robustness under real-world domain shifts. Submissions from 13 international teams delivered measurable gains over state-of-the-art in all three areas, offering both
methodological advances and insight into the design choices
that influence performance. # Team A
ffiliation(s) Architecture Temporal
Component
Auxiliary Objective Pretraining
1 TUE-VCA Eindhoven University of Technology
(Netherlands)
PVT-v2 (x5 ensemble) None L2 reconstruction loss Self-supervised (DINO on SurgeNet and
GenSurgery datasets)
2 Farm Stanford University (USA) DINOv2 (giant, large), SigLIP, ConvNeXt2-L, InternImage Stacked dilated
TCN
None Self-supervised (DINO, SigLIP)
3 CVS HUST Huazhong University of Science and
Technology (China)
ConvNeXt + MLP LSTM ℓ1 loss for confidence
prediction
Supervised (natural image weights)
4 SDS-HD German Cancer Research Center
(DKFZ) (Germany)
EVA02-Large Transformer None Segmentation loss on
YOLOv8 pseudolabels
Endoscapes-pretrained Endoscapes)
Self-supervised (MoCov2 on Cholec80, HeiChole,
5 SRV-WEISS University College London (UK) EndoViT + Dense Prediction Transformer, E
fficientNet, ConvNeXt None Segmentation +
classification fusion
Self-Supervised (EndoViT) + Supervised
E
(EndoViT on Endoscapes segmentation;
fficientNet on Endoscapes classes)
6 Caresyntax Caresyntax DenseNet-121 (5× ensemble) None None Supervised (ImageNet weights)
8 HUFT-MedIA Hefei University of Technology (China) Vision Transformer (ViT)
7 FightTumor University Health Network (Canada) ConvNeXt-B None None Supervised (ImageNet weights)
+ MLP None None Supervised (ImageNet weights)
9 IRCV-URV Universitat Rovira i Virgili (Spain) E
fficientNet-B5 + FPN + spatial attention (5x ensemble) None None Supervised (ImageNet weights)
10 mmll NAAMII (Nepal) 1. ResNet-50 + Mask R-CNN + Transformer encoder (temporal 10-frame)
2.Where the paper says it
Where the paper says it
Key Methodological Insights
Analysis revealed that while architectures like Transformer or hybrid transformer–CNN models dominated the very top, leveraging surgical or self-supervised surgical pretraining was common among accuracy/robustness leaders. Ensembling was frequently present among Subchallenge A and C leaders but rare among calibration leaders. The results underscored that optimizing for discriminative performance does not inherently yield confidence alignment, indicating that calibration requires targeted choices. p. 7 This design reflected real-world deployment constraints: in surgical AI systems, raw predictive performance, reliability under distribution shifts, and appropriate uncertainty estimation are all critical, yet not inherently aligned. Optimizing for one may compromise another.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI system can achieve high performance in predicting majority expert assessment for each of the three CVS criteria by utilizing a mode-of-raters label incorporating annotator confidence: y = 1/3 sum(li - 0.5) · ci, where li is the binary label and ci is the confidence score provided by annotator i.
-
Improved AI system can reduce Brier error by over 80% compared to the LG-DG baseline in uncertainty quantification tasks through the use of auxiliary objectives or multi-task formulations, as seen in top performers like SDS-HD and Theator.
-
Improved AI system can enhance domain shift robustness by achieving a 17% relative gain in robustness over state-of-the-art by utilizing methods that explicitly account for the inherent ambiguity in surgical quality assessments under distribution shifts related to device characteristics, case complexity, and regional differences in technique.
Abstract
Advances in artificial intelligence (AI) for surgical quality assessment promise to democratize access to expertise, with applications in training, guidance, and accreditation. This study presents the SAGES Critical View of Safety (CVS) Challenge, the first AI competition organized by a surgical society, using the CVS in laparoscopic cholecystectomy, a universally recommended yet inconsistently performed safety step, as an exemplar of surgical quality assessment. A global collaboration across 54 institutions in 24 countries engaged hundreds of clinicians and engineers to curate 1,000 videos annotated by 20 surgical experts according to a consensus-validated protocol. The challenge addressed key barriers to real-world deployment in surgery, including achieving high performance, capturing uncertainty in subjective assessment, and ensuring robustness to clinical variability. To enable this scale of effort, we developed EndoGlacier, a framework for managing large, heterogeneous surgical video and multi-annotator workflows. Thirteen international teams participated, achieving up to a 17% relative gain in assessment performance, over 62% reduction in calibration error, and a 17% relative improvement in robustness over the state-of-the-art. Analysis of results highlighted methodological trends linked to model performance, providing guidance for future research toward robust, clinically deployable AI for surgical quality assessment.
Sources
- 2018 Robotic Scene Segmentation Challenge
- 2017 Robotic Instrument Segmentation Challenge
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- FetReg: Placental Vessel Segmentation and Registration in Fetoscopy Challenge Dataset
- Feature Mixing Approach for Detecting Intraoperative Adverse Events in Laparoscopic Roux-en-Y Gastric Bypass Surgery
- Improved Baselines with Momentum Contrastive Learning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- 2020 CATARACTS Semantic Segmentation Challenge
- Surgical data science for safe cholecystectomy: a protocol for segmentation of hepatocystic anatomy and assessment of the critical view of safety
- MOSaiC: a Web-based Platform for Collaborative Medical Video Assessment and Annotation
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark
- DINOv2: Learning Robust Visual Features without Supervision
- Optimizing Latent Graph Representations of Surgical Scenes for Zero-Shot Domain Transfer
- General surgery vision transformer: A video pre-trained foundation model for general surgery
- Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs