CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme
summary
The gist
Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC)…
In short
The CURVAS-PDACVI challenge tested AI models to accurately segment pancreatic tumors and assess vascular invasion using a dataset annotated by five independent experts per scan. Results showed that high overall segmentation accuracy does not guarantee reliable detection of critical tumor-vessel interfaces. The study concludes that uncertainty-aware methods, which explicitly model expert disagreement, are more robust in ambiguous clinical cases.
Key concepts
- Vascular Invasion (VI)
- This refers to whether a pancreatic tumor has grown into or invaded nearby blood vessels. It is a crucial factor in determining if the cancer is surgically resectable. Accurate identification of these interfaces is vital for surgical planning.
- CURVAS-PDACVI Dataset
- This benchmark dataset was created by gathering CT scans and having five different board-certified radiologists annotate them independently. This multi-rater approach provides a richer, more realistic ground truth than single expert annotations alone.
- Uncertainty-Aware AI
- This involves developing AI models that don't just give a single answer but also quantify how confident they are in that answer. By modeling the uncertainty derived from multiple expert opinions, these models can better reflect the true ambiguity present in medical images.
Terminology used across episodes
This episode discusses
- CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme · Paper Radio
- TwinTrack: Post-hoc Multi-Rater Calibration for Medical Image Segmentation
The paper
CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme · Read on arXiv
M. Riera-Marána, O. K. Sikhab, J. Rodríguez-Comasa, M. S. Mayc, T. Kirschere, X. Coubezf, P. Meyere, S. Faisane, Í Pangh
Sycaitechnologies SL · BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · Ániversit¨atsklinikum Erlangen, Department of Radiology of the Uniklinikum Erlangen (UKER), Erlangen, Germany · University Hospital Erlangen, Imaging Science Institute, Erlangen, Germany · ICUBE Laboratory, CNRS UMR-7357, University of Strasbourg · CLCC Institut-Strauss, Strasbourg · Shenzhen Institute of Advanced Technology Chinese Academy of Sciences Shenzhen China · University of Chinese Academy of Sciences Beijing China · Í Université de Rennes 1 CLCC Eug`ene Marquis and INSERM UMR 1099 LTSI Rennes France · German Cancer Research Center DKFZ Medical Image Computing E230 Heidelberg Germany · IMT Atlantique LaTIM UMR 1101 Inserm Brest France · Hospital de Mataróo Mataróo Spain · Hospital de Sant Pau i la Santa Creu Diagnostic Imaging Department Barcelona Spain · Í Institut de Recerca Sant Pau CERCA Advanced Medical Imaging Artificial Intelligence and Imaging-Guided Therapy Research Group Barcelona Spain · Institució Catalana de Recerca i Estudis Avançats ICREA Barcelona Spain · TECNALIA Basque Research and Technology Alliance BRTA Bizkaia Spain
Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI), i.e., tumor extension into adjacent critical vessels. Despite its importance for preoperative staging and surgical planning, computational VI assessment remains underexplored. Two major challenges are the lack of public datasets and the diagnostic ambiguity at the tumor-vessel interface, which leads to substantial inter-rater variability even among expert radiologists. To address these limitations, we introduce the CURVAS-PDACVI Dataset and Challenge, an open benchmark for uncertainty-aware AI in PDAC staging based on a densely annotated dataset with five independent expert annotations per scan. We also propose a multi-metric evaluation framework that extends beyond spatial overlap to include probabilistic calibration and VI assessment. Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. In particular, methods optimized for binary segmentation perform competitively on average overlap metrics, but often degrade in high-complexity cases with low expert consensus, either collapsing in volume or overextending at uncertain boundaries. In contrast, methods that model inter-rater disagreement produce better calibrated probabilistic maps and show greater robustness in these ambiguous cases. The benchmark highlights the limitations of volumetric accuracy as a proxy for localized surgical utility, motivating uncertainty-aware probabilistic models for preoperative decision-making.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme".
Paper summary: Tom: So we’re looking at this paper today called CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme. The main thing here is that surgical resection is the only potentially curative treatment for pancreatic ductal adenocarcinoma, but eligibility depends on accurately assessing vascular invasion, or VI <ref:2604.27582#pg2>. This paper sets up a benchmark for uncertainty-aware AI in staging PDAC based on a dataset with five independent expert annotations per scan <ref:2604.27582#pg3>.
Jane: That sounds really important because, even though we know VI is key for surgical planning, computational assessment of it is still underexplored <ref:2604.27582#pg2>. So what this challenge introduces is this open benchmark using a densely annotated dataset where five different experts annotate every scan <ref:2604.27582#pg3>.
Lu: The core idea they are pushing is that strong global volumetric overlap doesn't necessarily mean reliable performance at the critical tumor-vessel interfaces <ref:2604.27582#pg3>. They found that this makes volumetric accuracy a poor proxy for what actually matters for local surgical utility <ref:2604.27582#pg3>.
Meng: From an engineering standpoint, I'm curious how they handle the sheer volume of expert input when you're trying to build a robust model <ref:2604.27582#pg3>. How do you manage all that multi-rater supervision at scale?
Lalam: The dataset construction involved taking raw CT data from the PANORAMA collection and adding initial single-rater annotations <ref:2604.27582#pg3>. They selected a subset of one hundred twenty-five CT volumes, making sure they had a diagnosis, and then distributed those to four more board certified radiologists from different hospitals like Universitatsklinikum Erlangen and Hospital de Sant Pau <ref:2604.27582#pg3>.
Tom: And then they brought in a fifth radiologist specifically to refine the vascular annotations, separating them into five distinct parts: the Aorta, Celiac Trunk, Porta, Superior Mesenteric Vein and Superior Mesenteric Artery <ref:2604.27582#pg3>. That level of detail is crucial for assessing VI correctly.
Jane: Right. When we look at how they measured performance, they used a few different families of metrics to check things, including segmentation quality, multi-rater calibration, probabilistic volume assessment, and vascular invasion analysis <ref:2604.27582#pg3>. They used Dice based measures like the conventional Dice Similarity Coefficient and a threshold averaged Dice score <ref:2604.27582#pg3>.
Lu: And for calibration, they extended the Expected Calibration Error by comparing each probabilistic prediction against each of the five expert annotations <ref:2604.27582#pg3>. That shows they are really focused on how well the model's confidence matches what the human experts are actually seeing.
Paper summary: Meng: What about volume estimation? I need to know if this is just a rough approximation or if it’s actually predictive of surgical risk <ref:2604.27582#pg3>? They used something called the Continuous Ranked Probability Score, or CRPS, which measures how well the predicted tumor volume agrees with the distribution of volumes induced by those multiple expert annotations <ref:2604.27582#pg3>.
Lalam: The methods they tested were all based on variants of the nnU-Net framework but they differed in how they used multi-rater supervision and modeled predictive uncertainty <ref:2604.27582#pg3>. For example, TwinTrack used a two stage nnU-Net cascade followed by post hoc calibration using isotonic regression to align predicted probabilities with the mean human consensus <ref:2604.27582#pg3>.
Tom: CorpuSeg used a low resolution nnU-Net for coarse localization and then five second stage full resolution models, getting the final predictions by voxel wise averaging of those five probabilistic outputs <ref:2604.27582#pg3>. That sounds like a lot of complexity to manage during inference.
Jane: BreizhSeg took a deterministic model and converted it into an Adaptable Bayesian Neural Network using Bayesian Normalization Layers, using multiple stochastic forward passes to approximate the posterior predictive distribution <ref:2604.27582#pg3>. That’s a heavy lift in terms of modeling uncertainty.
Lu: MIC DKFZ pretrained an nnU-Net ResEnc L model and fine tuned an ensemble of five models on CURVAS-PDACVI, applying temperature scaling before ensembling <ref:2604.27582#pg3>. And ROISeg trained a full resolution nnU-Net directly on the STAPLE consensus labels, which avoids multi stage processing but discards explicit information about inter rater disagreement <ref:2604.27582#pg3>.
Meng: OrdSTAPLE explicitly modeled rater disagreement through an ordinal formulation, combining a standard STAPLE model with a second model trained to predict increasing levels of annotator agreement <ref:2604.27582#pg3>. That seems like the most direct way to capture the human confusion.
Tom: So what did they find when they compared all these approaches? The benchmark revealed that strong global segmentation performance doesn't translate into accurate assessment of localized vascular involvement, especially for venous structures <ref:2604.27582#pg3>.
Jane: They specifically pointed out that methods like BreizhSeg and MIC DKFZ performed strongly under global overlap metrics, whereas TwinTrack and CorpuSeg were more competitive at the vascular interface <ref:2604.27582#pg3>. That means the way they look globally isn't what matters for the actual tumor boundary <ref:2604.27582#pg3>.
Paper summary: Lu: When they looked at high complexity cases with low human inter rater agreement, OrdSTAPLE became the top performing method on that cohort <ref:2604.27582#pg3>. This suggests that explicitly modeling disagreement seems to improve robustness in those most ambiguous cases where rigid binary segmentation tends to either collapse toward under segmentation or overextend at critical tumor vessel boundaries <ref:2604.27582#pg3>.
Tom: So what does this mean for a surgeon who only listens to the show? It means an over confident false positive vascular contact might incorrectly suggest unresectability, whereas disagreement aware approaches generate more diffuse probability maps that reflect the uncertainty present in those annotations <ref:2604.27582#pg3>.
Jane: Exactly. The challenge here isn't just recovering the tumor volume, but doing so in a way that remains calibrated and anatomically reliable at those critical boundaries <ref:2604.27582#pg3>. It moves the focus from just getting a big number to getting a trustworthy map of where we can actually operate safely.
Lu: The paper also flags some limitations, including the relatively small size of the final test cohort which limits statistical power for pairwise comparisons <ref:2604.27582#pg3>. Plus, they note that because the ground truth comes from expert radiological annotations and not surgical mapping, that reference standard inherits its own uncertainty and potential bias <ref:2604.27582#pg3>.
Meng: And another thing is that the limited architectural diversity in submissions means the benchmark primarily captures differences in training strategy and uncertainty modeling rather than the broader design space of modern segmentation architectures <ref:2604.27582#pg3>. So we’re mostly seeing how they train, not a complete picture of what deep learning can do for this task.
Tom: It sounds like the main challenge in VI focused PDAC segmentation isn't just recovering tumor volume, but doing so in a way that remains calibrated and anatomically reliable at clinically critical boundaries <ref:2604.27582#pg3>. This paper supports evaluating methods not only through aggregate overlap but also through uncertainty aware multimetric frameworks that better reflect downstream clinical goals <ref:2604.27582#pg3>.
Jane: That points toward a future where we need evaluation protocols that better link radiological ambiguity to actual clinical decision making <ref:2604.27582#pg3>. So the next step is moving beyond just aggregate overlap and focusing on those uncertainty aware frameworks <ref:2604.27582#pg3>.
Conclusion: Tom: So we're wrapping up our discussion on this CURVAS PDACVI challenge paper <ref:2604.27582#pg3> and what it actually means for pancreatic cancer surgery right now <ref:2604.27582#pg3>.
Jane: It boils down to this: we're talking about getting the AI right on where the blood vessels are invading a tumor <ref:2604.27582#pg3>, which is the key for surgeons to decide if they can actually remove it safely <ref:2604.27582#pg3>.
Lu: The authors are showing that just getting a good overall picture of the tumor volume isn't enough; you need the AI to be precise about those tiny spots where the vessel touches the tumor boundary <ref:2604.27582#pg3>.
Meng: From my side, what this means is that we can't just train models on general scans and expect them to handle that specific, tricky interface reliably in a real hospital setting <ref:2604.27582#pg3>.
Lalam: My literature sweep confirms what the authors found: methods that explicitly model how different experts disagree tend to be more robust when the cases are really hard and confusing <ref:2604.27582#pg3>.
Tom: So the main point of this whole benchmark is showing a clear split between models that just get high overlap scores and models that actually work when it comes to those critical vessel interfaces <ref:2604.27582#pg3>.
Jane: Exactly, and it suggests we need to look at these uncertainty aware frameworks rather than just chasing the highest Dice score on a global measure <ref:2604.27582#pg3>.
Lu: The implication is that for preoperative decisions, an overconfident prediction of vascular contact could be a problem, whereas a model showing high uncertainty in those exact spots gives the surgeon more caution <ref:2604.27582#pg3>.
Meng: But we have to remember the authors themselves point out the limitations, like having a relatively small test group which limits how much we can really compare these different strategies directly <ref:2604.27582#pg3>.
Lalam: And they also flag that because their ground truth comes from human expert annotations and not from surgery or pathology, the reference standard itself carries some inherent uncertainty with it <ref:2604.27582#pg3>.
Tom: It really puts the focus on building tools that are calibrated and reliable at those hard boundaries, not just tools that give a big volume number <ref:2604.27582#pg3>.
Jane: Right, so the next thing we need to watch is how we translate this uncertainty into something a surgeon can actually use in their workflow <ref:2604.27582#pg3>.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound