CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme".
Paper summary: Tom: So we’re looking at this paper today called CURVAS-PDACVI: A Multi-Rater CT Dataset and Challenge for Pancreatic Cancer Segmentation and Vascular Involvement Assessme. The main thing here is that surgical resection is the only potentially curative treatment for pancreatic ductal adenocarcinoma, but eligibility depends on accurately assessing vascular invasion, or VI <ref:2604.27582#pg2>. This paper sets up a benchmark for uncertainty-aware AI in staging PDAC based on a dataset with five independent expert annotations per scan <ref:2604.27582#pg3>.
Jane: That sounds really important because, even though we know VI is key for surgical planning, computational assessment of it is still underexplored <ref:2604.27582#pg2>. So what this challenge introduces is this open benchmark using a densely annotated dataset where five different experts annotate every scan <ref:2604.27582#pg3>.
Lu: The core idea they are pushing is that strong global volumetric overlap doesn't necessarily mean reliable performance at the critical tumor-vessel interfaces <ref:2604.27582#pg3>. They found that this makes volumetric accuracy a poor proxy for what actually matters for local surgical utility <ref:2604.27582#pg3>.
Meng: From an engineering standpoint, I'm curious how they handle the sheer volume of expert input when you're trying to build a robust model <ref:2604.27582#pg3>. How do you manage all that multi-rater supervision at scale?
Lalam: The dataset construction involved taking raw CT data from the PANORAMA collection and adding initial single-rater annotations <ref:2604.27582#pg3>. They selected a subset of one hundred twenty-five CT volumes, making sure they had a diagnosis, and then distributed those to four more board certified radiologists from different hospitals like Universitatsklinikum Erlangen and Hospital de Sant Pau <ref:2604.27582#pg3>.
Tom: And then they brought in a fifth radiologist specifically to refine the vascular annotations, separating them into five distinct parts: the Aorta, Celiac Trunk, Porta, Superior Mesenteric Vein and Superior Mesenteric Artery <ref:2604.27582#pg3>. That level of detail is crucial for assessing VI correctly.
Jane: Right. When we look at how they measured performance, they used a few different families of metrics to check things, including segmentation quality, multi-rater calibration, probabilistic volume assessment, and vascular invasion analysis <ref:2604.27582#pg3>. They used Dice based measures like the conventional Dice Similarity Coefficient and a threshold averaged Dice score <ref:2604.27582#pg3>.
Lu: And for calibration, they extended the Expected Calibration Error by comparing each probabilistic prediction against each of the five expert annotations <ref:2604.27582#pg3>. That shows they are really focused on how well the model's confidence matches what the human experts are actually seeing.
Paper summary: Meng: What about volume estimation? I need to know if this is just a rough approximation or if it’s actually predictive of surgical risk <ref:2604.27582#pg3>? They used something called the Continuous Ranked Probability Score, or CRPS, which measures how well the predicted tumor volume agrees with the distribution of volumes induced by those multiple expert annotations <ref:2604.27582#pg3>.
Lalam: The methods they tested were all based on variants of the nnU-Net framework but they differed in how they used multi-rater supervision and modeled predictive uncertainty <ref:2604.27582#pg3>. For example, TwinTrack used a two stage nnU-Net cascade followed by post hoc calibration using isotonic regression to align predicted probabilities with the mean human consensus <ref:2604.27582#pg3>.
Tom: CorpuSeg used a low resolution nnU-Net for coarse localization and then five second stage full resolution models, getting the final predictions by voxel wise averaging of those five probabilistic outputs <ref:2604.27582#pg3>. That sounds like a lot of complexity to manage during inference.
Jane: BreizhSeg took a deterministic model and converted it into an Adaptable Bayesian Neural Network using Bayesian Normalization Layers, using multiple stochastic forward passes to approximate the posterior predictive distribution <ref:2604.27582#pg3>. That’s a heavy lift in terms of modeling uncertainty.
Lu: MIC DKFZ pretrained an nnU-Net ResEnc L model and fine tuned an ensemble of five models on CURVAS-PDACVI, applying temperature scaling before ensembling <ref:2604.27582#pg3>. And ROISeg trained a full resolution nnU-Net directly on the STAPLE consensus labels, which avoids multi stage processing but discards explicit information about inter rater disagreement <ref:2604.27582#pg3>.
Meng: OrdSTAPLE explicitly modeled rater disagreement through an ordinal formulation, combining a standard STAPLE model with a second model trained to predict increasing levels of annotator agreement <ref:2604.27582#pg3>. That seems like the most direct way to capture the human confusion.
Tom: So what did they find when they compared all these approaches? The benchmark revealed that strong global segmentation performance doesn't translate into accurate assessment of localized vascular involvement, especially for venous structures <ref:2604.27582#pg3>.
Jane: They specifically pointed out that methods like BreizhSeg and MIC DKFZ performed strongly under global overlap metrics, whereas TwinTrack and CorpuSeg were more competitive at the vascular interface <ref:2604.27582#pg3>. That means the way they look globally isn't what matters for the actual tumor boundary <ref:2604.27582#pg3>.
Paper summary: Lu: When they looked at high complexity cases with low human inter rater agreement, OrdSTAPLE became the top performing method on that cohort <ref:2604.27582#pg3>. This suggests that explicitly modeling disagreement seems to improve robustness in those most ambiguous cases where rigid binary segmentation tends to either collapse toward under segmentation or overextend at critical tumor vessel boundaries <ref:2604.27582#pg3>.
Tom: So what does this mean for a surgeon who only listens to the show? It means an over confident false positive vascular contact might incorrectly suggest unresectability, whereas disagreement aware approaches generate more diffuse probability maps that reflect the uncertainty present in those annotations <ref:2604.27582#pg3>.
Jane: Exactly. The challenge here isn't just recovering the tumor volume, but doing so in a way that remains calibrated and anatomically reliable at those critical boundaries <ref:2604.27582#pg3>. It moves the focus from just getting a big number to getting a trustworthy map of where we can actually operate safely.
Lu: The paper also flags some limitations, including the relatively small size of the final test cohort which limits statistical power for pairwise comparisons <ref:2604.27582#pg3>. Plus, they note that because the ground truth comes from expert radiological annotations and not surgical mapping, that reference standard inherits its own uncertainty and potential bias <ref:2604.27582#pg3>.
Meng: And another thing is that the limited architectural diversity in submissions means the benchmark primarily captures differences in training strategy and uncertainty modeling rather than the broader design space of modern segmentation architectures <ref:2604.27582#pg3>. So we’re mostly seeing how they train, not a complete picture of what deep learning can do for this task.
Tom: It sounds like the main challenge in VI focused PDAC segmentation isn't just recovering tumor volume, but doing so in a way that remains calibrated and anatomically reliable at clinically critical boundaries <ref:2604.27582#pg3>. This paper supports evaluating methods not only through aggregate overlap but also through uncertainty aware multimetric frameworks that better reflect downstream clinical goals <ref:2604.27582#pg3>.
Jane: That points toward a future where we need evaluation protocols that better link radiological ambiguity to actual clinical decision making <ref:2604.27582#pg3>. So the next step is moving beyond just aggregate overlap and focusing on those uncertainty aware frameworks <ref:2604.27582#pg3>.
Conclusion: Tom: So we're wrapping up our discussion on this CURVAS PDACVI challenge paper <ref:2604.27582#pg3> and what it actually means for pancreatic cancer surgery right now <ref:2604.27582#pg3>.
Jane: It boils down to this: we're talking about getting the AI right on where the blood vessels are invading a tumor <ref:2604.27582#pg3>, which is the key for surgeons to decide if they can actually remove it safely <ref:2604.27582#pg3>.
Lu: The authors are showing that just getting a good overall picture of the tumor volume isn't enough; you need the AI to be precise about those tiny spots where the vessel touches the tumor boundary <ref:2604.27582#pg3>.
Meng: From my side, what this means is that we can't just train models on general scans and expect them to handle that specific, tricky interface reliably in a real hospital setting <ref:2604.27582#pg3>.
Lalam: My literature sweep confirms what the authors found: methods that explicitly model how different experts disagree tend to be more robust when the cases are really hard and confusing <ref:2604.27582#pg3>.
Tom: So the main point of this whole benchmark is showing a clear split between models that just get high overlap scores and models that actually work when it comes to those critical vessel interfaces <ref:2604.27582#pg3>.
Jane: Exactly, and it suggests we need to look at these uncertainty aware frameworks rather than just chasing the highest Dice score on a global measure <ref:2604.27582#pg3>.
Lu: The implication is that for preoperative decisions, an overconfident prediction of vascular contact could be a problem, whereas a model showing high uncertainty in those exact spots gives the surgeon more caution <ref:2604.27582#pg3>.
Meng: But we have to remember the authors themselves point out the limitations, like having a relatively small test group which limits how much we can really compare these different strategies directly <ref:2604.27582#pg3>.
Lalam: And they also flag that because their ground truth comes from human expert annotations and not from surgery or pathology, the reference standard itself carries some inherent uncertainty with it <ref:2604.27582#pg3>.
Tom: It really puts the focus on building tools that are calibrated and reliable at those hard boundaries, not just tools that give a big volume number <ref:2604.27582#pg3>.
Jane: Right, so the next thing we need to watch is how we translate this uncertainty into something a surgeon can actually use in their workflow <ref:2604.27582#pg3>.
M. Riera-Marána, O. K. Sikhab, J. Rodríguez-Comasa, M. S. Mayc, T. Kirschere, X. Coubezf, P. Meyere, S. Faisane, Í Pangh
Sycaitechnologies SL · BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · Ániversit¨atsklinikum Erlangen, Department of Radiology of the Uniklinikum Erlangen (UKER), Erlangen, Germany · University Hospital Erlangen, Imaging Science Institute, Erlangen, Germany · ICUBE Laboratory, CNRS UMR-7357, University of Strasbourg · CLCC Institut-Strauss, Strasbourg · Shenzhen Institute of Advanced Technology Chinese Academy of Sciences Shenzhen China · University of Chinese Academy of Sciences Beijing China · Í Université de Rennes 1 CLCC Eug`ene Marquis and INSERM UMR 1099 LTSI Rennes France · German Cancer Research Center DKFZ Medical Image Computing E230 Heidelberg Germany · IMT Atlantique LaTIM UMR 1101 Inserm Brest France · Hospital de Mataróo Mataróo Spain · Hospital de Sant Pau i la Santa Creu Diagnostic Imaging Department Barcelona Spain · Í Institut de Recerca Sant Pau CERCA Advanced Medical Imaging Artificial Intelligence and Imaging-Guided Therapy Research Group Barcelona Spain · Institució Catalana de Recerca i Estudis Avançats ICREA Barcelona Spain · TECNALIA Basque Research and Technology Alliance BRTA Bizkaia Spain
cs.CV
Submitted: 2026-04-30
Updated: 2026-11-18
Code: https://github.com/SYCAI-Technologies/curvas-challenge
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 84/100
The gist: Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC)…
Key concepts
- Vascular Invasion (VI)
- This refers to whether a pancreatic tumor has grown into or invaded nearby blood vessels. It is a crucial factor in determining if the cancer is surgically resectable. Accurate identification of these interfaces is vital for surgical planning.
- CURVAS-PDACVI Dataset
- This benchmark dataset was created by gathering CT scans and having five different board-certified radiologists annotate them independently. This multi-rater approach provides a richer, more realistic ground truth than single expert annotations alone.
- Uncertainty-Aware AI
- This involves developing AI models that don't just give a single answer but also quantify how confident they are in that answer. By modeling the uncertainty derived from multiple expert opinions, these models can better reflect the true ambiguity present in medical images.
Terminology
Summary
Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI) <ref:2604. p. 0 Gonzalez-Ballester ´
b,o, A. Galdranp
aSycai Technologies SL, Scientific and Technical Department, Barcelona, Spain
bBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain
cUniversit¨atsklinikum Erlangen, Department of Radiology of the Uniklinikum Erlangen (UKER), Erlangen, Germany
dUniversity Hospital Erlangen, Imaging Science Institute, Erlangen, Germany
e
ICUBE Laboratory, CNRS UMR-7357, University of Strasbourg, Strasbourg, France
fCLCC Institut-Strauss, Strasbourg, France
gShenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
hUniversity of Chinese Academy of Sciences, Beijing, China
iUniversit´e de Rennes 1, CLCC Eug`ene Marquis, and INSERM UMR 1099 LTSI, Rennes, France
jGerman Cancer Research Center (DKFZ), Medical Image Computing (E230), Heidelberg, Germany
k
IMT Atlantique, LaTIM UMR 1101, Inserm, Brest, France
lHospital de Matar´o, Matar´o, Spain
mHospital de Sant Pau i la Santa Creu, Diagnostic Imaging Department, Barcelona, Spain
n
Institut de Recerca Sant Pau - CERCA, Advanced Medical Imaging, Artificial Intelligence, and Imaging-Guided Therapy Research Group, Barcelona, Spain
o
Instituci´o Catalana de Recerca i Estudis Avan¸cats (ICREA), Barcelona, Spain
pTECNALIA, Basque Research and Technology Alliance (BRTA), Bizkaia, Spain
Abstract
Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI), i. , tumor extension into adjacent critical vessels. Despite its importance
for preoperative staging and surgical planning, computational VI assessment remains underexplored. Two major challenges are
the lack of public datasets and the diagnostic ambiguity at the tumor-vessel interface, which leads to substantial inter-rater variability even among expert radiologists. To address these limitations, we introduce the CURVAS-PDACVI Dataset and Challenge,
an open benchmark for uncertainty-aware AI in PDAC staging based on a densely annotated dataset with five independent expert
annotations per scan. We also propose a multi-metric evaluation framework that extends beyond spatial overlap to include probabilistic calibration and VI assessment. Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does
not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. In contrast, methods that model inter-rater
disagreement produce better calibrated probabilistic maps and show greater robustness in these ambiguous cases. The benchmark
highlights the limitations of volumetric accuracy as a proxy for localized surgical utility, motivating uncertainty-aware probabilistic
models for preoperative decision-making.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Dataset Construction and Annotation Process
The CURVAS-PDACVI dataset was constructed using raw CT data from the PANORAMA collection together with initial single-rater annotations <ref:2604.27582#pg2> A subset of 125 CT volumes was selected, ensuring no overlap with other public datasets, discarding any case with no pathologically confirmed diagnosis, and keeping only fully-manual tumor and pancreas annotations <ref:2604. p. 2 Multi-Rater Annotations in CURVAS-PDACVI
Raw CT data for this challenge was sourced from the
PANORAMA collection [22], together with initial single-rater
annotations. A subset of 125 CT volumes was selected, ensuring no overlap with other public datasets, discarding any
case with no pathologically confirmed diagnosis, and keeping
only fully-manual tumor and pancreas annotations. A subset of 125 CT volumes was selected, ensuring no overlap with other public datasets, discarding any
case with no pathologically confirmed diagnosis, and keeping
only fully-manual tumor and pancreas annotations. After curation, data was distributed to four additional board-certified
radiologists with varying levels of experience from Universitatsklinikum Erlangen, Hospital de Sant Pau, and Hospital de ¨
Mataro. In addition to re-annotating pancreas and tumoral lesions,
a fifth radiologist refined the semi-automatic vascular annotations originally provided in the PANORAMA dataset. Specifically, the vascular structures were separated into five distinct
anatomical components: the Aorta, Celiac Trunk, Porta, Superior Mesenteric Vein (SMV), and Superior Mesenteric Artery
(SMA), and manual completion of partial segments was conducted. This is critical to enable a detailed assessment of
vascular invasion (VI) for each vessel independently, allowing
structure-specific quantitative evaluation. Eventually, participants were provided with the original CT
volume and PANORAMA annotations (PDAC and pancreatic
parenchyma), together with the four additional PDAC annotations, five vascular delineations, and a STAPLE consensus label
of the tumoral area.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Evaluation Framework Metrics
The performance assessment utilized four complementary metric families: segmentation quality, multi-rater calibration, probabilistic volume assessment, and vascular invasion (VI) analysis <ref:2604.27582#pg2> Segmentation quality was quantified using two Dice-based measures: a conventional Dice Similarity Coefficient (DSC) and a threshold-averaged Dice score (Thr-DSC) <ref:2604. p. 3 Segmentation quality was quantified using two Dice-based
measures. First, a conventional Dice Similarity Coefficient
(DSC) was computed between each binarized prediction and a
binarized STAPLE consensus mask derived from the five expert
annotations. To assess calibration, we used a multi-rater extension of the
Expected Calibration Error (MR-ECE), computed by comparing each probabilistic prediction against each of the five expert annotations and averaging the resulting calibration errors. In addition, probabilistic volume estimation was evaluated using the Continuous Ranked Probability Score (CRPS), which
measures how well the predicted tumor volume agrees with the
distribution of volumes induced by the multiple expert annotations. To assess calibration, we used a multi-rater extension of the
Expected Calibration Error (MR-ECE), computed by comparing each probabilistic prediction against each of the five expert annotations and averaging the resulting calibration errors. In addition, probabilistic volume estimation was evaluated using the Continuous Ranked Probability Score (CRPS), which
measures how well the predicted tumor volume agrees with the
distribution of volumes induced by the multiple expert annotations. Finally, because the clinical relevance of PDAC segmentation lies largely in the assessment of tumor-vessel interfaces,
vascular invasion was evaluated independently for the Porta,
SMV, SMA, Celiac Trunk, and Aorta. For each structure, predicted and reference invasion scores were represented as distributions derived from multiple thresholds and multiple annotators, respectively, and compared using a Wasserstein-based distance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Methodological Approaches in the Challenge
Six unique algorithms were evaluated, all based on variants of the nnU-Net framework but differing substantially in how they incorporated multi-rater supervision and modeled predictive uncertainty <ref:2604.27582#pg2> TwinTrack adopted a two-stage nnU-Net cascade followed by post-hoc calibration using isotonic regression to align predicted probabilities with the mean human consensus <ref:2604. p. 3 A
first low-resolution stage localized the pancreas, tumor, and
surrounding vascular anatomy, while a second high-resolution
stage refined the prediction within a region of interest using a
deep ensemble [25]. Rather than integrating multiple annotations during training, TwinTrack explicitly modeled inter-rater
variability through post-hoc calibration, using isotonic regression to align predicted probabilities with the mean human consensus [26, 27]. Two-stage nnU-Net framework
with coarse localization followed
by full-resolution cropped
refinement. Prediction-level fusion through
voxel-wise averaging of
expert-specific models. The method built on a pretrained ResidualEncoderUNet and converted it into an Adaptable Bayesian Neural Network [28] by introducing stochastic
perturbations into normalization layers. Multiple stochastic forward passes were then used at inference time to approximate the
posterior predictive distribution [29], and the resulting probabilistic segmentations were fused into a final consensus prediction. MIC DKFZ - German Cancer Research Center
MIC DKFZ used a pretrained nnU-Net ResEnc-L ensemble,
subsequently fine-tuned on the CURVAS-PDACVI training set
using all available annotations, including the STAPLE consensus. Uncertainty was estimated from ensemble softmax outputs
and further calibrated through temperature scaling. PANORAMA
Batch 4
ROISeg Multi-rater labels fused into a
single STAPLE consensus prior
to training. Direct full-resolution nnU-Net
without explicit multi-stage
refinement. OrdSTAPLE - Universitat Pompeu Fabra, Sycai Medical
OrdSTAPLE, submitted by the organizers as a reference
baseline, explicitly modeled rater disagreement through an ordinal formulation. The method combined a standard STAPLEtrained binary model with a second model trained to predict
increasing levels of annotator agreement using an ordinal objective.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Findings and Clinical Implications
The benchmark revealed that strong global segmentation performance does not necessarily translate into accurate assessment of localized vascular involvement, especially for venous structures <ref:2604.27582#pg2> For instance, methods like BreizhSeg and MIC DKFZ performed strongly under global overlap metrics, whereas TwinTrack and CorpuSeg were more competitive at the vascular interface <ref:2604. p. 7 A second central finding is that uncertainty-aware and
disagreement-aware strategies lead to qualitatively different
model behavior from standard binary-target segmentation. Methods such as BreizhSeg and MIC DKFZ performed
strongly under global overlap metrics, whereas TwinTrack and
CorpuSeg were more competitive at the vascular interface. OrdSTAPLE, which ranked
lower under the global evaluation, became the top-performing
method on the high-complexity cohort, achieving the best mean
rank across metrics while maintaining competitive overlap and
vascular invasion scores. In contrast, methods that performed
strongly on the full benchmark, particularly BreizhSeg and
MIC DKFZ, showed a marked drop in spatial overlap in these
ambiguous cases, despite remaining competitive on selected
vascular structures. These results highlight a clinically important distinction between methods optimized for average-case volumetric accuracy
and methods that remain reliable when expert consensus breaks
down. In particular, explicitly modeling disagreement appears
to improve robustness in the most ambiguous cases, where rigid
binary segmentation strategies tend either to collapse toward
under-segmentation or to overextend at critical tumor-vessel
boundaries. In ambiguous cases, several
binary-target methods tended to produce sharp and visually
confident boundaries even when expert agreement was weak,
whereas disagreement-aware approaches more often generated
diffuse probability maps that better reflected the uncertainty
present in the annotations, see Fig. From a clinical perspective, this difference is meaningful: an overconfident falsepositive vascular contact may incorrectly suggest unresectability, whereas an overconfident false-negative prediction may underestimate local invasion.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Directions
Several limitations should be considered when interpreting our benchmark results, including the relatively small size of the final test cohort which limits statistical power for pairwise comparisons <ref:2604.27582#pg2> Furthermore, the ground truth is derived from expert radiological annotations rather than surgical or pathological mapping, meaning the reference standard inherits its uncertainty and potential bias <ref:2604. p. 8 Second, the benchmark ground truth is derived from expert
radiological annotations rather than from surgical or pathological mapping of the true tumor extent and vascular involvement. While this reflects the reality of preoperative imaging practice,
it also means that the reference standard remains conditional on
expert interpretation and therefore inherits its uncertainty and
potential bias. Finally, the participating methods showed limited architectural diversity, with all submissions relying on variants of the
nnU-Net framework. As a result, the benchmark primarily
captures differences in training strategy, uncertainty modeling,
and annotation handling, rather than the broader design space
of modern segmentation architectures. As a result, the benchmark primarily
captures differences in training strategy, uncertainty modeling,
and annotation handling, rather than the broader design space
of modern segmentation architectures. Future editions should
therefore expand both cohort size and methodological diversity,
while also exploring evaluation protocols that can better link radiological ambiguity to downstream clinical decision-making.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The CURVAS-PDACVI benchmark demonstrates a distinction between average-case segmentation performance and robustness under diagnostic ambiguity <ref:2604.27582#pg2> Our results indicate that explicitly modeling inter-rater ambiguity can improve robustness in the most clinically challenging scenarios, where rigid binary segmentation strategies tend to fail through either boundary overextension or volume collapse <ref:2604. p. 8 Our
benchmark shows that strong global segmentation performance
does not necessarily translate into reliable characterization of
pancreatic tumor-vessel interfaces, particularly in cases where
expert agreement is low. Our results indicate that explicitly modeling inter-rater ambiguity can improve robustness in
the most clinically challenging scenarios, where rigid binary
segmentation strategies tend to fail through either boundary
overextension or volume collapse. In particular, OrdSTAPLE
moved from a lower position in the global ranking to the best
mean rank on the high-complexity cohort, suggesting that preserving the structure of inter-rater ambiguity can be more robust
when expert consensus breaks down. Overall, these results indicate that the main challenge in
VI-focused PDAC segmentation is not simply recovering tumor volume, but doing so in a way that remains calibrated
and anatomically reliable at clinically critical boundaries. Overall, these results indicate that the main challenge in
VI-focused PDAC segmentation is not simply recovering tumor volume, but doing so in a way that remains calibrated
and anatomically reliable at clinically critical boundaries. Our
benchmark supports evaluating methods not only through aggregate overlap, but also through uncertainty-aware, multimetric frameworks that better reflect downstream clinical goals.Where the paper says it
Where the paper says it
Where the paper says it
Reference List
[1] H. Sun, H. Ma, G. Hong, H. Sun, J. Wang, Survival improvement in patients with pancreatic cancer by decade: A period analysis of the SEER database, 1981–2010, Scientific Reports 4 (2014) 6747 <ref:2604. p. 9 References
[1] H. Wang, Survival improvement in
patients with pancreatic cancer by decade: A period analysis of the
SEER database, 1981–2010, Scientific Reports 4 (2014) 6747.Where the paper says it
[2] M. Brugel, E. J. Rummeny, M. p. 9 gov/articles/PMC5381379/
[2] M. Rummeny, M. Rummeny, M. Dobritz, Vascular invasion in pancre- ¨
atic cancer, Abdominal Imaging 29 (2) (2004) 239–245.Where the paper says it
Where the paper says it
[3] L. Yao, Z. Zhang, E. Keles, C. Yazici, T. Tirkes, U. Bagci, A review of deep learning and radiomics approaches for pancreatic cancer diagnosis from medical imaging, Current Opinion in Gastroenterology 39 (5) (2023) 436–447 <ref:2604. p. 9 1007/s00261-003-0102-2
[3] L. Bagci, A review
of deep learning and radiomics approaches for pancreatic cancer diagnosis from medical imaging, Current Opinion in Gastroenterology 39 (5)
(2023) 436–447.Where the paper says it
[4] Z. Alidina, A. p. 9 0000000000000966. Alidina, A. Alidina, A. Hussain, I. Hussain, I. Pawlik, Radiomics for early detection of pancreatic cancer: a systematic review and
meta-analysis, Journal of Gastrointestinal Surgery 30 (5) (2026) 102374.Where the paper says it
Where the paper says it
Where the paper says it
[5] Y. Xia, Q. Yu, W. Shen, Y. Zhou, E. K. Fishman, A. p. 9 com/science/article/pii/
S1091255X26000557
[5] Y. Fishman, A. Fishman, A. Yuille, Detecting
Pancreatic Ductal Adenocarcinoma in Multi-phase CT Scans via Alignment Ensemble, in: A. Yuille, Detecting
Pancreatic Ductal Adenocarcinoma in Multi-phase CT Scans via Alignment Ensemble, in: A. Abolmaesumi, D. Abolmaesumi, D. Stoyanov, D. Stoyanov, D. Zuluaga, S. Zuluaga, S. Racoceanu, L. Racoceanu, L. Joskowicz (Eds. Joskowicz (Eds. ),
Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Springer International Publishing, Cham, 2020, pp.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[6] J. Zhao, J. Wang, Y. Gu, X. Huang, L. Wang, Diagnostic methods for pancreatic cancer and their clinical applications (Review), Oncology Letters 30 (1) (2025) 370 <ref:2604. p. 9 1007/978-3-030-59716-0_28. Wang, Diagnostic methods for pancreatic cancer and their clinical applications (Review), Oncology Letters
30 (1) (2025) 370.Where the paper says it
[7] F. Isensee, P. p. 9 gov/articles/PMC12150009/
[7] F. Isensee, P. Isensee, P. Petersen, K. Petersen, K. Maier-Hein, nnUNet: a self-configuring method for deep learning-based biomedical image
segmentation, Nature Methods 18 (2) (2021) 203–211.Where the paper says it
Where the paper says it
Where the paper says it
[8] J. I. Bereska, B. p. 9 1038/
s41592-020-01008-z. Bereska, B. Bereska, B. Janssen, C. Janssen, C. Kazemier,
O. Besselink,
I. Verpalen, for the Pancreatobiliary and Hepatic Artificial Intelligence Research (PHAIR) consortium, Artificial intelligence for assessment of vascular involvement and tumor resectability on CT in patients
with pancreatic cancer, European Radiology Experimental 8 (1) (2024)
18.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[9] A. Jungo, R. Meier, E. Ermis, M. Blatti-Moreno, E. p. 9 1186/s41747-023-00419-9
[9] A. Blatti-Moreno, E. Herrmann, R. Reyes, On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation, in: A. Fichtinger (Eds. ), ´
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, Springer International Publishing, Cham, 2018, pp.Where the paper says it
Where the paper says it
Where the paper says it
[10] L. Joskowicz, D. p. 9 Racoceanu, L. Joskowicz (Eds. Joskowicz, D. Sosna, Inter-observer variability
of manual contour delineation of structures in CT, European Radiology
29 (3) (2019) 1391–1399.Where the paper says it
Where the paper says it
[11] N. C. Buchs, M Chilcott, P-A Poletti, L H Buhler, P Morel Vascular invasion in pancreatic cancer: Imaging modalities, preoperative diagnosis and surgical management World Journal of Gastroenterology: WJG 16 (7) (2010) 818–831 <ref:2604. p. 9 Poletti, L. Morel, Vascular
invasion in pancreatic cancer: Imaging modalities, preoperative diagnosis and surgical management, World Journal of Gastroenterology : WJG
16 (7) (2010) 818–831.Where the paper says it
[12] R. K. G Do, A Kambadakone, Radiomics for CT Assessment of Vascular Contact in Pancreatic Adenocarcinoma, Radiology 301 (3) (2021) 623–624 <ref:2604. p. 9 gov/articles/PMC2825328/
[12] R. Kambadakone, Radiomics for CT Assessment of
Vascular Contact in Pancreatic Adenocarcinoma, Radiology 301 (3)
(2021) 623–624, publisher: Radiological Society of North America.Where the paper says it
[13] S. Warfield, K Zou, W Wells Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation, IEEE Transactions on Medical Imaging 23 (7) (2004) 903–921 <ref:2604. p. 9 Warfield, K. Wells, Simultaneous truth and performance level
estimation (STAPLE): an algorithm for the validation of image segmentation, IEEE Transactions on Medical Imaging 23 (7) (2004) 903–921.Where the paper says it
[14] M. H Jensen, D R Jørgensen, R Jalaboi, M E Hansen, M A Olsen Improving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement, in: D Shen T Liu T M Peters L H Staib C Essert S Zhou P T Yap A Khan (Eds. p. 9 Jalaboi, M. Olsen,
Improving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement, in: D. Olsen,
Improving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement, in: D. ), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Springer
International Publishing, Cham, 2019, pp. Olsen,
Improving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement, in: D. ), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Springer
International Publishing, Cham, 2019, pp.Where the paper says it
Where the paper says it
Where the paper says it
[15] C H Sudre, B G Anson, S Ingala, C D Lane D Jimenez L Haider T Varsavsky R Tanno L Smith S Ourselin R H Jager M J Car-doso Let’s Agree to Disagree: Learning Highly Debatable Multirater Labelling, in: D Shen T Liu T M Peters L H Staib C Essert S Zhou P T Yap A Khan (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Springer International Publishing, Cham, 2019 <ref:2604.27582#pg2>
[16] J Zhang, Y Zheng, Y Shi A Soft Label Method for Medical Image Segmentation with Multirater Annotations Computational Intelligence and Neuroscience 2023 (2023) 1883597 <ref:2604.27582#pg2>
[17] M Islam, B Glocker, Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations, in: A Feragen S Sommer J Schnabel M Nielsen (Eds.), Information Processing in Medical Imaging, Springer International Publishing, Cham, 2021 <ref:2604. p. 9 Nielsen (Eds. ), Information Processing in Medical Imaging,
Springer International Publishing, Cham, 2021, pp.Where the paper says it
[18] M Riera-Mar´ın, J G Lopez, J Rodr´ıguez-Comas, M A G Ballester A Galdran Multi-Rater Calibration Error Estimation, in: C H Sudre M I Hoque R Mehta C Ouyang C Qin M Rakic W M Wells (Eds.), Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, Springer Nature Switzerland, Cham, 2026 <ref:2604. p. 9 Wells
(Eds. ), Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, Springer Nature Switzerland, Cham, 2026, pp.Where the paper says it
[19] S Tuijn, F Janssens P Robben H van den Bergh Reducing interrater variability and improving health care: a meta-analytical review Journal of Evaluation in Clinical Practice 18 (4) (2012) 887–895 <ref:2604.27582#pg2>
[20] H K Yang, M-S Park, M Choi J Shin S S Lee W K Jeong S H Hwang S H Choi Systematic review and meta-analysis of diagnostic performance of CT imaging for assessing resectability of pancreatic ductal adenocarcinoma after neoadjuvant therapy: importance of CT criteria, Abdominal Radiology (New York) 46 (11) (2021) 5201–5217 <ref:2604.27582#pg2>
[21] Y N Shen, X L Bai G G Li T B Liang Review of radiological classifications of pancreatic cancer with peripancreatic vessel invasion: are new grading criteria required? Cancer Imaging 17 (2017) 14 <ref:2604. p. 9 Liang, Review of radiological
classifications of pancreatic cancer with peripancreatic vessel invasion:
are new grading criteria required? , Cancer Imaging 17 (2017) 14.Where the paper says it
[22] N Alves, M Schuurmans D Rutkowski D Yakar I Haldorsen M Liedenbaum A Molven P Vendittelli G Litjens J Hermans H Huisman The PANORAMA Study Protocol: Pancreatic Cancer Diagnosis - Radiologists Meet AI, Tech. rep., Zenodo (Jan. p. 9 Huisman, The PANORAMA Study Protocol: Pancreatic Cancer Diagnosis - Radiologists Meet AI, Tech. , Zenodo (Jan. Besselink,
I. Verpalen, for the Pancreatobiliary and Hepatic Artificial Intelligence Research (PHAIR) consortium, Artificial intelligence for assessment of vascular involvement and tumor resectability on CT in patients
with pancreatic cancer, European Radiology Experimental 8 (1) (2024)
18.Where the paper says it
Where the paper says it
[23] M Riera-Mar´ın S O K M M Duh A Aubanell R de Figueiredo Cardoso S Egger-Hackenschmidt M S May S Bernaus Tome J Rodr´ıguez-Comas M A Gonz alez Ballester J Garcia L opez Curvas-pdacvi dataset (Nov. 2025) <ref:2604. p. 0 A notable but often neglected shortcoming in this context is the failure to model annotation variability and data uncertainty, both of
which are inherent to PDAC and VI analysis [11, 12]. The aleatoric uncertainty induced by annotator variability,
as well as its impact on medical image segmentation modPreprint submitted to Elsevier May 1, 2026
arXiv:2604.Where the paper says it
[30] curvas-challenge/CURVAS-PDACVI 2025 at main · SYCAITechnologies/curvas-challenge (2025)
Appendix Details
The original PANORAMA CT scans are available at https://zenodo.org/records/11034178, and the CURVAS-PDACVI benchmark release is available at https://zenodo. p. 9 URL https://zenodo. org/records/10599559
[23] M. URL https://zenodo. org/records/10599559
[23] M. Jeremias Traub: Software, Investigation. Code availability
To support users with the evaluation of the algorithms, we
have made the evaluation code used in this manuscript publicly
available in the Challenge’s GitHub repository [30]. Separate rankings were generated for DSC, Thr-DSC, MRECE, CRPS, and each of the five vessel-specific VI scores. The
final leaderboard was obtained by averaging team ranks across
all metrics, with higher values indicating better performance for
DSC and Thr-DSC, and lower values indicating better performance for the remaining metrics.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Figure Analysis
Figure 5 summarizes the relationships among the benchmark metrics, showing that overlap-based measures were strongly correlated with each other, but only weakly associated with vessel-specific vascular invasion errors Figure 6 shows that most pairwise comparisons for vessel-specific vascular invasion metrics did not reach significance, indicating that case-wise variance in these structures is strongly influenced by a subset of highly ambiguous studies shared across methods Figure 7 summarizes standardized case-wise performance signatures, revealing recurrent hard studies that degrade performance across multiple architectures Figure 8 shows that methods exhibited distinct failure profiles, but also shared a subset of recurrent hard studies that degraded performance across architectures The results indicate that the most severe benchmark errors are driven less by isolated architectural weaknesses than by intrinsically ambiguous clinical cases.
Final Summary
The CURVAS-PDACVI benchmark shows a distinction between average-case segmentation performance and robustness under diagnostic ambiguity Strong global segmentation performance does not necessarily translate into reliable characterization of pancreatic tumor-vessel interfaces, particularly in cases where expert agreement is low Uncertainty-aware and disagreement-aware strategies lead to qualitatively different model behavior from standard binary-target segmentation Explicitly modeling inter-rater ambiguity can improve robustness in the most clinically challenging scenarios, where rigid binary segmentation strategies tend either to fail through boundary overextension or volume collapse The main challenge in VI-focused PDAC segmentation is not simply recovering tumor volume, …
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Uncertainty-aware probabilistic models can be developed to support preoperative decision-making by moving beyond deterministic predictions that
collapse toward under-segmentation or to overextend at critical tumor-vessel boundaries.
-
AI systems can be improved by adopting multi-metric evaluation frameworks that extend beyond spatial overlap to include
probabilistic calibration and VI assessment,
allowing models to be evaluated on metrics like the Wasserstein distance for vascular invasion. -
Models can be designed with uncertainty modeling, such as
Bayesian uncertainty modeling via Adaptable Bayesian Neural Network and Monte Carlo inference,
which enables them to generatemore diffuse predictions that better reflect inter-rater variability
in ambiguous cases. -
Systems can be enhanced by incorporating strategies that explicitly model disagreement, such as
OrdSTAPLE Joint modeling of STAPLE consensus and ordinal levels of annotator agreement,
to createrobustness in the most ambiguous cases.
-
The improved system can provide clinicians with confidence measures by generating
diffuse probability maps that better reflected the uncertainty present in the annotations,
which is clinically meaningful aspoorly calibrated confidence can be as clinically relevant as geometric error itself.
Abstract
Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI), i.e., tumor extension into adjacent critical vessels. Despite its importance for preoperative staging and surgical planning, computational VI assessment remains underexplored. Two major challenges are the lack of public datasets and the diagnostic ambiguity at the tumor-vessel interface, which leads to substantial inter-rater variability even among expert radiologists. To address these limitations, we introduce the CURVAS-PDACVI Dataset and Challenge, an open benchmark for uncertainty-aware AI in PDAC staging based on a densely annotated dataset with five independent expert annotations per scan. We also propose a multi-metric evaluation framework that extends beyond spatial overlap to include probabilistic calibration and VI assessment. Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. In particular, methods optimized for binary segmentation perform competitively on average overlap metrics, but often degrade in high-complexity cases with low expert consensus, either collapsing in volume or overextending at uncertain boundaries. In contrast, methods that model inter-rater disagreement produce better calibrated probabilistic maps and show greater robustness in these ambiguous cases. The benchmark highlights the limitations of volumetric accuracy as a proxy for localized surgical utility, motivating uncertainty-aware probabilistic models for preoperative decision-making.
Sources
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs