Different Models, Shared Failures: Auditing Frozen-Encoder Pools for Medical Segmentation
cs.CV, cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: Accepted at NeurIPS 2026, Track on Evaluations and Datasets. 47 pages, 8 figures. Code and data: https://github.com/eunseobchoi/CoFailMed
Code: https://github.com/eunseobchoi/CoFailMed
License: http://creativecommons.org/licenses/by/4.0/
The gist: Average segmentation accuracy and model count do not reveal whether models fail on the same cases.
Terminology
Abstract
Average segmentation accuracy and model count do not reveal whether models fail on the same cases. We present CoFailMed, an audit of correlated failures in pretrained-encoder pools for medical segmentation. Across five 2D tasks, we evaluate 11 public pretrained encoders (a controlled sample, not a census) using frozen features and separately trained 1 times 1 readouts with four seeds per encoder. Shared-checkpoint pairs have the highest mean per-case Dice correlation. An expanded 17-encoder analysis shows that distinct checkpoints within the same encoder family remain more correlated than cross-family pairs. At the pooled 33rd-percentile Dice threshold, four-member pools drawn from distinct encoders have mean effective sizes of 1.56-2.92, estimated from failure correlations in pools with enough failures and passes per member. In the primary pool, richer readouts and fine-tuning narrow the Dice-correlation gap between pairs sharing a checkpoint (or the DINOv2-B/S release) and all other pairs. Yet dependence between different encoders can increase, and fine-tuning does not consistently increase effective pool size at 33rd-percentile thresholds recomputed for each training recipe. A smaller between-group gap can coexist with greater dependence. Pool evaluation should therefore report absolute dependence and threshold-specific effective size alongside accuracy and nominal pool size. The CoFailMed release provides aligned per-case traces, analysis scripts and study code to reproduce the central audit results.
Sources
- Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks
- On the Opportunities and Risks of Foundation Models
- MONAI: An open-source framework for deep learning in healthcare
- SAM-Med2D
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- Current Pathology Foundation Models are unrobust to Medical Center Differences
- The 2024 Brain Tumor Segmentation (BraTS) Challenge: Glioma Segmentation on Post-treatment MRI
- How we won BraTS 2023 Adult Glioma challenge? Just faking it! Enhanced Synthetic Data Augmentation and Model Ensemble for brain tumour segmentation
- Deep Ensembles: A Loss Landscape Perspective
- The Subjectivity of Monoculture
- MedSAM2: Segment Anything in 3D Medical Images and Videos
- Deep Ensembles for Low-Data Transfer Learning
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms
- Medical SAM 2: Segment medical images as video via Segment Anything Model 2
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs