Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models

The paper addresses whether 3D CT foundation models can detect incidental findings without any fine-tuning. The authors note that Routine CT interpretation is inherently comprehensive: Regardless of the primary indication, radiologists inspect the entire volume and report any anatomical deviations, including incidental findings. They argue that while hospitals may run inference locally, they rarely have the labeled datasets necessary to fine-tune task heads for hundreds of rare conditions. The central questions posed are: Are current CT foundation models capable of incidental-finding detection without any fine-tuning, does a consistently best-performing model emerge, and where do their representational limitations lie? The authors emphasize that assessing this requires isolating the latent space and benchmarking their frozen representations directly.

Datasets. Three thoracic CT cohorts were used: CT-RATE (1,564 scans after filtering one reconstruction per study, 18 labels), RAD-ChestCT (2,284 single-scan patients, 92 labels aggregated from fine-grained location annotations), and an unseen in-house hospital cohort (3,001 scans from 2,087 patients, 87 labels). The internal cohort is guaranteed to be unseen by any model. Labels were derived from radiology reports using different NLP pipelines per cohort: a fine-tuned text model for CT-RATE, the RATE framework for the internal cohort, and a radiologist-validated rule-based extractor for RAD-ChestCT.

Models. Ten 3D CT encoders were evaluated strictly as frozen feature extractors using author-prescribed preprocessing: three image-only self-supervised models (CT-FM, Curia-2, VoxelFM), one supervised model (CT-SSG), and six vision–language aligned models (CT-CLIP, COLIPRI, SPECTRE, Merlin, Pillar-0, FlexiCT). Notably, Five models were exposed to the CT-RATE dataset during pretraining. Domain overlap with the chest-focused evaluation cohorts differed markedly: some models were chest-specific (COLIPRI, CT-CLIP, CT-SSG, Pillar-0), thoraco-abdominal (SPECTRE), broad multi-region (FlexiCT, VoxelFM, Curia-2, CT-FM), or exclusively abdomen–pelvis with no chest exposure (Merlin).

Evaluation readouts. Three readout methods were used without fine-tuning: cosine-weighted k-nearest neighbors (k=5), zero-shot classification using fixed prompt pairs (A chest CT scan showing finding. vs. A chest CT scan showing no finding.), and linear probing (one-vs-rest l2-regularized logistic regression). All readouts used five-fold cross-validation strictly grouped by patient to prevent leakage, with 95% confidence intervals from 1,000 paired patient-level bootstrap resamples. Performance was measured using AUROC and prevalence-normalized PR-AUC, also known as skill: for a finding of prevalence π, skill = (AP − π)/(1 − π).

Finding taxonomy. Labels were grouped into six radiological phenotypes: calcification, devices, fluid collections/morphometric changes, diffuse texture patterns, focal space-occupying lesions, and skeletal findings. Consistency of difficulty ordering was measured with Kendall's coefficient of concordance.

The authors report: We find no universal state-of-the-art, with rankings fluctuating significantly depending on the evaluation context. The best model depends on the readout method: "Under cosine kNN, the top three encoders on every cohort are FlexiCT, COLIPRI, and Pillar-0, which are the transformers with the finest image tokenizer (8-voxel patches)… whereas the coarsest-tokenizer transformer (CT-CLIP, 20×20×10) ranks last. Under linear probing, COLIPRI maximizes AUROC and FlexiCT maximizes skill, at which point the strict patch-granularity ordering dissolves and the self-supervised VoxelFM rises to the front. CT-CLIP and CT-FM consistently underperform."

The authors state that architectures utilizing fine-grained image tokenizers combined with report-aligned pre-training consistently lead performance. This patch-granularity advantage strongly co-occurs with report-aligned pretraining. Importantly, Pillar-0 never saw CT-RATE yet remains in the top three on the two held-out cohorts, indicating that these shared architectural traits confer genuine generalization.

The supervised encoder CT-SSG, which has the smallest embedding (512-d), "stays highly competitive across both kNN and linear probing on all three cohorts. It matches or beats every self-supervised encoder and both convolutional models, suggesting that explicit labels can effectively substitute for scale."

Data efficiency curves reveal that Pillar-0 rapidly hits a representational ceiling and saturates as labeled data increases, eventually being overtaken by CT-SSG. Leading representations scale log-linearly with available data, but Pillar-0's ceiling is notable.

The paper's central finding is that diagnostic difficulty is governed not by the choice of encoder, but by characteristics of the findings like its spatial extent and contrast. The authors observe: Performance differences across distinct findings are substantially larger than the differences between models on any given finding. The easy-to-hard ordering across the six finding types barely moves from one encoder to the next (Kendall W = 0.89): support devices are easy, diffuse fluid and texture changes are moderate, and focal lesions are hardest.

Through controlled within-organ comparisons isolating contrast and extent from prevalence and anatomy, they found: "Holding extent fixed while increasing contrast (e.g., pleural and pericardial effusion vs. thickening; solid vs. subsolid nodule) improves macro AUROC by +0.18 on average. Conversely, holding contrast fixed and increasing extent (e.g., bulk vs. faint calcification; diffuse vs. focal ground-glass) yields a +0.16 improvement. These effects were robust: All seven within-organ comparisons are positive and bootstrap-significant, with the ten evaluated encoders demonstrating near-universal agreement on this directionality (68 of 70 comparison × encoder pair)."

The limitation is representational, not an artifact of the readout: The linear probe extracts more signal than kNN or zero-shot, yet focal performance stays low even under the probe. It is also selective within the focal class: larger solid lesions stay retrievable (solid nodule skill value 0.26, mediastinal lymphadenopathy 0.29) while small subsolid ones sink toward chance (ground-glass nodule skill 0.05, part-solid nodule 0.02). The authors attribute this to the inherent limitations of globally pooled embeddings, noting that A globally pooled embedding inherently attenuates the subtle spatial signals of small lesions.

"For the rarest findings, zero-shot prompting yields a paired advantage of approximately 0.10 AUROC over kNN. However, this advantage disappears when evaluated using prevalence-normalized skill. Under the skill metric, both zero-shot and kNN readouts score near zero on the rarest findings."

The authors conclude that no single encoder consistently wins across all cohorts and readouts. Instead, the fundamental factor that transfers across cohorts is a physical bottleneck. Because "diagnostic difficulty is governed by contrast and extent rather than model capacity, overcoming this will likely require region- or lesion-level pretraining objectives that explicitly preserve small, low-contrast structures. Furthermore, the performance gap between linear probing and zero-shot retrieval suggests that relevant visual signals are present in the embeddings but remain inaccessible to the text encoder. Improving fine-grained vision–language alignment, perhaps coupled with lightweight, pattern-specific detection heads, offers a more promising route to robust clinical translation than relying on a single global vector."

The authors acknowledge that evaluating frozen embeddings bounds achievable peak performance, and the NLP-based extraction of labels across all three cohorts introduces noise that disproportionately affects the rarest findings. However, they argue the difficulty ordering's consistency (Kendall W = 0.89) across three independently constructed label pipelines makes label noise an unlikely cause of the focal-lesion failure, indicating the pattern is representational. The vertical banding in their results (e.g., CT-CLIP systematically collapsing toward chance) and the persistence of difficulty patterns in the unseen internal cohort demonstrate that these representational bottlenecks are fundamental to the models themselves, rather than artifacts of the public pretraining datasets.

• Improvement: Modify 3D CT encoders to output per-patch feature maps (e.g., from 8-voxel tokenizers) instead of a single globally pooled vector. Attach lightweight convolutional or attention-based detection heads directly to the dense tokens, trained with region-level supervision.

• What it can do: Detect small, low-contrast focal lesions (ground-glass nodules, part-solid nodules) that currently sink toward chance (skill 0.02–0.05). It preserves spatial extent and contrast information that global averaging inherently attenuates.

• Improvement: Add pretraining tasks that explicitly manipulate lesion boundaries, simulate low-contrast and variable-size findings, and use hard-negative mining on subtle abnormalities. Use a multi-scale feature pyramid where small lesions are represented at native resolution rather than downsampled.

• What it can do: Lift performance on physically difficult findings (focal lesions, faint calcifications) by making the representation invariant to contrast/extent variation, while maintaining strong performance on easy findings (devices, fluid collections).

• Improvement: Align local image regions with corresponding phrases in radiology reports (e.g., ground-glass opacity in right upper lobe) instead of global image–report alignment. Use contrastive losses on token-level or sentence-level correspondences, with attention masking to prevent shortcut learning.

• What it can do: Improve zero-shot retrieval of specific findings, especially rare ones, because the text encoder can access local visual evidence. This addresses the paper's finding that visual signals are present in embeddings but inaccessible to a global text head.

• Improvement: For frozen encoders, train multiple small detection heads—each specialized for one radiological phenotype (calcification, focal lesions, texture patterns, etc.)—on dense features, rather than one global logistic-regression readout. Optionally combine with an ensembling mechanism that selects the best readout per finding.

• What it can do: Better separate easy and hard findings, reduce reliance on any single global vector, and improve AUROC/skill on categories like subsolid nodules and ground-glass opacities without full fine-tuning.

• Improvement: Adjust zero-shot prompt logits using prevalence priors, and switch between zero-shot, kNN, and linear-probe readouts based on finding rarity. For prevalence-normalized skill, fall back to linear probing on dense features when zero-shot confidence is low.

• What it can do: Maintain true positive rate for rare incidental findings without being inflated by class imbalance; the paper shows zero-shot's AUROC advantage over kNN for rare findings disappears under skill normalization—this fix retains sensitivity where it matters.

• Improvement: During pretraining, add a supervised auxiliary task (e.g., predicting coarse anatomy, organ presence, or known pathology labels) alongside self-supervised reconstruction/contrastive loss. This mimics CT-SSG's advantage, where explicit labels substitute for scale.

• What it can do: Achieve competitive or better representations with smaller models and less data, especially for linear probing on new cohorts—useful in hospitals with limited labeled data.

• Improvement: For models like Pillar-0 that rapidly hit a ceiling, use curriculum learning that progressively introduces harder examples (lower contrast, smaller lesions) after easy-data saturation. Combine with log-linear scaling of labeled data only on examples where the model has not saturated.

• What it can do: Push past the observed performance ceiling and make additional labeled data continue to improve detection, rather than plateauing.

• Improvement: Structure model development around the six radiological phenotypes (calcification, devices, fluid, texture, focal lesions, skeletal) and report skill per phenotype. Train separate heads or loss weighting for the hardest class (focal lesions) to address the near-universal difficulty ordering.

• What it can do: Make model improvement efforts target the actual bottleneck (contrast/spatial extent) instead of chasing leaderboard rankings, yielding more robust clinical deployment across heterogeneous cohorts and label pipelines.