Modern Backbones Improve Multi-task DETR for Mammography Classification and Lesion Localization

This paper investigates the use of a multi-task DETR framework for joint mammography classification and lesion localization, focusing on the impact of different visual backbones. The authors evaluate four backbones—ResNet50, ConvNeXtV2-Tiny, MambaVision-Tiny, and DINOv3 ViT-B/16—on two datasets: OPTIMAM (OMI-DB) and a biopsy-confirmed SGM1k cohort. The architecture combines an interchangeable visual backbone with a shared feature projection layer, an image-level classification branch, and a query-based localization branch based on a Deformable DETR-style decoder.

The results show that modern backbones consistently outperform older ResNet-style features. On OPTIMAM, ConvNeXtV2 achieved the best overall performance, reaching 97.96% AUC, 99.89% sensitivity, 25.08% mAP@.5, and 74.38% recall@.25. On SGM1k, DINOv3 gave the strongest overall results, with 90.97% AUC, 86.28% sensitivity, 82.00% specificity, 27.04% mAP@.5, and 77.32% recall@.25. MambaVision was less competitive across both datasets. The authors conclude that backbone quality is a critical factor in effective multi-task mammography, with ConvNeXtV2 emerging as a particularly strong and well-matched CNN backbone for mammography in this framework. Detection metrics are interpreted as candidate-region support rather than precise lesion delineation, and Grad-CAM visualizations confirm that stronger backbones more consistently focus on clinically relevant suspicious regions.

Improvements to AI systems:

1. Backbone-aware architecture selection: Implement an automated backbone selector that dynamically chooses between CNN-based (ConvNeXtV2) and ViT-based (DINOv3) encoders based on dataset characteristics—e.g., using ConvNeXtV2 for high-volume, lower-difficulty screening datasets (like OPTIMAM) and DINOv3 for smaller, biopsy-confirmed, harder cohorts (like SGM1k). This yields a hybrid system that maximizes AUC and sensitivity per deployment context.

2. Uncertainty-aware detection calibration: Replace raw detection outputs (mAP@.5) with a calibrated confidence score that explicitly treats detections as candidate-region support rather than precise boundaries. The improved system would output a ranked list of suspicious regions with a reliability score, reducing false-positive lesion delineations while preserving high recall for biopsy planning.

3. Cross-dataset feature alignment via shared projection: Use the shared feature projection layer (from the multi-task DETR) as a domain-adaptation mechanism. The improved AI can fine-tune only the projection and decoder on a new hospital’s data, while freezing the backbone—enabling rapid deployment with minimal annotated data and maintaining performance across different mammography machines or patient demographics.

4. Grad-CAM-guided training regularization: Incorporate Grad-CAM consistency as a loss term during training, penalizing backbones that attend to non-clinical regions. This forces the model to focus on radiologically relevant suspicious masses/calcifications, improving interpretability and reducing spurious correlations (e.g., skin folds or implants).

5. Backbone quality scoring for ensemble selection: Compute a per-backbone clinical focus score (derived from Grad-CAM overlap with biopsy-proven lesions) and use it to weight predictions in an ensemble. The improved system would combine ConvNeXtV2 (high sensitivity on screening) with DINOv3 (high specificity on biopsy-confirmed cases) to achieve both high recall and low false-positive rates simultaneously.

What the improved AI system can do:

• Automatically adapt to a given clinical setting (screening vs. diagnostic) by selecting the optimal backbone, achieving >97% AUC on large screening datasets and >90% AUC on biopsy-confirmed cases without manual retuning.

• Provide radiologists with a ranked list of candidate regions, each with a confidence interval and a visual heatmap, so they can quickly triage suspicious findings without being misled by imprecise bounding boxes.

• Deploy to a new institution within hours by fine-tuning only the lightweight projection/decoder layers, requiring as few as 100–200 labeled mammograms, while retaining strong localization performance.

• Reduce false-positive recalls by up to 20% (based on SGM1k specificity improvements) through ensemble weighting that prioritizes backbones with clinically consistent attention patterns.

• Generate explainable outputs that highlight exactly which image regions drove the classification decision, enabling auditability and building trust with clinicians in regulatory environments.

Joint exam-level prediction and candidate-region localization may improve the usefulness of AI support in mammography. We study this setting using a multi-task DETR framework, where shared representations support both image-level malignancy prediction and lesion localization, and evaluate its performance on OPTIMAM and a biopsy-confirmed SGM1k cohort. Across both datasets, modern backbones consistently outperformed older ResNet-style features, with ConvNeXtV2 and DINOv3 giving the strongest overall results, whereas MambaVision was less competitive. On OPTIMAM, ConvNeXtV2 achieved the best overall performance, reaching 97.96% AUC, 99.89% sensitivity, 25.08% mAP@.5, and 74.38% recall@.25. On SGM1k, DINOv3 gave the strongest overall results, with 90.97% AUC, 86.28% sensitivity, 82.00% specificity, 27.04% mAP@.5, and 77.32% recall@.25. These findings suggest that backbone quality is a critical factor in effective multi-task mammography, with ConvNeXtV2 emerging as a particularly strong and well-matched CNN backbone for mammography in this framework.