Benchmarking CNNs, Vision Transformers, and Foundation Models for Explainable Brain Tumor MRI Classification: Calibration, Decision Analysis, and External Validation
cs.CV
Submitted: 2026-11-09
Updated: 2026-11-09
Comments: Under review at Scientific Report Journal, Springer
License: http://creativecommons.org/licenses/by/4.0/
The gist: Deep learning models for brain tumor MRI classification are usually compared by accuracy alone, on public datasets whose independence is rarely checked.
Terminology
Abstract
Deep learning models for brain tumor MRI classification are usually compared by accuracy alone, on public datasets whose independence is rarely checked. We benchmarked six architectures: two convolutional networks (EfficientNetV2-M and ConvNeXt-V2-Tiny), two vision transformers (Swin-V2-S and ViT-L/16), and two foundation models (DINOv2-L and MedSigLIP-448), for four-class classification of 7,216 public MRI images. We assessed accuracy, calibration, explainability, ensembles, and decision curves, with every fitted quantity estimated on validation data only. Pixel-verified duplicate screening showed that 36% of the test images had a near-duplicate in the training data. Test accuracy ranged from 94.5% to 95.6%, with no pairwise difference after Holm correction, and fell to 90.3% to 91.7% on the 773 test images free of duplicates and shared patients. Ensembles did not outperform the best single model. Attribution maps passed a model-randomization test but localized expert-annotated tumors poorly. On 1,675 external BDNeuro-MRI images that did not duplicate the training data, accuracy ranged from 77.6% (MedSigLIP-448) to 95.0% (ViT-L/16), and 12 of 15 pairwise differences were significant. Internal accuracy on this public collection therefore did not distinguish the architectures, whereas the duplicate-filtered external test did.
Sources
- PerceptronCARE: A Deep Learning-Based Intelligent Teleophthalmology Application for Diabetic Retinopathy Diagnosis
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Swin Transformer V2: Scaling Up Capacity and Resolution
- A ConvNet for the 2020s
- ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
- DINOv2: Learning Robust Visual Features without Supervision
- Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset
- MedGemma Technical Report
- Densely Connected Convolutional Networks
- Foundation Models in Medical Imaging: A Review and Outlook
- Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classification
- MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis
- Axiomatic Attribution for Deep Networks
- Sanity Checks for Saliency Maps
- On Calibration of Modern Neural Networks
- EfficientNetV2: Smaller Models and Faster Training
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- E-ConvNeXt: A Lightweight and Efficient ConvNeXt Variant with Cross-Stage Partial Connections
- Convolutional Neural Networks and Vision Transformers for Fashion MNIST Classification: A Literature Review
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs