Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology
cs.CV
Submitted: 2026-09-16
Updated: 2026-09-16
Comments: 13 pages, 5 figures, plus supplementary material. Accepted at DAGM GCPR 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pathologists diagnose diseases by first locating suspicious tissue and then examining it at higher magnification, whereas self-supervised vision transformers (ViTs) allocate the same spatial
Terminology
Abstract
Pathologists diagnose diseases by first locating suspicious tissue and then examining it at higher magnification, whereas self-supervised vision transformers (ViTs) allocate the same spatial resolution to every image region despite diagnostic evidence being sparse and spanning multiple biological scales. Recent pathology foundation models have substantially improved representation quality by scaling training data and model capacity, but largely retain uniform tokenization. We instead investigate whether pathology representations can be improved by learning where to allocate spatial resolution during self-supervised learning. To this end, we propose CRAFT (Coarse-to-fine Region-Adaptive Feature Tokenization), a DINO-based framework that learns image-dependent mixed-scale representations by using self-supervised attention to selectively refine informative regions while preserving coarse context, together with a symmetric cross-scale regularization objective that encourages complementary coarse and fine representations. Across CAMELYON16, TCGA-Lung subtype classification, and TCGA-LUAD survival prediction, CRAFT consistently outperforms comparable-scale self-supervised methods while requiring lower inference computation. Despite using only a compact 22M parameter backbone trained on comparatively small pathology datasets, CRAFT remains competitive with, and often surpasses, substantially larger pathology foundation models.
Sources
- Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis
- Token Merging: Your ViT But Faster
- Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images
- Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning
- A General-Purpose Self-Supervised Model for Computational Pathology
- Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology
- An Empirical Study of Training Self-Supervised Vision Transformers
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling foundation models for robust and efficient models in digital pathology
- RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank
- Attention-based Deep Multiple Instance Learning
- PLUTO: Pathology-Universal Transformer
- Benchmarking Self-Supervised Learning on Diverse Pathology Datasets
- Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations
- Self-Supervision Enhances Instance-based Multiple Instance Learning Methods in Digital Pathology: A Benchmark Study
- Representation Learning with Contrastive Predictive Coding
- DINOv2: Learning Robust Visual Features without Supervision
- Learning Transferable Visual Models From Natural Language Supervision
- ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification
- How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models