Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness
cs.CV, cs.AI, cs.LG
Submitted: 2026-03-09
Updated: 2026-09-26
Comments: CVPR 2026 ; Project page: https://yonisgit.github.io/concept-ft/
Code: https://github.com/yonisGit/cft
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations, such as background cues, rather than semantically meaningful features.
Terminology
Abstract
Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations, such as background cues, rather than semantically meaningful features. Existing regularization methods, typically relying on simple foreground-background masks, which fail to capture the fine-grained semantic concepts that define an object (e.g., ``long beak'' and ``wings'' for a ``bird''). As a result, these methods provide limited robustness to distribution shifts. To address this limitation, we introduce a novel finetuning framework that steers model reasoning toward concept-level semantics. Our approach optimizes the model's internal relevance maps to align with spatially grounded concept masks. These masks are generated automatically, without manual annotation: class-relevant concepts are first proposed using an LLM-based, label-free method, and then segmented using a VLM. The finetuning objective aligns relevance with these concept regions while simultaneously suppressing focus on spurious background areas. Notably, this process requires only a minimal set of images and uses half of the dataset classes. Extensive experiments on five out-of-distribution benchmarks demonstrate that our method improves robustness across multiple ViT-based models. Furthermore, we show that the resulting relevance maps exhibit stronger alignment with semantic object parts, offering a scalable path toward more robust and interpretable vision models. Finally, we confirm that concept-guided masks provide more effective supervision for model robustness than conventional segmentation maps, supporting our central hypothesis.
Sources
- GPT-4 Technical Report
- Fidelity-Aware Recommendation Explanations via Stochastic Path Integration
- Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Rethinking Saliency Maps: A Cognitive Human Aligned Taxonomy and Evaluation Framework for Explanations
- Label-Free Concept Bottleneck Models
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
- Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
- GradMask: Reduce Overfitting by Regularizing Saliency
- SmoothGrad: removing noise by adding noise
- Saliency is a Possible Red Herring When Diagnosing Poor Generalization
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models