Diagnosing Zero-Shot Forgetting in CLIP Fine-Tuning: Attention Drift Characterizes Adaptation, Representation Drift Predicts Transfer Degradation

arXiv:2611.00042 · cs.CV, cs.LG · Submitted 2026-10-08 · Read on arXiv

cs.CV, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted to the ICDM 2026 Workshops (ICDMW), Teen Research Track

Code: https://github.com/xiaruize0911/Attention_Collapse_in_CLIP_Fine-tuning_repo

License: http://creativecommons.org/licenses/by/4.0/

The gist: Fine-tuning CLIP on a downstream task typically reduces its zero-shot accuracy on other tasks, and quantifying this loss ordinarily requires a labeled transfer benchmark.

Terminology

Abstract

Fine-tuning CLIP on a downstream task typically reduces its zero-shot accuracy on other tasks, and quantifying this loss ordinarily requires a labeled transfer benchmark. We study whether the loss can be anticipated from unlabeled target images that are already available during fine-tuning. Four signals are compared: the entropy of CLS token attention, linear centered kernel alignment (CKA) with the pretrained model, embedding drift, and relative weight change. ViT-B/32 is adapted by full fine-tuning (Full FT) and by LoRA on EuroSAT and Oxford-IIIT Pets over four learning rates and three seeds, with an additional ViT-B/16 grid; DTD and GTSRB are held out as confirmation targets, and forgetting is measured as zero-shot retention on CIFAR-100, Food-101, and ImageNet-R. Within a fixed configuration, the change in attention entropy is strongly associated with forgetting (mean ρ=0.860). Its sign, however, depends on the dataset, the adaptation method, and the backbone, and the pooled correlation falls to 0.361. A block decomposition shows that this sign is determined almost entirely by blocks 7 to 12. CKA ranks runs by retention at ρ=0.960, and the ranking is already established after the first of six epochs (ρ=0.959). On the held-out targets, CKA reaches ρ=0.95, and the block account correctly predicts the sign of the attention change for both datasets. These results support a simple decision rule: CKA is suited to comparisons across configurations, whereas attention entropy is informative only within a single configuration. The evidence is limited to one model family and short training schedules.

Related papers