Attention Transfer Is Not Universally Effective for Vision Transformers
Attention Transfer, which copies attention patterns from a pre-trained teacher Vision Transformer to a new student model, works for some models but fails for others. The study found that architectural mismatch between the teacher and student is the main reason for failure. Fixing this by adding only the teacher's native components rescues performance in all failing cases, proving that attention transfer requires matching architectures.