Attention Transfer Is Not Universally Effective for Vision Transformers

arXiv:2605.07191 · cs.CV, cs.LG · Submitted 2026-05-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Attention Transfer Is Not Universally Effective for Vision Transformers".

Jane: The gist Attention Transfer is not universally effective for Vision Transformers because its success depends critically on architectural compatibility between the teacher and student models,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up this discussion on "Attention Transfer Is Not Universally Effective for Vision Transformers," the paper really hammers home that attention transfer isn't a magic solution that works everywhere <ref:2605.07191#pg4>.

Jane: It means the success of transferring attention is fundamentally tied to whether the student architecture matches the teacher architecture, and that this is what unlocks the full benefit of those pre-trained weights <ref:2605.07191#pg3>.

Lu: The main implication for research is that we need to move beyond just looking at attention patterns in isolation and start considering the full architectural context when thinking about knowledge transfer <ref:2605.07191#pg8>.

Meng: For those of us shipping models, it’s a warning to be very specific about the teacher-student architecture before we even start the transfer process <ref:2605.07191#pg9>.

Lalam: The paper suggests that restoring the native architectural components, when they are randomly initialized in a student model, is enough to make those transferred attention patterns usable <ref:2605.07191#pg9>.

Jane: So, if you’re thinking about using this kind of transfer method, you need to ensure that the structural compatibility between your teacher and student models is a prerequisite for any success <ref:2605.07191#pg3>.

Conclusion: Tom: So, we’re wrapping up this look at "Attention Transfer Is Not Universally Effective for Vision Transformers." Basically, the paper shows that you can't just grab attention patterns from one pre-trained vision transformer and slap them onto a random student model and expect it to work.

Jane: That’s right. It means that what used to seem like a simple knowledge transfer trick has some serious limitations when you try it across different types of models.

Lu: The authors tested this across twenty teachers from eleven different families of vision transformers, and they found that the success rate wasn't consistent at all.

Meng: I’m curious about the numbers here. How much did they fall off compared to just training from scratch? That’s what matters for us when we’re trying to deploy something reliable.

Lalam: They saw failures in four different families, and these failures weren't random; they were consistent across different model sizes and even after training for a long time or testing on new data.

Tom: So it's not just one specific architecture that fails; it’s a whole class of them that don't play nicely with this transfer method.

Jane: And the core issue isn't just about the attention maps themselves, but about the structure of the student model not matching what was learned in the teacher.

Lu: Exactly. The authors pinpointed it to specific architectural mismatches, like missing components in certain models that they had to add back into the student for those families to even have a chance.

Meng: So if I’m building a model, this suggests I can't just take a pre-trained attention block and drop it in; I need the whole surrounding structure to be compatible.

Tom: It basically means attention transfer isn't a universal fix, so we can’t rely on it working for every single vision transformer out there.

Jane: It refines our idea of how attention works in these models—attention only transfers use when the student architecture is structurally similar to the teacher one.

Lalam: The paper shows that restoring those missing native components alone is enough to fix the failure for those four families, which confirms that structure is what unlocks the attention's usability.

Tom: It’s a warning for anyone doing transfer research to be very careful about architectural assumptions when they try this path.

Jane: And it opens up a new direction where we have to think more deeply about how the student and teacher models interact at a structural level before we even look at the attention patterns.

Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Peng Hu, Chen Gong, Xi Peng, Hongyuan Zhu

Institute for Infocomm Research (I2R), A*STAR, Singapore · Sichuan University · Shanghai Jiao Tong University

cs.CV, cs.LG

Submitted: 2026-05-08

Updated: 2026-10-14

Comments: Published in NeurIPS 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist Attention Transfer is not universally effective for Vision Transformers because its success depends critically on architectural compatibility between the teacher and student models, which

Key concepts

Attention Transfer: This process involves taking the attention maps learned by a large, pre-trained 'teacher' Vision Transformer and applying them to a randomly initialized 'student' model. The goal is to use the teacher's knowledge without retraining the student from scratch.
Architectural Mismatch: This occurs when the structure of the pre-trained teacher model and the standard student model are not identical. The paper shows that this structural difference is a critical barrier, causing attention patterns to become non-functional for certain models.
Attention-Routing Channel: This specific part of the network is where the failure originates. It's not just about matching the overall attention pattern; it's about ensuring that the specific routing mechanisms within this channel are compatible and functional for the student model.

Terminology

Transformer — 2× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Vision Transformer (ViT) — 1× in this paper · explained in 2 episodes

Summary

The gist

Attention Transfer is not universally effective for Vision Transformers because its success depends critically on architectural compatibility between the teacher and student models, which prevents it from being a simple, architecture-agnostic knowledge transfer mechanism.

Failure Discovery

The study reveals that Attention Transfer is not universally effective across different Vision Transformer families because 4 families consistently fail to transfer successfully under the standard teacher-student protocol, falling up to 5.1% below the from-scratch no-transfer baseline <ref:2605.07191#pg4>. This failure is family-consistent across model sizes, persists under extended training durations, different transfer datasets, and out-ofdistribution evaluations <ref:2605.07191#pg5>. The researchers provide the first comprehensive benchmark of Attention Transfer across 20 teachers from 11 well-known ViT families, uncovering a critical validity boundary where 4 major families consistently fail to transfer to a standard student architecture <ref:2605.07191#pg4>.

Three-Level Localization

The failure of Attention Transfer is localized specifically to the attention-routing channel, indicating that the key issue is whether the matched patterns remain functional for the student <ref:2605.07191#pg5>. A component-level decomposition analysis showed that MLP-only initialization was consistently beneficial for both success and failure teachers, while Attention-only initialization preserved the success/failure split <ref:2605.07191#pg7>. Furthermore, QKV decomposition confirmed that transferring any of Q, K, or V alone remains beneficial for the success teacher DeiT-S but is harmful for the failure teacher DINOv2-S, with V producing the largest drop <ref:2605.07191#pg7>.

Architectural Compatibility as Mechanism & Fix

The primary mechanism behind Attention Transfer failure is identified as architectural mismatch between the pre-trained teacher and the standard student <ref:2605.07191#pg8>. The researchers identify specific missing native architectural components in failure families, such as LayerScale in DINOv2 [2] and DINOv2-wr [17], PreLayerNorm in CLIP [4], and RelativePositionBias in BEiTv2 [14] <ref:2605.07191#pg8>. By adding only the teacher’s native architectural components to the student in a randomly initialized state, the failure is completely reversed for all 4 families, confirming that these components specifically unlock the usability of the teacher’s attention <ref:2605.07191#pg9>.

Elimination of Alternative Explanations

The analysis systematically ruled out potential alternative explanations for the observed failure boundary <ref:2605.07191#pg10>. The study showed that different losses with sufficient magnitude can induce similar supervision signals, as increasing the loss magnitude further hurts performance for failure families <ref:2605.07191#pg9>. Similarly, testing various pre-training recipe properties—including different pre-training signals, data sources, and special attributes like attention sinks—found none that predict the family-level failure boundary <ref:2605.07191#pg10>. This strengthens the finding that architectural mismatch is the governing cause behind Attention Transfer failure <ref:2605.07191#pg9>.

Conclusion

The findings refine the understanding of attention in ViT representations, concluding that attention is sufficient only when the student architecture matches the teacher <ref:2605.07191#pg9>. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue demonstrates that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacher-student architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:2605.07191#pg9>. The native-architecture rescue shows that restoring these components in randomly initialized form is sufficient to make the transferred attention usable <ref:2605.07191#pg9>. This work encourages more careful reporting of teacherstudent architectural assumptions in transfer and distillation studies. The architectural compatibility principle concerns whether the student ViT preserves the architectural context under which the teacher learned its attention patterns <ref:

Improvements for AI systems

Attention Transfer Failure Mitigation: Implement an architectural compatibility check during model initialization to identify missing native components like LayerScale in DINOv2 [2] and DINOv2-wr [17], PreLayerNorm in CLIP [4], and RelativePositionBias in BEiTv2 [14].

This system will ensure that the transferred attention remains usable only when the student architecture provides the same attention-routing as the teacher’s, thereby preventing performance degradation seen in failure families.

Targeted Attention Routing Correction: Employ a dynamic mechanism to adjust attention patterns specifically within the attention-routing channel when architectural mismatch is detected, addressing the finding that the key issue is not simply whether the student can match the teacher’s attention patterns, but whether the matched patterns remain functional in the student’s attention-routing.
Loss Function Robustness Tuning: Instead of relying solely on standard Cross-Entropy (CE) loss for distillation, dynamically adjust the distillation loss weight λ based on model family characteristics to avoid larger drops with increasing λ observed in failure families under CE or MSE losses.

This ensures that the transfer signal is beneficial for success families while preventing it from causing harm in architectures that lack necessary components.

Component-Level Knowledge Preservation: When performing knowledge transfer, prioritize initializing and fine-tuning MLP blocks over attention blocks to maintain performance across different teacher families, as Table 3 shows MLP-only initialization is consistently beneficial for both the success teacher (DeiT-S) and the failure teachers (DINOv2-S and CLIP-B).