Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring
Dyna-DINO introduces Adaptive Representation Anchoring, a training curriculum for distilling large Vision Transformers into smaller models. It solves the teacher-student gap by adaptively selecting intermediate teacher features based on Centered Kernel Alignment (CKA) similarity. This method guides student learning through a structured progression, leading to faster convergence and better performance than static distillation schedules.