SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
SWARD transfers knowledge from large vision foundation models to small convolutional networks for dense prediction tasks like segmentation. It uses Multi-Scale Windowed Attention Distillation (MWAD) to align teacher and student attention patterns across different spatial scales, and Prototype Discriminative Regularization (PDR) to shape the student's feature space for better class separation, achieving state-of-the-art results.