Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study

arXiv:2610.11205 · cs.CV, cs.LG · Submitted 2026-10-08 · Read on arXiv

cs.CV, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-09

Comments: Accepted in IEEE VCIP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Transformer — 27× in this paper · explained in 2 episodes
• Generalization — 16× in this paper · explained in 13 episodes
• Accuracy — 12× in this paper · explained in 1 episode
• Neural Architecture Search — 6× in this paper
• Transformer Architecture — 5× in this paper · explained in 2 episodes
• Feature Selection — 3× in this paper · explained in 4 episodes
• Distillation — 2× in this paper · explained in 2 episodes
• Entropy Collapse — 2× in this paper · explained in 2 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Vision Transformer (ViT) — 2× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Attention Entropy — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Core — 1× in this paper · explained in 2 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Feature Collapse
• TVT: Training-Free Vision Transformer Search on Tiny Datasets
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models