Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores
Vision Transformers face quadratic costs from dense self-attention. VECA proposes replacing this with an efficient core-periphery structure using a small set of learned 'core tokens.' This architecture allows patches to communicate indirectly through these cores, achieving linear computational scaling while maintaining high visual representation quality.