Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs
Gaze Attention introduces a mechanism for Multimodal Large Language Models (MLLMs) to selectively focus on relevant visual areas instead of attending to all pixels. It achieves this by grouping visual tokens into compact 'gaze regions' and dynamically routing attention based on these regions. This results in more focused attention, significantly reducing computational overhead and the memory required for processing images.