Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
cs.LG, cs.AI, cs.CL, cs.CV
Submitted: 2026-09-26
Updated: 2026-09-26
Comments: 58 pages, 15 figures, 43 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large vision-language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the typical magnitude.
Terminology
Abstract
Large vision-language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the typical magnitude. The text spike systematically appears in early layers at a fixed initial position, independently of input content. Visual spikes vary across images, but whether their formation follows a consistent pattern across LVLMs and how they respond to image perturbations remain open questions. We find that some LVLMs do not form visual spikes, while others spike at different rates, typically in deeper layers. We identify the trigger direction from model weights and an interpretable location rule: before the language model decoder runs, eventual spike tokens are largely restricted to those sharing least with the rest of the image. Crucially, visual spikes are strikingly brittle. Common corruptions frequently create and relocate spikes, and less often remove them, raising overall incidence. Our trigger-guided spike attack deliberately creates or removes spikes under a small infinity budget, with 1/255 enough in nine of the ten models that spike. Finally, our preventive intervention removes only the trigger component before spikes erupt, eliminating or substantially reducing spikes on clean and perturbed images while leaving the other image tokens nearly unchanged. Our study spans 25 adapter-based LVLMs built on 18 released text-only bases from 10 families, ranging from 2B to 72B parameters.
Sources
- Pixtral 12B
- Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
- ActiveMark: on watermarking of visual foundation models via massive activations
- When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
- Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Gemma 2: Improving Open Language Models at a Practical Size
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- The Llama 3 Herd of Models
- Building and better understanding vision-language models: insights and future directions
- A More Word-like Image Tokenization for MLLMs
- LLaVA-OneVision: Easy Visual Task Transfer
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
- LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks