Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
cs.CV
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: 29 pages including references and appendices, 10 figures. Submitted to ICLR 2027
License: http://creativecommons.org/licenses/by/4.0/
The gist: Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses.
Terminology
Abstract
Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.
Sources
- LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
- JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
- Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
- On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models