Structural Anchor Pruning: Training-Free Multi-Vector Compression for Visual Document Retrieval
cs.CV, cs.CL, cs.IR
Submitted: 2026-01-27
Updated: 2026-08-29
Comments: Accepted to EMNLP 2026 (Main Track)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive multi-vector index storage overhead.
Terminology
Abstract
Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive multi-vector index storage overhead. Existing training-free pruning methods either rely on heuristic layer choices or degrade sharply under aggressive compression, leading prior work to argue that effective high-compression pruning requires query-dependent training. We challenge this view with Structural Anchor Pruning (SAP), a self-calibrating, training-free, query-agnostic index-time framework combining (i) Score Retention (SR), a white-box per-layer compression diagnostic; (ii) SR-guided window selection, which automatically locates the structural pruning region of any backbone with no per-model hyperparameters; and (iii) a visual in-degree centrality scorer that identifies anchor patches within that window. On ViDoRe v1/v2 across three architectures spanning 18, 28, and 36 backbone layers, SAP retains 93--96% of NDCG@5 on v1 and 88--90% on the harder v2 while pruning 90% of visual tokens; at 20 times compression it retains 85--90% and 76--79% respectively. Our layer-resolved SR analysis reveals an Alignment-Aggregation Divergence: visual structure is preserved as a stable ``Structural Plateau'' within the backbone, while the final layers reshape it into a sparse, query-aligned form unsuitable for pruning. Probing the pre-retrieval base backbones shows that contrastive fine-tuning sharpens this boundary three- to eight-fold, explaining why final-layer methods fail.
Sources
- Qwen2.5-VL Technical Report
- PaliGemma: A versatile 3B VLM for transfer
- Token Merging: Your ViT But Faster
- ColPali: Efficient Document Retrieval with Vision Language Models
- Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings
- ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model
- Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
- DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models