TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
cs.AI
Submitted: 2026-06-25
Updated: 2026-10-15
Comments: 27 pages, 18 figures
Code: https://github.com/haotian-liu/LLaVAhttps:
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial
Terminology
Abstract
Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.
Sources
- GPT-4 Technical Report
- AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
- Qwen Technical Report
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Token Merging: Your ViT But Faster
- GPT-4o System Card
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- The Llama 3 Herd of Models
- VideoPoet: A Large Language Model for Zero-Shot Video Generation
- LLaVA-OneVision: Easy Visual Task Transfer
- Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations
- Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model
- Instruction Tuning with GPT-4
- OpenAI GPT-5 System Card
- IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems