Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models
cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: 21 pages including references and appendix
License: http://creativecommons.org/licenses/by/4.0/
The gist: The inference efficiency of Multimodal Large Language Models (MLLMs) is severely constrained by massive visual token sequences induced by high-resolution inputs, with computational cost scaling
Terminology
Abstract
The inference efficiency of Multimodal Large Language Models (MLLMs) is severely constrained by massive visual token sequences induced by high-resolution inputs, with computational cost scaling quadratically. Existing approaches primarily focus on downstream token compression, while overlooking a fundamental upstream inefficiency: input resolution is treated as a static, task-agnostic hyperparameter. We propose Task-Conditioned Resolution Routing (TCRR), which formulates visual compression as a task-conditioned decision and employs a lightweight cross-modal router that conditions backbone visual representations on textual semantics via feature-wise modulation and cross-attention to predict the minimal sufficient compression level per query. To support this, we curate a dataset of 500k samples across 12 task categories, labeled via a teacher-oracle pipeline to approximate Pareto-optimal compression scales. Extensive experiments across diverse architectures show that TCRR achieves a superior efficiency frontier, specifically reducing visual FLOPs by 40.9% and latency by 53.7% on Qwen3-VL-8B while preserving competitive performance. Further analysis of scaling behavior confirms that dynamically routing visual compression enables optimal resource allocation without modifying the MLLM backbone.
Sources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Token Merging: Your ViT But Faster
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
- GPT-4 Technical Report
- Qwen3-VL Technical Report
- HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
- FineVision: Open Data Is All You Need
- LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models