MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs".
Jane: Here's the gist of the paper. For multimodal large language models (MLLMs), visual information is relatively sparse compared with text, and this paper introduces MI-Pruner to efficiently prune visual tokens by directly computing Mutual Information between visual and textual features prior to their interaction This allows for a surgical approach that explicitly measures crossmodal dependency at the feature levels Experimental results demonstrate that this approach outperforms previous attention-based pruning methods with minimal latency Pruning redundant reasoning tokens The MI-Pruner is introduced as a simple, efficient and non-intrusive approach that directly quantifies token relevance through their Mutual Information It is built on the principle of maximizing crossmodal Mutual Information in the projection space to efficiently capture visual cues aligned with the textual query The method can be easily applied to a wide range of MLLMs following the Enc-MLP-Dec paradigm, making it model-agnostic How the training signal is built The process begins after the projection layer where visual embeddings V = v i and textual embeddings T = t j are extracted Similarity matrices are computed based on normalized embeddings using Boltzmann distributions to obtain crossmodal similarity rho cross ij and self-similarity rho self ij These similarity scores are then used to define conditional probabilities p(x j v i) which quantify the semantic alignment between (v i, t j) and the structural diversity among (v i, v j) Finally, Pointwise Mutual Information PMI(v i; x j) is adopted to measure token-level relevancy between embeddings Where the gains come from The selection of visual tokens is performed using a greedy search strategy based on a scoring function Si which balances crossmodal relevance and intra-modal redundancy The scoring function is defined as Si = λScross i − (1 − λ) · S self i where lambda balances the relevance and redundancy factors This greedy search holds a complexity of O(NV Nkeep) for classic methods but MI-Pruner achieves a complexity of O(NV + Nkeep log NV) due to the independence assumption The maximal aggregation serves as a tight lower bound for the sum of PMI, capturing the most salient semantic tokens and preserving alignment measure from noise Efficiency analysis The efficiency analysis shows that MI-Pruner’s complexity is O(NV · max(NT, Nkeep)) for full scoring and O(NV NT) for relevance-based sorting This superior computational efficiency is demonstrated in Table 9 where MI-Pruner achieves lower latency than methods like CDPruner which incurs a complexity of O(NV N 2keep) The method's non-intrusive nature and lack of reliance on attention collection make it very efficient with minimal latency Video understanding results Following previous work, MI-Pruner achieves average 93.19% accuracy across three datasets with merely 114 visual tokens in the Video-LLaVA-7B model This demonstrates the method's effectiveness in video domain tasks The results on Qwen3VL series show that MI-Pruner achieves SOTA performance for both scales when keeping 50% of visual tokens We propose MI-Pruner as a plug-in visual pruner for MLLMs by formulating token pruning as a subset selection problem leveraging both crossmodal and intra-modal Mutual Information The surgical manner enables the method to be applied to any off-the-shelf MLLMs with the most favorable efficiency Limitations The paper notes that the assumption of equal visual probability and conditional independence is made for tractability Future work involving an appropriate visual prior is expected to refine results Furthermore, the textual token pruning plays a significant role in “needle in a haystack” and “conflict detection” tasks Related work Attention-based pruning methods often suffer from heuristics and implementation bottlenecks such as attention sinks and multi-head diversity Diversity-based pruning methods prune tokens by similarity or conditional diversity measures MI-Pruner is theoretically sound and efficient compared to methods like MMToK which only provide empirical coverage The authors adopt softmax scores for a probabilistic MI proxy, achieving robust performance with best efficiency Impact statement This paper aims to prune visual tokens in MLLMs for efficient inference in latency- and resource-constrained scenarios The work is expected to contribute to significant social benefits by enabling efficient inference References Alvar, S.
Tom: R., Singh, G., Akbari, M., and Zhang, Y.
Jane: Divprune: Diversity-based visual token pruning for large multimodal models Bai, S., Cai, Y., Chen, R., et al.
Tom: Flashvlm: Text-guided visual token selection for large multimodal models Cao, J., Ye, P., Li, S., et al.
Paper summary: Tom: So we're looking at this paper called "MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs". The main idea here is that visual info in these models is pretty sparse compared to text.
Jane: Right, so they've developed MI-Pruner to efficiently cut down on those visual tokens by directly calculating the Mutual Information between the visual and textual features before they interact.
Tom: That’s the core thesis, and what they claim is that this lets them surgically measure crossmodal dependency right at the feature level. It’s supposed to be a simple, efficient, and non-intrusive way to prune tokens in these massive models without needing any architectural changes.
Lu: What I find interesting is how it’s built on maximizing crossmodal Mutual Information in the projection space to capture visual cues that line up with the textual query. It’s aiming for a principled objective for identifying tokens that hold shared and semantically aligned information, which information theory gives us as Mutual Information.
Meng: So if it works like that, what does that mean for us at the engineering side? Does this actually translate into something practical when we're talking about deployment costs and what breaks at scale?
Jane: That's a fair question, Meng. The paper claims this approach can be easily applied to a wide range of MLLMs following the Enc-MLP-Dec paradigm, which suggests it’s model agnostic. It’s supposed to preserve query relevant information while achieving state of the art performance with minimal latency.
Tom: Minimal latency is key for inference, and they show that MI-Pruner outperforms previous attention based pruning methods with very little delay. It’s a surgical approach instead of relying on internal attention maps or architectural modifications to decide which tokens to keep.
Lu: They define a scoring function that balances crossmodal relevance and intra modal redundancy using a lambda parameter, balancing the two factors we need to consider when selecting visual tokens. The complexity they achieve is O n v plus n keep log n v, which is much better than classic methods that are O n v N squared keep.
Meng: But what's the actual process for building that signal? How do they get those similarity scores and then derive the final relevance measure for each token?
Jane: They start after the projection layer where they extract visual embeddings, say V equals v i, and textual embeddings T equals t j. Then they compute similarity matrices using Boltzmann distributions to get crossmodal similarity rho cross ij and self similarity rho self ij.
Paper summary: Tom: And those similarities are used to define conditional probabilities p of x j given v i, which quantifies the semantic alignment between that pair, and finally they use point wise Mutual Information PMI of v i; x j to measure token level relevancy.
Lu: The maximal aggregation serves as a tight lower bound for the sum of PMI, which helps them capture the most salient semantic tokens while preserving that alignment measure from noise. That’s a strong theoretical anchor for what they're selecting.
Meng: So, if we look at the practical implementation—how do we actually deploy this on production traffic? What does this mean for my pipeline?
Jane: It means it’s non-intrusive because it doesn't require access to internal attention maps or any architectural changes, so it should be pretty straightforward to plug in if you follow the Enc-MLP-Dec paradigm.
Tom: I agree with that. The fact that they demonstrate this method on video understanding tasks, achieving average ninety-three point nineteen percent accuracy across three datasets with only one hundred fourteen visual tokens in the Video LLaVA seven billion model is quite impressive.
Lu: They also showed SOTA performance on Qwen three V L series when keeping fifty percent of the visual tokens, which really shows it’s effective in the video domain. It’s not just theoretical; it's hitting solid numbers on real models.
Meng: But we need to be careful—what are the limitations? What assumptions are they making that might not hold up when we run this on our specific model or data stream?
Jane: The paper notes that the assumption of equal visual probability and conditional independence is made for tractability, so future work involving an appropriate visual prior is expected to refine their results. They also mention that textual token pruning plays a significant role in "needle in a haystack" and "conflict detection" tasks.
Tom: So, while they’ve got this efficient approach, we have to keep in mind those assumptions about the data distribution when we try to generalize it outside of their specific setup. It’s not a universal fix for every problem.
Lu: I think the biggest shift here is moving away from attention based pruning that relies on heuristics, which often leads to things like attention sinks and multi head diversity issues. MI-Pruner offers a theoretically sound and efficient alternative compared to methods like MMToK which only provide empirical coverage.
Paper summary: Meng: So for me, what this means practically is that if we can implement this plug-in visual pruner efficiently, it could help us cut down on the computational load for inference in latency and resource constrained scenarios.
Jane: Exactly. The paper frames this as a way to prune visual tokens in MLLMs by formulating token pruning as a subset selection problem using both crossmodal and intra modal Mutual Information, enabling this surgical manner that's applicable to any off the shelf MLLM with the most favorable efficiency.
Tom: So we’ve seen the thesis, we’ve seen they build it on MI scores from similarity matrices, and we’ve heard about the speed gains compared to O n v N squared keep methods. It seems like a solid contribution to making visual pruning more principled rather than just a heuristic guess.
Lu: The paper focuses heavily on how information theory provides the rigorous mathematical framework for quantifying information transmission, and Mutual Information emerges as that natural metric for measuring statistical dependence between tokens.
Meng: I still need to see if this translates into real world savings, but theoretically, replacing complex attention based pruning with something governed by crossmodal MI sounds like it could reduce the operational overhead significantly.
Jane: The paper’s conclusion is that MI-Pruner is a plug in visual pruner for MLLMs by using both crossmodal and intra modal Mutual Information to select tokens, enabling this surgical manner applicable to any off the shelf MLLM with the most favorable efficiency.
Tom: It seems like they’ve given us a way to quantify token relevance based on shared information between modalities, which is a solid step forward from just looking at attention scores.
Lu: The main point is that it directly computes Mutual Information between visual and textual features prior to their interaction, which allows explicit measurement of crossmodal dependency at the feature levels.
Meng: So, for production deployment, we’d need to test this thoroughly against our actual traffic patterns to see if those O n v plus n keep log n v complexities hold up under real load.
Jane: That’s exactly where the next steps lie—testing that assumption of tractability and independence in a real operational setting. It seems like a method that could genuinely help us make inference faster without sacrificing the quality of the visual understanding.
Conclusion: Tom: So we’re wrapping up on MI-Pruner, this paper by Alvar and Singh et al., which is trying to prune visual tokens in large multimodal models for better efficiency, right?
Jane: Right, so they developed this method using Mutual Information between visual and textual features to decide which visual tokens matter most.
Tom: The title itself says it's a crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs, which basically tells us they’re focusing on that information overlap between the text and the visuals.
Lu: What they do is take those feature embeddings, V and T, extract similarity matrices based on Boltzmann distributions to get crossmodal similarity and self-similarity scores.
Meng: So you’re measuring how much the visual tokens actually share meaning with the textual query before they even fully interact in the model. That sounds like a solid way to find relevant stuff.
Jane: Exactly, and they use those scores to define conditional probabilities about which visual token aligns best with a specific text input.
Tom: And then they use point wise Mutual Information between the visual token and the text query to get that final relevance measure for pruning. It’s all about quantifying that statistical dependency at the feature level.
Lu: The whole goal is to find a way to surgically prune those redundant visual tokens without messing up the model's performance too much, which they call O n v plus n keep log n v complexity instead of O n v N squared keep.
Jane: So for us listening, it means they found a principled way to cut down on the visual token count based on actual information sharing rather than just looking at generic similarity measures.
Tom: It’s about taking that whole process and making it plug-in so we can apply it to any existing MLLM, which is a big deal for practical deployment, right?
Meng: Yeah, if they can do that with minimal latency and without changing the core architecture of the model itself, that could really help us get these large models running cheaper on our infrastructure.
Jane: And their results show it works well on video tasks too, hitting top performance levels on Qwen three V L series when they keep fifty percent of the visual tokens.
Tom: It’s a strong demonstration that this information-theoretic approach isn't just theoretical; it actually delivers better accuracy in complex vision tasks.
Lu: The authors admit they made some assumptions about equal visual probability for tractability, but they point to future work exploring an appropriate visual prior to refine those results.
Jane: So, while it’s effective now, the next steps involve making sure their assumptions hold up when we apply this to different kinds of visual data.
Tom: They also mention that pruning the textual tokens has a big impact on tasks like finding needles in a haystack or detecting conflicts in reasoning, which is another area they are looking at.
Meng: That’s interesting because it shows the method isn't just about visuals; it touches on how the whole multimodal reasoning process works.
Jane: So, MI-Pruner offers a way to make visual token pruning more principled by focusing on actual crossmodal information sharing, and we’ll be looking at how this idea fits into our deployment strategy next.
Jiameng Li, Aleksei Tiulpin, Matthew B. Blaschko
KU Leuven · Weill Cornell Medicine
cs.CV
Submitted: 2026-04-03
Updated: 2026-11-16
Comments: 9 pages
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: For multimodal large language models (MLLMs), visual information is relatively sparse compared with text…
Key concepts
- Mutual Information (MI)
- MI measures the statistical dependence between two sets of data, in this case, visual and textual features. It quantifies how much knowing one set of features tells you about the other. High MI indicates a strong, relevant connection between a visual token and a text query.
- Crossmodal Similarity ($ ho_{cross}$)
- This score is computed by comparing normalized embeddings from both the visual and textual modalities. It quantifies the semantic alignment or relevance between specific pairs of visual and textual tokens, helping to identify which visual tokens are most useful for a given text.
- Scoring Function ($S_i$)
- This function balances two factors: crossmodal relevance ($S_{cross}$) and intra-modal redundancy ($S_{self}$). By tuning the balance parameter ($\lambda$), the method selects visual tokens that are both highly relevant to the text and not redundant with other visual tokens, ensuring efficient pruning.
- Complexity Improvement
- MI-Pruner achieves a significantly better computational complexity of $O(NV + Nkeep ext{ log } NV)$ compared to previous methods like CDPruner's $O(NV N^2keep)$. This efficiency stems from using independence assumptions during the greedy search for token selection, making it much faster for large models.
Terminology
Summary
For multimodal large language models (MLLMs), visual information is relatively sparse compared with text, and this paper introduces MI-Pruner to efficiently prune visual tokens by directly computing Mutual Information between visual and textual features prior to their interaction This allows for a surgical approach that explicitly measures crossmodal dependency at the feature levels Experimental results demonstrate that this approach outperforms previous attention-based pruning methods with minimal latency
Pruning redundant reasoning tokens
The MI-Pruner is introduced as a simple, efficient and non-intrusive approach that directly quantifies token relevance through their Mutual Information It is built on the principle of maximizing crossmodal Mutual Information in the projection space to efficiently capture visual cues aligned with the textual query The method can be easily applied to a wide range of MLLMs following the Enc-MLP-Dec paradigm, making it model-agnostic
How the training signal is built
The process begins after the projection layer where visual embeddings V = v i and textual embeddings T = t j are extracted Similarity matrices are computed based on normalized embeddings using Boltzmann distributions to obtain crossmodal similarity rho cross ij and self-similarity rho self ij These similarity scores are then used to define conditional probabilities p(x j v i) which quantify the semantic alignment between (v i, t j) and the structural diversity among (v i, v j) Finally, Pointwise Mutual Information PMI(v i; x j) is adopted to measure token-level relevancy between embeddings
Where the gains come from
The selection of visual tokens is performed using a greedy search strategy based on a scoring function Si which balances crossmodal relevance and intra-modal redundancy The scoring function is defined as Si = λScross i − (1 − λ) · S self i where lambda balances the relevance and redundancy factors This greedy search holds a complexity of O(NV Nkeep) for classic methods but MI-Pruner achieves a complexity of O(NV + Nkeep log NV) due to the independence assumption The maximal aggregation serves as a tight lower bound for the sum of PMI, capturing the most salient semantic tokens and preserving alignment measure from noise
Efficiency analysis
The efficiency analysis shows that MI-Pruner’s complexity is O(NV · max(NT, Nkeep)) for full scoring and O(NV NT) for relevance-based sorting This superior computational efficiency is demonstrated in Table 9 where MI-Pruner achieves lower latency than methods like CDPruner which incurs a complexity of O(NV N 2keep) The method's non-intrusive nature and lack of reliance on attention collection make it very efficient with minimal latency
Video understanding results
Following previous work, MI-Pruner achieves average 93.19% accuracy across three datasets with merely 114 visual tokens in the Video-LLaVA-7B model This demonstrates the method's effectiveness in video domain tasks The results on Qwen3VL series show that MI-Pruner achieves SOTA performance for both scales when keeping 50% of visual tokens
Conclusion
We propose MI-Pruner as a plug-in visual pruner for MLLMs by formulating token pruning as a subset selection problem leveraging both crossmodal and intra-modal Mutual Information The surgical manner enables the method to be applied to any off-the-shelf MLLMs with the most favorable efficiency
Limitations
The paper notes that the assumption of equal visual probability and conditional independence is made for tractability Future work involving an appropriate visual prior is expected to refine results Furthermore, the textual token pruning plays a significant role in “needle in a haystack” and “conflict detection” tasks
Related work
Attention-based pruning methods often suffer from heuristics and implementation bottlenecks such as attention sinks and multi-head diversity Diversity-based pruning methods prune tokens by similarity or conditional diversity measures MI-Pruner is theoretically sound and efficient compared to methods like MMToK which only provide empirical coverage The authors adopt softmax scores for a probabilistic MI proxy, achieving robust performance with best efficiency
Impact statement
This paper aims to prune visual tokens in MLLMs for efficient inference in latency- and resource-constrained scenarios The work is expected to contribute to significant social benefits by enabling efficient inference
References
Alvar, S. R., Singh, G. p. 5, Akbari, M. p. 8, and Zhang, Y. p. 0 Divprune: Diversity-based visual token pruning for large multimodal models Bai, S. p. 8, Cai, Y., Chen, R., et al. Flashvlm: Text-guided visual token selection for large multimodal models Cao, J. p. 8, Ye, P., Li, S., et al. Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer Chen, L. p. 8, Zhao, H., Liu, T., et al. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large visionlanguage models Chen, Z. p. 8, Wang, W., Cao, Y., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks Dao, T. p. 8 FlashAttention-2: Faster attention with better parallelism and work partitioning Darcet, T. p. 8, Oquab, M. p. 8, Mairal, J. p. 8, and Bojanowski, P. p. 8 Vision transformers need registers Dong, S. p. 8, Hu, J., Zhang, M. p. 0, et al. Mmtok: Multimodal coverage maximization for efficient inference of vlms Fan, C. p. 8, Li, J., Zhang, T. p. 0, et al. Visipruner: Decoding discontinuous cross-modal dynamics for efficient multimodal llms Fujishige, S. p. 8 Submodular functions and optimization Gray, R. p. 8 M. Entropy and information theory Gunjal, A. p. 9, Yin, J., and Bas, E. p. 9 Detecting and preventing hallucinations in large vision language models Hudson, D. p. 9 A. and Manning, C. p. 5 D. Gqa: A new dataset for real-world visual reasoning and compositional question answering Iyer, R. p. 9, Khargoankar, N. p. 9, Bilmes, J. p. 9, and Asanani, H. p. 9 Submodular combinatorial information measures with applications in machine learning Jang, Y. p. 9, Song, Y., Yu, Y., et al. Tgifqa: Toward spatio-temporal reasoning in visual question answering Jaynes, E. p. 9 T. Information theory and statistical mechanics Jeddi, A. p. 9, Baghbanzadeh, N. p. 9, Dolatabadi, E. p. 9, and Taati, B. p. 9 Similarity-aware token pruning: Your vlm but faster Kang, S. p. 9, Kim, J., Kim, J., and Hwang, S. p. 9 See what you are told: Visual attention sink in large multimodal models Krasin, I. p. 9, Duerig, T. p. 9, Alldrin, N. p. 9, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification Krause, A. p. 9, Singh, A. p. 5, and Guestrin, C. p. 9 Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies Laurenc¸on, H. p. 9, Tronchon, L. p. 9, Cord, M., and Sanh, V. p. 9 What matters when building vision-language models? p. 9 Leng, S., Zhang, H. p. 0, Chen, G., et al. Mitigating object hallucinations in large visionlanguage models through visual contrastive decoding Li, B. p. 9, Zhang, Y. p. 0, Guo, D., et al. Llavaonevision: Easy visual task transfer Li, J. p. 9, Li, D., Savarese, S. p. 9, and Hoi, S. p. 9 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models Liu, H. p. 9, Li, C., Li, Y., and Lee, Y. p. 9 J. Llavanext: Improved reasoning, ocr, and world knowledge Lu, P. p. 9, Mishra, S. p. 9, Xia, T., et al. Learn to explain: Multimodal reasoning via thought chains for science question answering Nemhauser, G. p. 9 L., Wolsey, L. p. 9 A., and Fisher, M. p. 9 L. An analysis of approximations for maximizing submodular set functions Open AI. p. 9 Gpt-3.5 turbo OpenAI. Gpt-4.1 Potkins, F. p. 0, Langton, O., Smirnov, V., et al. Improve the accuracy and efficiency of large language models via dynamic token compression and adaptive layer pruning Singh, A., Natarajan, V., Shah, M., et al. Towards vqa models that can read Spadaro, G. p. 0, Renzulli, R., Bragagnolo, A., et al. Shannon strikes again! entropy-based pruning in deep neural networks for transfer learning under extreme memory and computation budgets Tang, L., Zhuang, X., Yang, B., et al. Not all tokens and heads are equally important: Dual-level attention intervention for hallucination mitigation Voita, E. p. 0, Talbot, D., Moiseev, F., et al. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned Wang, H. p. 0, Zhang, Z. p. 0, and Han, S. Spatten: Efficient sparse attention architecture with cascade token and head pruning Wang, P. p. 0, Bai, S., Tan, S., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution Wang, T. p. 9, Zhou, W., Zeng, Y., and Zhang, X. p. 0 Efficientvlm: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning Wen, Z., Gao, Y., Wang, S., et al. Token pruning in multimodal large language models: Are we solving the right problem? p. 0 Xiao, G., Tian, Y., Chen, B., et al. Efficient streaming language models with attention sinks Xu, D. p. 0, Zhao, Z., Xiao, J., et al. Video question answering via gradually refined attention over appearance and motion Yan, Z. p. 9, Chen, Y., Zhou, C., et al. Vision transformers with self-distilled registers Yang, S. p. 8, Chen, Y., Tian, Z., et al. Visionzip: Longer is better but not necessary in vision language models Yang, T. p. 7, Zhang, L. p. 0, Lin, J., et al. D-leaf: Localizing and correcting hallucinations in multimodal llms via layer-to-head attention diagnostics Ye, X. p. 9, Gan, Y., Ge, Y., et al. Atp-llava: Adaptive token pruning for large vision language models Yin, H. p. 9, Si, G., and Wang, Z. p. 2 Lifting the veil on visual information flow in mllms: Unlocking pathways to faster inference Zhang, Q. p. 0, Cheng, A. p. 8, Lu, M., et al. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms Zhang, Q. p. 7, Liu, M., Li, L., et al. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms Zhang, X. p. 0, Quan, Y., Shen, C., et al. Shallow focus, deep fixes: Enhancing shallow layers vision attention sinks to alleviate hallucination in lvlms Zhang, Y. p. 7, Fan, C.-K., Ma, J., et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference Zhang, Z. p. 9, Yadav, S., Han, F., and Shutova, E. Cross-modal information flow in multimodal large language models Zhao, S. p. 0, Wang, Z., Juefei-Xu, F., et al.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Crossmodal Mutual Information guided token pruning allows for a
surgical token pruning approach in the projection space, enabling reliable and non-intrusive token pruning which preserves queryrelevant information.
This means that instead of relying oninternal attention maps or architectural modifications,
the system directly computes Mutual Information between visual and textual features to select tokens. -
The system can achieve state-of-the-art performance with minimal latency by using a greedy search strategy defined by a scoring function Si which balances crossmodal relevance and intra-modal redundancy, such as
Si = λScrossi − (1 − λ) · Sselfi.
-
The MI-Pruner can be integrated into various off-the-shelf MLLMs following the Enc-MLP-Dec paradigm, making it a
modelagnostic module
that is easily applied to a wide range of MLLMs. -
The system can perform dynamic budget selection by allowing the pruning algorithm to
automatically decides the trade-off between accuracy and efficiency
given a minimum performance threshold and maximum computation limit, which is described as a future work direction.
Sources
- Qwen3-VL Technical Report
- Hallucination of Multimodal Large Language Models: A Survey
- FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- Similarity-Aware Token Pruning: Your VLM but Faster
- LLaVA-OneVision: Easy Visual Task Transfer
- Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs
- Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
- Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
- Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment