OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

The gist: OmniSelect, a training-free, modality-adaptive token pruning framework, dynamically selects appropriate compression strategies for multimodal inputs to maintain strong performance while achieving inference speedup and memory reduction.

OmniSelect Framework Overview

OmniSelect is a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs and performs fine-grained token pruning within each temporal group, adaptively allocating pruning ratios to preserve informative tokens across modalities The paper proposes OmniSelect as a training-free framework that utilizes a two-stage strategy to dynamically compress visual and audio tokens This framework is designed to avoid the pitfalls of one-size-fits-all compression by explicitly modeling modality preference and enabling dynamic strategy selection

Modality-Aware Dynamic Ratio Allocation

The first stage of OmniSelect involves calculating similarity scores between visual and audio representations using AudioCLIP to define the pruning ratio ρi for each temporal group i The process categorizes the strategy into three distinct types: Uniform Pruning, Video-Centric Pruning, and Audio-Centric Pruning The similarity scores are computed as s(v)i = vi·t∥vi∥∥t∥ and s(a)i = ai·t∥ai∥∥t∥, where vi, ai, and t denote the visual, audio, and textual embeddings respectively Based on the score difference and a threshold θ, the process divides pruning into three cases: Uniform Pruning if s¯(v) − s¯(a) ≤ θ, Video-Centric Pruning when s¯(v) > s¯(a), and Audio-Centric Pruning otherwise An adaptive allocation strategy is then used to satisfy the global pruning budget while preserving score-based importance, where ρi,m is defined based on the modality and similarity scores in Equation 4

Temporal Group Pruning Pipeline (TGP2)

The second stage employs the Temporal Group Pruning Pipeline (TGP2) to prune redundant multimodal tokens within each temporal group i For audio token pruning, the method computes an attention matrix from the last layer of the audio encoder Ea and selects salient audio tokens using a Top-K selection based on the average attention score For vision token pruning, it computes the cosine similarity matrix of visual embeddings Vi and selects salient tokens via a Bottom-K selection to retain visually distinctive tokens The retained token number for each temporal group is then computed as Ki,m = max (1, ⌊(1 − ρi,m%) · ni⌋)

Experimental Results and Efficiency

Experimental results demonstrate that OmniSelect delivers exceptional performance on audio-visual understanding tasks and achieves competitive performance among existing training-free token compression methods OmniSelect achieves an inference speedup of 1.19× to 1.33× while reducing GPU memory consumption by 2.61GB to 2.81GB Specifically, when compressing 70% of the audio and video tokens, OmniSelect achieves a speedup of 1.33× during inference Furthermore, OmniSelect maintains strong robustness under aggressive compression, preserving 94.3%–97.4% of the original model accuracy on the WorldSense benchmark

Conclusion

OmniSelect presents a training-free, modality-adaptive token compression framework for Omni-LLMs that dynamically selects appropriate compression strategies and performs fine-grained token pruning within each temporal group Extensive experimental results on audio-visual understanding tasks with two OmniLLMs (3B and 7B parameters) demonstrate that OmniSelect achieves competitive performance among existing training-free token compression approaches OmniSelect achieves an inference speedup of 1.19× to 1.33× while reducing GPU memory consumption by 2.58G–2.77G when compressing 70% of the audio and video tokens Despite these substantial resource savings, OmniSelect retains 94% to 99% of the original full-token accuracy The paper concludes that OmniSelect achieves a favorable trade-off between inference efficiency and reasoning quality

How it works

OmniSelect is fully training-free and utilizes a two-stage strategy to dynamically compress visual and audio tokens First, it calculates the similarity scores of visual-text and audio-text pairs using AudioCLIP to define the pruning ratio ρi for each temporal group i This step also categorizes the pruning strategy into three distinct types: Uniform Pruning, Video-Centric Pruning, and Audio-Centric Pruning Second, it employs the Temporal Group Pruning Pipeline (TGP2) to prune audio and video tokens based on attention scores and cosine similarity matrix within each temporal group i

More Experimental Details

The threshold θ controls the transition between modality-centric pruning and Uniform pruning, with θ = 0 for small modality gaps and θ = 5 for large gaps The significance of pruning strategy classification is validated by evaluating performance against Audio-Centric, Video-Centric, and Uniform pruning strategies across different compression constraints The results show that OmniSelect consistently achieves the best performance across all settings in the comparison of different token compression strategies

Limitations and Future Work

The current strategy selection mechanism relies on manually designed threshold-based rules, which may not fully capture the complex and continuously changing relationships between audio and visual modalities across different queries The framework also depends on AudioCLIP to estimate cross-modal relevance, which cannot always accurately distinguish fine-grained modality importance Future work could explore stronger yet lightweight modality-aware models specifically optimized for multimodal importance estimation and dynamic strategy prediction The paper suggests that the optimal strategy often exists within the proposed strategy space, and the main limitation lies in strategy selection rather than representation capacity

A Preliminary Analysis

Preliminary analysis is provided in Appendix A to show how visual and audio tokens are grouped into temporal chunks before entering the LLM backbone The similarity scores are computed as s(v)i = vi·t∥vi∥∥t∥ and s(a)i = ai·t∥ai∥∥t, where vi, ai, and t denote the visual, audio, and textual embeddings respectively The averaged scores s¯(v) and s¯(a) reflect the overall relevance of each modality to the query Based on these scores, pruning becomes modality-aware by adopting Video-Centric Pruning when s¯(v) > s¯(a), and Audio-Centric Pruning otherwise The adaptive allocation strategy is designed to satisfy the global pruning budget while preserving score-based importance through iterative refinement

Ablation Study of Threshold θ

The ablation study of threshold θ reveals a U-shaped performance trend as θ varies, with modality-centric pruning at θ = 0 achieving 41.93% and 42.

1. Bold header: Dynamic Modality-Aware Pruning Strategy Implementation

This improvement involves replacing fixed pruning ratios with a dynamic allocation mechanism defined in Equation 3, which iteratively refines pruning ratios to satisfy a global pruning budget while preserving score-based importance. This allows the system to dynamically select appropriate compression strategies for multimodal inputs and adaptively allocating pruning ratios to preserve informative tokens across modalities.

2. Bold header: Cross-Modal Relevance Estimation via AudioCLIP

The system can be improved by leveraging AudioCLIP [7] not just for initial categorization, but as a continuous relevance estimator during inference. This allows the framework to move beyond static threshold decisions by using the similarity scores (Equation 1) to determine if visual or audio information is more informative in real-time for every temporal group.

3. Bold header: Attention-Guided Token Selection for Salient Visual Tokens

The token pruning pipeline can be enhanced by explicitly adopting the Bottom-K strategy for visual tokens, as detailed in Algorithm 1 (Step 21). This ensures that the system retains visually distinctive tokens that are more likely to align with textual semantics, which is crucial for maintaining high reasoning performance under aggressive compression.

4. Bold header: Robust Strategy Selection via Any-Correct Evaluation

The framework can be made more robust by adopting the relaxed evaluation setting described in Section 3.3, where a sample is correct if any one of the three strategies produces a correct prediction. This mitigates reliance on potentially inaccurate threshold estimations, as shown by the result that this setting achieves performance very close to the Full Tokens setting.

5. Bold header: Performance Gains in Efficiency and Accuracy Trade-off

The improved system will be able to achieve a inference speedup of 1.19× to 1.33× while simultaneously reducing GPU memory consumption by 2.58GB to 2.77GB, maintaining 94% to 97.4% of the original model accuracy on the WorldSense benchmark.

Omnimodal large language models (OmniLLMs) have recently gained increasing attention for unified audio-video understanding. However, processing long multimodal token sequences introduces substantial computational overhead, making efficient token compression crucial. Existing methods typically rely on fixed, modality-specific guidance, which fails to account for the varying importance of modalities across different queries. To address this limitation, we propose OmniSelect, a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs. Specifically, we leverage a lightweight AudioCLIP model to estimate cross-modal relevance and categorize each input into three pruning regimes: Audio-Centric, Video-Centric, and Uniform pruning. Based on these relevance scores, OmniSelect further performs fine-grained token pruning within each temporal group, adaptively allocating pruning ratios to preserve informative tokens across modalities. By explicitly modeling modality preference and enabling dynamic strategy selection, OmniSelect effectively avoids the pitfalls of one-size-fits-all compression. Extensive experiments demonstrate that our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.