R squared-Omni: Reducing Structural and Precision Redundancy in Omni-LLMs for Long-Form Generation
cs.CV
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: 32 pages, 9 figures
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Omni-modal large language models (Omni-LLMs) typically process large numbers of multimodal tokens from audio-visual inputs…
Terminology
Abstract
Omni-modal large language models (Omni-LLMs) typically process large numbers of multimodal tokens from audio-visual inputs, incurring substantial computational costs and a large memory footprint for the key-value (KV) cache. Aggressive token pruning can alleviate this problem while maintaining competitive performance on multiple-choice tasks, which require only a single option as output. However, we observe substantial quality degradation when generating long-form, open-ended responses. We attribute this gap to query-guided selection, which can retain sufficient evidence for answer selection while discarding details needed for comprehensive generation. This motivates reducing overlapping information across tokens, termed structural redundancy, while preserving diverse content. However, the retained representations may still contain precision redundancy due to unnecessarily high numerical precision in their cached values. To address these two levels of redundancy in token representations, we introduce a training-free framework named R squared-Omni. First, Structure-Aware Token Consolidation (STC) consolidates redundant representations using spatiotemporal similarity in video and local continuity in audio. Then, Semantic-Sensitivity Guided Precision Allocation (SSPA) allocates value precision to the retained representations according to the estimated reduction in attention-output error under quantized keys, with query attention and audio-visual alignment providing a semantic prior. Extensive experiments on long-form generation benchmarks demonstrate an average 5.0 times reduction in the memory footprint while preserving 97.98% of baseline generation quality and achieving a 1.48 times end-to-end speedup, highlighting the benefits of jointly reducing structural and precision redundancy.
Sources
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
- MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
- Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence
- Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
- video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
- TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
- OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models
- Stage-adaptive Token Selection for Efficient Omni-modal LLMs
- OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
- LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
- Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
- ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
- EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs
- Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models
- Towards Joint Quantization and Token Pruning of Vision-Language Models
- QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs