Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models

Omni2LoRA is a two-stage framework for efficient parametric memory compression in omnimodal language models (OLMs), designed to bypass the token bottleneck entirely by internalizing audio-visual context into model weights rather than pruning input tokens. The paper states: We introduce Omni2LoRA, a two-stage framework for efficient parametric memory compression via coherence-preserving context distillation that bypasses the token bottleneck entirely.

In Stage 1, a Perceiver hypernetwork processes intermediate representations from a frozen OLM to encode the multimodal context into a full-rank Low-Rank Adaptation (LoRA) adapter in a single forward pass. This enables zero-token inference by mapping an input audio-visual recording... into a discrete, instance-specific Low-Rank Adaptation (LoRA) adapter generated by a hypernetwork in a single forward pass.

To prevent the parameter footprint from scaling linearly with recording length, Stage 2 introduces a reinforcement-learned, discrete allocation policy that dynamically determines which subset of the generated rank directions to retain under a fixed, sub-linear adapter budget. This policy is optimized via Group Relative Policy Optimization (GRPO) using a novel advantage shaping mechanism based on unimodal counterfactuals, which explicitly penalizes the loss of audio-visual coherence, forcing the model to allocate its fixed sub-linear rank budget to synergistic cross-modal anchors rather than isolated visual features.

The main results show that "Omni2LoRA operating at a 30% rank budget outperforms direct full-context inference and strong token-compression baselines (OmniZip, OMAC, O-MARC) on four audio-visual question answering benchmarks, improving average accuracy by 8-12% over the strongest baseline and remaining stable under compression ratios as tight as 75%, where token-pruning methods degrade sharply. For example, on Qwen2.5-Omni-7B, our approach achieves a 53.2 average accuracy, exceeding O-MARC (51.1) and the full-context ceiling (47.2). On UGC-AVQA, By achieving 68.0% overall accuracy on this dataset, Omni2LoRA proves that parametric memory compression effectively bypasses the cross-modal information loss inherent to spatial-temporal token pruning."

The ablation analysis shows that "the O2L-GRPO policy entirely bridges this gap. By learning to allocate ranks specifically where audio and video intersect, it filters out query-irrelevant noise, allowing a sparse 30% subset to consistently outperform dense uncompressed representations. Under extreme compression, Omni2LoRA demonstrates exceptional stability, maintaining accuracy of 60.7 even at 75% compression."

For video frame scaling, "Omni2LoRA completely avoids the active token-space bottleneck by internalizing the temporal context directly into parametric memory. Rather than collapsing under high frame counts, our approach uniquely capitalizes on the expanded cross-modal evidence, scaling gracefully and monotonically to reach a peak accuracy of 46.2 at 1024 frames."

Regarding inference efficiency, "By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero, cutting per-query Time to First Token (TTFT) by up to 12× relative to full-context inference and amortizing to under 0.5s after a handful of queries. Specifically, On the 7B backbone, it averages 0.49s per question, outperforming both the full-context baseline (6.03s) and the strongest token-compression baseline, OMARC (3.45s). After five queries, effective latency falls to 0.82s (7B) and 0.72s (3B), ultimately plateauing near 0.43s."

The paper concludes: By eliminating multimodal tokens from the active context at inference time, our method reduces per-query TTFT by an order of magnitude and amortizes to sub-second latency across repeated queries.

Improvements to AI systems:

1. Zero-token multimodal inference: Replace token-pruning or token-compression modules in existing omnimodal LLMs (e.g., Qwen2.5-Omni, LLaVA-Next, GPT-4o-style systems) with a Perceiver hypernetwork that generates a task-specific LoRA adapter from raw audio-visual input in a single forward pass. This eliminates the active-context token bottleneck entirely, enabling the model to process arbitrarily long recordings without quadratic attention costs.

2. Reinforcement-learned adaptive rank allocation: Integrate a GRPO-based discrete allocation policy (with unimodal counterfactual advantage shaping) into any multimodal system that uses adapters or parameter-efficient fine-tuning. This allows the model to dynamically select which LoRA rank directions to keep under a fixed sub-linear budget, prioritizing synergistic cross-modal anchors over redundant unimodal features. The system learns to filter query-irrelevant noise automatically, improving accuracy even with only 30% of the full rank.

3. Parametric memory for long-form temporal context: Replace video frame sampling or frame-drop strategies with parametric memory compression. The system internalizes temporal context into weights, so performance scales gracefully with frame count (e.g., peak accuracy at 1024 frames) rather than degrading due to token limits. This enables processing hour-long videos or multi-hour audio streams without memory overflow.

4. Sub-second amortized inference for repeated queries: Convert multimodal context into a reusable, fixed-budget parameter state (the generated LoRA adapter). After the first query, subsequent questions on the same recording reuse this state, cutting per-query TTFT by up to 12× (from 6s to 0.5s) and amortizing to under 0.5s after five queries. This is ideal for interactive systems like virtual assistants, meeting summarizers, or real-time video Q&A bots.

5. Coherence-preserving compression under extreme constraints: For deployment on edge devices or bandwidth-limited environments, the system can operate at 75% compression with only 2-3% accuracy drop (e.g., 60.7% vs. 62% full), whereas token-pruning baselines degrade sharply. This enables high-accuracy multimodal reasoning on low-memory hardware.

6. Cross-modal counterfactual training signal: Use the unimodal counterfactual advantage shaping to train any multimodal fusion layer to explicitly preserve audio-visual coherence. This improves robustness to missing or noisy modalities, as the system learns to allocate capacity to where audio and video intersect, not just to dominant visual features.

What the improved AI system can do:

• Answer complex audio-visual questions (e.g., Why did the speaker pause after showing the chart?) with higher accuracy (8-12% better than the strongest token-compression baseline) while using only 30% of the parameter budget.

• Process a 1-hour lecture video or a 30-minute podcast with no token limit, maintaining accuracy that improves with more frames (up to 1024) instead of collapsing.

• Respond to a series of follow-up questions on the same video in under half a second each, after an initial 0.5s setup—enabling real-time interactive tutoring, video search, or surveillance analysis.

• Run on a single consumer GPU (7B model) with sub-linear memory growth, compressing multimodal context into a few hundred MB of adapter weights, rather than storing thousands of tokens in the KV cache.

• Maintain stable performance even when the input is extremely long or noisy, because the GRPO policy filters out irrelevant cross-modal noise and focuses on synergistic anchors—making it suitable for unconstrained user-generated content (e.g., TikTok videos, vlogs, meeting recordings).

Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent token compression methods attempt to alleviate this burden, compressing modalities in isolation often destroys the temporal cross-modal anchors necessary for coherent reasoning. We introduce Omni2LoRA, a two-stage framework for efficient parametric memory compression via coherence-preserving context distillation that bypasses the token bottleneck entirely. First, a Perceiver hypernetwork processes intermediate representations from a frozen OLM to encode the multimodal context into a full-rank Low-Rank Adaptation (LoRA) adapter in a single forward pass. To prevent the resulting parameter footprint from scaling linearly with recording length, we optimize a discrete rank allocation policy via Group Relative Policy Optimization (GRPO) that uses a modality-ablated counterfactual reward to explicitly penalize the loss of audio-visual coherence, forcing the model to allocate its fixed sub-linear rank budget to synergistic cross-modal anchors rather than isolated visual features. Across three omnimodal backbones, Omni2LoRA operating at a 30% rank budget outperforms direct full-context inference and strong token-compression baselines (OmniZip, OMAC, O-MARC) on four audio-visual question answering benchmarks, improving average accuracy by 8-12% over the strongest baseline and remaining stable under compression ratios as tight as 75%, where token-pruning methods degrade sharply. By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero, cutting per-query Time to First Token (TTFT) by up to 12x relative to full-context inference and amortizing to under 0.5s after a handful of queries.