OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation

arXiv:2605.30519 · cs.CV · Submitted 2026-05-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation".

Jane: Here's the gist of the paper. OmniMem proposes an explicit full-range memory retrieval framework that performs sparse KV retrieval over the historical cache to address computational challenges in long video generation The framework improves Dynamic Degree by 52.3% and preserves strong consistency over strong baselines while maintaining comparable memory usage Problem Formulation The core problem addressed is the growing KV cache in autoregressive (AR) video generation, which increases both memory consumption and attention computation as video length increases A simple solution of discarding distant KV states weakens long-range dependencies and causes the generated subject to gradually lose its identity, resulting in noticeable drift across frames Existing methods often lead to repetitive motion patterns or may lose fine-grained details needed for long-horizon consistency OmniMem formulates long-range memory access as an explicit retrieval problem over the historical KV cache, aiming to preserve informative long-range dependencies while avoiding dense attention over the full history Addressing Local Bias Local bias arises because attention scores favor nearby KV blocks, concentrating selection around the current chunk and failing to retrieve long-range historical memory This is caused by strong spatiotemporal correlation among nearby video tokens, amplification under AR video DiTs, and a granularity mismatch between KV blocks and video chunks OmniMem mitigates this with Adaptive Window Exclusion, which removes KV blocks covered by the sliding window from selection candidates when sufficient long-range history is available This exclusion is conditional on the availability of outside-window blocks to prevent destabilization when history is short Mitigating Union Explosion Union Explosion occurs because thousands of query tokens are processed in parallel, and different queries or attention heads may select different historical regions, leading to a large union of selected KV blocks This divergence manifests as cross-query divergence across queries and cross-head divergence across attention heads To handle this, OmniMem introduces Query-Shared KV Selection to reduce cross-query selection diversity by assigning a single shared selection to adjacent query tokens It also employs Per-Head Scattered KV Access, which avoids materializing a large merged cache across attention heads by allowing each head to directly retrieve its own selected non-contiguous KV blocks Experimental Results and Efficiency Experiments on long-video generation show that OmniMem sets a new state of the art for long video generation, simultaneously improving motion dynamics and long-range consistency over prior designs while incurring only 1.7% additional memory over LongLive OmniMem improves the Dynamic Degree score by 52.3% over the strongest baseline on VBench-Long while preserving visual consistency with only 1.7% additional memory overhead In a multi-prompt setting, OmniMem maintains stable CLIP scores as the video length grows, with only a 2.

Paper summary: Tom: The paper starts by framing long-range memory access as an explicit retrieval problem over the historical key value cache, which is a big conceptual move away from just truncating or compressing the data implicitly <ref:2605.30519#pg1>.

Jane: They are addressing two main issues upfront: first, local bias, and second, union explosion when dealing with thousands of query tokens running in parallel.

Lu: Local bias happens because attention scores naturally favor nearby key value blocks, meaning the system tends to only look at recent history rather than digging deep into older parts of the video <ref:2605.30519#pg2>.

Meng: That’s a real problem for quality; if you only look at the last few seconds, you lose the context of what happened earlier in the sequence.

Tom: To fix that, OmniMem uses something called Adaptive Window Exclusion to remove key value blocks covered by the sliding window from selection candidates if there is enough long-range history available <ref:2605.30519#pg3>.

Jane: That sounds like a smart way to force the system to look further back when it needs to, provided it can find something useful outside the immediate view.

Lalam: They also introduced Query-Shared KV Selection, which reduces how many different historical regions different queries select by assigning one shared selection to adjacent query tokens <ref:2605.30519#pg3>.

Tom: So they’re not just looking at things independently; they are trying to coordinate the retrieval across the parallel queries to make sure everyone is looking at relevant history.

Jane: And on top of that, Per-Head Scattered KV Access lets each attention head directly grab its own selected non-contiguous key value blocks instead of merging them all into one big cache for every single head <ref:2605.30519#pg3>.

Conclusion: Tom: So, looking at the OmniMem paper, the authors are proposing this full-range retrieval framework to explicitly handle long-range dependencies in video generation by solving local bias with adaptive window exclusion and union explosion with query sharing and scattered access <ref:2605.30519#pg3>.

Jane: What this means for us is that we can keep generating longer videos without the subject losing its identity or the motion becoming repetitive, even if we are still using a chunk-based generation process <ref:2605.30519#pg2>.

Lu: The implication is that memory doesn't have to be just about aggressive truncation anymore; we can afford to keep more explicit history and still run efficiently by retrieving only what's necessary for each step <ref:2605.30519#pg3>.

Meng: From a production standpoint, if this works as well as the benchmarks suggest, it means we can deploy models that generate longer, more coherent video content without needing massive upfront memory or relying on risky implicit compression techniques <ref:2605.30519#pg1>.

Tom: I think the authors successfully showed a way to improve motion dynamics and long-range consistency over prior designs while only adding one point seven percent memory overhead compared to LongLive, which is a key comparison point <ref:2605.30519#pg3>.

Jane: Overall, OmniMem is about making the retrieval process smarter and more explicit so that we don't have to sacrifice long-range detail for short-term speed or memory savings.

Lin Zhao, Yushu Wu, Yifan Gong, Yanzhi Wang, Pu Zhao

Northeastern University · Adobe Research

cs.CV

Submitted: 2026-05-28

Updated: 2026-11-06

Comments: 31 pages, 13 figures; project page: https://wuyushuwys.github.io/OmniMem/

Project page: https://wuyushuwys.github.io/OmniMem

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: OmniMem proposes an explicit full-range memory retrieval framework that performs sparse KV retrieval over the historical cache to address computational challenges in long video generation The…

Key concepts

KV Cache
The Key-Value (KV) cache stores information about previous parts of a video sequence to help the AI model generate the next frame. As videos get longer, this cache grows very large, causing high memory usage and slow computation.
Dynamic Degree
This metric measures how effectively the model can utilize its available memory resources during generation. OmniMem aims to increase this score by 52.3%, meaning the system is much better at using the existing memory efficiently without needing more hardware.
Local Bias
This occurs when attention mechanisms only look at nearby parts of the historical video data, ignoring important information from much earlier in time. OmniMem uses Adaptive Window Exclusion to prevent this by selectively retrieving distant, relevant memory blocks.

Terminology

Summary

OmniMem proposes an explicit full-range memory retrieval framework that performs sparse KV retrieval over the historical cache to address computational challenges in long video generation The framework improves Dynamic Degree by 52. p. 03% and preserves strong consistency over strong baselines while maintaining comparable memory usage

Problem Formulation

The core problem addressed is the growing KV cache in autoregressive (AR) video generation, which increases both memory consumption and attention computation as video length increases A simple solution of discarding distant KV states weakens long-range dependencies and causes the generated subject to gradually lose its identity, resulting in noticeable drift across frames Existing methods often lead to repetitive motion patterns or may lose fine-grained details needed for long-horizon consistency OmniMem formulates long-range memory access as an explicit retrieval problem over the historical KV cache, aiming to preserve informative long-range dependencies while avoiding dense attention over the full history

Addressing Local Bias

Local bias arises because attention scores favor nearby KV blocks, concentrating selection around the current chunk and failing to retrieve long-range historical memory This is caused by strong spatiotemporal correlation among nearby video tokens, amplification under AR video DiTs, and a granularity mismatch between KV blocks and video chunks OmniMem mitigates this with Adaptive Window Exclusion, which removes KV blocks covered by the sliding window from selection candidates when sufficient long-range history is available This exclusion is conditional on the availability of outside-window blocks to prevent destabilization when history is short

Mitigating Union Explosion

Union Explosion occurs because thousands of query tokens are processed in parallel, and different queries or attention heads may select different historical regions, leading to a large union of selected KV blocks This divergence manifests as cross-query divergence across queries and cross-head divergence across attention heads To handle this, OmniMem introduces Query-Shared KV Selection to reduce cross-query selection diversity by assigning a single shared selection to adjacent query tokens It also employs Per-Head Scattered KV Access, which avoids materializing a large merged cache across attention heads by allowing each head to directly retrieve its own selected non-contiguous KV blocks

Experimental Results and Efficiency

Experiments on long-video generation show that OmniMem sets a new state of the art for long video generation, simultaneously improving motion dynamics and long-range consistency over prior designs while incurring only 1.7% additional memory over LongLive OmniMem improves the Dynamic Degree score by 52. p. 63% over the strongest baseline on VBench-Long while preserving visual consistency with only 1. p. 27% additional memory overhead In a multi-prompt setting, OmniMem maintains stable CLIP scores as the video length grows, with only a 2. p. 7

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive Window Exclusion improves block selection by remov[ing] local-window blocks from the selection candidates when sufficient long-range history is available, preserving the sparse budget for informative long-range retrieval. This prevents local bias in sparse KV selection by ensuring that selection attention retrieves complementary long-range memory instead of being dominated by nearby KV blocks.

  2. Query Shared KV Selection reduces cross query diversity by letting nearby query tokens share the same sparse KV selection, which mitigates the worst-case selected block union from S times K to R times K while preserving diversity across query groups.

  3. Per-Head Scattered KV Access avoids expanding head-specific selections into a large selected KV buffer by allowing each attention head to retrieve non-contiguous KV blocks according to its own selection pattern, which scales the memory footprint with the sum of per-head selections rather than the union across heads.

Sources

Related papers