SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation

summary

The gist

Streaming video generation models typically rely on temporal-centric memory, which organizes historical context as raw frames, chunk segments…

In short

SlotMemory introduces an object-centric Key-Value memory for streaming video generation to solve identity drift and semantic inconsistency. Instead of using raw temporal data, it decomposes transformer memory into discrete, reusable semantic slots based on 'what' is being represented. This approach allows the model to maintain consistent visual entities and prompt adherence across long, interactive narratives.

Key concepts

Temporal-centric Memory
Traditional models organize history as raw frames or tokens in time. This causes identity drift when objects leave the frame or during prompt changes because the memory is organized by 'when' an event happened rather than 'what' the entity is.
SlotMemory (Object-Centric KV Memory)
This mechanism shifts memory abstraction to discrete semantic slots. It treats memory as a Key-Value system where each slot represents a specific object or concept, allowing the model to retrieve relevant visual information based on its content rather than its chronological position.
Slot Attention Mechanism
During writing, this mechanism partitions the high-dimensional transformer manifold into discrete semantic regions. Low-dimensional slot embeddings act as routing addresses to identify and extract the specific KV tokens corresponding to a desired object's representation.

Terminology used across episodes

This episode discusses

The paper

SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation · Read on arXiv

Fudan University · Meta Superintelligence Labs · Baidu

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation".

Jane: Here's the gist of the paper. Streaming video generation models typically rely on temporal-centric memory, which organizes historical context as raw frames, chunk segments, or unclustered tokens This organization frequently leads to identity drift and semantic inconsistency when entities exit the frame or during interactive prompt transitions SlotMemory proposes an object-centric Key-Value (KV) memory mechanism for streaming video diffusion that shifts the memory abstraction from “when” an event occurred to “what” is being represented by decomposing the transformer’s KV manifold into discrete, reusable semantic slots Evaluated on 60-second interactive narratives using the Wan2.1-T2V-1.3B backbone, SlotMemory achieves a state-of-the-art quality score of 81.61 and a 22.8% relative improvement in dynamic consistency over the strongest existing streaming baseline.

Tom: Problem Formulation and Memory Abstraction The paper frames long-form, multi-prompt video synthesis as a boundedmemory streaming generation problem The objective is to synthesize a video V1:T in an autoregressive manner across C discrete chunks of length Tc The system maintains three distinct states: a short-term local KV cache Cc for local context, a text-conditioning cache Ctextc for prompt-dependent features, and a long-term memory bank Bc for persistent history The primary challenge is the management of the long-term bank Bc under a fixed capacity constraint Nbudget The optimization problem is formulated as max Bc: Bc ≤Nbudget E Consistency(V1:T) + PromptAdherence(V1:T, P) − λ Cost(Bc).

Jane: Architectural Mechanism of SlotMemory SlotMemory redefines the fundamental memory primitive as a slot-conditioned semantic storage The system operates via a recursive read-write-update cycle synchronized with the autoregressive generation of video chunks During the writing phase, it utilizes a temporally-initialized Slot Attention mechanism to partition highdimensional transformer manifolds into discrete, reusable semantic regions The low-dimensional slot embeddings are utilized solely as semantic routing addresses rather than the primary storage medium For each slot s, the model identifies the Rt tokens with the highest assignment weights, Icl,s = TopR(Acl,s,:), and extracts their corresponding KV tokens.

Tom: Retrieval and Adaptation Strategy Before denoising chunk c, the model retrieves a subset of memory items whose semantic content aligns with the current prompt and visual state For each item mi, a fused relevance score is computed: r(mi) = γ · rtext(mi, pc) + (1 −γ) · rvis(mi, qc), where pc is the active prompt The top-k ranked items are then integrated into the attention mechanism AttnQc l,

Klocal l;Kslot l: ,

Vlocal l;Vslot l: Following generation, new slot-conditioned items are written to the bank Bc′, and a relevance-driven eviction policy manages capacity under a fixed budget.

Title and authors: Tom: So, SlotMemory is basically changing how video models remember things by switching from remembering raw time slices to remembering specific objects through these key value slots <ref:2605.31033#pg1>.

Jane: That’s the core idea, right? Instead of the model just seeing a stream of frames, it’s organizing that memory into discrete semantic slots that represent things like "that person," or "that specific chair."

Lu: Exactly. The paper frames long video generation as a bounded memory problem, and they optimize for balancing how consistent the video is with how well it follows the prompt, all while keeping the size of this object memory bank under a certain budget <ref:2605.31033#pg1>.

Meng: And what I’m hearing that sounds like they are building a system that can keep track of entities across sixty seconds, which is way longer than what we usually see in these demos <ref:2605.31033#pg2>.

Tom: Right, and they show that this structured memory gives them a state of the art quality score of eighty-one point six one on interactive narratives <ref:2605.31033#pg2>. That’s a big number when you’re talking about maintaining consistency during a long story.

Jane: It means that even if the scene changes or someone walks out of frame, the model can still refer back to what that person looked like earlier, because it has a specific slot for them <ref:2605.31033#pg2>. That’s entity persistence in action.

Lu: The authors detail how they do this by using a recursive read-write update cycle, where they use these slot embeddings only as addresses to route the high dimensional transformer features <ref:2605.31033#pg1>. It’s not about storing everything; it’s about smart retrieval.

Meng: But what happens when you try to deploy this on real traffic? They have a fixed budget constraint, Nbudget, so they have to decide what gets kicked out of memory, which is a real operational challenge <ref:2605.31033#pg1>.

Tom: The way they handle that eviction policy by using relevance scores based on both the text prompt and the visual state seems like a smart way to manage that capacity constraint <ref:2605.31033#pg2>.

Jane: And before they start generating a new video chunk, the model looks at its current prompt and retrieves only those memory items whose content matches what it needs right now <ref:2605.31033#pg2>. It filters out everything else.

Lu: To make sure this works during training, they use a two-stage approach. First, Stage One uses a contrastive loss to ensure that identical entities map to the same slot index over time <ref:2605.31033#pg1>. Then, Stage Two uses a reconstruction loss to keep those abstract slot addresses grounded in the actual high fidelity feature space <ref:2605.31033#pg2>.

Meng: So they are training it to be both consistent and visually grounded simultaneously, which sounds like a lot of tuning for production pipelines <ref:2605.31033#pg1>.

Tom: And the results show that adding that Stage One loss actually gave them a better dynamic score, pushing it up to seventy-nine point six eight <ref:2605.31033#pg1>. That’s solid evidence of why the structure matters.

Jane: It shows that by focusing on object-centric memory instead of just temporal continuity, we get better alignment between what the text asks for and what the video actually shows during those long sixty-second sequences <ref:2605.31033#pg2>.

Lu: The main thing they flag as a limitation is attribute leakage, where a fine detail from one object might bleed over into another neighboring slot <ref:2605.31033#pg4>. That’s something we need to work on next.

Tom: So, to sum up, SlotMemory takes the complexity of long video generation and solves it by structuring memory around objects instead of raw time, delivering better quality and consistency on those long-form tasks <ref:2605.31033#pg2>.

Jane: It really puts the focus on building a system that understands what things are, rather than just recording when things happen. We’ve got a lot to think about with how they plan to scale that memory bank for real-world use next.

The paper's summary: Tom: So, we’re looking at how they suggest they can actually make this slot memory better for production <ref:2605.31033#pg1>.

Jane: What are the specific suggestions for improvement here? Are they just tweaking the training losses, or is there a bigger architectural change needed?

Lu: They’re focusing on disentanglement. They want to figure out how to stop those fine details from leaking between slots, which they call attribute leakage <ref:2605.31033#pg4>.

Meng: Attribute leakage sounds like it’s going to cause problems when we try to reuse these memory slots for different tasks later on, because the same slot might hold conflicting information <ref:2605.31033#pg4>.

Tom: Right, and they are looking into more rigorous disentanglement objectives to solve that specific problem <ref:2605.31033#pg4>. It’s a necessary step if you want this system to be robust across different types of video synthesis <ref:2605.31033#pg4>.

Jane: That makes sense because right now, when an object moves, the model might confuse its identity with another one <ref:2605.31033#pg4>. So getting that separation right is key for long-term coherence.

Lu: They’re also looking at learned manifold aware transition operators <ref:2605.31033#pg4>. That means they want the model to learn how to smoothly move between those different semantic slots in a way that makes sense physically or visually, rather than just snapping between them <ref:2605.31033#pg4>.

Meng: A learned transition operator sounds expensive to train. If we need that kind of specialized learning for every video type, the cost to run this system might get really high <ref:2605.31033#pg1>.

Tom: The hope is that it’s worth the cost because they want to avoid those transient scene regression artifacts where the video briefly reverts to an old environment <ref:2605.31033#pg4>. That kind of glitch is just not good for a polished output.

Jane: So, if we look at the big picture, this isn't just about making it work for sixty seconds; it’s about building memory that understands the meaning of what things are <ref:2605.31033#pg1>.

Lu: Exactly. It’s moving beyond just storing pixels and moving toward storing semantic concepts, which is a huge step in representation learning <ref:2605.31033#pg1>.

Meng: I’m still worried about the operational overhead of training these complex operators, but if it stabilizes the memory bank size better than their fixed budget of six blocks, that could be worth exploring <ref:2605.31033#pg1>.

Tom: So we have a solid foundation with the slot mechanism and the results are strong on quality and consistency for long narratives <ref:2605.31033#pg2>. But fixing those leakage issues is what will take them to the next level <ref:2605.31033#pg4>.

Jane: And once they solve that, this object-centric memory could really change how we build persistent digital experiences across any kind of media.

The paper's improvements: Tom: So, to wrap up, SlotMemory is using an object-centric key value memory system to solve long video generation problems by organizing historical context around specific visual entities <ref:2605.31033#pg1>.

Jane: It really shows that instead of just remembering raw time slices, the model can maintain those specific things consistently over very long sequences <ref:2605.31033#pg2>.

Lu: The paper concludes that this object-centric KV memory mechanism enables persistent representation of visual entities across sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg1>.

Meng: It does deliver state of art quality scores, but they’ve already pointed out that attribute leakage and transient scene regression are real problems for deployment <ref:2605.31033#pg4>.

Tom: That’s fair. The results are impressive on a benchmark, but those operational hurdles need to be cleared before we see it running reliably at scale <ref:2605.31033#pg1>.

Jane: It’s a big step toward building narratives that actually feel coherent over time, even when the prompt changes in the middle of a story <ref:2605.31033#pg2>.

Lu: The paper sets up the next steps by suggesting they need more rigorous disentanglement objectives to handle those leakage issues and learned transition operators to manage scene changes <ref:2605.31033#pg4>.

Meng: I’m still waiting on them to show how they plan to scale that memory bank dynamically instead of sticking to that fixed budget of six blocks <ref:2605.31033#pg1>.

Tom: So the takeaway is that structured semantic representation is a powerful primitive for high fidelity streaming diffusion, even if we still have some refinement work to do <ref:2605.31033#pg2>.

Jane: We’ve seen how they moved from raw temporal capacity to structured semantic slots to solve the drift problem in streaming video <ref:2605.31033#pg1>.

Lu: This object-centric KV memory mechanism is about achieving persistent representation across extended sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg2>.

Meng: For me, the marginal utility of this slot-conditioned memory scales with the duration and complexity of the synthesis task <ref:2605.31033#pg2>.

Tom: And it’s a significant result because it proves that organizing memory around what things *are* is better than just organizing it by when things *happened* <ref:2605.31033#pg1>.

Jane: We’ve got some interesting ideas for how they plan to improve it next, and I think we need to keep watching those disentanglement objectives closely <ref:2605.31033#pg4>.

Conclusion: Tom: So we’ve been looking at SlotMemory, which is essentially using an object-centric key value memory system to solve long video generation problems by organizing historical context around specific visual entities <ref:2605.31033#pg1>.

Jane: It really shows that instead of just remembering raw time slices, the model can maintain those specific things consistently over very long sequences <ref:2605.31033#pg2>.

Lu: The paper concludes that this object-centric KV memory mechanism enables persistent representation of visual entities across sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg1>.

Meng: It does deliver state of the art quality scores, but they’ve already pointed out that attribute leakage and transient scene regression are real problems for deployment <ref:2605.31033#pg4>.

Tom: That’s fair. The results are impressive on a benchmark, but those operational hurdles need to be cleared before we see it running reliably at scale <ref:2605.31033#pg1>.

Jane: It’s a big step toward building narratives that actually feel coherent over time, even when the prompt changes in the middle of a story <ref:2605.31033#pg2>.

Lu: The paper sets up the next steps by suggesting they need more rigorous disentanglement objectives and learned transition operators to handle those leakage issues and scene changes <ref:2605.31033#pg4>.

Meng: I’m still waiting on them to show how they plan to adapt this memory bank size dynamically rather than sticking to that fixed budget of six blocks <ref:2605.31033#pg1>.

Tom: So the takeaway is that structured semantic representation is a powerful primitive for high fidelity streaming diffusion, even if we still have some refinement work to do <ref:2605.31033#pg2>.

Jane: We’ve seen how they moved from raw temporal capacity to structured semantic slots to solve the drift problem in streaming video <ref:2605.31033#pg1>.

Lu: This object-centric KV memory mechanism is about achieving persistent representation across extended sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg2>.

Meng: For me, the marginal utility of this slot-conditioned memory scales with the duration and complexity of the synthesis task <ref:2605.31033#pg2>.

Tom: And it’s a significant result because it proves that organizing memory around what things *are* is better than just organizing it by when things *happened* <ref:2605.31033#pg1>.

Jane: We’ve got some interesting ideas for how they plan to improve it next, and I think we need to keep watching those disentanglement objectives closely <ref:2605.31033#pg4>.

More episodes

← Home