SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation

arXiv:2605.31033 · cs.CV · Submitted 2026-05-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation".

Jane: Here's the gist of the paper. Streaming video generation models typically rely on temporal-centric memory, which organizes historical context as raw frames, chunk segments, or unclustered tokens This organization frequently leads to identity drift and semantic inconsistency when entities exit the frame or during interactive prompt transitions SlotMemory proposes an object-centric Key-Value (KV) memory mechanism for streaming video diffusion that shifts the memory abstraction from “when” an event occurred to “what” is being represented by decomposing the transformer’s KV manifold into discrete, reusable semantic slots Evaluated on 60-second interactive narratives using the Wan2.1-T2V-1.3B backbone, SlotMemory achieves a state-of-the-art quality score of 81.61 and a 22.8% relative improvement in dynamic consistency over the strongest existing streaming baseline.

Tom: Problem Formulation and Memory Abstraction The paper frames long-form, multi-prompt video synthesis as a boundedmemory streaming generation problem The objective is to synthesize a video V1:T in an autoregressive manner across C discrete chunks of length Tc The system maintains three distinct states: a short-term local KV cache Cc for local context, a text-conditioning cache Ctextc for prompt-dependent features, and a long-term memory bank Bc for persistent history The primary challenge is the management of the long-term bank Bc under a fixed capacity constraint Nbudget The optimization problem is formulated as max Bc: Bc ≤Nbudget E Consistency(V1:T) + PromptAdherence(V1:T, P) − λ Cost(Bc).

Jane: Architectural Mechanism of SlotMemory SlotMemory redefines the fundamental memory primitive as a slot-conditioned semantic storage The system operates via a recursive read-write-update cycle synchronized with the autoregressive generation of video chunks During the writing phase, it utilizes a temporally-initialized Slot Attention mechanism to partition highdimensional transformer manifolds into discrete, reusable semantic regions The low-dimensional slot embeddings are utilized solely as semantic routing addresses rather than the primary storage medium For each slot s, the model identifies the Rt tokens with the highest assignment weights, Icl,s = TopR(Acl,s,:), and extracts their corresponding KV tokens.

Tom: Retrieval and Adaptation Strategy Before denoising chunk c, the model retrieves a subset of memory items whose semantic content aligns with the current prompt and visual state For each item mi, a fused relevance score is computed: r(mi) = γ · rtext(mi, pc) + (1 −γ) · rvis(mi, qc), where pc is the active prompt The top-k ranked items are then integrated into the attention mechanism AttnQc l,

Klocal l;Kslot l: ,

Vlocal l;Vslot l: Following generation, new slot-conditioned items are written to the bank Bc′, and a relevance-driven eviction policy manages capacity under a fixed budget.

Title and authors: Tom: So, SlotMemory is basically changing how video models remember things by switching from remembering raw time slices to remembering specific objects through these key value slots <ref:2605.31033#pg1>.

Jane: That’s the core idea, right? Instead of the model just seeing a stream of frames, it’s organizing that memory into discrete semantic slots that represent things like "that person," or "that specific chair."

Lu: Exactly. The paper frames long video generation as a bounded memory problem, and they optimize for balancing how consistent the video is with how well it follows the prompt, all while keeping the size of this object memory bank under a certain budget <ref:2605.31033#pg1>.

Meng: And what I’m hearing that sounds like they are building a system that can keep track of entities across sixty seconds, which is way longer than what we usually see in these demos <ref:2605.31033#pg2>.

Tom: Right, and they show that this structured memory gives them a state of the art quality score of eighty-one point six one on interactive narratives <ref:2605.31033#pg2>. That’s a big number when you’re talking about maintaining consistency during a long story.

Jane: It means that even if the scene changes or someone walks out of frame, the model can still refer back to what that person looked like earlier, because it has a specific slot for them <ref:2605.31033#pg2>. That’s entity persistence in action.

Lu: The authors detail how they do this by using a recursive read-write update cycle, where they use these slot embeddings only as addresses to route the high dimensional transformer features <ref:2605.31033#pg1>. It’s not about storing everything; it’s about smart retrieval.

Meng: But what happens when you try to deploy this on real traffic? They have a fixed budget constraint, Nbudget, so they have to decide what gets kicked out of memory, which is a real operational challenge <ref:2605.31033#pg1>.

Tom: The way they handle that eviction policy by using relevance scores based on both the text prompt and the visual state seems like a smart way to manage that capacity constraint <ref:2605.31033#pg2>.

Jane: And before they start generating a new video chunk, the model looks at its current prompt and retrieves only those memory items whose content matches what it needs right now <ref:2605.31033#pg2>. It filters out everything else.

Lu: To make sure this works during training, they use a two-stage approach. First, Stage One uses a contrastive loss to ensure that identical entities map to the same slot index over time <ref:2605.31033#pg1>. Then, Stage Two uses a reconstruction loss to keep those abstract slot addresses grounded in the actual high fidelity feature space <ref:2605.31033#pg2>.

Meng: So they are training it to be both consistent and visually grounded simultaneously, which sounds like a lot of tuning for production pipelines <ref:2605.31033#pg1>.

Tom: And the results show that adding that Stage One loss actually gave them a better dynamic score, pushing it up to seventy-nine point six eight <ref:2605.31033#pg1>. That’s solid evidence of why the structure matters.

Jane: It shows that by focusing on object-centric memory instead of just temporal continuity, we get better alignment between what the text asks for and what the video actually shows during those long sixty-second sequences <ref:2605.31033#pg2>.

Lu: The main thing they flag as a limitation is attribute leakage, where a fine detail from one object might bleed over into another neighboring slot <ref:2605.31033#pg4>. That’s something we need to work on next.

Tom: So, to sum up, SlotMemory takes the complexity of long video generation and solves it by structuring memory around objects instead of raw time, delivering better quality and consistency on those long-form tasks <ref:2605.31033#pg2>.

Jane: It really puts the focus on building a system that understands what things are, rather than just recording when things happen. We’ve got a lot to think about with how they plan to scale that memory bank for real-world use next.

The paper's summary: Tom: So, we’re looking at how they suggest they can actually make this slot memory better for production <ref:2605.31033#pg1>.

Jane: What are the specific suggestions for improvement here? Are they just tweaking the training losses, or is there a bigger architectural change needed?

Lu: They’re focusing on disentanglement. They want to figure out how to stop those fine details from leaking between slots, which they call attribute leakage <ref:2605.31033#pg4>.

Meng: Attribute leakage sounds like it’s going to cause problems when we try to reuse these memory slots for different tasks later on, because the same slot might hold conflicting information <ref:2605.31033#pg4>.

Tom: Right, and they are looking into more rigorous disentanglement objectives to solve that specific problem <ref:2605.31033#pg4>. It’s a necessary step if you want this system to be robust across different types of video synthesis <ref:2605.31033#pg4>.

Jane: That makes sense because right now, when an object moves, the model might confuse its identity with another one <ref:2605.31033#pg4>. So getting that separation right is key for long-term coherence.

Lu: They’re also looking at learned manifold aware transition operators <ref:2605.31033#pg4>. That means they want the model to learn how to smoothly move between those different semantic slots in a way that makes sense physically or visually, rather than just snapping between them <ref:2605.31033#pg4>.

Meng: A learned transition operator sounds expensive to train. If we need that kind of specialized learning for every video type, the cost to run this system might get really high <ref:2605.31033#pg1>.

Tom: The hope is that it’s worth the cost because they want to avoid those transient scene regression artifacts where the video briefly reverts to an old environment <ref:2605.31033#pg4>. That kind of glitch is just not good for a polished output.

Jane: So, if we look at the big picture, this isn't just about making it work for sixty seconds; it’s about building memory that understands the meaning of what things are <ref:2605.31033#pg1>.

Lu: Exactly. It’s moving beyond just storing pixels and moving toward storing semantic concepts, which is a huge step in representation learning <ref:2605.31033#pg1>.

Meng: I’m still worried about the operational overhead of training these complex operators, but if it stabilizes the memory bank size better than their fixed budget of six blocks, that could be worth exploring <ref:2605.31033#pg1>.

Tom: So we have a solid foundation with the slot mechanism and the results are strong on quality and consistency for long narratives <ref:2605.31033#pg2>. But fixing those leakage issues is what will take them to the next level <ref:2605.31033#pg4>.

Jane: And once they solve that, this object-centric memory could really change how we build persistent digital experiences across any kind of media.

The paper's improvements: Tom: So, to wrap up, SlotMemory is using an object-centric key value memory system to solve long video generation problems by organizing historical context around specific visual entities <ref:2605.31033#pg1>.

Jane: It really shows that instead of just remembering raw time slices, the model can maintain those specific things consistently over very long sequences <ref:2605.31033#pg2>.

Lu: The paper concludes that this object-centric KV memory mechanism enables persistent representation of visual entities across sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg1>.

Meng: It does deliver state of art quality scores, but they’ve already pointed out that attribute leakage and transient scene regression are real problems for deployment <ref:2605.31033#pg4>.

Tom: That’s fair. The results are impressive on a benchmark, but those operational hurdles need to be cleared before we see it running reliably at scale <ref:2605.31033#pg1>.

Jane: It’s a big step toward building narratives that actually feel coherent over time, even when the prompt changes in the middle of a story <ref:2605.31033#pg2>.

Lu: The paper sets up the next steps by suggesting they need more rigorous disentanglement objectives to handle those leakage issues and learned transition operators to manage scene changes <ref:2605.31033#pg4>.

Meng: I’m still waiting on them to show how they plan to scale that memory bank dynamically instead of sticking to that fixed budget of six blocks <ref:2605.31033#pg1>.

Tom: So the takeaway is that structured semantic representation is a powerful primitive for high fidelity streaming diffusion, even if we still have some refinement work to do <ref:2605.31033#pg2>.

Jane: We’ve seen how they moved from raw temporal capacity to structured semantic slots to solve the drift problem in streaming video <ref:2605.31033#pg1>.

Lu: This object-centric KV memory mechanism is about achieving persistent representation across extended sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg2>.

Meng: For me, the marginal utility of this slot-conditioned memory scales with the duration and complexity of the synthesis task <ref:2605.31033#pg2>.

Tom: And it’s a significant result because it proves that organizing memory around what things *are* is better than just organizing it by when things *happened* <ref:2605.31033#pg1>.

Jane: We’ve got some interesting ideas for how they plan to improve it next, and I think we need to keep watching those disentanglement objectives closely <ref:2605.31033#pg4>.

Conclusion: Tom: So we’ve been looking at SlotMemory, which is essentially using an object-centric key value memory system to solve long video generation problems by organizing historical context around specific visual entities <ref:2605.31033#pg1>.

Jane: It really shows that instead of just remembering raw time slices, the model can maintain those specific things consistently over very long sequences <ref:2605.31033#pg2>.

Lu: The paper concludes that this object-centric KV memory mechanism enables persistent representation of visual entities across sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg1>.

Meng: It does deliver state of the art quality scores, but they’ve already pointed out that attribute leakage and transient scene regression are real problems for deployment <ref:2605.31033#pg4>.

Tom: That’s fair. The results are impressive on a benchmark, but those operational hurdles need to be cleared before we see it running reliably at scale <ref:2605.31033#pg1>.

Jane: It’s a big step toward building narratives that actually feel coherent over time, even when the prompt changes in the middle of a story <ref:2605.31033#pg2>.

Lu: The paper sets up the next steps by suggesting they need more rigorous disentanglement objectives and learned transition operators to handle those leakage issues and scene changes <ref:2605.31033#pg4>.

Meng: I’m still waiting on them to show how they plan to adapt this memory bank size dynamically rather than sticking to that fixed budget of six blocks <ref:2605.31033#pg1>.

Tom: So the takeaway is that structured semantic representation is a powerful primitive for high fidelity streaming diffusion, even if we still have some refinement work to do <ref:2605.31033#pg2>.

Jane: We’ve seen how they moved from raw temporal capacity to structured semantic slots to solve the drift problem in streaming video <ref:2605.31033#pg1>.

Lu: This object-centric KV memory mechanism is about achieving persistent representation across extended sixty second horizons and complex, multi-prompt transitions <ref:2605.31033#pg2>.

Meng: For me, the marginal utility of this slot-conditioned memory scales with the duration and complexity of the synthesis task <ref:2605.31033#pg2>.

Tom: And it’s a significant result because it proves that organizing memory around what things *are* is better than just organizing it by when things *happened* <ref:2605.31033#pg1>.

Jane: We’ve got some interesting ideas for how they plan to improve it next, and I think we need to keep watching those disentanglement objectives closely <ref:2605.31033#pg4>.

Fudan University · Meta Superintelligence Labs · Baidu

cs.CV

Submitted: 2026-05-29

Updated: 2026-11-03

Comments: Code is available at https://github.com/tj12323/SlotMemory

Project page: https://tj12323.github.io/SlotMemory

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Streaming video generation models typically rely on temporal-centric memory, which organizes historical context as raw frames, chunk segments…

Key concepts

Temporal-centric Memory
Traditional models organize history as raw frames or tokens in time. This causes identity drift when objects leave the frame or during prompt changes because the memory is organized by 'when' an event happened rather than 'what' the entity is.
SlotMemory (Object-Centric KV Memory)
This mechanism shifts memory abstraction to discrete semantic slots. It treats memory as a Key-Value system where each slot represents a specific object or concept, allowing the model to retrieve relevant visual information based on its content rather than its chronological position.
Slot Attention Mechanism
During writing, this mechanism partitions the high-dimensional transformer manifold into discrete semantic regions. Low-dimensional slot embeddings act as routing addresses to identify and extract the specific KV tokens corresponding to a desired object's representation.

Terminology

Summary

Streaming video generation models typically rely on temporal-centric memory, which organizes historical context as raw frames, chunk segments, or unclustered tokens This organization frequently leads to identity drift and semantic inconsistency when entities exit the frame or during interactive prompt transitions SlotMemory proposes an object-centric Key-Value (KV) memory mechanism for streaming video diffusion that shifts the memory abstraction from “when” an event occurred to “what” is being represented by decomposing the transformer’s KV manifold into discrete, reusable semantic slots Evaluated on 60-second interactive narratives using the Wan2. p. 01-T2V-1.3B backbone, SlotMemory achieves a state-of-the-art quality score of 81. p. 061 and a 22.8% relative improvement in dynamic consistency over the strongest existing streaming baseline. p. 0

Problem Formulation and Memory Abstraction

The paper frames long-form, multi-prompt video synthesis as a boundedmemory streaming generation problem The objective is to synthesize a video V1:T in an autoregressive manner across C discrete chunks of length Tc The system maintains three distinct states: a short-term local KV cache Cc for local context, a text-conditioning cache Ctextc for prompt-dependent features, and a long-term memory bank Bc for persistent history The primary challenge is the management of the long-term bank Bc under a fixed capacity constraint Nbudget The optimization problem is formulated as max Bc: Bc ≤Nbudget E Consistency(V1:T) + PromptAdherence(V1:T, P) − λ Cost(Bc).

Architectural Mechanism of SlotMemory

SlotMemory redefines the fundamental memory primitive as a slot-conditioned semantic storage The system operates via a recursive read-write-update cycle synchronized with the autoregressive generation of video chunks During the writing phase, it utilizes a temporally-initialized Slot Attention mechanism to partition highdimensional transformer manifolds into discrete, reusable semantic regions The low-dimensional slot embeddings are utilized solely as semantic routing addresses rather than the primary storage medium For each slot s, the model identifies the Rt tokens with the highest assignment weights, Icl,s = TopR(Acl,s,:), and extracts their corresponding KV tokens.

Retrieval and Adaptation Strategy

Before denoising chunk c, the model retrieves a subset of memory items whose semantic content aligns with the current prompt and visual state For each item mi, a fused relevance score is computed: r(mi) = γ · rtext(mi, pc) + (1 −γ) · rvis(mi, qc), where pc is the active prompt The top-k ranked items are then integrated into the attention mechanism AttnQc l, [Klocal l;Kslot l], [Vlocal l;Vslot l] Following generation, new slot-conditioned items are written to the bank Bc′, and a relevance-driven eviction policy manages capacity under a fixed budget.

Training Objective and Regularization

To align the memory module with its inference-time behavior, SlotMemory is trained using a streaming rollout trajectory The global optimization objective is defined as Lgen = LDMD + λslotLslot, where LDMD represents the distribution matching distillation loss for adapting the diffusion backbone The auxiliary slot loss Lslot is partitioned into two components: Lcon enforces temporal identity consistency and Lrec facilitates token-level grounding during the second training stage The contrastive loss for temporal identity consistency encourages the model to map identical entities to the same slot index over time The reconstruction regularization Lrec prevents semantic drift by grounding abstract slot addresses in the high-fidelity transformer feature space.

Experimental Results and Performance

SlotMemory achieves a state-of-the-art Quality Score of 81.61 in interactive narratives and a 74. p. 029 dynamic score at the 30-second horizon–a 22. p. 18% relative improvement over the existing approach In interactive multi-prompt generation, SlotMemory achieves a state-of-the-art Quality Score of 81. p. 561 The marginal utility of slot-conditioned memory scales with the duration and complexity of the synthesis task. p. 5 p. 5

Ablation Study Insights

The introduction of the Slot Module without temporal constraints leads to a decline in the Dynamic Score, suggesting that naive feature compression disrupts motion continuity The addition of the Stage-I Contrastive Loss provides the most significant performance gain, with the Dynamic Score increasing to 79.68 and CLIP alignment reaching 26. p. 606 The final inclusion of the Stage-II Reconstruction Loss further refines Quality to 81. p. 661, albeit with a marginal trade-off in motion intensity. p. 6 The optimal memory bank size for persistent streaming synthesis is identified as 6 blocks. p. 7

Limitations and Future Directions

Failure cases of SlotMemory include attribute leakage, where fine-grained properties migrate between adjacent semantic slots Furthermore, the transition logic can induce transient scene regression artifacts where the synthesis may momentarily revert to a prior environmental state Future work will focus on investigating more rigorous disentanglement objectives and learned, manifold-aware transition operators to address attribute leakage and scene regression. The framework is envisioned toward dynamic slot allocation mechanisms to adaptively scale its representational capacity. p. 7

Conclusion

SlotMemory establishes structured semantic decomposition–rather than raw temporal capacity–as the critical primitive for high-fidelity streaming diffusion Both quantitative and qualitative results demonstrate that SlotMemory achieves state-of-the-art performance in both longrange consistency and prompt adherence. The paper concludes that the object-centric KV memory mechanism enables the persistent representation of visual entities across extended 60-second horizons and complex, multi-prompt transitions. p. 7 The advantages of structured semantic representation become pronounced at the 30-second horizon. p. 5 The marginal utility of slot-conditioned memory scales with the duration and complexity of the synthesis task.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Object-centric KV memory for streaming video generation enables entity-level persistence across long horizons, addressing identity drift when entities exit and re-enter frames by utilizing structured semantic representation, rather than raw temporal capacity.

  2. The system can maintain visual entities and key props coherently across scene changes and temporary occlusions in 60-second rollouts, as shown in the teaser examples where the main subject identity and key visual anchors remain consistent over time.

  3. It supports interactive multi-prompt generation by using a promptaware scoring function to calibrate textual alignment against visual relevance, ensuring that core entities are preserved across narrative transitions while obsolete temporal context is suppressed.

  4. The architecture allows for the synthesis of persistent long-form narratives by framing it as a bounded memory streaming generation problem with an optimization objective balancing consistency and adherence, formulated as max Bc: Bc ≤ Nbudget E Consistency(V1:T) + PromptAdherence(V1:T, P) − λ Cost(Bc).

  5. The training process uses a two-stage approach where Stage-I optimizes adapters under slot regularization and Stage-II utilizes reconstruction loss to ensure semantic slots remain grounded in the high-fidelity transformer feature space.

Sources

Related papers