Learning-to-Memorize: Dynamic Context Management for Long-Horizon Autoregressive Video Generation
cs.CV
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autoregressive video generation has achieved impressive results on short clips, yet generating long videos remains challenging due to error accumulation over extended horizons…
Abstract
Autoregressive video generation has achieved impressive results on short clips, yet generating long videos remains challenging due to error accumulation over extended horizons, where each predicted frame depends on potentially imperfect previous frames. Existing approaches typically rely on static context management strategies, such as sliding-window caches that discard early frames or fixed sink frames that permanently preserve initial content. These designs either lose critical long-range context or anchor generation to outdated information, which can lead to degraded fidelity, motion stagnation, and reduced diversity. In this paper, we propose Learning to Memorize (L2M), a learned context memory management framework that dynamically retains, evicts, and evolves historical frames according to predicted importance. The framework consists of an importance prediction router that estimates the relevance of each historical frame, degradation-aware training that improves robustness to noisy or corrupted context, and a dynamic memory initialization and evolution mechanism that updates long-term memory using exponential moving average (EMA) importance scores. This allows high-quality frames to replace stale anchors and enables memory to adapt naturally to scene evolution. Trained with a teacher-student distillation objective and sparsity regularization, L2M makes efficient use of a fixed KV-cache budget while preserving the most informative contextual information. Extensive experiments demonstrate that L2M improves long-horizon video generation quality and temporal consistency over existing baselines, providing a learned strategy for context memory management in long autoregressive video generation.
Sources
- Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
- Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
- LTX-Video: Realtime Video Latent Diffusion
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
- Categorical Reparameterization with Gumbel-Softmax
- Train Short, Inference Long: Training-free Horizon Extension for Autoregressive Video Generation
- SnapKV: LLM Knows What You are Looking for Before Generation
- Flow Matching for Generative Modeling
- Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Decoupled Weight Decay Regularization
- Movie Gen: A Cast of Media Foundation Models
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling
- Measuring Social Biases in Grounded Vision and Language Embeddings
- MotionStream: Real-Time Video Generation with Interactive Motion Controls
- Make-A-Video: Text-to-Video Generation without Text-Video Data
- History-Guided Video Diffusion
- Wan: Open and Advanced Large-Scale Video Generative Models
- Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs