What Remains in Sight? Autoregressive Video Decoding as Representation-Guided Context Rewriting
cs.CV, cs.AI, cs.LG
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/lixuan27/ReCR
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Long-horizon video generation is increasingly moving toward autoregressive rollout, where each newly generated segment becomes part of the evidence used for future prediction.
Abstract
Long-horizon video generation is increasingly moving toward autoregressive rollout, where each newly generated segment becomes part of the evidence used for future prediction. Recent causal decoders, long-context training, and KV-cache mechanisms have extended feasible duration, but most inference pipelines still decide the visible past implicitly through recent windows, compressed caches, or external memory. This raises a direct question: can the choice of which past frames or segments to show to the decoder be treated as an explicit test-time decision? We answer this question with ReCR, a training-free method that formulates autoregressive video decoding as representation-guided context rewriting. At each step, ReCR builds a selected visible context from the generated history by using internal decoder representations to favor context units that match the global rollout state, avoid redundancy with the recent continuation, preserve temporal boundary coverage, and widen the range of spatial layouts in view. The selected units are then mapped onto a compact causal axis before decoding the next block. Across long-horizon text-to-video benchmarks, ReCR consistently improves multiple autoregressive backbones under matched visible-token budgets. Empirical analyses such as representation-space diagnostics and transition-count prompt switching further show that improving what remains visible is an effective path toward more stable long-video decoding.
Sources
- MonarchRT: Efficient Attention for Real-Time Video Generation
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- SkyReels-V2: Infinite-length Film Generative Model
- Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion
- Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models
- Adaptive Begin-of-Video Tokens for Autoregressive Video Diffusion Models
- Adapting VACE for Real-Time Autoregressive Video Diffusion
- Long-Context Autoregressive Video Modeling with Next-Frame Prediction
- Efficient Autoregressive Video Diffusion with Dummy Head
- Entropy-Guided k-Guard Sampling for Long-Horizon Autoregressive Video Generation
- Scaling Image and Video Generation via Test-Time Evolutionary Search
- Imagen Video: High Definition Video Generation with Diffusion Models
- RELIC: Interactive Video World Model with Long-Horizon Memory
- Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft
- Planning with Sketch-Guided Verification for Physics-Aware Video Generation
- VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction
- MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Train Short, Inference Long: Training-free Horizon Extension for Autoregressive Video Generation
- PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs