Recency Forcing: Bridging the Long-Horizon Gap in Autoregressive Video Generation
cs.CV
Submitted: 2026-09-17
Updated: 2026-09-17
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autoregressive (AR) video generation degrades over long horizons due to an overlooked train-inference discrepancy we term KV eviction mismatch: models train on short clips where all context frames
Terminology
Abstract
Autoregressive (AR) video generation degrades over long horizons due to an overlooked train-inference discrepancy we term KV eviction mismatch: models train on short clips where all context frames reside in the KV cache, but at inference, memory constraints force distant frames to be evicted from the KV cache - removing context the model was conditioned on. Rather than simulating eviction via context truncation - which discards temporal information the model still needs and degrades motion coherence - we keep the context but while progressively reducing the influence of distant frames, making their eventual eviction negligible. To guide this design, we introduce the positional response R(Δ,, t denoise), a perturbation-based sensitivity measure revealing that context influence decays steeply with temporal distance and varies systematically across denoising steps. Motivated by this analysis, we propose Recency Forcing, which applies a non-positive, timestep-dependent bias, termed Temporal Response Bias (TRB), on pre-softmax attention logits derived directly from R, closing the train-inference gap without modifying context length or training objectives. We further introduce Biased Attention Reparameterization (BAR), an exact reformulation that moves the bias outside the softmax, making TRB a standard FlashAttention call at zero overhead. Recency Forcing operates in both training-free mode and training-based mode. Experiments on VBench and VBench-Long demonstrate state-of-the-art long-horizon generation quality at no additional inference cost.
Sources
- Cosmos World Foundation Model Platform for Physical AI
- MAGI-1: Autoregressive Video Generation at Scale
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- SkyReels-V2: Infinite-length Film Generative Model
- Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion
- Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
- LoL: Longer than Longer, Scaling Video Generation to Hour
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Autoregressive Video Generation without Vector Quantization
- Flex Attention: A Programming Model for Generating Optimized Attention Kernels
- End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
- LTX-Video: Realtime Video Latent Diffusion
- RELIC: Interactive Video World Model with Long-Horizon Memory
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
- FIFO-Diffusion: Generating Infinite Videos from Text without Training
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion
- Train Short, Inference Long: Training-free Horizon Extension for Autoregressive Video Generation
- Unified Video Action Model
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models