HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion
cs.CV, cs.AI, cs.LG
Submitted: 2026-07-03
Updated: 2026-11-05
Comments: Accepted by NeurIPS 2026. Project Page: https://leedongyeun.github.io/hypervattention-page/
Code: https://github.com/Tencent-Hunyuan/HunyuanVideo
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation.
Terminology
Abstract
Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation. However, their ability to produce long-duration videos is fundamentally constrained by the quadratic complexity of the self-attention mechanism. Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks: substantial clustering overhead and low CTA utilization caused by irregular cluster-induced blocks. We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly. To reduce clustering overhead, we introduce 3D local-window clustering, which exploits the spatio-temporal locality of video tokens to restrict centroid search to fixed local neighborhoods, and implement it with a custom Triton kernel for efficient execution. We further propose a hybrid clustering strategy that performs full clustering only at anchor steps and updates only subset tokens at intermediate steps, leveraging the temporal stability of cluster assignments across denoising steps. To improve CTA utilization, we present hardware-aware cluster merging that minimizes CTA-aligned execution cost through parallel agglomerative merging, improving block density and approximation fidelity by utilizing idle tile capacity. Together, these components reduce clustering overhead, avoid redundant updates, and better align sparse attention with the fixed tile structure of modern GPU kernels. Experiments on video generation show that HVA advances the Pareto frontier of training free sparse attention, achieving higher fidelity than SVG2 at comparable latency and a 2.13 times speedup over full attention on HunyuanVideo, or 2.23 times with its faster variant HVA-Flash.
Sources
- MonarchRT: Efficient Attention for Real-Time Video Generation
- Kandinsky 5.0: A Family of Foundation Models for Image and Video Generation
- Mixture of Contexts for Long Video Generation
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
- Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
- Flex Attention: A Programming Model for Generating Optimized Attention Kernels
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
- VMonarch: Efficient Video Diffusion Transformers with Structured Attention
- BitsFusion: 1.99 bits Weight Quantization of Diffusion Model
- DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Gemini: A Family of Highly Capable Multimodal Models
- Wan: Open and Advanced Large-Scale Video Generative Models
- SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
- SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
- SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs