Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation

arXiv:2508.12969 · cs.CV · Submitted 2025-08-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation".

Jane: Here's the gist of the paper. The paper proposes Compact Attention, a hardware-aware acceleration framework that exploits structured spatio-temporal sparsity in video diffusion transformers to achieve significant speedup while maintaining high visual quality Structured Sparsity Patterns The analysis of video diffusion transformers (DiT) reveals that attention matrices exhibit structured, yet heterogeneous sparsity patterns where specialized heads dynamically attend to distinct spatiotemporal regions such as local pattern, cross-shaped pattern, or global pattern These patterns include spatial categories like Local Pattern, Cross-shaped Pattern, and Global Pattern across the spatial dimension Additionally, temporal patterns are identified as Time-Variant or Time-Invariant depending on their correlation with temporal relative distance.

Tom: Compact Attention Innovations The proposed framework introduces three key innovations to exploit these patterns efficiently These include adaptive tiling strategies that approximate diverse spatial interaction patterns via dynamic tile grouping, temporally varying windows that adjust sparsity levels based on frame proximity, and an automated configuration search algorithm that optimizes sparse patterns while preserving critical attention pathways.

Jane: Offline Configuration Search The mask search framework decouples pattern discovery from runtime execution through an offline configuration pipeline This process involves guided boundary contraction along hierarchical dimensions, starting with full attention coverage and iteratively tightening window boundaries based on recall loss per computational unit.

Tom: The search is governed by dual thresholds: a minimum recall threshold τ to preserve critical interactions and a maximum cost threshold λ to balance reduction against accuracy loss.

Title and authors: Tom: We’re moving into the specifics now about what exactly this paper proposes. Basically, they’re introducing Compact Attention to speed up video generation by using structured sparsity patterns found in video diffusion transformers.

Jane: So, before we get into the mechanics, I want to make sure we’ve got the context clear: these attention matrices aren't just random noise; they have a specific organization related to where things are spatially and temporally within a video frame sequence.

Lu: That structure is what makes this paper interesting because previous sparse methods either force us into rigid patterns or introduce too much extra computation for the benefit we get.

Meng: So, what’s the core insight they found about these attention matrices that sets this apart from older techniques? I need to know why it’s worth our time now.

Lalam: The key is that specialized attention heads emerge with distinct functional roles, focusing on local spatial regions, forming cross-shaped spatial interactions, or exhibiting a global focus across the whole input sequence.

Tom: That's the pattern discovery part. But how do they actually turn that structural knowledge into a faster computation? That’s where their three main innovations come in.

Jane: They have three things: first, adaptive tiling strategies to approximate those diverse spatial interaction patterns by grouping adjacent tiles dynamically; second, temporally varying windows that adjust how sparse the attention is based on how close the frames are; and third, an automated configuration search algorithm to optimize those sparse patterns while making sure critical pathways stay intact.

Lu: The automated configuration search is a big deal because it’s designed to optimize the mask itself offline, which saves us from having to run that expensive search every single time we want to generate a video.

Meng: I’m thinking about the cost of that optimization pipeline. How much computational overhead does building this optimal mask framework add before we even start generating?

Tom: They describe an offline configuration search where they use guided boundary contraction along dimensions, starting with full attention coverage and tightening the boundaries based on recall loss per computational unit.

Jane: They balance this search using dual thresholds: one minimum recall threshold to make sure critical interactions are preserved, and a maximum cost threshold to control how much computation we cut back.

Lalam: It sounds like a very controlled way to find the best sparsity level without just throwing away everything that might be important for quality.

The paper's summary: Tom: So what does the overall summary of "Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation" boil down to? It’s essentially a hardware-aware acceleration framework targeting structured sparsity in video diffusion transformers.

Jane: In simple terms, they found that video diffusion transformers have inherent organizational patterns—local, cross, global—in their attention mechanisms, and they built a system called Compact Attention to exploit these patterns for massive speedups.

Lu: They are essentially proposing three ways to approximate those complex interaction patterns: dynamic tiling for space, time-varying windows for temporal relevance, and an automated search algorithm to find the best sparse pattern configuration.

Meng: So, if I’m just looking at the numbers on paper without reading every line, what’s the immediate impact of this summary? Is it a simple speedup or something more nuanced?

Tom: It's more nuanced than just a simple speedup. It's about intelligently pruning computation based on learned structure, which they claim leads to up to two point five times end-to-end speedup while keeping quality high.

Jane: The summary emphasizes that their method works because it respects the natural spatiotemporal organization of video data, rather than just applying generic sparsity masks.

Lalam: They highlight that these patterns are not just spatial; they involve time too, with temporal patterns being either time-variant or time-invariant depending on how correlated they are with the distance between frames.

Lu: That insight into the temporal dimension is crucial because it shows how these models model the sequence of video data itself, which changes how we need to apply attention pruning.

The paper's improvements: Tom: Let’s dig into those specific improvements they detail. They outline four key innovations that make up the Compact Attention framework. I want to walk through what each one actually does mechanically.

Jane: First, the adaptive tiling strategy: it uses dynamic tile grouping to approximate local, cross-shaped, and global spatial patterns efficiently in the attention maps.

Lu: That’s how they handle the spatial dimension without forcing a fixed grid; it lets them capture those complex interactions naturally through grouping tokens that are adjacent in both space and time.

Meng: Does this tiling affect memory usage? If you’re tiling dynamically, you have to manage a lot of dynamic data structures on the fly, which sounds like a potential runtime bottleneck.

Tom: They address that by focusing on reordering tokens into spacetime tiles—clusters adjacent in both spatial and temporal domains—to construct computational units that align with natural video locality.

Jane: Second, the temporally varying windows: this adjusts sparsity levels based on frame proximity, letting them suppress attention to distant frames or focus intensely on very close ones as needed.

Lalam: That directly addresses the temporal pattern discovery they mentioned earlier; it lets the system be flexible enough to handle both time-variant and time-invariant dependencies.

Lu: And finally, the automated configuration search algorithm is designed to optimize these sparse patterns while specifically preserving those critical attention pathways that maintain fidelity with full attention baselines.

Tom: So, the takeaway here is that they’re not just applying a fixed mask; they are dynamically reconfiguring how the computation runs based on what their analysis of the attention structure tells them is important.

Conclusion: Jane: We’ve covered the mechanics and results. To wrap up, this paper, "Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation," shows that understanding the inherent structure in video diffusion transformers unlocks a way to accelerate them effectively.

Tom: The main implication is that we can achieve significant speedup—up to two point five times—without sacrificing the high visual quality expected from these models. It’s a practical, hardware-aware acceleration method for long-video generation.

Lu: It provides valuable new insights into how video Transformers model spatiotemporal relationships, which should improve our understanding of the underlying representation learning process.

Meng: From an engineering standpoint, this means lower memory costs and faster inference times for deploying these models at scale, which is a huge win for production environments.

Lalam: I’m just adding that the method shows stability across various layers and heads in a video generation model, which suggests it’s robust to different types of input conditions.

Jane: It really moves the conversation from just "making models bigger" to "making models smarter about how they compute," by exploiting their structure. That’s what this paper is all about.

College of Computer Science & Technology, Zhejiang University · Huawei Technologies

cs.CV

Submitted: 2025-08-18

Updated: 2026-11-03

Comments: Updated to the accepted version of IEEE Transactions on Multimedia. Minor corrections and formatting improvements were made. The main conclusions remain unchanged

Project page: https://yo-ava.github.io/Compact-Attention.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 98/100

The gist: The paper proposes Compact Attention, a hardware-aware acceleration framework that exploits structured spatio-temporal sparsity in video diffusion transformers to achieve significant speedup while…

Key concepts

Structured Spatio-Temporal Sparsity
Video diffusion transformers have attention matrices that are not random; they follow predictable structures. This means different attention heads specialize in looking at specific areas of the video (like local details or global context) and specific time relationships (like nearby frames). Identifying these patterns is key to making the model run faster.
Adaptive Tiling Strategies
This innovation groups tokens into dynamic tiles based on their spatial interaction needs. Instead of treating every connection equally, the system dynamically organizes these tiles to approximate complex spatial attention patterns, allowing for efficient computation that respects how video data is naturally organized spatially.
Offline Configuration Search
The framework uses a search process done beforehand to find the best sparse pattern. It iteratively tightens boundaries around attention windows by balancing two goals: maintaining enough important connections (recall) and reducing computational cost. This ensures the final sparse configuration maintains high visual fidelity.
Temporally Varying Windows
The system uses windows whose size or sparsity level changes depending on how close the frames are in time. This allows the attention mechanism to focus more intensely on immediately relevant temporal distances while ignoring less important, distant frame correlations, optimizing processing based on frame proximity.

Terminology

Summary

The paper proposes Compact Attention, a hardware-aware acceleration framework that exploits structured spatio-temporal sparsity in video diffusion transformers to achieve significant speedup while maintaining high visual quality

Structured Sparsity Patterns

The analysis of video diffusion transformers (DiT) reveals that attention matrices exhibit structured, yet heterogeneous sparsity patterns where specialized heads dynamically attend to distinct spatiotemporal regions such as local pattern, cross-shaped pattern, or global pattern These patterns include spatial categories like Local Pattern, Cross-shaped Pattern, and Global Pattern across the spatial dimension Additionally, temporal patterns are identified as Time-Variant or Time-Invariant depending on their correlation with temporal relative distance.

Compact Attention Innovations

The proposed framework introduces three key innovations to exploit these patterns efficiently These include adaptive tiling strategies that approximate diverse spatial interaction patterns via dynamic tile grouping, temporally varying windows that adjust sparsity levels based on frame proximity, and an automated configuration search algorithm that optimizes sparse patterns while preserving critical attention pathways.

Offline Configuration Search

The mask search framework decouples pattern discovery from runtime execution through an offline configuration pipeline This process involves guided boundary contraction along hierarchical dimensions, starting with full attention coverage and iteratively tightening window boundaries based on recall loss per computational unit. The search is governed by dual thresholds: a minimum recall threshold τ to preserve critical interactions and a maximum cost threshold λ to balance reduction against accuracy loss. p. 5

Performance Validation

Experiments on the Wan2.1 and Hunyuan models demonstrate that Compact Attention achieves up to 2. p. 65× end-to-end speedup with negligible degradation in generated video quality In comparative analysis, Compact Attention shows superior acceleration ratios compared to methods like STA and Sparse VideoGen, maintaining high PSNR values under higher sparsity levels. p. 1 The method consistently delivers high-quality outputs with limited variance across stochastic or semantic input perturbations. p. 8

Broader Impacts and Stability

The work reduces computational barriers for deploying long-video generation models through accelerated inference and lower memory costs The discovery of hierarchical attention patterns provides new insights into how video Transformers model spatiotemporal relationships, improving model interpretability. Furthermore, the method exhibits stability across various layers and heads within a video generation model, indicating robustness to input conditions. p. 3 The analysis suggests that maintaining full attention for the initial 15 denoising timesteps is essential for preserving generation quality. p. 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive Tiling Strategies for Spatial Interaction Approximation: The system can approximate diverse spatial interaction patterns like local pattern, cross-shaped pattern, and global pattern via dynamic tile grouping to efficiently capture complex attention distributions in video data.

  2. Temporally Varying Windows for Frame Proximity Adjustment: The system can adjust sparsity levels based on frame proximity using temporally varying windows, allowing it to suppress distant frames or focus on specific relevant distances as required by the attention head's functional role.

  3. Automated Configuration Search Algorithm for Sparse Pattern Optimization: The system can employ an automated configuration search algorithm that optimizes sparse patterns while preserving critical attention pathways, ensuring that the resulting computation maintains high fidelity with full-attention baselines.

  4. Deformable Attention via Tile-based Computation: The system can reorder tokens into spacetime tiles—clusters adjacent in both spatial and temporal domains—to construct computational units aligned with natural video locality, achieving acceleration while respecting the intrinsic structure of video data.

  5. Optimized Mask Search Framework: The system can utilize an offline configuration pipeline that decouples pattern discovery from runtime execution, employing boundary contraction processes governed by recall and cost thresholds to iteratively tighten window boundaries for maximal sparsity-quality trade-off.

Sources

Related papers