Streaming-Aware Diffusion for Real-Time Video Super-Resolution via Cross-Step Attention

arXiv:2610.11746 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Streaming-Aware Diffusion for Real-Time Video Super-Resolution via Cross-Step Attention".

Tom: The gist: This framework proposes a streaming-aware diffusion framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the sequential structure of video streams.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we've been looking at this paper about "Streaming-Aware Diffusion for Real-Time Video Super-Resolution via Cross-Step Attention."

Jane: Yeah, basically, they took those big diffusion models that make great still images and figured out how to make them work for video super resolution in a way that’s fast enough for real time.

Lu: The core of it is exploiting the sequential nature of video streams instead of treating each frame as totally separate data.

Meng: So, they're trying to keep the computational load manageable when you're dealing with a whole sequence of frames instead of just one picture at a time.

Lalam: They introduce two main things: Cross-Step Attention and Trajectory-Coupled Diffusion Scheduling to help the model reuse information across frames and between diffusion steps.

Tom: Cross-Step Attention reuses intermediate denoising features between adjacent frames and different diffusion steps without needing any explicit temporal modeling, which is pretty neat.

Jane: And that goes hand-in-hand with Trajectory-Coupled Diffusion Scheduling, which aligns those intermediate states together along the noise trajectory to make everything more consistent.

Lu: From an engineering view, this reuses structural information across the diffusion process in a way that's tailored for video sequences rather than just single images.

Meng: That leads to their streaming inference pipeline where they structure the latent states in a FIFO buffer aligned with the diffusion steps, which really helps reduce complexity from O of N times S down to O of N plus S.

Tom: And the results on REDS and YouHQ show that this setup gives real improvements in both how good the picture looks and how stable it is across consecutive frames.

Jane: It seems like they managed to get high perceptual quality while maintaining frame-wise stability, which is a big hurdle in video tasks.

Lu: The authors say their method works even with different base diffusion models, which suggests it’s not locked into just one specific architecture.

Meng: For someone building a production system, the main thing is that this approach lets you get high quality output at over forty frames per second after a cold start, which is fast enough for practical use.

Lalam: This means we can use these powerful generative models for real-time video enhancement without having to do all that complicated motion estimation upfront.

Tom: So, in conclusion, this paper shows a practical way to inject temporal feature reuse into diffusion restoration models just by using the structure of the diffusion process itself rather than building a whole new temporal architecture from scratch.

Jane: It's about making these complex tools usable for real time video processing by focusing on how the data flows sequentially.

Lu: The implication is that we might not always need to model time explicitly when working with sequential data like video, but instead let the diffusion trajectory do some of the heavy lifting for temporal coherence.

Meng: It opens up a lot of possibilities for applying these diffusion models directly to live feeds because it solves the latency issue in a very direct way.

Tom: We'll keep digging into how this streaming reorganization impacts other types of generative tasks next, so stick around.

Conclusion: Tom: So, wrapping up this look at "Streaming-Aware Diffusion for Real-Time Video Super-Resolution via Cross-Step Attention," we've seen how they used a streaming approach to make those large diffusion models work for video without needing complicated motion estimation or extra temporal modules.

Jane: Right, essentially they are showing us how to get high fidelity video upscaling that actually runs in real time by leveraging the way video data flows sequentially.

Lu: The authors, they focus on this mechanism called Cross-Step Attention and Trajectory-Coupled Diffusion Scheduling as the main tools for reusing information between frames.

Meng: What this means practically is that we can take these powerful diffusion models and use them directly on live video feeds because it solves the speed problem right there at the inference stage.

Lalam: It’s about making high-quality restoration accessible for real time applications, improving how we process and enhance video content across different platforms.

Tom: So the authors show that exploiting the diffusion trajectory structure is a simple and effective way to bridge high-quality diffusion restoration with real time video processing.

Jane: It really shows that you can introduce temporal feature reuse into pretrained models without needing a dedicated multi-frame architecture built from scratch.

Lu: They found that this method works even with different base diffusion models, which suggests this is a general technique for video applications rather than just a fix for one specific model.

Meng: That opens up a lot of possibilities for applying these generative models directly to live feeds because it solves the latency issue in a very direct way.

Tom: We'll keep digging into how this streaming reorganization impacts other types of generative tasks next, so stick around.

Harris Partaourides, Sotirios Chatzis

Cyprus University of Technology

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: The gist: This framework proposes a streaming-aware diffusion framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the

Key concepts

Cross-Step Attention
This mechanism allows the model to look at intermediate denoising features from adjacent frames and diffusion steps simultaneously. It enables temporal coordination by letting the current frame's denoising process attend to information from a slightly later state of the previous frame, reusing structural details without needing explicit motion modeling.
Trajectory-Coupled Diffusion Scheduling
This technique models dependencies along the noise trajectory by pairing consecutive diffusion steps. It creates a vectorized representation for adjacent states, allowing the model to access coupled states (step s and s+1) simultaneously. This improves temporal coherence during training and inference by ensuring related intermediate representations are aligned.
Streaming-Aware Diffusion Pipeline
Inference is structured as a FIFO pipeline where frames move sequentially through a buffer of size S, matching the diffusion horizon. Instead of running S steps for every frame, each frame advances one step per time iteration. This reorganization reduces computational complexity and ensures temporal consistency in the output stream.

Terminology

Summary

The gist: This framework proposes a streaming-aware diffusion framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the sequential structure of video streams.

Cross-Step Attention

This mechanism is a model-agnostic attention mechanism that reuses intermediate denoising features across adjacent frames and diffusion steps, enabling temporal coordination without explicit temporal modeling This allows the current frame at diffusion step s to attend to the previous frame at step s + 1 for efficient feature reuse and temporal consistency By leveraging intermediate denoising states further along the denoising trajectory of the previous frame, the model reuses progressively refined structural information to guide the current denoising process This formulation naturally interacts with the denoising trajectory, motivating a Trajectory-Coupled Diffusion Scheduling strategy that aligns intermediate states across diffusion steps for both training and inference

Trajectory-Coupled Diffusion Scheduling

This formulation models local dependencies along the denoising trajectory by lifting the scalar noise schedule into a paired representation over adjacent diffusion steps Given a standard scalar noise schedule, it constructs a vectorized schedule by stacking consecutive values, which explicitly couples adjacent diffusion steps s and s + 1 This induces a corresponding paired latent representation along the denoising trajectory, enabling simultaneous access to two consecutive states within the same trajectory without modifying the underlying diffusion process The model is trained using the standard diffusion denoising objective L(θ) = E(z s n, z s+1 n-1),ϵ hϵ - ϵθ(z s n, zs+1 n-1) with ϵ ∼ N (0, I) At inference, this formulation enables consistent access to coupled states z˜s n = (z s n, zs+1 n), reducing redundancy across diffusion steps and frames while complementing Cross-Step Attention

Streaming-Aware Diffusion Pipeline

The streaming inference pipeline reformulates inference as a streaming FIFO pipeline with buffer size S, matching the diffusion horizon At time n, the system maintains a structured latent state Zn = 0. (z k n-k k = 1,..., S), where each element corresponds to a frame progressing through a specific diffusion depth Each incoming frame is encoded into its initial latent state and enters the buffer at diffusion step 0 At every iteration, all latents advance by one diffusion step under the trajectory-coupled denoising process with Cross-Step Attention Consequently, each frame traverses the buffer over S updates before being decoded at the final stage This induces a pipeline latency of S steps, after which the system produces one output frame per iteration in steady state Computationally, each diffusion update is executed once per time step rather than S times per frame, yielding amortized complexity: O(N + S) The FIFO-aligned design ensures temporal consistency, efficient state reuse, and full compatibility with the underlying diffusion process

Experimental Results

Experiments on REDS4 and YouHQ40-Test demonstrate improved perceptual quality and temporal realism while maintaining frame-wise stability Cross-Step Attention improves perceptual and distribution-level temporal metrics over vanilla LDM in the zero-shot setting, although this comes with a degradation in distortion-based and frame-wise temporal metrics Fine-tuning substantially improves this trade-off, for instance on REDS4, LDM∗CS improves PSNR from 24.66 to 25.26 relative to vanilla LDM The streaming variants substantially increase throughput; for example, SDx4stream achieves 17 FPS at 1024 squared resolution These results show that the main efficiency gain comes from streaming reorganization, which amortizes diffusion computation across consecutive frames

Conclusion

The proposed framework provides a practical way to introduce temporal feature reuse into pretrained diffusion restoration models without explicit motion estimation or a dedicated multi-frame temporal architecture Overall, our results show that exploiting the structure of the diffusion trajectory provides a simple and effective way to bridge high-quality diffusion restoration and real-time video processing The combined framework achieves over 40 FPS at 512 squared after cold start, demonstrating the feasibility of real-time diffusion-based VSR

--- Page 1 ---

The gist: This framework proposes a streaming-aware diffusion framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the sequential structure of video streams.

--- Page 2 ---

The gist: This framework proposes a streaming-aware diffusion framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the sequential structure of video streams.

Improvements for AI systems

  1. Bold header: Cross-Step Attention for Temporal Feature Reuse

This mechanism reuses intermediate denoising features across adjacent frames and diffusion steps, enabling temporal information exchange without explicit temporal modeling. This allows the system to achieve improved perceptual quality and temporal realism while maintaining frame-wise stability by implicitly coordinating adjacent video frames.

  1. Bold header: Trajectory-Coupled Diffusion Scheduling

This formulation introduces a coupling mechanism where z˜sn = (zsn, zs+1n−1) to create trajectory-aligned training pairs, which results in a cross-step clean prior that improves temporal coherence while remaining compatible with standard diffusion training.

  1. Bold header: Streaming-Aware Diffusion Pipeline

The pipeline transforms inference from independent processing into a continuous evolution, achieving an efficiency gain by reducing complexity from O(N · S) to O(N + S) for N frames and S diffusion steps, enabling real-time VSR with over 40 FPS at 512 × 512 resolution after cold start.

  1. Bold header: Enhanced Sample Efficiency

Cross-Step Attention enables the model to achieve comparable or better perceptual quality at substantially fewer steps, meaning it can utilize only one-third of the denoising steps compared to vanilla LDM, improving sample efficiency of the diffusion process.

  1. Bold header: Fine-Tuned Temporal Stability

Fine-tuning mitigates instability by showing that on REDS4, tLPIPS/tDISTS improve from 0.044/0.027 for LDMCS to 0.028/0.023 for LDM∗CS, suggesting it can achieve stability close to the vanilla model while gaining perceptual benefits without explicit motion estimation or dedicated multi-frame temporal architectures.

Abstract

Real-time video super-resolution requires high spatio-temporal fidelity under strict latency constraints, challenging diffusion models due to their iterative sampling cost and limited temporal coordination. We propose a streaming-aware framework that adapts pretrained single-image latent diffusion models for efficient video super-resolution (VSR) by exploiting the sequential structure of video streams. Our Cross-Step Attention mechanism reuses intermediate denoising features across adjacent frames and diffusion steps, enabling temporal information exchange without explicit temporal modeling. We further introduce Trajectory-Coupled Diffusion Scheduling, which aligns adjacent diffusion states and provides cleaner intermediate representations for cross-step conditioning, improving temporal coherence. These components are integrated into a streaming inference pipeline that incrementally propagates latent states across frames, reducing the effective computational complexity from O(N times S) to O(N + S) for N frames and S diffusion steps. Experiments on REDS4 and YouHQ40-Test demonstrate improved perceptual quality and temporal realism while maintaining frame-wise stability. Our method achieves over 40 FPS at 512 times 512 resolution after cold start, enabling real-time VSR without explicit temporal modeling.

Sources

Related papers