Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

summary

The gist

Frontier video generation models work in a VAE latent space that compresses video in both space and time The paper presents PixVid…

In short

PixVid lifts a pretrained latent video diffusion model into pixel space to generate every frame directly as pixels. It replaces the VAE decoder with a convolutional head and uses frame-wise tokenization, overcoming structural limitations like resolution scaling, control alignment under fast motion, and low latency.

Key concepts

VAE Latent Space
Video generation models compress video into a compressed latent space using a Variational Autoencoder (VAE). This compression reduces the complexity of the video data by encoding both spatial dimensions (pixels) and temporal dimensions (time), making it manageable for diffusion models.
Frame-wise Tokenization
PixVid ensures that each token in the model corresponds to exactly one frame, covering a 1x p x p patch. This design prevents conditions or actions from spanning multiple frames within a single token, which is crucial for precise control and accurate frame generation.
Detailer Head
This is a lightweight 3D convolutional decoder that reads the diffusion transformer tokens and renders them into actual pixels. It suppresses artifacts like checkerboard patterns by using convolutions across neighboring tokens, ensuring clean local structure in the final pixel output.

Terminology used across episodes

This episode discusses

The paper

Back to Pixels: Lifting Latent Video Diffusion into Pixel Space · Read on arXiv

Yanke Song, Shuo Chen, Chris Rockwell, Guande He, Nan Liu, Desai Xie, Tianpei Gu, Yong-Hyun Park, Aryan V S1,  Dennis Yuren Menn,  Ming-Hsuan Yang, ©Xun Huang

Roblox · University of California, Merced · University of Pennsylvania · University of Texas at Austin

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Back to Pixels".

Jane: Here's the gist of the paper. Frontier video generation models work in a VAE latent space that compresses video in both space and time The paper presents PixVid, a simple yet effective method that lifts a pretrained latent video model into pixel space to generate every frame directly as pixels This approach overcomes structural limitations of latent models such as resolution scaling, control alignment under fast motion, and control-to-screen latency.

Tom: Pruning redundant reasoning tokens PixVid keeps the diffusion transformer (DiT) and replaces the VAE decoder with a convolutional head that renders its outputs into pixels The method achieves frame-wise tokenization where each token covers a 1 × p × p patch of exactly one frame, ensuring no condition or action spans multiple frames.

Jane: The detailer head is implemented as a lightweight 3D convolutional decoder that reads only the DiT tokens and suppresses checkerboard artifacts through convolutions across neighboring tokens How the training signal is built PixVid finetunes end-to-end in pixel space from a pretrained latent model, initializing the DiT and input patchification to start from close to its pretrained latent distribution.

Tom: The key techniques identified include predicting clean video (x-prediction) rather than velocity, which is essential for the detailer to render clean local structure Additionally, a larger noise shift is used because redundant pixels retain more signal at the same noise level.

Paper summary: Tom: So to summarize "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space," the authors have effectively removed the compression step by finetuning a latent video diffusion model directly into pixel space. They claim this successfully overcomes resolution scaling issues, better motion handling in editing, and low latency response for controls >

Jane: The paper suggests that while latent models save on training costs for short clips, they are hitting limits when you need real-time interactive world models or precise frame-by-frame control alignment >

Lu: It shows that by predicting clean video instead of velocity and using specific noise shifts, they can maintain competitive quality while achieving frame-wise pixel generation >

Meng: From a deployment angle, the latency reduction to thirty-four milliseconds for control to screen is significant because it means actions are responsive on the very next frame, which is what we need for interactive applications >

Tom: And they show that this direct pixel generation method can achieve higher total quality scores than both baselines at different resolutions, showing gains in scene consistency and imaging quality >

Jane: Overall, "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space" suggests that finetuning a latent video diffusion model into a frame-wise pixel space is a competitive method for achieving high-quality video generation, especially when you need those specific performance improvements in motion and speed >

Conclusion: Tom: So we’re wrapping up our look at this paper, "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space." Basically, they took a video model that was super compressed in latent space and found a way to generate every single frame directly as raw pixels.

Jane: That sounds pretty wild when you think about how much data that actually is. So, what does this whole "lifting" process actually do for us?

Lu: It’s about taking this existing, trained model—the diffusion transformer—and replacing the part that squashes everything into a compressed latent code with something new. They use a convolutional head to turn those tokens back into pixels, instead of using the old VAE decoder.

Meng: From my side at the startup, I care about what this means for deployment. The authors claim they fixed some major problems with those latent models, like scaling resolution or handling fast motion better than before.

Tom: Exactly. They show that by doing this finetuning in pixel space, you can double the resolution—say from a certain size to twice that size—and it still uses the same amount of computation for the main model part.

Jane: That’s really interesting because I always thought latent models were just a clever shortcut for training speed. So, what about those things they specifically mentioned fixing, like control alignment under fast motion?

Lu: They found that this method keeps ninety-eight percent of the source's fast motion quality when compared to the original latent model's eighty-three percent. It’s pretty close to what you get if you trained a pixel-space model from scratch.

Meng: But we gotta ask about latency. Latent models wait for whole blocks of frames, right? This method generates one frame block at a time, so the delay for an action to show up on the screen drops dramatically.

Tom: Right, that’s where it gets practical. They cut the control-to-screen latency from four hundred ninety-five milliseconds down to thirty-four milliseconds after they did some distillation work. That’s real time responsiveness.

Jane: So, to sum up, this paper is showing us that you can actually get high quality in pixel space by intelligently adapting a model that was originally designed for latent spaces, and it does so while making the generation much faster and more responsive for real-time use.

Lu: It opens up new training possibilities too. They suggest generating directly in pixels lets you train things in ways that a frozen VAE just wouldn't allow.

Tom: Yeah, it seems like they’ve found a way to get the best of both worlds here—using the power of pretraining but giving you the direct pixel output and speed we actually need right now.

Jane: And this leads us to wonder how much further this specific technique can be pushed before we see something even more different in video generation.

More episodes

← Home