Back to Pixels: Lifting Latent Video Diffusion into Pixel Space
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Back to Pixels".
Jane: Here's the gist of the paper. Frontier video generation models work in a VAE latent space that compresses video in both space and time The paper presents PixVid, a simple yet effective method that lifts a pretrained latent video model into pixel space to generate every frame directly as pixels This approach overcomes structural limitations of latent models such as resolution scaling, control alignment under fast motion, and control-to-screen latency.
Tom: Pruning redundant reasoning tokens PixVid keeps the diffusion transformer (DiT) and replaces the VAE decoder with a convolutional head that renders its outputs into pixels The method achieves frame-wise tokenization where each token covers a 1 × p × p patch of exactly one frame, ensuring no condition or action spans multiple frames.
Jane: The detailer head is implemented as a lightweight 3D convolutional decoder that reads only the DiT tokens and suppresses checkerboard artifacts through convolutions across neighboring tokens How the training signal is built PixVid finetunes end-to-end in pixel space from a pretrained latent model, initializing the DiT and input patchification to start from close to its pretrained latent distribution.
Tom: The key techniques identified include predicting clean video (x-prediction) rather than velocity, which is essential for the detailer to render clean local structure Additionally, a larger noise shift is used because redundant pixels retain more signal at the same noise level.
Paper summary: Tom: So to summarize "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space," the authors have effectively removed the compression step by finetuning a latent video diffusion model directly into pixel space. They claim this successfully overcomes resolution scaling issues, better motion handling in editing, and low latency response for controls >
Jane: The paper suggests that while latent models save on training costs for short clips, they are hitting limits when you need real-time interactive world models or precise frame-by-frame control alignment >
Lu: It shows that by predicting clean video instead of velocity and using specific noise shifts, they can maintain competitive quality while achieving frame-wise pixel generation >
Meng: From a deployment angle, the latency reduction to thirty-four milliseconds for control to screen is significant because it means actions are responsive on the very next frame, which is what we need for interactive applications >
Tom: And they show that this direct pixel generation method can achieve higher total quality scores than both baselines at different resolutions, showing gains in scene consistency and imaging quality >
Jane: Overall, "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space" suggests that finetuning a latent video diffusion model into a frame-wise pixel space is a competitive method for achieving high-quality video generation, especially when you need those specific performance improvements in motion and speed >
Conclusion: Tom: So we’re wrapping up our look at this paper, "Back to Pixels: Lifting Latent Video Diffusion into Pixel Space." Basically, they took a video model that was super compressed in latent space and found a way to generate every single frame directly as raw pixels.
Jane: That sounds pretty wild when you think about how much data that actually is. So, what does this whole "lifting" process actually do for us?
Lu: It’s about taking this existing, trained model—the diffusion transformer—and replacing the part that squashes everything into a compressed latent code with something new. They use a convolutional head to turn those tokens back into pixels, instead of using the old VAE decoder.
Meng: From my side at the startup, I care about what this means for deployment. The authors claim they fixed some major problems with those latent models, like scaling resolution or handling fast motion better than before.
Tom: Exactly. They show that by doing this finetuning in pixel space, you can double the resolution—say from a certain size to twice that size—and it still uses the same amount of computation for the main model part.
Jane: That’s really interesting because I always thought latent models were just a clever shortcut for training speed. So, what about those things they specifically mentioned fixing, like control alignment under fast motion?
Lu: They found that this method keeps ninety-eight percent of the source's fast motion quality when compared to the original latent model's eighty-three percent. It’s pretty close to what you get if you trained a pixel-space model from scratch.
Meng: But we gotta ask about latency. Latent models wait for whole blocks of frames, right? This method generates one frame block at a time, so the delay for an action to show up on the screen drops dramatically.
Tom: Right, that’s where it gets practical. They cut the control-to-screen latency from four hundred ninety-five milliseconds down to thirty-four milliseconds after they did some distillation work. That’s real time responsiveness.
Jane: So, to sum up, this paper is showing us that you can actually get high quality in pixel space by intelligently adapting a model that was originally designed for latent spaces, and it does so while making the generation much faster and more responsive for real-time use.
Lu: It opens up new training possibilities too. They suggest generating directly in pixels lets you train things in ways that a frozen VAE just wouldn't allow.
Tom: Yeah, it seems like they’ve found a way to get the best of both worlds here—using the power of pretraining but giving you the direct pixel output and speed we actually need right now.
Jane: And this leads us to wonder how much further this specific technique can be pushed before we see something even more different in video generation.
Yanke Song, Shuo Chen, Chris Rockwell, Guande He, Nan Liu, Desai Xie, Tianpei Gu, Yong-Hyun Park, Aryan V S1, Dennis Yuren Menn, Ming-Hsuan Yang, ©Xun Huang
Roblox · University of California, Merced · University of Pennsylvania · University of Texas at Austin
cs.CV
Submitted: 2026-11-12
Updated: 2026-11-12
Comments: Project Page: https://chenshuo20.github.io/PixVid/
Project page: https://chenshuo20.github.io/PixVid
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 92/100
The gist: Frontier video generation models work in a VAE latent space that compresses video in both space and time The paper presents PixVid…
Key concepts
- VAE Latent Space
- Video generation models compress video into a compressed latent space using a Variational Autoencoder (VAE). This compression reduces the complexity of the video data by encoding both spatial dimensions (pixels) and temporal dimensions (time), making it manageable for diffusion models.
- Frame-wise Tokenization
- PixVid ensures that each token in the model corresponds to exactly one frame, covering a 1x p x p patch. This design prevents conditions or actions from spanning multiple frames within a single token, which is crucial for precise control and accurate frame generation.
- Detailer Head
- This is a lightweight 3D convolutional decoder that reads the diffusion transformer tokens and renders them into actual pixels. It suppresses artifacts like checkerboard patterns by using convolutions across neighboring tokens, ensuring clean local structure in the final pixel output.
Terminology
Summary
Frontier video generation models work in a VAE latent space that compresses video in both space and time The paper presents PixVid, a simple yet effective method that lifts a pretrained latent video model into pixel space to generate every frame directly as pixels This approach overcomes structural limitations of latent models such as resolution scaling, control alignment under fast motion, and control-to-screen latency. p. 7
Pruning redundant reasoning tokens
PixVid keeps the diffusion transformer (DiT) and replaces the VAE decoder with a convolutional head that renders its outputs into pixels The method achieves frame-wise tokenization where each token covers a 1 × p × p patch of exactly one frame, ensuring no condition or action spans multiple frames. The detailer head is implemented as a lightweight 3D convolutional decoder that reads only the DiT tokens and suppresses checkerboard artifacts through convolutions across neighboring tokens
How the training signal is built
PixVid finetunes end-to-end in pixel space from a pretrained latent model, initializing the DiT and input patchification to start from close to its pretrained latent distribution. The key techniques identified include predicting clean video (x-prediction) rather than velocity, which is essential for the detailer to render clean local structure Additionally, a larger noise shift is used because redundant pixels retain more signal at the same noise level. p. 1 Auxiliary losses include a loss on frame differences to learn temporal smoothness, denoted as Ltemp. p. 2
Where the gains come from
PixVid demonstrates several benefits across different applications, including resolution scaling, control alignment under fast motion, and low-latency autoregressive generation Specifically, in high-resolution generation (960 × 1664), doubling the patch size from p to 2p adds one up block to the detailer head, keeping the token grid unchanged while achieving 2× resolution at the same DiT cost Furthermore, for precise video editing, PixVid keeps 98% of the source’s fast motion compared to 83% for an identically finetuned latent model.
Latency and speed
PixVid achieves low latency by generating single-frame blocks, allowing each action to take effect on the next frame rather than waiting for a full block of frames as in latent models After autoregressive distillation, the median control-to-screen latency is cut from 495 ms to 34 ms while still streaming in real time. This contrasts with latent models where a control cannot be generated until all controls for the full block are received, resulting in a minimum control-to-screen latency proportional to the block size. p. 1
Conclusion
Latent video diffusion models compress video in space and time with a VAE, but PixVid removes this compression by efficiently finetuning a pretrained latent model to generate every frame directly in pixels It successfully overcomes resolution scaling, control alignment under fast motion, and control-to-screen latency. The work shows that finetuning is competitive with the pretrained latent model and compares favorably with concurrent pixel-space models. p. 4 The overall quality dimensions show that PixVid achieves a higher total score than both baselines at both resolutions. p. 4 The largest gains are in dynamic degree, spatial relationship, scene, and imaging quality. p. 4 The paper suggests that generating directly in pixels opens up training options that a frozen VAE rules out
--- Page 9 ---
Acknowledgments. We thank Yicong Hong, Zhanpeng Luo, and Wei-An Lin for their assistance in human evaluation
--- Page 5 ---
Overall Quality dimensions Semantic dimensions Model Size Total Quality Semantic Subject consist. Backgr. consist. Temporal flicker. p. 4 Motion smooth. p. 4 Dynamic degree Aesthetic quality Imaging quality Object class Multiple objects Human action Color Spatial relation Scene Appear style Temporal style Overall consist. p. 4
--- Page 10 ---
Table 1 shows that PixVid reaches a VBench total of 84.78, above both Wan2. p. 01 (84.05) and PixelUMM (83. p. 472).
--- Page 6 ---
Table 2 shows that PixVid keeps the scene intact under fast motion, where the latent model morphs and blurs moving objects (Fig. 5).
--- Page 4 ---
Figure 5 shows that when several frames share one latent, fast motion blurs and objects deform.
--- Page 8 ---
Figure 2 shows the comparison between the pretrained latent model and PixVid, illustrating how a linear pixel-latent alignment is fused with the pretrained patch embedding to obtain a pixel patch embedding.
--- Page 4 ---
Figure 9 shows that v-prediction fills the video with high-frequency, flickering noise at 80 times the noise level of x-prediction.
--- Page 8 ---
Figure 8 shows that our convolutional head removes most seams, lowering the seam measure from 118.6 to 13.3 compared to a linear head. p. 1
--- Page 4 ---
Figure 9 shows that x-prediction produces clean ones, whereas v-prediction produces grainy frames at 80× the estimated noise of x-prediction.
--- Page 5 ---
Table 7 shows that PixVid has lower motion error than the finetuned la16 model across all motion bins.
--- Page 5 ---
Table 8 shows that PixVid achieves the highest semantic score of 82.15.
--- Page 4 ---
Figure 4 shows that raters prefer PixVid across all comparisons, showing that finetuning is competitive with the pretrained latent model and compares favorably with PixelUMM.
--- Page 5 ---
Table 6 shows that the total scores change by at most 0.10 (84.07 and 83.82), so scoring the first 1. p. 43 s does not favor PixVid. p. 0
--- Page 2 ---
The first followed latent diffusion into the latent space of a pretrained variational autoencoder (VAE) The second extended the compression to time, using 3D VAEs to merge every few frames into a single latent that corresponds to no individual frame.
--- Page 2 ---
For generating short clips offline at moderate resolution, this was a favorable trade: compression made training tractable at scale, at the cost of a frozen VAE that ruled out end-to-end training and capped quality at its reconstruction.
--- Page 2 ---
Video generation models have since outgrown this setting: they now power real-time interactive world models through causal few-step distillation and serve as geometry-conditioned renderers for editing and games.
--- Page 2 ---
Under a real-time compute budget, doubling the resolution yields 4× more tokens and 16× more attention compute, while cutting tokens requires a more aggressive VAE that further reduces quality Precise editing requires conditions aligned to individual frames, but when several frames share one latent, fast motion blurs and objects deform (Fig. 5).
--- Page 2 ---
Since common latent world models autoregressively generate frames in blocks, the frame corresponding to a control cannot be generated until all controls for the full block are received, resulting in a minimum control-to-screen latency proportional to the block size.
--- Page 2 ---
These limitations motivate removing the VAE altogether and generating every frame directly as pixels, which naively would bring back the token counts and training costs that latent models avoid
--- Page 2 ---
In this work, we present PixVid, a recipe that lifts a pretrained latent video diffusion model into a frame-wise pixelspace model which keeps the diffusion transformer and replaces both the encoder and the decoder from the VAE
--- Page 2 ---
At input, a patch embedding that initializes from the VAE encoder allows the DiT to start training from close to its pretrained latent distribution At output, a lightweight detailer head renders the DiT tokens into pixels; it is convolutional to suppress the patch-grid artifacts of a linear head
--- Page 4 ---
We find that predicting clean video rather than velocity, together with a larger noise shift, is key to quality PixVid converges within 1.3M training clips sampled from 64K videos, or about 4K B200 GPU-hours. p. 1
--- Page 5 ---
PixVid generates at 2× higher resolution with the same per-frame token count, keeps depth-conditioned video editing aligned under fast motion, and responds to each action at the next frame
--- Page 7 ---
The paper identifies three key techniques: x-prediction, using a larger noise shift, and using the Euler sampler as key techniques for quality improvement.
--- Page 8 ---
PixVid converges within 1.3M training clips sampled from 64K videos, or about 4K B200 GPU-hours
--- Page 9 ---
The recipe transfers across Wan2.1 and Wan2. p. 02-based models with different VAEs. p. 1
--- Page 4 ---
We propose PixVid, a recipe that lifts a pretrained latent video diffusion model into a frame-wise pixel-space model, in which every token covers a 1 × p × p patch of exactly one frame
--- Page 4 ---
We identify the design choices that make this lift work: a causal convolutional detailer head with 9× fewer patch seams than a linear head, x-prediction with 80× less pixel noise than v-prediction, and a larger noise shift than the latent model
--- Page 4 ---
We show frame-wise pixel generation overcomes three structural limitations of latent models: resolution scaling, control alignment under fast motion, and control-toscreen latency.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Precision in video editing under fast motion: PixVid keeps 98% of source motion compared to 83% for an identically finetuned latent model, and lowers motion error by 40% on fast clips.
-
High-resolution generation with token efficiency: PixVid generates 960 × 1664 videos with the same number of tokens per frame as a smaller model (480 × 832), achieving competitive quality by doubling the patch size and keeping the token grid unchanged.
-
Low latency interactive video generation: By generating single-frame blocks, PixVid cuts median control-to-screen latency from 495 ms to 34 ms, allowing actions to take effect at the next frame instead of waiting for a full block of frames.
-
Semantic fidelity in complex scenes: PixVid achieves a higher semantic score (82.15) than the latent baseline, indicating that lifting generation directly into pixels does not degrade quality and improves scores in dynamic degree and spatial relationship.
-
Robustness to motion artifacts: The convolutional detailer head suppresses checkerboard artifacts at token boundaries, resulting in significantly lower seam measures (from 118.6 to 13.3) compared to linear heads, which prevents visible grid-like artifacts during pixel generation.
Sources
- ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- SkyReels-V2: Infinite-length Film Generative Model
- Asymmetric Flow Models
- PixelFlow: Pixel-Space Generative Models with Flow
- Context Forcing: Consistent Autoregressive Video Generation with Long Context
- L2P: Unlocking Latent Potential for Pixel Generation
- Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers
- Matrix-game 2.0: An open-source, real-time, and streaming interactive world model
- Classifier-Free Diffusion Guidance
- Imagen Video: High Definition Video Generation with Diffusion Models
- RELIC: Interactive Video World Model with Long-Horizon Memory
- ViPE: Video Pose Engine for 3D Geometric Perception
- ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU
- MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions
- EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
- Sekai: A Video Dataset towards World Exploration
- Flow Matching for Generative Modeling
- Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos
Related papers
- Efficient All-in-One Weather Restoration using Spectral Harmonization
- Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment
- Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
- WithEveryone: Unified Planning and Identity Grounding for Group Image Generation
- SciFigure2Code: An AI-Reconstructed Benchmark for Scientific Figure-to-Code
- RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space