When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution

Summary

This paper introduces PGSR (Pixel-Grounded Super-Resolution), a framework for generative image super-resolution using diffusion transformers (DiTs). The authors identify a critical limitation in latent generative models: the VAE compression bottleneck weakens fine-grained spatial information, leading to hallucinated details that are weakly grounded in the input image. They state: latent generative models operate on compressed representations that can weaken fidelity-critical pixel evidence and the compression bottleneck of the VAE weakens fine-grained spatial information, leading to hallucinated details that are weakly grounded in the input image.

To address this, PGSR preserves LR-observed pixel evidence before VAE compression and reuses it throughout restoration via two complementary mechanisms:

1. Condition-Side Trajectory Guidance: This fuses LR-derived pixel evidence with the latent LR condition to guide the latent restoration trajectory. The fusion is formulated as: z̃lr = zlr + λ σ(g) Wp (p̄lr), where p̄lr is the pre-VAE pixel feature, Wp is a 1×1 projection, g is a learnable scalar gate, and λ controls pixel-conditioning strength.

2. Decoder-Side Pixel Grounding: This injects multi-scale pixel features into the frozen VAE decoder to ground the final rendering. The method applies: f̃l = fl + Zl (SiLU(Al(sl))), where sl denotes scale-matched pixel taps, Al aligns pixel features to decoder width, and Zl is a zero-initialized output projection.

The authors note: "our design differs from existing pixel-guided SR methods in that pixel evidence is not only appended as an auxiliary condition. Instead, it is used to ground both what content is generated in latent space and how that content is decoded back into pixels."

The framework is built on FLUX.1-dev with a frozen latent autoencoder and flow-matching backbone, training only lightweight modules: a ControlNet restoration branch, LoRA adapters, a pre-VAE pixel encoder, gated trajectory-guidance fusion, and decoder-side pixel-grounding adapters. For high-resolution efficiency, they adopt local-window attention.

The training objective combines latent flow supervision with pixel-grounded fidelity: L = Lfm + λperc Lperc + λpg Lpg, where image-space losses are applied only at low noise levels.

Experiments on DIV2K-Val, RealSR, and DRealSR benchmarks show PGSR achieves the best PSNR, LPIPS, DISTS, FID, and NIQE on DIV2K-Val; best PSNR, DISTS, FID, and NIQE on DRealSR; and best LPIPS, DISTS, and NIQE on RealSR. Ablation studies confirm that removing either pixel-grounded pathway degrades performance, and the full design gives the best results.

The authors conclude: PGSR improves the realism–fidelity trade-off and produces more faithful, visually convincing results than existing latent generative SR approaches.

Improvements to AI Systems Based on PGSR:

1. Pixel-Grounded Latent Diffusion for Medical Imaging: Integrate the dual-pathway grounding (trajectory guidance + decoder-side injection) into latent diffusion models for MRI/CT super-resolution. This would reduce hallucinated anatomical details, enabling more reliable diagnosis from low-resolution scans.

2. Real-Time Video Enhancement with Temporal Consistency: Apply the gated fusion mechanism to video SR models, where the pre-VAE pixel features from adjacent frames are used to ground latent trajectories. The improved system can upscale low-quality surveillance or satellite footage in real-time with fewer flickering artifacts and better edge preservation.

3. Text-to-Image Editing with Faithful Region Preservation: Use the decoder-side pixel grounding to inject original image pixels into the VAE decoder during text-guided edits. This allows the system to modify only the requested regions (e.g., changing a car color) while keeping the rest of the image pixel-perfect, avoiding unintended global changes common in current latent editing models.

4. Compressed-Domain Image Restoration for Low-Bandwidth Applications: Build a system that operates directly on heavily compressed latent representations (e.g., from JPEG/HEVC) by fusing pre-compression pixel evidence via the trajectory guidance pathway. This enables high-fidelity restoration on edge devices with limited compute, improving photo recovery in mobile and IoT cameras.

5. Adaptive Pixel-Conditioning Strength Control: Implement a learned, input-dependent gating scalar (λ) that automatically adjusts pixel evidence influence based on noise level and image content. The improved system can dynamically balance between hallucination (for perceptual quality) and fidelity (for detail preservation) across heterogeneous inputs, outperforming fixed-weight baselines in both clean and noisy conditions.

6. Cross-Modal Grounding for Audio-Visual Super-Resolution: Extend the pixel-grounding concept to audio-visual SR, where low-resolution video frames are grounded by high-resolution audio spectrograms (and vice versa). The system uses the gated fusion to inject cross-modal evidence into the latent trajectory, improving lip-reading and speech enhancement from low-quality video feeds.