RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space
summary
The gist
RepFusion presents a method that uses pretrained Multimodal Large Language Models (MLLMs) as noisy representation encoders to provide strong priors for denoising visual representations in diffusion…
In short
RepFusion uses frozen pretrained Multimodal Large Language Models (MLLMs) as noisy representation encoders to provide strong priors for denoising visual representations in diffusion models. By feeding noisy visual latents into an MLLM and fine-tuning only its projector, the method outperforms static text conditioning and allows test-time compute scaling by making the MLLM condition evolve across denoising steps.
Key concepts
- Repurposing the MLLM as a Noisy Encoder
- The core idea is to use a pretrained MLLM not just for text, but to encode noisy visual data. RepFusion feeds noisy visual representations into the MLLM using its existing MLP projector while keeping the main LLM backbone frozen. This allows the model to focus on modeling structured visual information from imperfect inputs.
- Training Signal Construction
- The conditioning signal is created by selecting the final hidden states (LastN) from the MLLM after it processes noisy visual tokens. Timestep information is added before projection, and this signal is recomputed at every denoising step because the visual latents change, ensuring the LLM remains causal.
- Allocating Parameters to a Frozen Encoder
- RepFusion shows that spending parameters on a frozen pretrained conditional encoder can beat static text embedding baselines. This is because the MLLM provides richer priors than newly initialized denoisers, and its conditioning becomes state-dependent, enabling more effective denoising.
- Scaling Behavior at Inference
- The method introduces two scaling axes: the frozen MLLM reading evolving noisy representations and the DiT denoiser predicting velocity. Analysis shows that allocating more compute to the DiT often outperforms allocating more to the MLLM, demonstrating an optimized way to use test-time compute.
Terminology used across episodes
This episode discusses
- RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space · Paper Radio
- Qwen3-VL Technical Report
- Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
- BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
- ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation
- Emerging Properties in Unified Multimodal Pretraining
- The Llama 3 Herd of Models · Paper Radio
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
- Transfer between Modalities with MetaQueries
- Movie Gen: A Cast of Media Foundation Models
- Gemma 3 Technical Report
- LLaMA: Open and Efficient Foundation Language Models
- Qwen-Image Technical Report
- Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation
The paper
RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space · Read on arXiv
Meta AI · New York University
Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. Reusing MLLM outputs across denoising steps further accelerates inference while retaining performance comparable to per-step MLLM recomputation. These results demonstrate that MLLMs provide strong priors for denoising visual representations.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space".
Jane: Here's the gist of the paper. RepFusion presents a method that uses pretrained Multimodal Large Language Models (MLLMs) as noisy representation encoders to provide strong priors for denoising visual representations in diffusion models The main contributions are showing that frozen pretrained MLLMs can encode noisy RAE latents and provide useful denoising priors beyond static text conditioning We demonstrate that allocating parameters to a frozen pretrained conditional encoder can outperform static text embedding baselines that spend comparable capacity on newly initialized denoisers Furthermore, we show that noisy representation inputs unlock a way to scale test-time compute by making MLLM conditioning evolve across denoising steps.
Tom: Repurposing the MLLM as a Noisy Encoder The core idea of RepFusion is to repurpose the pretrained MLLM itself as a noisy representation encoder This mechanism extends from clean to noisy inputs, using the resulting MLLM outputs as the conditioning signal for a diffusion transformer.
Jane: Specifically, RepFusion feeds noisy visual representations into an off-the-shelf MLLM by reusing its MLP projector.
Tom: The LLM backbone is kept frozen and only its projector is fine-tuned, allowing the pretrained MLLM to focus on modeling structured visual representations.
Paper summary: Jane: We've been talking about RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space. To recap, the core thesis here is repurposing a pretrained multimodal large language model, or MLLM, to act as a noisy representation encoder for diffusion models. The paper claims that this approach provides strong priors for denoising visual representations when working in the latent space of representation autoencoders or RAEs.
Tom: So it’s not just about using an MLLM to read text prompts anymore; they are showing that frozen pretrained MLLMs can encode noisy RAE latents and provide useful denoising priors beyond just static text conditioning. This means we get a prior that's more semantically meaningful because we're working in a space designed for structured visual representations.
Lu: They introduce the idea of feeding noisy visual representations into an off-the-shelf MLLM by reusing its MLP projector. The LLM backbone itself stays frozen, only that projector gets fine-tuned to handle the noisy inputs effectively. This is how they extend the mechanism from clean to noisy inputs, which is what makes it a new way of conditioning a diffusion transformer.
Meng: What's the specific signal they use for this? How do you actually feed that noisy data into the model during generation? Is it just concatenating things together?
Lalam: The conditioning signal, called ct, is defined as LastN of the MLLM output given the noisy visual tokens and timestep information. This means they select a final number N of hidden states from the MLLM's output corresponding to those noisy visual tokens. And they inject the timestep information before applying that projection P psi, following Zhou et al., two thousand twenty-five <ref:2606.14700#pg2>.
Jane: So during sampling, this conditioning signal ct is recomputed at every single denoising step because the visual representation itself changes at each step, and the language model stays causal so it can process it correctly. That’s a key mechanism for making this work dynamically.
Tom: It's about making that conditioning signal change along the denoising trajectory, which makes per-step MLLM recomputation useful instead of just using a single fixed condition at the start. This is what unlocks that scaling behavior we talked about earlier.
Lu: They compare this against other methods like Transfusion, and they found that even when those baselines are upgraded to operate in the RAE latent space, the gains are smaller than explicitly repurposing a frozen MLLM as a noisy encoder. This shows the value of keeping those pretrained priors intact.
Jane: So what does this mean for someone who only listens to the show? It means we can get better results by letting the language model learn to interpret visual noise directly, rather than just being told what text is in it.
Conclusion: Tom: So, looking at the paper "RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space," the authors argue that many modern T2I systems already spend a lot of capacity on huge LLM text encoders. RepFusion suggests that by letting a frozen MLLM read noisy visual representations, the encoder becomes an active component of the denoising loop instead of just being a static text encoder.
Jane: The main implication is that this method offers a practical recipe for future models: take pretrained MLLM priors and expose them to evolving noisy representations. Then, spend test-time compute on repeated conditioning only when that input carries information dependent on the current denoising state, while preserving those strong priors.
Lu: They’re suggesting we should think about how the community views the role of MLLMs in T2I generation differently—not just as a fixed text encoder, but as something that can actively participate in refining visual representations during generation.
Meng: From a deployment standpoint, it seems like they're pointing toward a system where test-time compute is managed dynamically based on how much input-dependent information the MLLM's condition actually provides at each step.
Lalam: The result shows that this approach yields high GenEval scores, reaching zero point seven zero when compared against various baselines, demonstrating that repurposing the MLLM as a noisy encoder is a solid strategy for improving denoising in representation space <ref:2606.14700#pg2>.
More episodes
- 2606.27866-FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- 2607.04680-A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- 2607.04919-When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters
- 2607.05025-Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencoder
- 2607.12304-What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
- 2608.02877-Graph-of-Thoughts for LLM Causal Discovery with a Fairness-Pathway Case Study
- 2110.08294-Coherence boosting: When your pretrained language model is not paying enough attention
- 2210.01293-ThinkSum: Probabilistic reasoning over sets using large language models
- 2504.15270-Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax
- 2508.10683-Neural Machine Translation for Coptic-French: Strategies for Low-Resource Ancient Languages