RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

arXiv:2606.14700 · cs.CV · Submitted 2026-06-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space".

Jane: Here's the gist of the paper. RepFusion presents a method that uses pretrained Multimodal Large Language Models (MLLMs) as noisy representation encoders to provide strong priors for denoising visual representations in diffusion models The main contributions are showing that frozen pretrained MLLMs can encode noisy RAE latents and provide useful denoising priors beyond static text conditioning We demonstrate that allocating parameters to a frozen pretrained conditional encoder can outperform static text embedding baselines that spend comparable capacity on newly initialized denoisers Furthermore, we show that noisy representation inputs unlock a way to scale test-time compute by making MLLM conditioning evolve across denoising steps.

Tom: Repurposing the MLLM as a Noisy Encoder The core idea of RepFusion is to repurpose the pretrained MLLM itself as a noisy representation encoder This mechanism extends from clean to noisy inputs, using the resulting MLLM outputs as the conditioning signal for a diffusion transformer.

Jane: Specifically, RepFusion feeds noisy visual representations into an off-the-shelf MLLM by reusing its MLP projector.

Tom: The LLM backbone is kept frozen and only its projector is fine-tuned, allowing the pretrained MLLM to focus on modeling structured visual representations.

Paper summary: Jane: We've been talking about RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space. To recap, the core thesis here is repurposing a pretrained multimodal large language model, or MLLM, to act as a noisy representation encoder for diffusion models. The paper claims that this approach provides strong priors for denoising visual representations when working in the latent space of representation autoencoders or RAEs.

Tom: So it’s not just about using an MLLM to read text prompts anymore; they are showing that frozen pretrained MLLMs can encode noisy RAE latents and provide useful denoising priors beyond just static text conditioning. This means we get a prior that's more semantically meaningful because we're working in a space designed for structured visual representations.

Lu: They introduce the idea of feeding noisy visual representations into an off-the-shelf MLLM by reusing its MLP projector. The LLM backbone itself stays frozen, only that projector gets fine-tuned to handle the noisy inputs effectively. This is how they extend the mechanism from clean to noisy inputs, which is what makes it a new way of conditioning a diffusion transformer.

Meng: What's the specific signal they use for this? How do you actually feed that noisy data into the model during generation? Is it just concatenating things together?

Lalam: The conditioning signal, called ct, is defined as LastN of the MLLM output given the noisy visual tokens and timestep information. This means they select a final number N of hidden states from the MLLM's output corresponding to those noisy visual tokens. And they inject the timestep information before applying that projection P psi, following Zhou et al., two thousand twenty-five <ref:2606.14700#pg2>.

Jane: So during sampling, this conditioning signal ct is recomputed at every single denoising step because the visual representation itself changes at each step, and the language model stays causal so it can process it correctly. That’s a key mechanism for making this work dynamically.

Tom: It's about making that conditioning signal change along the denoising trajectory, which makes per-step MLLM recomputation useful instead of just using a single fixed condition at the start. This is what unlocks that scaling behavior we talked about earlier.

Lu: They compare this against other methods like Transfusion, and they found that even when those baselines are upgraded to operate in the RAE latent space, the gains are smaller than explicitly repurposing a frozen MLLM as a noisy encoder. This shows the value of keeping those pretrained priors intact.

Jane: So what does this mean for someone who only listens to the show? It means we can get better results by letting the language model learn to interpret visual noise directly, rather than just being told what text is in it.

Conclusion: Tom: So, looking at the paper "RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space," the authors argue that many modern T2I systems already spend a lot of capacity on huge LLM text encoders. RepFusion suggests that by letting a frozen MLLM read noisy visual representations, the encoder becomes an active component of the denoising loop instead of just being a static text encoder.

Jane: The main implication is that this method offers a practical recipe for future models: take pretrained MLLM priors and expose them to evolving noisy representations. Then, spend test-time compute on repeated conditioning only when that input carries information dependent on the current denoising state, while preserving those strong priors.

Lu: They’re suggesting we should think about how the community views the role of MLLMs in T2I generation differently—not just as a fixed text encoder, but as something that can actively participate in refining visual representations during generation.

Meng: From a deployment standpoint, it seems like they're pointing toward a system where test-time compute is managed dynamically based on how much input-dependent information the MLLM's condition actually provides at each step.

Lalam: The result shows that this approach yields high GenEval scores, reaching zero point seven zero when compared against various baselines, demonstrating that repurposing the MLLM as a noisy encoder is a solid strategy for improving denoising in representation space <ref:2606.14700#pg2>.

Meta AI · New York University

cs.CV

Submitted: 2026-06-12

Updated: 2026-11-10

Comments: NeurIPS 2026; Project page: https://xichenpan.com/repfusion

Project page: https://xichenpan.com/repfusion

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 100/100

The gist: RepFusion presents a method that uses pretrained Multimodal Large Language Models (MLLMs) as noisy representation encoders to provide strong priors for denoising visual representations in diffusion…

Key concepts

Repurposing the MLLM as a Noisy Encoder
The core idea is to use a pretrained MLLM not just for text, but to encode noisy visual data. RepFusion feeds noisy visual representations into the MLLM using its existing MLP projector while keeping the main LLM backbone frozen. This allows the model to focus on modeling structured visual information from imperfect inputs.
Training Signal Construction
The conditioning signal is created by selecting the final hidden states (LastN) from the MLLM after it processes noisy visual tokens. Timestep information is added before projection, and this signal is recomputed at every denoising step because the visual latents change, ensuring the LLM remains causal.
Allocating Parameters to a Frozen Encoder
RepFusion shows that spending parameters on a frozen pretrained conditional encoder can beat static text embedding baselines. This is because the MLLM provides richer priors than newly initialized denoisers, and its conditioning becomes state-dependent, enabling more effective denoising.
Scaling Behavior at Inference
The method introduces two scaling axes: the frozen MLLM reading evolving noisy representations and the DiT denoiser predicting velocity. Analysis shows that allocating more compute to the DiT often outperforms allocating more to the MLLM, demonstrating an optimized way to use test-time compute.

Terminology

Summary

RepFusion presents a method that uses pretrained Multimodal Large Language Models (MLLMs) as noisy representation encoders to provide strong priors for denoising visual representations in diffusion models The main contributions are showing that frozen pretrained MLLMs can encode noisy RAE latents and provide useful denoising priors beyond static text conditioning We demonstrate that allocating parameters to a frozen pretrained conditional encoder can outperform static text embedding baselines that spend comparable capacity on newly initialized denoisers Furthermore, we show that noisy representation inputs unlock a way to scale test-time compute by making MLLM conditioning evolve across denoising steps. p. 2

Where the paper says it

• We demonstrate that allocating parameters to a frozen pretrained conditional encoder can outperform static text embedding baselines that spend comparable capacity on newly initialized denoisers. • We show that noisy representation inputs unlock a way to scale test-time compute by making MLLM conditioning evolve across denoising steps.

Repurposing the MLLM as a Noisy Encoder

The core idea of RepFusion is to repurpose the pretrained MLLM itself as a noisy representation encoder This mechanism extends from clean to noisy inputs, using the resulting MLLM outputs as the conditioning signal for a diffusion transformer. Specifically, RepFusion feeds noisy visual representations into an off-the-shelf MLLM by reusing its MLP projector. p. 1

Where the paper says it

We present RepFusion, a T2I model that treats a pretrained MLLM as a noisy representation encoder. In addition to text inputs, we feed noisy RAE latents into an off-the-shelf MLLM by reusing its MLP projector.

The LLM backbone is kept frozen and only its projector is fine-tuned, allowing the pretrained MLLM to focus on modeling structured visual representations. p. 1
Where the paper says it

We then use the MLLM’s output to condition a DiT that denoises in the same latent space. Conceptually, this design allows the pretrained MLLM to focus on what it does best: modeling structured visual representations.

How the Training Signal is Built

The conditioning signal ct is defined as LastN (ELLM ([y, Pψ(zt + et)])) (2) where [·, ·] denotes sequence concatenation, and LastN selects the final N hidden states corresponding to the noisy visual tokens. Timestep information is injected before the MLLM by adding embedding et to zt before applying projection Pψ, following Zhou et al. p. 4

Where the paper says it

The conditioning ct is defined as ct = LastN (ELLM ([y, Pψ(zt + et)])) (2) where [·, ·] denotes sequence concatenation, and LastN selects the final N hidden states corresponding to the noisy visual tokens. We inject timestep information before the MLLM by adding the embedding et to zt before applying the projection Pψ, following Zhou et al.

(2025). During sampling, ct is recomputed accordingly because zt evolves at each denoising step, and the LLM remains causal. p. 4
Where the paper says it

During sampling, zt evolves at each denoising step, so ct is recomputed accordingly. The LLM remains causal.

Where the Gains Come From

The performance gains stem from several factors related to representation space and conditioning strategy

  1. The transition from VAEs to RAEs provides a denoising space that is both easier to optimize and more semantically meaningful. p. 0

    Where the paper says it

    , 2021) or DINO (Caron et al. , 2021) features, RAEs provide a denoising space that is both easier to optimize and more semantically meaningful.

  2. Allocating parameters to a frozen pretrained conditional encoder can outperform static text embedding baselines that spend comparable capacity on newly initialized denoisers

  3. Noisy representation inputs make RepFusion’s condition depend on the denoising state, enabling useful test-time compute scaling through repeated MLLM conditioning and improving GenEval to 0. p. 5

    Where the paper says it

    54), because the MLLM still does not observe the evolving noisy representation. In contrast, noisy representation inputs make RepFusion’s condition depend on the denoising state, enabling useful test-time compute scaling through repeated MLLM conditioning and improving GenEval to 0.

    70.

  4. Multimodal perception pretraining improves denoising in RAE space, but the best performance is achieved when the LLM backbone remains frozen, preserving the pretrained prior. p. 6

    Where the paper says it

    Jointly training the LLM with an LM loss and bidirectional attention gives a minor gain to 0. Finally, adding multimodal perception pretraining improves denoising in RAE space, but the best performance is achieved when the LLM backbone remains frozen, preserving the pretrained prior.

Scaling Behavior and Interface Design

RepFusion introduces two scaling axes at inference time: the frozen MLLM that repeatedly reads evolving noisy representations, and the DiT denoiser that predicts the velocity. Table 4 shows that at around 280T FLOPs, allocating more compute to the DiT outperforms the configuration that allocates more compute to the MLLM across all metrics. p. 9

Where the paper says it

We further study how to allocate inference compute across these two components under iso-FLOPs settings in Table 4. At around 280T FLOPs, allocating more compute to the DiT outperforms the configuration that allocates more compute to the MLLM across all metrics.

The choice of conditioning interface is also optimized, as AdaLN-Single achieves a slightly higher GenEval score with fewer parameters compared to Cross Attention. p. 9
Where the paper says it

As shown in Table 3, AdaLN-Single (Chen et al. , 2024) achieves a slightly higher GenEval score with fewer parameters, and we therefore use it as the default conditioning interface.

Breaking Down the Improvement

The improvement path from TextEmbed to RepFusion involves several steps Starting from a standard text embedding baseline with a GenEval score of 0.47, feeding noisy VAE latents into the LLM improves the score to 0. p. 6

Where the paper says it

As shown in Figure 6a, starting from a standard text embedding baseline with a GenEval score of 0. 47, feeding noisy VAE latents into the LLM improves the score to 0.

54. Replacing VAE latents with RAE latents further improves the score to 0. p. 6
Where the paper says it

47, feeding noisy VAE latents into the LLM improves the score to 0. Replacing VAE latents with RAE latents, which are easier to denoise and more compatible with LLMs, further improves the score to 0.

64 Finally, adding multimodal perception pretraining improves denoising in RAE space, but the best performance is achieved when the LLM backbone remains frozen. p. 6
Where the paper says it

Jointly training the LLM with an LM loss and bidirectional attention gives a minor gain to 0. Finally, adding multimodal perception pretraining improves denoising in RAE space, but the best performance is achieved when the LLM backbone remains frozen, preserving the pretrained prior.

The choice of decoder does not affect prompt-following ability because both decoders are optimized for reconstruction. p. 2
Where the paper says it

, 2021) popularized a key design choice in modern T2I models: instead of diffusing in pixel space, models denoise in the latent space of an autoencoder, making high-resolution generation tractable. Most systems adopt VAEs (Kingma, 2014) for this purpose, but VAE latents are heavily compressed and optimized for reconstruction, which limits their semantic expressiveness.

The overall layout and colors are largely determined by the CLIP feature, while only fine-grained textures differ slightly. The choice of decoder mainly affects fine-grained appearance (e. p. 2
Where the paper says it

From VAEs to RAEs Latent diffusion (Rombach et al. , 2021) popularized a key design choice in modern T2I models: instead of diffusing in pixel space, models denoise in the latent space of an autoencoder, making high-resolution generation tractable.

g., textures), and thus has little impact on object-level prompt-following. p. 9
Where the paper says it

† denotes rewritten prompts. For GenEval2, we report the prompt-level metric Soft-TIFAGM.

The best performance is achieved when the LLM backbone remains frozen, preserving the pretrained prior. p. 6
Where the paper says it

Jointly training the LLM with an LM loss and bidirectional attention gives a minor gain to 0. Finally, adding multimodal perception pretraining improves denoising in RAE space, but the best performance is achieved when the LLM backbone remains frozen, preserving the pretrained prior.

Conclusion

We study a simple but underused degree of freedom in modern T2I systems: the conditional encoder By allowing a frozen MLLM to read noisy visual representations, the encoder becomes an active component of the denoising loop rather than a static text encoder. The resulting gains suggest a practical recipe for future models: expose pretrained MLLM priors to evolving noisy representations, spend test-time compute on repeated conditioning only when it carries input-dependent information, and preserve those priors. p. 0

Where the paper says it

In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.

We hope this perspective helps shift how the community thinks about the role of MLLMs in T2I generation. p. 2
Where the paper says it

In other words, moving from VAE to RAE helps, but by itself it does not unlock the full benefit of pretrained language priors. In summary, this paper argues for a simple shift in perspective: many modern T2I systems already allocate substantial capacity to huge LLM text encoders, and RAEs provide a representation space where these encoders can do more than encode text.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. RepFusion allows a frozen Multimodal Large Language Model (MLLM) to act as a noisy representation encoder, conditioning a diffusion transformer on evolving noisy visual representations during denoising. This enables test-time compute to be productively spent on repeated MLLM conditioning in modern T2I systems.

  2. The system demonstrates that allocating substantial model capacity to a frozen pretrained conditional encoder can outperform static text embedding baselines that spend comparable capacity on newly initialized denoisers. This suggests repurposing frozen MLLMs is a more effective use of parameters than scaling newly initialized denoisers.

  3. RepFusion introduces a distinct axis for scaling at test time, where the conditioning signal changes along the denoising trajectory, making per-step MLLM recomputation useful. This allows for a way to scale test-time compute by making MLLM conditioning evolve across denoising steps.

  4. The system leverages multimodal perception pretraining in the MLLM backbone, showing that perception pretraining provides a transferable prior for diffusion in RAE space, meaning an MLLM pretrained to perceive clean visual representations can better support encoding their noisy counterparts.

Abstract

Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. Reusing MLLM outputs across denoising steps further accelerates inference while retaining performance comparable to per-step MLLM recomputation. These results demonstrate that MLLMs provide strong priors for denoising visual representations.

Sources

Related papers