Follow the Mean: Reference-Guided Flow Matching

summary

Video file (mp4)

The gist

The gist The paper shows that flow matching admits a different control interface: adaptation through examples, which allows steering pretrained models by changing the reference set they follow

In short

The paper introduces a new control method for flow matching called adaptation through examples. Instead of fine-tuning, it shows that steering pretrained models can be achieved by changing the reference set they follow. This is demonstrated via training-free Reference-Mean Guidance and a semi-parametric approach, proving that generative models can adapt to data without needing parameter updates.

Key concepts

Flow Matching
A method used for controllable generation where the velocity field of a diffusion process is directly related to the endpoint mean. This allows control by manipulating this mean.
Reference-Mean Guidance (RMG)
A training-free technique that steers frozen models by applying a closed-form correction to the endpoint mean derived from a reference bank. It enables control over style and structure without retraining the model.
Semi-Parametric Guidance (SPG)
A method that builds reference-mean guidance into a trained model using an explicit mean anchor and a learned residual refiner. This allows inference-time swapping of reference sets while maintaining high generation quality.

Terminology used across episodes

This episode discusses

The paper

Follow the Mean: Reference-Guided Flow Matching · Read on arXiv

Pedro M. P. Curvo, Maksim Zhdanov, Floor Eijkelboom, Jan-Willem van de Meent

University of Amsterdam

Existing approaches to controllable generation typically rely on fine-tuning, auxiliary networks, or test-time search. We show that flow matching admits a different control interface: adaptation through examples. For deterministic interpolants, the velocity field is solely governed by a conditional endpoint mean, so shifting this mean shifts the flow itself, and a pretrained model can be steered simply by changing the reference set it follows. We instantiate this idea in two forms. Reference-Mean Guidance is training-free. It computes a closed-form correction to the endpoint mean from a reference bank and applies it to a frozen FLUX.2-klein (4B) model, controlling color, identity, style, and structure while the prompt, seed, and weights stay fixed. Semi-Parametric Guidance builds the same principle into a trained model through an explicit mean anchor and a learned residual refiner, matching the quality of an unconditional DiT-B/4 on AFHQv2 while allowing the reference set to be swapped at inference time. Together, these results point to a broader direction: generative models that adapt through data rather than parameter updates.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Follow the Mean: Reference-Guided Flow Matching".

Jane: Here's the gist of the paper. The gist The paper shows that flow matching admits a different control interface: adaptation through examples, which allows steering pretrained models by changing the reference set they follow Reference-Guided Flows The core insight is that "the velocity field in flow matching points toward the endpoint mean" and "shifting this mean shifts the flow itself".

Tom: This leads to a simple principle for controllable generation: "steer a pretrained model by changing the reference set it follows".

Jane: This idea is instantiated in two forms: 1.

Tom: Reference-Mean Guidance (RMG) is training-free because it "computes a closed-form endpoint-mean correction from a reference bank and applies it to a frozen FLUX.2-klein (4B) model, enabling control of color, identity, style, and structure while keeping the prompt, seed, and weights fixed".

Paper summary: Lu: To wrap up, the authors of "Follow the Mean: Reference-Guided Flow Matching" are arguing that control in this area can be framed simply as a problem of shifting endpoint means. They found that you don't need to change the model parameters or train a new network for every style or concept.

Tom: So, if you boil it down, instead of fine-tuning or search methods, the paper proposes steering generation just by changing the reference set over which the model implicitly or explicitly aggregates information. It’s a different way to think about control.

Jane: What does this mean for someone who only listens to our show? It means that for pretrained models like FLUX.two-klein, you can gain control over output attributes by just feeding it a different set of reference data at the moment you want to generate something new <ref:2605.10302#pg1>.

Meng: From an engineering standpoint, this is less about modifying the massive model and more about manipulating external data inputs at inference time. That’s actually easier to manage in production environments where you don't have access to retraining infrastructure constantly.

Lu: And that’s the ambition of it. They show that this mechanism works whether you're doing it with a simple closed-form correction for RMG or a more complex learned residual refiner for SPG, both of which aim to match high quality baselines like DiT-B/four on AFHQv2 <ref:2605.10302#pg1>.

Tom: So we have these two distinct paths: training-free control with RMG and higher quality adaptation with SPG. It shows the direction toward generative models that adapt through data rather than constantly requiring parameter updates.

Jane: That’s what they found. They proved that adaptation through examples is a viable interface for flow matching, opening up a new way to interact with these powerful generation tools we rely on every day.

Conclusion: Tom: So, to wrap up, this paper shows that controlling flow matching isn't about tweaking the model weights directly anymore; it’s about changing what reference set you feed it during generation time.

Jane: Right, so instead of retraining for every new style or identity you want, you just pick a different bank of examples and tell the model to follow those instead. That sounds way more practical for real-time use.

Lu: Yeah, that’s the core idea here: we found that if you shift the endpoint mean—which is essentially where the model settles after some noise—you directly shift the entire generation flow. You don't need auxiliary networks or complex loss functions to get this effect for deterministic interpolants.

Meng: But what about quality? If you just swap reference sets, does it degrade the output? We’ve seen that RMG works on a frozen model and we still see good structural transfer in things like keyhole shapes.

Tom: That's the thing about Reference-Mean Guidance, or RMG—it's training-free. They figured out a closed-form way to calculate the correction needed for that mean, so you can apply it without any extra gradient computation.

Jane: So it’s training-free control for a frozen model? That’s huge because it means you don't need massive GPU time just to get a new look or style. But then there’s Semi-Parametric Guidance, SPG, which builds that idea into the model itself using an explicit anchor and a learned refiner.

Lu: SPG is what makes it really interesting for quality because it matches the output of a much bigger model like DiT-B/four on AFHQv2. It lets you swap those reference sets at inference time, which means you get both control and high fidelity.

Tom: So we have this simple, fast method for quick steering with RMG, and then SPG which is more complex but matches the quality of a much bigger unconditional model. It really frames the problem as controlling that endpoint mean.

Jane: That’s exactly it: framing control as shifting that mean. It moves us away from constantly updating parameters and toward adapting through data inputs, which is a really different way to think about generative models at scale.

More episodes

← Home