Follow the Mean: Reference-Guided Flow Matching

Existing approaches to controllable generation typically rely on fine-tuning, auxiliary networks, or test-time search. We show that flow matching admits a different control interface: adaptation through examples. For deterministic interpolants, the velocity field is solely governed by a conditional endpoint mean, so shifting this mean shifts the flow itself, and a pretrained model can be steered simply by changing the reference set it follows. We instantiate this idea in two forms. Reference-Mean Guidance is training-free. It computes a closed-form correction to the endpoint mean from a reference bank and applies it to a frozen FLUX.2-klein (4B) model, controlling color, identity, style, and structure while the prompt, seed, and weights stay fixed. Semi-Parametric Guidance builds the same principle into a trained model through an explicit mean anchor and a learned residual refiner, matching the quality of an unconditional DiT-B/4 on AFHQv2 while allowing the reference set to be swapped at inference time. Together, these results point to a broader direction: generative models that adapt through data rather than parameter updates.