Conditional Flow Matching for Visually-Guided Acoustic Highlighting
The authors reframe visually-guided acoustic highlighting as a generative problem and introduce Conditional Flow Matching (CFM) to create coherent audio-visual experiences. They learn continuous transformations between poorly balanced and well-balanced audio distributions by modeling a time-dependent velocity field, using an improved conditioning module and a rollout loss for stability. This generative approach outperforms previous methods.