Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing
cs.SD, cs.AI
Submitted: 2026-09-24
Updated: 2026-09-24
Comments: Submitted to ICASSP 2027. Project page https://sse-ai.notion.site
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement.
Terminology
Abstract
We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources, and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments. Project page: https://sse-ai.notion.site
Sources
- Conditional Flow Matching for Visually-Guided Acoustic Highlighting
- Movie Gen: A Cast of Media Foundation Models
- Perception Encoder: The best visual embeddings are not at the output of the network
- SAM Audio: Segment Anything in Audio
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Qwen2-Audio Technical Report
- Audiobox: Unified Audio Generation with Natural Language Prompts
- Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
- WASIL: In-the-Wild Arabic Spoken Interactions with LLMs