TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

arXiv:2412.01488 · eess.AS, cs.LG, eess.IV · Submitted 2024-12-02 · Read on arXiv

eess.AS, cs.LG, eess.IV

Submitted: 2024-12-02

Updated: 2026-09-28

Terminology

• Generalization — 2× in this paper · explained in 13 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes

Sources

• OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models
• Neural Network Matrix Factorization
• Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation
• SAM 2: Segment Anything in Images and Videos

Related papers

• Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation
• Conditional Flow Matching for Visually-Guided Acoustic Highlighting
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization