Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
The episode discusses a paper introducing ST-OmniQA, a benchmark of 40,000 panoramic videos with spatial audio and 400,000 QA pairs, and ST-Omni-R1, a model that tracks and binds sound sources to visible objects. The hosts explain the perception, binding, and reasoning gaps, and highlight the model's 77.83% accuracy versus 37.28% for baselines.