WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations".

Jane: Here's the gist of the paper. The gist: This work introduces WoWBench, a new benchmark specifically designed to rigorously evaluate the fine-grained, low-level listening, and cognitive capabilities of acoustics-language modeling using marine mammal vocalizations.

Tom: Large audio language models (LALMs) extend language understanding into the auditory domain, yet their ability to perform low-level listening, such as pitch and duration detection, remains underexplored Current evaluations of LALMs’ perceptual capabilities predominantly rely on audio classification tasks (Gemmeke et al., 2017; Chen et al., 2020; Piczak, 2015; Yang et al., 2024), which map acoustic signals to predefined semantic labels such as “dog barking” or “siren” These benchmarks offer limited insight into whether models are attending to fine-grained acoustic features, such as pitch and duration Low-level listening is defined as the perceptual ability to detect and differentiate elementary acoustic attributes prior to semantic interpretation or categorization This gap is critical, as perceptual errors often account for a substantial portion of failures in reasoning tasks World-of-Whale Benchmark Structure WoWBench is composed of two main components designed to assess auditory perception and cognition First, it includes a Perception benchmark for categorizing novel sounds and a Cognition benchmark inspired by Bloom’s taxonomy to assess abilities like remembering, understanding, applying, and analyzing sound events The Perception benchmark tests the model’s ability to categorize unfamiliar sound events based on low-level listening and internal knowledge It includes tasks such as the Species task, which requires models to classify each audio clip of a vocalization into the correct species category, analogous to typical audio event classification The Vocalization task involves selecting the most appropriate natural language description for a given vocalization, requiring models to capture low-level acoustic details and map them to unfamiliar labels The Both task requires models to select the option that correctly describes both the species and the vocalization type for a given audio clip, assessing compositional auditory perception Cognition Benchmark Details The Cognition benchmark is structured into four subtask types based on Bloom’s taxonomy to evaluate different aspects of how LALMs process low-level acoustic details and perceived audio events These levels are Remember, Understand, Apply, and Analyze The Remember level involves recognizing and recalling previously encountered information by identifying which of the subsequent sounds is identical to the reference sound The Understand level evaluates whether a model comprehends underlying acoustic properties by selecting the description that most accurately represents low-level acoustic features, including perceptual pitch labels and frequency ranges The Apply level tests the capacity to utilize acquired knowledge in new situations by comparing audio clips based on a core acoustic property, either pitch or duration Finally, the Analyze level involves breaking information into parts and examining their relationships by investigating transitions within complex acoustic sequences, such as identifying shifts between sounds Distractor Design and Evaluation To evaluate whether models are truly solving problems through listening rather than relying on other heuristics, distractor questions are introduced to the Cognition benchmark These distractors incorporate audio that is qualitatively distinct from regular questions for the same type, aiming to directly test whether the model is truly solving the task through listening For instance, in a comparison task like Apply–Frequency, three identical sounds are presented as input as a distractor audio The design process involves systematically designing distractors by inverting the expected pattern for each question type to encourage reliance on listening A detailed analysis of incorrect answers reveals that Semantic Categorization (S) is often the most frequent strategy employed by models, reflecting a "category-first behavior" rather than genuine perceptual grounding Experimental Results and Findings Extensive experiments with existing LALMs reveal that both their perceptual generalization and cognitive processing based on low-level listening are limited Overall, all models perform poorly on the WoWBench, with even the best performing commercial model remaining below 50% accuracy A key finding is that models often perform worse on Cognition than Perception tasks, despite the former requiring no prior knowledge about marine mammals Furthermore, performance varies substantially across tasks, reflecting strengths and weaknesses in each model’s low-level listening capabilities The performance gap between distractor and non-distractor questions highlights a limitation in auditory grounding Qualitative analysis shows that models often adopt a classify-first strategy, which can lead to failure when perceptual grounding is required Acoustic Range and Model Performance The benchmark spans a broad acoustic range from low-frequency sounds of 20 Hz to signals exceeding 20 kHz, covering the entire human auditory spectrum Performance fluctuates with both duration and sample rate, but no sharp increase or drop is observed at any specific threshold These findings suggest that current models do not exhibit significantly improved low-level listening performance even within their nominal frequency range WoWBench provides a challenging testbed for advancing the next generation of multimodal models The results highlight a critical gap in current LALM architectures: despite impressive progress in general audio understanding and instruction following, robust low-level perception remains elusive Our findings emphasize the necessity for future research to focus on improved auditory grounding and enhanced sensitivity to acoustic detail in order to close the gap between machine and human auditory cognition The benchmark is currently restricted to marine mammal vocalizations, but this focus enables a more precise evaluation of models’ low-level listening capabilities in genuinely out-of-domain settings Limitations One limitation is that the benchmark is currently restricted to marine mammal vocalizations, which represents only a narrow segment of the complex diversity found in natural acoustic environments A further limitation concerns the task format; while multiple-choice questions enable standardized comparisons, they may not fully capture the open-ended reasoning or generative abilities of modern audio-language models Ethical Considerations The creation and use of WoWBench raise several ethical considerations, particularly regarding the potential deployment of audio-language models in ecological monitoring or conservation contexts Understanding marine mammal vocalizations is crucial for ecological monitoring and marine conservation, but machine listening can help bridge the gap in this under-resourced field Acknowledgments This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) We gratefully acknowledge the New Bedford Whaling Museum for granting permission to use their database for research purposes Gunhee Kim is the corresponding author References Lorin W Anderson and David R Krathwohl.

Tom: A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives: complete edition.

Title and authors: Tom: So we're looking at this paper called WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations. It’s pretty focused on the idea that large audio language models are good at understanding language, but they still struggle with the very low-level stuff like just hearing the pitch or how long a sound lasts.

Jane: Exactly, Tom. It’s about how these models handle listening to sounds that aren't in their usual training data, which is super important because real-world situations often throw us at unfamiliar audio. The authors introduce this benchmark specifically to test that fine-grained acoustic perception and cognition using marine mammal vocalizations.

Lu: What's interesting here is that they define low-level listening as detecting and differentiating elementary acoustic attributes before you even try to figure out what the sound means or categorize it, which is a really critical distinction. This gap is where perceptual errors often cause failures in reasoning tasks.

Meng: So how do they test this? I mean, most benchmarks just ask the model to classify a sound, like telling it if it’s a dog bark or a siren. This benchmark seems to go deeper than that by separating perception into two main parts: Perception and Cognition.

Tom: Right. The Perception benchmark is for categorizing novel sounds, while the Cognition benchmark is inspired by Bloom’s taxonomy, using levels like remembering, understanding, applying, and analyzing sound events. It’s testing if the model can actually remember or analyze those acoustic details without just relying on a category label.

Jane: That makes sense. For instance, in the Perception part of WoW-Bench, they have tasks like the Species task where models have to classify a vocalization into a specific species category, and another one called the Vocalization task where they pick the best natural language description for an unfamiliar sound.

Lu: And then there's this Both task which requires selecting both the species and the vocalization type, which really tests compositional auditory perception. It’s not just about one feature; it’s about piecing together multiple low-level acoustic details to get the right answer.

Meng: Speaking of those details, they have a whole section on Cognition that breaks down into four subtasks based on Bloom’s taxonomy—Remember, Understand, Apply, and Analyze. This gives them a structured way to see exactly where the model is succeeding or failing in processing those acoustic properties.

Title and authors: Tom: And they introduce some really interesting elements for testing how models think. They put distractor questions into the Cognition benchmark to see if the model is actually listening or just guessing based on its general knowledge of sound categories.

Jane: That’s where it gets really sharp. The distractor design is specific; they use audio that sounds qualitatively different from the actual question, trying to force a test on whether the model is truly solving a problem through listening or if it’s using other shortcuts.

Lu: I noticed in their analysis of those distractors, they found that semantic categorization, or S, was often the most frequent strategy models used, suggesting a tendency toward category-first behavior instead of genuine perceptual grounding. That’s a key finding for understanding where the current models are falling short.

Meng: From an engineering standpoint, if models are defaulting to semantic classification when they should be listening perceptually, that means their internal representations aren't sensitive enough to those fine acoustic features like pitch or duration. That could mean we need better ways to embed those features into the model architecture.

Tom: The overall results show that both the perceptual generalization and the cognitive processing based on low-level listening are limited across all existing large audio language models. Even their best commercial model, according to their experiments, stayed below fifty percent accuracy on WoW-Bench <ref:2508.20976#pg2>.

Jane: And what's interesting is that they found models often perform worse on the Cognition tasks than they do on the Perception tasks, even though perception doesn't require any prior knowledge about marine mammals. That points to a weakness in how these models handle complex acoustic reasoning.

Lu: Performance varies quite a bit across different cognitive tasks, which just shows that there are real strengths and weaknesses in each model’s low-level listening capabilities, rather than one single failure mode across the board.

Meng: They also looked at the acoustic range they covered—from low frequencies of twenty Hertz up to signals over twenty kHz—and found no sharp increase or drop in performance even within that whole range <ref:2508.20976#pg2>. That suggests current models aren't getting significantly better at low-level listening just by expanding the frequency spectrum.

Title and authors: Tom: So, to wrap up what we've covered about WoW-Bench: it’s a tough testbed that clearly shows a gap between general audio understanding and robust low-level perception in these models. The implication is that future work really needs to focus on improving auditory grounding and sensitivity to acoustic detail so they can close that gap.

Jane: It highlights the necessity for research to focus more on how models develop a true sense of acoustic detail, moving beyond just semantic labels. This benchmark provides a precise way to measure that exact capability using marine mammal sounds, which is useful because it’s an out-of-domain setting.

Lu: For future work, I think we need to move toward implementing low-level auditory grounding modules directly into the LALMs themselves so they can get better at detecting those pitch and duration features from the start. That would be a fundamental shift in how these models process sound.

Meng: If we look at deployment, this benchmark shows that relying on existing architectures alone isn't enough for complex, real-world tasks where sounds are novel. We need mechanisms that force the model to listen first before it tries to categorize based on what it already knows about a general category.

Tom: So the big picture here is that while LALMs are advancing in general audio understanding, they haven't mastered the fine-grained perception needed for real reasoning with unfamiliar sounds. WoW-Bench gives us a concrete way to measure that specific deficit.

Jane: It’s a solid foundation for making sure future models aren't just memorizing labels but actually learning to perceive the acoustic reality of those vocalizations. That’s what this work is all about when we talk about WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations.

Lu: And for us, it confirms that even with massive scale, the architecture still needs a dedicated mechanism to handle those subtle acoustic cues properly.

Meng: It’s a lot of data and complexity just to measure this one aspect of perception, but it sets a clear bar for what true auditory grounding means in these systems.

Tom: We’ll keep an eye on how researchers start incorporating these kinds of benchmarks into the training loops to see if we can see any real lift in those low-level abilities.

Jane: And that’s where we’ll be next time, when we look at what the next set of audio language models are doing to tackle these perception challenges.

The paper's summary: Tom: So, basically, this paper introduces WoWBench, which is a new benchmark designed to really test how well large audio language models can perform low-level listening and cognitive tasks using recordings of marine mammal sounds.

Jane: That’s right, Tom. It’s not just about telling the model what sound it is; it’s about whether the model can actually hear the fine details like pitch or duration before it tries to categorize anything.

Lu: The core idea here is that current evaluations mostly rely on simple audio classification, and that doesn't tell us much about whether models are actually paying attention to those subtle acoustic features.

Meng: What’s interesting is how they structure this benchmark into two main parts: a perception test for new sounds, and a cognition test based on Bloom’s taxonomy.

Tom: Exactly. The cognition part specifically breaks down memory, understanding, applying, and analyzing sound events to see if the model can process those acoustic properties correctly.

Jane: And they introduce these specific tasks like the Species task or the Vocalization task where models have to pick descriptions based on what they hear, rather than just what they've been trained on.

Lu: The authors design a section specifically for testing how models handle distractor questions, which is meant to see if the model is genuinely listening or just using some other shortcut.

Tom: They found that models often fall into a "classify-first" strategy, which means they’re trying to categorize the sound before they’ve actually processed the acoustic details themselves.

Jane: That suggests a big issue: these models aren't always doing the necessary perceptual grounding required for real reasoning.

Lu: The experimental results show that even the best commercial models struggle, performing below fifty percent accuracy on WoWBench across both perception and cognition tasks.

Meng: And they found that performance is inconsistent; it varies a lot depending on which specific acoustic feature you’re testing, like pitch or duration.

Tom: They also looked at the audio range from twenty hertz up to twenty kilohertz, and there wasn't any big jump in performance even when they expanded the frequency spectrum.

Jane: So what this means for us is that we need better ways to train these models so they develop a more robust sense of acoustic detail, not just general audio awareness.

Lu: The paper points toward needing low-level auditory grounding modules built directly into the model architecture to improve out-of-distribution performance.

Tom: And that’s where I think we should look next: how do we actually build those modules so they don't break when you deploy them at scale?

Jane: Exactly. The challenge isn't just measuring the problem; it’s figuring out how to make these models actually learn to hear like humans do, before they try to reason about what those sounds mean.

The paper's improvements: Tom: So, this paper isn't just pointing out problems; they’re proposing concrete ways to fix them in these audio models.

Jane: Right, Tom. They suggest three main things: first, you need to implement low-level auditory grounding modules directly into the AI architecture.

Lu: That means models need mechanisms that focus on detecting those fine acoustic features like pitch and duration right from the start so they don't just rely on semantic labels later.

Meng: From an engineering standpoint, that would mean adding new layers or processing steps specifically tuned to capture those physical properties of sound. It sounds complex to deploy at scale.

Tom: I get the complexity, but if we can make that grounding module work well on these specific tasks, it could drastically improve performance on out-of-distribution sounds.

Jane: They also suggest developing a cognitive processing layer inspired by Bloom’s taxonomy to explicitly evaluate memory and analysis of those sound events.

Lu: That helps test whether the model is genuinely solving the problem through listening or just relying on some other learned heuristic, which is what they did with those tricky distractor questions.

Meng: Testing that at scale would be tough, though. We’d need a way to run those cognitive assessments without massive computational overhead on every audio input.

Tom: And they also talk about making model reasoning chains smarter by penalizing that "classify-first" strategy and rewarding responses that actually include specific acoustic feature terms when answering.

Jane: That's a big deal because it directly targets the failure we saw: models infer acoustic properties based on categories instead of truly listening to them.

Lu: If they can get the model to output those feature terms, it shows that the internal representation is actually sensitive enough to those details.

Meng: I'd be interested in seeing how much computational cost that specific feature refinement adds compared to just using a standard classification layer.

Tom: And they also mention expanding model capacity by training on more diverse, rare vocalizations from databases like the Watkins Marine Mammal Sound Database.

Jane: That’s important because it addresses the finding that current models don't get better performance even when you increase the frequency range to twenty kilohertz.

Lu: Expanding that training set is a solid approach for generalization, but we have to make sure those rare sounds are actually diverse enough to teach the model something new.

Meng: If we feed it too much noise or irrelevant data, the performance might just stay flat or even degrade because of the noise floor.

Tom: So, they’re looking at these three improvements: dedicated grounding modules, structured cognitive evaluation with distractors, and better reasoning chains that reward feature detection.

Jane: It sounds like the next step is moving from general audio understanding to a system that can actually perform deep acoustic perception on unfamiliar sounds.

Lu: The paper suggests this benchmark is a good way to pinpoint exactly where these models fall short, which helps guide the architectural changes needed for future iterations.

Meng: For those of us shipping models, it’s helpful because it gives us a metric to track when our new layers actually improve that fine-grained listening ability under stress.

Tom: So, the move forward is clearly toward building better auditory grounding and making sure the AI learns to hear the acoustic reality of sounds instead of just guessing based on what it already knows.

Conclusion: Tom: So we've covered how WoWBench tests if large audio language models can actually hear the fine acoustic details of marine mammal sounds, and they found that most of them are struggling with low-level perception compared to their general language skills.

Jane: That’s right, Tom. The main implication is that we really need to focus on improving auditory grounding before these models can do complex reasoning in real-world acoustic environments.

Lu: The authors suggest building in those low-level modules and using cognitive evaluations with distractors as a way to force the model to listen instead of just guessing based on categories.

Meng: For deployment, it means we need better ways to run these models that can handle that extra processing without making the system too slow or expensive at scale.

Tom: I think what this means for the field is that we can’t just focus on bigger model sizes anymore; we have to focus on architectural improvements in how they process sound at a basic level.

Jane: It also gives us a very concrete testbed, even though it’s limited to marine mammals, which makes it easier for researchers to see exactly what's missing.

Lu: The limitation they point out is that the benchmark is currently restricted to marine mammal vocalizations, which means we still need to figure out how these concepts apply to much more diverse and complex natural acoustic environments.

Meng: And another thing they flag is that while the models can process a wide frequency range, performance doesn't show a sharp increase even when you expand that spectrum.

Tom: So the big picture is that achieving true perceptual ability in AI isn't just about scaling up parameters; it’s about fundamentally changing how we teach them to listen to subtle acoustic cues.

Jane: Exactly, Tom. The paper on WoWBench shows us where the gap is between what we can tell an AI and what an actual human can perceive acoustically.

Lu: For future work, I think the next step needs to be figuring out how to integrate those grounding modules efficiently so they don't just add more complexity without adding real perceptual power.

Meng: If we can figure out a way to make that process efficient enough for production, then this benchmark becomes a really useful tool for testing our industrial anomaly detection models later on.

Tom: Agreed. We need to see those modules in action on real traffic, and WoWBench gives us the initial metrics to know if we're even on the right track before we spend time building them out fully.

Jane: So that’s the summary for WoWBench: a tough test showing that without better low-level listening, our audio language models are missing a crucial piece of real-world perception.