VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing".
Jane: Here's the gist of the paper. Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input <ref:2609.04241#pg2>.
Tom: VocalCoachBench introduces a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing <ref:2609.04241#pg2>.
Jane: It contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims <ref:2609.04241#pg3>.
Tom: Benchmark Composition VocalCoachBench comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions <ref:2609.04241#pg2>.
Paper summary: Tom: So, wrapping up VocalCoachBench: it’s this new benchmark focused specifically on evaluating audio-language models on expert vocal coaching feedback for singing >
Jane: It’s a framework built around separating deterministic structured targets from the open-ended assessment of free-form diagnosis and corrective guidance >
Lu: The authors used eighteen professional vocal trainers to create this data, yielding one thousand fifty-six expert submissions and twelve thousand fifty one atomic coaching claims across two distinct subsets > <ref:2609.04241#pg1>
Meng: The big implication is that while models can identify general problem areas, they consistently miss the specific fine-grained issues experts prioritize and their proposed corrections don't always align with those expert claims by more than seventy percent >
Lalam: This shows that simply processing audio isn't enough; the AI needs a deeper grasp of vocal coaching itself to produce reliable, actionable advice for singers >
Tom: The authors are clear that this is a first step toward benchmarking audiogrounded expert feedback, and they caution that model-generated vocal guidance should not replace professional instruction >
Jane: So, while it’s a valuable tool for testing current models against expert claims on singing, we still need to work on making these evaluations more comprehensive and applicable across different scenarios >
Conclusion: Tom: So, VocalCoachBench is this new way to test how good audio models are at actually giving expert singing advice >
Jane: It's basically taking all that expert feedback—the really detailed critiques singers get from vocal coaches—and turning it into a structured test for these language models >
Lu: The core idea is setting up two different kinds of tests, one where the answers are pretty specific, and another where the model has to guess what kind of correction an expert would suggest >
Meng: And they used eighteen professional vocal trainers to create this data set, which gives you a lot of human expertise to judge the models against >
Lalam: So what we're seeing is that while these models can grasp general singing problems, they struggle when it comes to pinpointing the exact technical issues or giving valid advice >
Tom: Right. The authors are showing that even with recent audio-language models, there's still a gap between just hearing a song and actually understanding how to coach someone on it >
Jane: It really highlights that being good at processing sound isn't the same as being able to provide real pedagogical feedback >
Lu: And they found that for the smaller, controlled tests, you can see some improvement in ranking songs based on expert opinions >
Meng: But they also pointed out that when you try to predict a specific technical problem, like a vocal placement issue, those models fall behind the established baselines by quite a bit >
Lalam: So the authors are basically saying we need more than just audio processing; we need models that understand the *why* behind expert coaching >
Tom: Exactly. They’re not claiming these models are ready to replace a human coach yet, but they're showing us exactly where they fall short right now >
Jane: It sets up a real challenge for the next generation of audio language models trying to move beyond just summarizing music into actually providing guidance >
Hayeon Bang, Hounsu Kim, Wonil Kim, Juhan Nam
KAIST
cs.SD, eess.AS
Submitted: 2026-08-06
Updated: 2026-11-08
Comments: NeurIPS 2026, Evaluations & Datasets Track
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 91/100
The gist: Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability…
Key concepts
- VocalCoachBench
- This is a new benchmark designed to evaluate audio-language models on their ability to generate expert vocal coaching feedback for singing. It uses recordings annotated by professional trainers to test if AI can provide useful, actionable guidance.
- Atomic Coaching Claims
- These are the specific, fine-grained diagnostic points made by human vocal coaches. Instead of just giving general advice, these claims pinpoint exact problems in technical production, musical accuracy, or delivery that need correction.
- Structured vs. Open-Ended Suite
- The benchmark has two testing methods: a structured suite with fixed tasks like ranking and classification, and an open-ended suite where models must assess the coverage and validity of expert claims against specific reference points.
Terminology
Summary
Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input <ref:2609. p. 0 nam}@kaist. kr
Abstract
Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different
capability: producing feedback that identifies problems and suggests corrective
actions grounded in the input. kr
Abstract
Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different
capability: producing feedback that identifies problems and suggests corrective
actions grounded in the input. We introduce VocalCoachBench, a benchmark for
evaluating audio-language models on expert vocal coaching feedback for singing. We introduce VocalCoachBench, a benchmark for
evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal
trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims.Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Composition
VocalCoachBench comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions <ref:2609.04241#pg2>. The same-song subset includes controlled triplet rankings with ranking rationales, while the diverse-song subset includes segment-grounded feedback with time spans <ref:2609. p. 1 In total, 18 expert vocal trainers provide ranked Top-3 issue
labels and natural-language coaching reviews covering the main problems, corrective directions,
and strengths. The same-song subset additionally includes controlled triplet rankings with ranking
rationales, and the diverse-song subset includes segment-grounded feedback with time spans. It
comprises a same-song subset for controlled comparison and a diverse-song subset
for segment-grounded feedback across varied songs and recording conditions. To
accommodate the open-ended nature of expert feedback, VocalCoachBench separates deterministic structured targets from claim-based assessment of free-form
diagnosis and corrective guidance.Where the paper says it
Where the paper says it
Annotation Protocol
Expert vocal trainers annotate two complementary subsets: a same-song subset for controlled comparison, and a diverse-song subset for diverse coaching scenarios and segment-grounded feedback <ref:2609.04241#pg2>. Annotation was carried out by 18 professional vocal trainers with formal vocal-performance training and at least three years of coaching experience <ref:2609. p. 4 Together, the two subsets provide complementary evaluation
regimes: same-song recordings support controlled comparison, while diverse-song recordings support
diverse and segment-grounded coaching analysis. 3 Expert annotation protocol
Annotation was carried out by 18 professional vocal trainers with formal vocal-performance training
and at least three years of coaching experience. To support later analysis and evaluation, annotators wrote their feedback in semantically coherent
feedback blocks and tagged each block with one or more
vocal issue categories. The final taxonomy consists of seven fine-grained issue
categories under three parent categories: Technical Production covers breath support, vocalization, and technique;
Musical Accuracy covers pitch and rhythm; and Delivery
covers diction and expression.Where the paper says it
Where the paper says it
Evaluation Suites
VocalCoachBench defines two complementary evaluation suites: a structured evaluation suite consisting of fully reproducible deterministic tasks and an open-ended coaching suite assessing model-generated feedback against expert atomic claims <ref:2609.04241#pg4>. The structured evaluation includes triplet ranking, Top-3 issue prediction, and segment-level issue classification <ref:2609. p. 7 All values are percentages except Kendall’s τ . Protocol Model Triplet ranking Top-3 issue prediction Segment
Pair τ Exact Cycle ↓ F1@3 Top-1 nDCG@3 Parent F1 Match
REF
Random baseline 50. Top-3 issue-label prediction, and segment-level issue-label classification. The open-ended coaching evaluation suite evaluates model-generated feedback against expert atomic claims, separating
diagnosis coverage from correction validity.Where the paper says it
Where the paper says it
Model Performance Findings
Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7% <ref:2609.04241#pg2>. Models often identify broad issue domains but miss expert-prioritized fine-grained problems in diagnosis, while correction validity is higher for more capable models <ref:2609. p. 8 Strict diagnosis hit rates stay below 7% across all models, while
Hit rates vary more widely (17–63%), indicating that models often identify broad issue domains
but miss expert-prioritized fine-grained problems. Correction validity is higher for more capable
models, but high Valid rates should be interpreted alongside diagnosis coverage: models can generate
plausible corrections even when their diagnoses remain loose or generic.Where the paper says it
Limitations and Future Work
VocalCoachBench is a first step toward benchmarking audiogrounded expert feedback, not a complete model of vocal pedagogy <ref:2609.04241#pg10>. Limitations include focusing on English solo singing, the same-song subset being limited to a single song, and the open-ended evaluation measuring reference-supported claim coverage rather than full pedagogical value <ref:2609. p. 8 The benchmark is audio-only and oneshot, whereas real coaching may use visual or bodily cues and adapt feedback over repeated attempts;
segment-grounded evaluation also assumes expert-provided segments rather than testing end-to-end
discovery. Our open-ended evaluation measures reference-supported claim coverage rather than the
full pedagogical value of a response, so it may under-credit valid feedback outside the available
expert claims. Because incorrect vocal guidance may encourage strain or unsafe self-training, model
outputs should not be treated as a substitute for professional instruction. Future work should extend
VocalCoachBench toward multilingual, multimodal, interactive, and longitudinal coaching evaluation.Where the paper says it
Where the paper says it
Data and Code Availability
The paper provides open access to the benchmark annotations, evaluation scripts, prompt templates, and documentation sufficient to reproduce the reported benchmark metrics <ref:2609.04241#pg32>. The authors release anonymized versions of the benchmark annotations, evaluation code, prompts, and documentation governed by the licenses and access terms of the underlying audio sources <ref:2609.04241#pg32>.
LLM Usage
LLMs are used as part of the methodology for claim decomposition, parsing/normalization, translation support, and judge-based evaluation. These uses are described in the dataset post-processing and evaluation sections with prompts and validation details in the appendix. p. 6 The main metric is the correction validity rate,
the fraction of correction claims labeled valid; we also report weak and invalid rates. Judge prompts,
label definitions, and human validation are described below. The main metric is the correction validity rate,
the fraction of correction claims labeled valid; we also report weak and invalid rates. Judge prompts,
label definitions, and human validation are described below.Where the paper says it
Where the paper says it
Human Subjects Ethics
The work involved only paid annotation rather than a behavioral intervention, and annotators were informed of the task content and compensation before participation <ref:2609.04241#pg32>. Formal IRB review was not required under the applicable institutional policy <ref:2609.04241#pg32>.
Broader Impacts
The introduction and conclusion discuss the positive goal of improving audiolanguage evaluation for expert feedback, while the limitations discuss risks from incorrect vocal guidance and the need not to treat model outputs as professional instruction <ref:2609.04241#pg31>. The authors warn that model-generated vocal guidance should not replace professional instruction <ref:2609. p. 8 Our open-ended evaluation measures reference-supported claim coverage rather than the
full pedagogical value of a response, so it may under-credit valid feedback outside the available
expert claims. Because incorrect vocal guidance may encourage strain or unsafe self-training, model
outputs should not be treated as a substitute for professional instruction.Where the paper says it
License Compliance
Existing audio datasets and models are cited in the paper, and source-license information for the audio assets is documented in the appendix <ref:2609.04241#pg32>. The name of the license should be included for each asset <ref:2609. p. 0 Evaluating vocal coaching ability,
therefore, is not a matter of audio description or singing quality assessment alone, but a problem of
identifying the main issues and generating corrective feedback grounded in the performance. arXiv:2609.Where the paper says it
New Assets
VocalCoachBench introduces new expert annotations, prompts, evaluation code, and documentation; the paper and appendix describe the annotation schema, taxonomy, examples, validation, and intended benchmark use <ref:2609.04241#pg32>. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates <ref:2609. p. 5 observed no omitted or hallucinated claims and only minor extraction errors; details are provided in
Appendix E. 5 Dataset statistics and annotation behavior
VocalCoachBench contains 515 recordings, 1,056 expert audio-level submissions, 4,443 feedback
blocks, and 12,051 atomic claims (Table 2).Where the paper says it
Experimental Reproducibility
Sections 4.1, 4.2, and 5.2 define the tasks, metrics, prompting protocols, inference setup, parsing, and judging procedures; additional prompts and validation details are provided in the appendix <ref:2609. p. 7 5-Omni
Flash and Plus variants [29]—are evaluated with the main protocol via public APIs. 2 Prompting protocols and inference setup
The main claim/objective protocol uses task-specific structured prompts for coaching claims, Top-3
issues, and segment classification.Where the paper says it
Resources
Appendix I reports that no models are trained, open-weight inference uses a single NVIDIA H100 80GB GPU, closed-source models use public APIs, and long-running inference and judge jobs are executed with resumable batch scripts because runtime varies by model and API latency. The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider. p. 3 Beyond music, recent embodied-skill
benchmarks evaluate expert-level performance understanding through proficiency cues, improvementrelated commentary, action differences, or expert-derived QA [9, 3, 44]. VocalCoachBench complements these efforts by moving from collecting or matching expert feedback to evaluating the internal
quality of generated feedback: whether its diagnosis is expert-aligned, and paired with correction
appropriate to the diagnosed issue.Where the paper says it
Statistical Significance
We report deterministic benchmark metrics and human/judge agreement statistics on fixed evaluation sets, but do not report bootstrap confidence intervals or statistical significance tests for model comparisons <ref:2609.04241#pg31>. The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests <ref:2609.04241#pg31>.
Safeguards
The paper does not release a generative model or other high-risk dual-use system <ref:2609.04241#pg32>. The benchmark release is limited to annotations, evaluation code, and documentation governed by the licenses and access terms of the underlying audio sources <ref:2609. p. 8 It focuses on English solo
singing, and its same-song subset is limited to a single song. The benchmark is audio-only and oneshot, whereas real coaching may use visual or bodily cues and adapt feedback over repeated attempts;
segment-grounded evaluation also assumes expert-provided segments rather than testing end-to-end
discovery.Where the paper says it
Licenses for Existing Assets
Existing audio datasets and models are cited in the paper, and source-license information for the audio assets is documented in the appendix <ref:2609.04241#pg32>. The license should be included for each asset <ref:2609. p. 0 Evaluating vocal coaching ability,
therefore, is not a matter of audio description or singing quality assessment alone, but a problem of
identifying the main issues and generating corrective feedback grounded in the performance. arXiv:2609. Evaluating vocal coaching ability,
therefore, is not a matter of audio description or singing quality assessment alone, but a problem of
identifying the main issues and generating corrective feedback grounded in the performance. arXiv:2609.Where the paper says it
Where the paper says it
Crowdsourcing and Human Subjects
The work does not use open crowdsourcing; annotations were collected from paid expert vocal coaches who consented to the annotation work <ref:2609.04241#pg32>. The paper describes the expert annotation protocol, quality-control process, annotation interface examples, and annotation labor details in the main text and Appendix <ref:2609. p. 5 observed no omitted or hallucinated claims and only minor extraction errors; details are provided in
Appendix E. 5 Dataset statistics and annotation behavior
VocalCoachBench contains 515 recordings, 1,056 expert audio-level submissions, 4,443 feedback
blocks, and 12,051 atomic claims (Table 2).Where the paper says it
IRB Approvals
Under our applicable institutional policy, formal IRB review was not required <ref:2609.04241#pg32>. Annotators were informed of the task content and compensation before participation, and no private personal information was collected for analysis <ref:2609.04241#pg32>.
LLM Usage Declaration
LLMs are used as part of the methodology for claim decomposition, parsing/normalization, translation support, and judge-based evaluation. These uses are described in the dataset post-processing and evaluation sections with prompts and validation details in the appendix.
Data Access
The paper is accompanied by benchmark annotations, evaluation scripts, prompt templates, and documentation sufficient to reproduce the reported benchmark metrics <ref:2609.04241#pg32>. The authors provide instructions on data access and preparation <ref:2609.04241#pg32>. The authors release anonymized versions of the benchmark annotations, evaluation code, prompts, and documentation governed by the licenses and access terms of the underlying audio sources <ref:2609.04241#pg32>.
Experimental Setting/Details
The dataset construction, splits, task definitions, model routing, prompting protocols, deterministic decoding policy, parsing, and judge validation are described in the paper and appendix <ref:2609.04241#pg32>. The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results <ref:2609.04241#pg32>.
Code of Ethics
We reviewed the NeurIPS Code of Ethics and believe the work conforms to it <ref:2609.04241#pg32>. The paper uses cited audio sources, expert annotation, documented licensing, and warns that model-generated vocal guidance should not replace professional instruction <ref:2609. p. 8 Our open-ended evaluation measures reference-supported claim coverage rather than the
full pedagogical value of a response, so it may under-credit valid feedback outside the available
expert claims. Because incorrect vocal guidance may encourage strain or unsafe self-training, model
outputs should not be treated as a substitute for professional instruction.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Structured evaluation suite for deterministic tasks can be used to evaluate model preferences over three same-song performances using direct pairwise comparison, which is preferred over scalar elicitation due to its robustness against score-tie collapse.
-
Open-ended coaching evaluation can assess free-form feedback by evaluating
diagnosis coverage
against atomic diagnosis claims andcorrection validity
against atomic correction claims, treating expert disagreement as a property of coaching feedback rather than noise. -
The system can be benchmarked on Top-3 issue prediction using fine-level F1@3, which measures the Top-3 set overlap divided by three, and also report parent-level F1 to capture partially aligned expert judgment.
-
Segment-level issue classification enables models to perform local issue recognition on expert consensus segments, where the main metric is any-match accuracy against a set of shared issue tags.
-
Models can be evaluated on response diversity diagnostics, such as Cat H (normalized entropy over the seven diagnosis categories), to indicate whether a model's diagnosis and correction claims vary across recordings or collapse to repeated category and claim patterns.
Abstract
Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,064 expert audio-level reviews and 12,216 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench separates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a persistent gap: while the strongest models can compare performances and identify broad issue domains in free-form feedback, the best Top-3 macro-F1 only slightly exceeds a label-prior baseline, and strict diagnosis hit rates remain below 8%. To our knowledge, VocalCoachBench provides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback. Data, code, and audio examples are available at https://hayeonbang.github.io/VocalCoachBench.
Sources
- FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- Qwen2-Audio Technical Report
- LP-MusicCaps: LLM-Based Pseudo Music Captioning
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations
- Kimi-Audio Technical Report
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
- Audiovisual Singing Voice Separation
- Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering
- The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation
- gpt-oss-120b & gpt-oss-20b Model Card
- Qwen3.5-Omni Technical Report
- MiMo-Audio: Audio Language Models are Few-Shot Learners
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems