VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing

arXiv:2609.04241 · cs.SD, eess.AS · Submitted 2026-08-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing".

Jane: Here's the gist of the paper. Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input <ref:2609.04241#pg2>.

Tom: VocalCoachBench introduces a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing <ref:2609.04241#pg2>.

Jane: It contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims <ref:2609.04241#pg3>.

Tom: Benchmark Composition VocalCoachBench comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions <ref:2609.04241#pg2>.

Paper summary: Tom: So, wrapping up VocalCoachBench: it’s this new benchmark focused specifically on evaluating audio-language models on expert vocal coaching feedback for singing >

Jane: It’s a framework built around separating deterministic structured targets from the open-ended assessment of free-form diagnosis and corrective guidance >

Lu: The authors used eighteen professional vocal trainers to create this data, yielding one thousand fifty-six expert submissions and twelve thousand fifty one atomic coaching claims across two distinct subsets > <ref:2609.04241#pg1>

Meng: The big implication is that while models can identify general problem areas, they consistently miss the specific fine-grained issues experts prioritize and their proposed corrections don't always align with those expert claims by more than seventy percent >

Lalam: This shows that simply processing audio isn't enough; the AI needs a deeper grasp of vocal coaching itself to produce reliable, actionable advice for singers >

Tom: The authors are clear that this is a first step toward benchmarking audiogrounded expert feedback, and they caution that model-generated vocal guidance should not replace professional instruction >

Jane: So, while it’s a valuable tool for testing current models against expert claims on singing, we still need to work on making these evaluations more comprehensive and applicable across different scenarios >

Conclusion: Tom: So, VocalCoachBench is this new way to test how good audio models are at actually giving expert singing advice >

Jane: It's basically taking all that expert feedback—the really detailed critiques singers get from vocal coaches—and turning it into a structured test for these language models >

Lu: The core idea is setting up two different kinds of tests, one where the answers are pretty specific, and another where the model has to guess what kind of correction an expert would suggest >

Meng: And they used eighteen professional vocal trainers to create this data set, which gives you a lot of human expertise to judge the models against >

Lalam: So what we're seeing is that while these models can grasp general singing problems, they struggle when it comes to pinpointing the exact technical issues or giving valid advice >

Tom: Right. The authors are showing that even with recent audio-language models, there's still a gap between just hearing a song and actually understanding how to coach someone on it >

Jane: It really highlights that being good at processing sound isn't the same as being able to provide real pedagogical feedback >

Lu: And they found that for the smaller, controlled tests, you can see some improvement in ranking songs based on expert opinions >

Meng: But they also pointed out that when you try to predict a specific technical problem, like a vocal placement issue, those models fall behind the established baselines by quite a bit >

Lalam: So the authors are basically saying we need more than just audio processing; we need models that understand the *why* behind expert coaching >

Tom: Exactly. They’re not claiming these models are ready to replace a human coach yet, but they're showing us exactly where they fall short right now >

Jane: It sets up a real challenge for the next generation of audio language models trying to move beyond just summarizing music into actually providing guidance >

Hayeon Bang, Hounsu Kim, Wonil Kim, Juhan Nam

KAIST

cs.SD, eess.AS

Submitted: 2026-08-06

Updated: 2026-11-08

Comments: NeurIPS 2026, Evaluations & Datasets Track

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 91/100

The gist: Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability…

Key concepts

VocalCoachBench
This is a new benchmark designed to evaluate audio-language models on their ability to generate expert vocal coaching feedback for singing. It uses recordings annotated by professional trainers to test if AI can provide useful, actionable guidance.
Atomic Coaching Claims
These are the specific, fine-grained diagnostic points made by human vocal coaches. Instead of just giving general advice, these claims pinpoint exact problems in technical production, musical accuracy, or delivery that need correction.
Structured vs. Open-Ended Suite
The benchmark has two testing methods: a structured suite with fixed tasks like ranking and classification, and an open-ended suite where models must assess the coverage and validity of expert claims against specific reference points.

Terminology

Summary

Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input <ref:2609. p. 0

Where the paper says it

nam}@kaist. kr Abstract Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input.

04241#pg2>. VocalCoachBench introduces a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing <ref:2609. p. 0
Where the paper says it

kr Abstract Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing.

04241#pg2>. It contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims <ref:2609. p. 0
Where the paper says it

We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims.

04241#pg3>.

Benchmark Composition

VocalCoachBench comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions <ref:2609.04241#pg2>. The same-song subset includes controlled triplet rankings with ranking rationales, while the diverse-song subset includes segment-grounded feedback with time spans <ref:2609. p. 1

Where the paper says it

In total, 18 expert vocal trainers provide ranked Top-3 issue labels and natural-language coaching reviews covering the main problems, corrective directions, and strengths. The same-song subset additionally includes controlled triplet rankings with ranking rationales, and the diverse-song subset includes segment-grounded feedback with time spans.

04241#pg4>. This design accommodates the open-ended nature of expert feedback by separating deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance <ref:2609. p. 0
Where the paper says it

It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench separates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance.

04241#pg4>.

Annotation Protocol

Expert vocal trainers annotate two complementary subsets: a same-song subset for controlled comparison, and a diverse-song subset for diverse coaching scenarios and segment-grounded feedback <ref:2609.04241#pg2>. Annotation was carried out by 18 professional vocal trainers with formal vocal-performance training and at least three years of coaching experience <ref:2609. p. 4

Where the paper says it

Together, the two subsets provide complementary evaluation regimes: same-song recordings support controlled comparison, while diverse-song recordings support diverse and segment-grounded coaching analysis. 3 Expert annotation protocol Annotation was carried out by 18 professional vocal trainers with formal vocal-performance training and at least three years of coaching experience.

04241#pg6>. The final taxonomy consists of seven fine-grained issue categories under three parent categories: Technical Production, Musical Accuracy, and Delivery <ref:2609. p. 4
Where the paper says it

To support later analysis and evaluation, annotators wrote their feedback in semantically coherent feedback blocks and tagged each block with one or more vocal issue categories. The final taxonomy consists of seven fine-grained issue categories under three parent categories: Technical Production covers breath support, vocalization, and technique; Musical Accuracy covers pitch and rhythm; and Delivery covers diction and expression.

04241#pg4>.

Evaluation Suites

VocalCoachBench defines two complementary evaluation suites: a structured evaluation suite consisting of fully reproducible deterministic tasks and an open-ended coaching suite assessing model-generated feedback against expert atomic claims <ref:2609.04241#pg4>. The structured evaluation includes triplet ranking, Top-3 issue prediction, and segment-level issue classification <ref:2609. p. 7

Where the paper says it

All values are percentages except Kendall’s τ . Protocol Model Triplet ranking Top-3 issue prediction Segment Pair τ Exact Cycle ↓ F1@3 Top-1 nDCG@3 Parent F1 Match REF Random baseline 50.

04241#pg8>. The open-ended suite evaluates diagnosis coverage and correction validity against atomic claims <ref:2609. p. 2
Where the paper says it

Top-3 issue-label prediction, and segment-level issue-label classification. The open-ended coaching evaluation suite evaluates model-generated feedback against expert atomic claims, separating diagnosis coverage from correction validity.

04241#pg8>.

Model Performance Findings

Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7% <ref:2609.04241#pg2>. Models often identify broad issue domains but miss expert-prioritized fine-grained problems in diagnosis, while correction validity is higher for more capable models <ref:2609. p. 8

Where the paper says it

Strict diagnosis hit rates stay below 7% across all models, while Hit rates vary more widely (17–63%), indicating that models often identify broad issue domains but miss expert-prioritized fine-grained problems. Correction validity is higher for more capable models, but high Valid rates should be interpreted alongside diagnosis coverage: models can generate plausible corrections even when their diagnoses remain loose or generic.

04241#pg9>.

Limitations and Future Work

VocalCoachBench is a first step toward benchmarking audiogrounded expert feedback, not a complete model of vocal pedagogy <ref:2609.04241#pg10>. Limitations include focusing on English solo singing, the same-song subset being limited to a single song, and the open-ended evaluation measuring reference-supported claim coverage rather than full pedagogical value <ref:2609. p. 8

Where the paper says it

The benchmark is audio-only and oneshot, whereas real coaching may use visual or bodily cues and adapt feedback over repeated attempts; segment-grounded evaluation also assumes expert-provided segments rather than testing end-to-end discovery. Our open-ended evaluation measures reference-supported claim coverage rather than the full pedagogical value of a response, so it may under-credit valid feedback outside the available expert claims.

04241#pg10>. Future work should extend VocalCoachBench toward multilingual, multimodal, interactive, and longitudinal coaching evaluation <ref:2609. p. 8
Where the paper says it

Because incorrect vocal guidance may encourage strain or unsafe self-training, model outputs should not be treated as a substitute for professional instruction. Future work should extend VocalCoachBench toward multilingual, multimodal, interactive, and longitudinal coaching evaluation.

04241#pg10>.

Data and Code Availability

The paper provides open access to the benchmark annotations, evaluation scripts, prompt templates, and documentation sufficient to reproduce the reported benchmark metrics <ref:2609.04241#pg32>. The authors release anonymized versions of the benchmark annotations, evaluation code, prompts, and documentation governed by the licenses and access terms of the underlying audio sources <ref:2609.04241#pg32>.

LLM Usage

LLMs are used as part of the methodology for claim decomposition, parsing/normalization, translation support, and judge-based evaluation. These uses are described in the dataset post-processing and evaluation sections with prompts and validation details in the appendix. p. 6

Where the paper says it

The main metric is the correction validity rate, the fraction of correction claims labeled valid; we also report weak and invalid rates. Judge prompts, label definitions, and human validation are described below.

p. 6
Where the paper says it

The main metric is the correction validity rate, the fraction of correction claims labeled valid; we also report weak and invalid rates. Judge prompts, label definitions, and human validation are described below.

Human Subjects Ethics

The work involved only paid annotation rather than a behavioral intervention, and annotators were informed of the task content and compensation before participation <ref:2609.04241#pg32>. Formal IRB review was not required under the applicable institutional policy <ref:2609.04241#pg32>.

Broader Impacts

The introduction and conclusion discuss the positive goal of improving audiolanguage evaluation for expert feedback, while the limitations discuss risks from incorrect vocal guidance and the need not to treat model outputs as professional instruction <ref:2609.04241#pg31>. The authors warn that model-generated vocal guidance should not replace professional instruction <ref:2609. p. 8

Where the paper says it

Our open-ended evaluation measures reference-supported claim coverage rather than the full pedagogical value of a response, so it may under-credit valid feedback outside the available expert claims. Because incorrect vocal guidance may encourage strain or unsafe self-training, model outputs should not be treated as a substitute for professional instruction.

04241#pg32>.

License Compliance

Existing audio datasets and models are cited in the paper, and source-license information for the audio assets is documented in the appendix <ref:2609.04241#pg32>. The name of the license should be included for each asset <ref:2609. p. 0

Where the paper says it

Evaluating vocal coaching ability, therefore, is not a matter of audio description or singing quality assessment alone, but a problem of identifying the main issues and generating corrective feedback grounded in the performance. arXiv:2609.

04241#pg32>. The authors state that they provide audio-access manifests rather than audio files for some sources <ref:2609.04241#pg32>.

New Assets

VocalCoachBench introduces new expert annotations, prompts, evaluation code, and documentation; the paper and appendix describe the annotation schema, taxonomy, examples, validation, and intended benchmark use <ref:2609.04241#pg32>. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates <ref:2609. p. 5

Where the paper says it

observed no omitted or hallucinated claims and only minor extraction errors; details are provided in Appendix E. 5 Dataset statistics and annotation behavior VocalCoachBench contains 515 recordings, 1,056 expert audio-level submissions, 4,443 feedback blocks, and 12,051 atomic claims (Table 2).

04241#pg32>. The paper describes how consent was obtained from people whose asset is used <ref:2609.04241#pg32>.

Experimental Reproducibility

Sections 4.1, 4.2, and 5.2 define the tasks, metrics, prompting protocols, inference setup, parsing, and judging procedures; additional prompts and validation details are provided in the appendix <ref:2609. p. 7

Where the paper says it

5-Omni Flash and Plus variants [29]—are evaluated with the main protocol via public APIs. 2 Prompting protocols and inference setup The main claim/objective protocol uses task-specific structured prompts for coaching claims, Top-3 issues, and segment classification.

04241#pg32>. The authors describe the steps taken to make their results reproducible or verifiable <ref:2609.04241#pg32>.

Resources

Appendix I reports that no models are trained, open-weight inference uses a single NVIDIA H100 80GB GPU, closed-source models use public APIs, and long-running inference and judge jobs are executed with resumable batch scripts because runtime varies by model and API latency. The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider. p. 3

Where the paper says it

Beyond music, recent embodied-skill benchmarks evaluate expert-level performance understanding through proficiency cues, improvementrelated commentary, action differences, or expert-derived QA [9, 3, 44]. VocalCoachBench complements these efforts by moving from collecting or matching expert feedback to evaluating the internal quality of generated feedback: whether its diagnosis is expert-aligned, and paired with correction appropriate to the diagnosed issue.

Statistical Significance

We report deterministic benchmark metrics and human/judge agreement statistics on fixed evaluation sets, but do not report bootstrap confidence intervals or statistical significance tests for model comparisons <ref:2609.04241#pg31>. The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests <ref:2609.04241#pg31>.

Safeguards

The paper does not release a generative model or other high-risk dual-use system <ref:2609.04241#pg32>. The benchmark release is limited to annotations, evaluation code, and documentation governed by the licenses and access terms of the underlying audio sources <ref:2609. p. 8

Where the paper says it

It focuses on English solo singing, and its same-song subset is limited to a single song. The benchmark is audio-only and oneshot, whereas real coaching may use visual or bodily cues and adapt feedback over repeated attempts; segment-grounded evaluation also assumes expert-provided segments rather than testing end-to-end discovery.

04241#pg32>.

Licenses for Existing Assets

Existing audio datasets and models are cited in the paper, and source-license information for the audio assets is documented in the appendix <ref:2609.04241#pg32>. The license should be included for each asset <ref:2609. p. 0

Where the paper says it

Evaluating vocal coaching ability, therefore, is not a matter of audio description or singing quality assessment alone, but a problem of identifying the main issues and generating corrective feedback grounded in the performance. arXiv:2609.

04241#pg32>. The authors provide audio-access manifests rather than audio files for some sources <ref:2609. p. 0
Where the paper says it

Evaluating vocal coaching ability, therefore, is not a matter of audio description or singing quality assessment alone, but a problem of identifying the main issues and generating corrective feedback grounded in the performance. arXiv:2609.

04241#pg32>.

Crowdsourcing and Human Subjects

The work does not use open crowdsourcing; annotations were collected from paid expert vocal coaches who consented to the annotation work <ref:2609.04241#pg32>. The paper describes the expert annotation protocol, quality-control process, annotation interface examples, and annotation labor details in the main text and Appendix <ref:2609. p. 5

Where the paper says it

observed no omitted or hallucinated claims and only minor extraction errors; details are provided in Appendix E. 5 Dataset statistics and annotation behavior VocalCoachBench contains 515 recordings, 1,056 expert audio-level submissions, 4,443 feedback blocks, and 12,051 atomic claims (Table 2).

04241#pg32>.

IRB Approvals

Under our applicable institutional policy, formal IRB review was not required <ref:2609.04241#pg32>. Annotators were informed of the task content and compensation before participation, and no private personal information was collected for analysis <ref:2609.04241#pg32>.

LLM Usage Declaration

LLMs are used as part of the methodology for claim decomposition, parsing/normalization, translation support, and judge-based evaluation. These uses are described in the dataset post-processing and evaluation sections with prompts and validation details in the appendix.

Data Access

The paper is accompanied by benchmark annotations, evaluation scripts, prompt templates, and documentation sufficient to reproduce the reported benchmark metrics <ref:2609.04241#pg32>. The authors provide instructions on data access and preparation <ref:2609.04241#pg32>. The authors release anonymized versions of the benchmark annotations, evaluation code, prompts, and documentation governed by the licenses and access terms of the underlying audio sources <ref:2609.04241#pg32>.

Experimental Setting/Details

The dataset construction, splits, task definitions, model routing, prompting protocols, deterministic decoding policy, parsing, and judge validation are described in the paper and appendix <ref:2609.04241#pg32>. The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results <ref:2609.04241#pg32>.

Code of Ethics

We reviewed the NeurIPS Code of Ethics and believe the work conforms to it <ref:2609.04241#pg32>. The paper uses cited audio sources, expert annotation, documented licensing, and warns that model-generated vocal guidance should not replace professional instruction <ref:2609. p. 8

Where the paper says it

Our open-ended evaluation measures reference-supported claim coverage rather than the full pedagogical value of a response, so it may under-credit valid feedback outside the available expert claims. Because incorrect vocal guidance may encourage strain or unsafe self-training, model outputs should not be treated as a substitute for professional instruction.

04241#pg32>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Structured evaluation suite for deterministic tasks can be used to evaluate model preferences over three same-song performances using direct pairwise comparison, which is preferred over scalar elicitation due to its robustness against score-tie collapse.

  2. Open-ended coaching evaluation can assess free-form feedback by evaluating diagnosis coverage against atomic diagnosis claims and correction validity against atomic correction claims, treating expert disagreement as a property of coaching feedback rather than noise.

  3. The system can be benchmarked on Top-3 issue prediction using fine-level F1@3, which measures the Top-3 set overlap divided by three, and also report parent-level F1 to capture partially aligned expert judgment.

  4. Segment-level issue classification enables models to perform local issue recognition on expert consensus segments, where the main metric is any-match accuracy against a set of shared issue tags.

  5. Models can be evaluated on response diversity diagnostics, such as Cat H (normalized entropy over the seven diagnosis categories), to indicate whether a model's diagnosis and correction claims vary across recordings or collapse to repeated category and claim patterns.

Abstract

Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,064 expert audio-level reviews and 12,216 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench separates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a persistent gap: while the strongest models can compare performances and identify broad issue domains in free-form feedback, the best Top-3 macro-F1 only slightly exceeds a label-prior baseline, and strict diagnosis hit rates remain below 8%. To our knowledge, VocalCoachBench provides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback. Data, code, and audio examples are available at https://hayeonbang.github.io/VocalCoachBench.

Sources

Related papers