TrueMuse: A Benchmark for Data Attribution in Text-to-Music Models

arXiv:2610.00835 · cs.LG · Submitted 2026-09-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "TrueMuse: A Benchmark for Data Attribution in Text-to-Music Models".

Tom: The gist The introduction proposes TRUEMUSE, a controlled dataset and benchmark for text-to-music data attribution, designed to systematically evaluate existing black-box attribution methods across four musical attributes.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Now let's talk about what the paper actually claims they did. They are addressing a big gap where existing attribution methods struggle because there isn't reliable ground truth for text-to-music generation yet.

Jane: So the core idea is that they propose TRUEMUSE as this controlled dataset and benchmark to systematically evaluate black-box attribution methods against these known fine-tuning source sets.

Lu: The paper explains that white-box methods try to approximate sample contributions through parameter estimation, which doesn't tell you if a sample really mattered.

Meng: But black-box methods operate on models trained on massive datasets where many samples influence the parameters at once, making it hard to isolate one specific contribution.

Tom: That entanglement makes it really difficult to figure out the exact influence of an individual sample on a given music generation. So they need this controlled benchmark to test their effectiveness.

Jane: They are constructing TRUEMUSE by fine-tuning three diffusion-based text-to-music models, AudioLDM2, Mustango, and Stable Audio Open, on these specific attribution samples they have curated.

Lu: The construction process adapts concept-specific fine-tuning from text-to-image diffusion models to text-to-music generation by binding the target musical attributes to selected audio samples.

Tom: It’s like they are taking that idea from image generation and applying it here, making sure those source clips are explicitly known targets for the attribution evaluation.

Jane: This setup allows them to generate samples using both contextual and style prompts, which is important because we need to test how the attribution works under different kinds of instructions.

Meng: So they are generating these ninety-five thousand five hundred forty-six samples across those two prompt types, and then using that output to see how well an attribution method can retrieve the right source clip <ref:2610.00835#pg1>.

Tom: And they set up this retrieval task where a generated sample is the query and it has to be matched against a gallery of clips from the designated source set.

The paper's summary: Jane: Moving into what they suggest are the improvements in their approach, they point out that existing methods can be rigorously evaluated across four dimensions.

Tom: Those dimensions are fine-tuning improvement, prompt-type difficulty, multitask training, and fine-tuning data size. They want to see which methods are more reliable and generalizable based on these tests.

Lu: They evaluate similarity-based black-box attribution using four different music encoders: MERTv0, MERTv0 public, DAC, and Music2Vec.

Meng: So they aren't just testing one tool; they are checking how these different types of encoders perform when trying to attribute music back to its sources.

Tom: And the results show that attribution remains challenging because existing methods vary quite a bit depending on which setting you test them in. It’s not straightforward.

Jane: The paper also introduces a new method where they use a mapper, which is essentially a linear projection head applied to the output embeddings of the frozen music encoder Fbase.

Lu: This mapper is trained with a contrastive loss using generated samples and their ground-truth clips to create embeddings that are better suited for attribution retrieval.

Tom: This mapper specifically improves performance for MERT-based encoders, showing gains like MERTv0 improving from zero point three five seven to zero point four zero zero in some cases.

Meng: But the gains aren't universal; they're largest on Mustango, where it gets a jump of about eight point four percent for both MERTv0 and MERTv0 public, while the improvement on Stable Audio Open is minimal, only about one point five percent.

The paper's improvements: Tom: So wrapping up this part of the paper, they show that style prompts actually outperform contextual prompts on average across all those different encoders they tested.

Jane: For example, for Melody-Centric attribution, style prompts increase retrieval performance from a score of zero point one seven eight under contextual to zero point three six five under style prompts.

Lu: This suggests that style prompts provide a cleaner and more attribute-focused training signal for the mapper they are training, which is quite useful for guiding the model's learning process.

Meng: It seems like they found that if you want a better attribution, you should be thinking about how the prompt is designed to focus on a specific musical feature rather than just describing what’s happening in context.

Tom: And they also looked at increasing the number of fine-tuning clips from three to six, which consistently reduced retrieval scores across the Instrument-Centric, Musician-Centric, and Genre-Centric settings.

Jane: That's because adding more clips expands the candidate source set, and this can make individual clips harder to retrieve when you increase that count.

Lu: So they conclude that source retrieval remains challenging and varies substantially across musical attributes and generators like Stable Audio Open compared to others like Mustango or AudioLDM2.

Meng: As an initial benchmark, TrueMuse focuses on a representative set of musical attributes, generators, and prompt conditions; future work could expand the coverage of models and concepts further.

Tom: This whole paper on TrueMuse gives us a controlled environment to study data attribution in text-to-music systems without reliable ground truth right now. It’s a solid foundation for where researchers can go next.

Jane: Indeed, it sets up a framework for more rigorous study of how we assign credit in these complex generative systems and supports developing methods that are more robust and interpretable.

Conclusion: Tom: So TrueMuse is this new benchmark for data attribution in text-to-music models, and what they did is build this controlled dataset where they know exactly which training samples are included in the fine-tuning.

Jane: Exactly, so it lets researchers systematically test how well existing black-box methods work when they have these known source sets to check against.

Lu: The construction is pretty interesting because they're taking that concept from text-to-image diffusion and applying it here to music generation.

Meng: It’s a controlled environment, which is what we need when trying to figure out if an attribution method actually points to the right training data contribution.

Lalam: If this works well, it could really help us understand how culture itself gets encoded in these generative models.

Tom: They are testing things across four major musical attributes: melodic structure, timbral characteristics, artist-level signatures, and genre patterns.

Jane: And they're using these generated samples to evaluate retrieval tasks where the sample is the query and the source clips are in a gallery based on which attribute it was fine-tuned on.

Lu: The results show that attribution remains pretty tough with current methods because there's so much variation depending on what you test them on.

Meng: They did look at how different encoders like MERTv0 and DAC perform, and they found that MERTv0 public was the strongest performer in some cases for R@ten scores.

Tom: And they also showed that the prompt type matters; style prompts are better than contextual ones on average across all those encoders.

Jane: That makes sense, because style prompts seem to give a cleaner signal for the mapper to learn from.

Lu: They also found that increasing the number of fine-tuning clips generally made retrieval harder in settings like Musician-Centric.

Meng: So it’s not just about getting more data; it’s about how that extra data affects the difficulty of isolating a single sample.

Lalam: It gives us a way to audit what's happening inside these massive models, which is super important for understanding their cultural output.

Tom: This whole TrueMuse benchmark is a huge step because it provides that controlled environment for evaluating attribution methods in text-to-music systems.

Jane: It helps move the conversation from just making cool music to actually understanding what's driving the music generation process.

Lu: It's a solid foundation, and they’ve laid out exactly where future research needs to go if we want better credit assignment.

Meng: I think seeing how these methods perform under different prompt types is really practical for deployment later on when we try to use AI for specific creative tasks.

Lalam: It opens up a lot of possibilities for how we can develop systems that are more transparent about their training data influences.

School of Computing, University of Georgia

cs.LG

Submitted: 2026-09-30

Updated: 2026-10-09

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: The gist The introduction proposes TRUEMUSE, a controlled dataset and benchmark for text-to-music data attribution, designed to systematically evaluate existing black-box attribution methods across

Key concepts

TRUEMUSE Construction: This is a controlled dataset built by fine-tuning three text-to-music models (AudioLDM2, Mustango, Stable Audio Open) on specific audio samples. These samples are chosen because their inclusion in the fine-tuning process is known, allowing researchers to precisely measure how well attribution methods can identify these specific musical features.
Attribution Settings: The benchmark tests attribution across four distinct musical dimensions: melodic structure, timbral characteristics (sound quality), artist-level stylistic signatures (like a specific singer's sound), and genre-level shared patterns. This allows for a comprehensive evaluation of attribution methods against different types of musical information.
Retrieval Task: The evaluation works like a search engine. A model generates music, and researchers use the generated audio as a query to search a gallery containing clips from known source sets corresponding to specific musical attributes. The success of this retrieval determines the attribution performance.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes

Summary

The gist The introduction proposes TRUEMUSE, a controlled dataset and benchmark for text-to-music data attribution, designed to systematically evaluate existing black-box attribution methods across four musical attributes.

TRUEMUSE Construction

TRUEMUSE is constructed by fine-tuning three diffusion-based text-to-music models: AudioLDM2 [18], Mustango [22], and Stable Audio Open [11] on carefully curated attribution samples, whose known inclusion in fine-tuning provides controlled attribution targets for evaluation. The benchmark covers four attribution settings spanning melodic structure, timbral characteristics, artist-level stylistic signatures, and genre-level shared patterns, including 133 attributes, 648 fine-tuned models, and 95,456 generated samples across two prompt types <ref:2610.00835#pg5>. The fine-tuning procedure adapts concept-specific fine-tuning from text-to-image diffusion models to text-to-music generation by binding target musical attributes to selected audio samples, which serve as controlled attribution targets because their inclusion in fine-tuning is explicitly known <ref:2610.00835#pg4>.

Evaluation Framework

TRUEMUSE enables the systematic evaluation of existing black-box attribution methods along four dimensions: fine-tuning improvement, prompt-type difficulty, multitask training, and fine-tuning data size <ref:2610.00835#pg3>. The benchmark covers four musical attribution sets spanning melodic structure, timbral characteristics, artist-level stylistic signatures, and genre-level shared patterns <ref:2610.00835#pg5>. For each generated sample xˆ produced by a model fine-tuned on attribute c, its corresponding Dc serves as the designated source set to be retrieved <ref:2610.00835#pg6>. Evaluation is formulated as a retrieval task where xˆ is the query and it is matched against a gallery containing clips from Dc <ref:2610.00835#pg6>.

Attribution Performance Analysis

The evaluation assesses similarity-based black-box attribution using four music encoders: MERT-v0 [16], MERT-v0-public [16], DAC [15], and Music2Vec [17] <ref:2610.00835#pg3>. The methods are assessed across four dimensions: attribution improvement from fine-tuned mappers over raw embeddings, attribution difficulty under contextual versus style prompts, the effect of joint multi-task training across melody, musician, instrument, and genre, and the impact of fine-tuning data size <ref:2610.00835#pg3>. Results show that attribution remains challenging with existing methods exhibiting substantial variation across evaluation settings <ref:2610.00835#pg3>.

Key Findings on Attribution Difficulty

The difficulty of retrieval varies significantly across different factors <ref:2610.00835#pg8>. For example, Instrument-Centric is the easiest at R@10 of 0.465, followed by Musician-Centric at 0.305, Melody-Centric at 0.239, and Genre-Centric at 0.101 <ref:2610.00835#pg8>. This ordering likely reflects the acoustic specificity of each attribute <ref:2610.00835#pg8>. Furthermore, attribution performance varies across generators; Stable Audio Open is the easiest to attribute, reaching R@10 of 0.487, compared to 0.205 for Mustango and 0.140 for AudioLDM2 <ref:2610.00835#pg8>.

Mapper Effectiveness

The second method introduced improves the baseline by introducing a mapper, a linear projection head applied to the output embeddings of the frozen music encoder Fbase, trained with a contrastive loss on generated samples and their ground-truth clips to produce embeddings better suited for attribution <ref:2610.00835#pg7>. Training mappers on top of the frozen encoders improves attribution mainly for MERT-based encoders, as shown in Figure 2, with MERT-v0 improving from 0.357 to 0.400 and MERT-v0-public from 0.397 to 0.441 <ref:2610.00835#pg7>. Gains are largest on Mustango, at 0.084 for both MERT-v0 and MERT-v0-public, and minimal on Stable Audio Open, at most 0.015 <ref:2610.00835#pg7>.

Prompt Type Influence

Style prompts outperform contextual prompts on average across all encoders <ref:2610.00835#pg5>. For Melody-Centric, style prompts increase R@10 from 0.178 under contextual to 0.365 under style prompts <ref:2610.00835#pg5>. This suggests that style prompts provide a cleaner and more attribute-focused training signal for the mapper <ref:2610.00835#pg5>.

Clip Count Impact

Increasing the number of fine-tuning clips from 3 to 6 consistently reduces R@10 across the Instrument-Centric, Musician-Centric, and Genre-Centric settings <ref:2610.00835#pg9>. The effect is most pronounced for Musician-Centric, where six clips drawn from different tracks introduce broader stylistic variation <ref:2610.00835#pg9>.

Conclusion

TRUEMUSE provides a controlled benchmark with known fine-tuning source sets for systematically evaluating text-to-music attribution methods <ref:2610.00835#pg2>. Our results show that source retrieval remains challenging and varies substantially across musical attributes and generators <ref:2610.00835#pg6>. As an initial benchmark, TRUEMUSE focuses on a representative set of musical attributes, text-to-music generators, and prompt conditions; future extensions may further expand the coverage of models, musical concepts, and generation scenarios <ref:2610.00835#pg6>. We hope this benchmark encourages more rigorous study of attribution in text-to-music systems and supports the development of methods that are more robust, interpretable, and suitable for real-world credit assignment.

--- Page 1 ---

TRUEMUSE Construction

TRUEMUSE is constructed by fine-tuning three diffusion-based text-to-music models: AudioLDM2 [18], Mustango [22], and Stable Audio Open [11] on carefully curated attribution samples, whose known inclusion in fine-tuning provides controlled attribution targets for evaluation <ref:2610.00835#pg5>. The benchmark covers four attribution settings spanning melodic structure, timbral characteristics, artist-level stylistic signatures, and genre-level shared patterns, including 133 attributes, 648 fine-tuned models, and 95,456 generated samples across two prompt types <ref:2610.00835#pg5>. The fine-tuning procedure adapts concept-specific fine-tuning from text-to-image diffusion models to text-to-music generation by binding target musical attributes to selected audio samples, which serve as controlled attribution targets because their inclusion in fine-tuning is explicitly known <ref:2610.00835#pg4>.

Attribution Performance Analysis

The evaluation assesses similarity-based black-box attribution using four music encoders: MERT-v0 [16], MERT-v0-public [16], DAC [15], and Music2Vec [17] <ref:2610.00835#pg3>.

Improvements for AI systems

Identification of data contribution via TRUEMUSE: The system can quantify the contribution of individual training samples by utilizing TRUEMUSE, a controlled dataset and benchmark for text-to-music data attribution, allowing researchers to evaluate existing black-box methods against known fine-tuning source sets.
Systematic evaluation across four dimensions: Existing methods can be rigorously assessed across fine-tuning improvement, prompt-type difficulty, multitask training, and fine-tuning data size using TRUEMUSE results to determine which methods are more reliable and generalizable.
Improved black-box attribution via mapper enhancement: The system can implement a contrastive learning framework to improve retrieval performance by training a mapper that learns a small linear correction that closes the gap between generated samples and ground-truth clips, as shown in Equation (3).
Encoder selection optimization: The system can select the most effective music encoder based on attribution needs; results show that MERT-v0-public is the strongest at R@10 of 0.397, indicating that selecting encoders with music-specific self-supervised objectives substantially outperform codec-based encoders.
Prompt strategy adaptation: The system can adapt prompt strategies based on the target attribute; for instance, style prompts outperform contextual prompts on average across all encoders, suggesting that style prompts provide a cleaner and more attribute-focused training signal for the mapper.
Attribution sensitivity analysis: The system can dynamically adjust prompting based on the attribute type; it can identify that Melody-Centric is the most sensitive, with R@10 increasing from 0.178 under contextual prompts to 0.365 under style prompts, informing prompt selection for specific musical features.
Clip count optimization: The system can optimize fine-tuning data size; it can determine that increasing the number of fine-tuning clips expands the candidate source set and may make individual clips harder to retrieve, showing that performance consistently degrades as clip counts increase in settings like Musician-Centric.

Abstract

Text-to-music generation models are trained on massive music collections, creating a growing need for data attribution methods that can quantify the contribution of individual training samples. However, existing attribution methods are difficult to rigorously evaluate due to the lack of reliable ground truth, making it challenging to reliably assess their actual effectiveness. To address this gap, we introduce TrueMuse, a controlled dataset and benchmark for text-to-music data attribution. TrueMuse is constructed by fine-tuning three diffusion-based text-to-music models on carefully curated attribution samples, whose known inclusion in fine-tuning provides controlled attribution targets for evaluation. The benchmark covers four attribution settings, spanning melodic structure, timbral characteristics, artist-level stylistic signatures, and genre-level shared patterns, and includes 133 attributes, 648 fine-tuned models, and 95,456 generated samples across two prompt types. Using TrueMuse, we systematically evaluate existing black-box attribution methods along four dimensions: fine-tuning improvement, prompt-type difficulty, multi-task training, and fine-tuning data size. Our results show that attribution remains challenging, with existing methods exhibiting substantial variation across evaluation settings, highlighting the need for more reliable and generalizable attribution methods for text-to-music generation.