MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation
cs.AI, cs.SD
Submitted: 2026-03-24
Updated: 2026-10-28
Comments: 10 Pages, 6 figures
Code: https://github.com/dgtql/MuQ-Eval
License: http://creativecommons.org/licenses/by/4.0/
The gist: Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments…
Terminology
Abstract
Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MUQ-EVAL, an open-source per-sample quality metric for AIgenerated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MUQ-EVAL, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.
Sources
- MusicLM: Generating Music From Text
- Stable Audio Open
- Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms
- Aligning Text-to-Music Evaluation with Human Preferences
- CLAP: Learning Audio Concepts From Natural Language Supervision
- From Aesthetics to Human Preferences: Comparative Perspectives of Evaluating Text-to-Music Systems
- The AudioMOS Challenge 2025
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation
- MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization
- MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
- A Survey on Evaluation Metrics for Music Generation
- Correlation of Fr'echet Audio Distance With Human Perception of Environmental Audio Is Embedding Dependant
- Adapting Frechet Audio Distance for Generative Music Evaluation
- Benchmarking Music Generation Models and Metrics via Human Preference Studies
- KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation
- Human-CLAP: Human-perception-based contrastive language-audio pretraining
- SCOREQ: Speech Quality Assessment with Contrastive Regression
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators
- PAM: Prompting Audio-Language Models for Audio Quality Assessment
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents