FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss".
Dev: Here's the gist of the paper. FUSEMOS proposes a dual-encoder fusion framework with ranking-aware loss for TTM perceptual evaluation to better capture both high-level semantic alignment and detailed musical characteristics for more accurate overall musical impression and textual alignment estimation Dual-Encoder Architecture The framework employs a dual-encoder fusion architecture that leverages complementary representations from CLAP and MERT, with an innovative ranking-aware composite loss The CLAP branch produces two L2-normalized embeddings, E Text CLAP and E Music CLAP, which reside in a common semantic space enabling direct comparison via dot product For coarse overall musical impression estimation, the music embedding E Music CLAP is used as input to a dedicated MLP The MERT branch extracts hidden states from the final layer of MERT, denoted as E0, and applies a learnable query-based attention pooling mechanism to obtain the final music representation EMERT Since the CLAP text encoder and the MERT model are trained independently and do not share a common representation space for text, they are concatenated to form a joint feature EMC The resulting features are passed through dedicated MLP to produce intermediate estimations of OMI and TA scores Late Fusion Strategy The authors adopt a late fusion strategy that combines predictions from both encoders at the score level, rather than fusing raw features early in the network This approach allows each encoder pathway to independently model perceptual aspects aligned with its inductive bias before integrating their outputs into a unified prediction The intermediate predictions, representing each branch’s preliminary assessment of perceptual quality, are concatenated and passed through a lightweight linear fusion head to produce the final scores This fusion head adaptively weights contributions from each branch Ranking-Aware Composite Loss Functions Effective training requires an objective function that captures relative preferences in human judgments by combining contrastive ranking loss and truncated regression loss The contrastive ranking loss Lcon explicitly models the relative quality ranking relationships between samples by constraining the consistency between predicted score differences and true score differences of sample pairs The contrastive ranking loss function is calculated as Lcon = 1/N(N − 1) X N i=1 X N j=1 j̸=i max (0, dij) The truncated regression loss Ltrun can prevent the model from overfitting to noisy labels or exact rating values by ignoring errors below a certain threshold τ By integrating contrastive ranking loss with truncated regression loss, the model can learn the relative quality of samples while maintaining the stability of the training process and preventing overfitting The final total loss is defined as Lsum = β · Lcon + γ · Ltrun Experimental Validation Experiments on the MusicEval benchmark demonstrate that FUSEMOS outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin For OMI, U-MSE and SMSE are substantially reduced by 53.6% and 87.6%, respectively, with U-SRCC reaching 0.811 and S-SRCC of 0.977 For TA, USRCC improves from 0.495 to 0.644 and S-SRCC from 0.861 to 0.
Paper summary: Rosa: So we're looking at this paper called FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss. Basically, they're trying to figure out how to properly evaluate music that an AI creates from text prompts.
Dev: Right, so the core idea here is that existing methods only use one model called CLAP for both audio and text features to predict scores like overall musical impression or textual alignment. But CLAP has a problem because it can't really capture the fine details of music, like timbre or expressiveness, you know?
Taro: That sounds like a limitation when you're testing real-world scenarios where those subtle musical qualities matter for how something sounds.
Rosa: Exactly. So FUSEMOS proposes a dual-encoder fusion framework that combines CLAP with another model called MERT, which is self-supervised and trained on large music audio datasets. The goal is to get better at catching both the big picture semantic alignment and those detailed musical characteristics for more accurate scores.
Dev: The architecture they propose uses two branches, one from CLAP and one from MERT, which are then fused together in a late fusion strategy. They use CLAP to create embeddings that live in a shared space so you can compare the music embedding directly with the text embedding using a dot product for things like textual alignment.
Taro: So they're trying to use two different kinds of learned representations—one focused on text-audio similarity and one focused on broad music characteristics—to get a fuller picture.
Rosa: And then they combine those results by passing them through separate layers and then using a lightweight linear fusion head to weight their contributions for the final overall musical impression or textual alignment predictions.
Dev: That sounds like they are trying to let each part of the model handle its own perceptual aspect independently before merging them into one final score.
Taro: What about how they actually train this thing? Because getting those scores right needs a good training signal, not just a simple regression loss.
Rosa: That's where they introduce a novel ranking-aware composite loss function. They combine a contrastive ranking loss with a truncated regression loss to train the model.
Dev: The contrastive part of that loss is supposed to explicitly model the relative quality rankings between different samples by checking for consistency in predicted score differences versus the actual score differences of pairs, which they call Lcon.
Taro: And then they add this truncated regression loss, Ltrun, which ignores errors below a certain threshold tau to stop the model from overfitting to noisy labels or exact rating values.
Rosa: By combining these two losses into a total loss, Lsum equals beta times the contrastive ranking loss plus gamma times the truncated regression loss, they aim to learn relative quality while keeping training stable and preventing that over-fitting issue.
Dev: The experimental validation on the MusicEval benchmark shows that this approach outperforms existing state-of-the-art methods in both mean squared error and rank correlation metrics by a large margin. For example, for overall musical impression, they see reductions in U-MSE and SMSE of fifty-three point six percent and eighty-seven point six percent, respectively, while the U-SRCC gets to zero point eight one one and the S-SRCC reaches zero point nine seven seven >
Taro: So if we're talking about textual alignment, their USRCC improves from a score of zero point four nine five to zero point six four four, and the S-SRCC goes from zero point eight six one up to zero point eight <ref:2611.07562#pg3>
Conclusion: Rosa: So FUSEMOS is about improving how we judge music made by text to music AI using a dual encoder setup and a new loss function.
Dev: Yeah, that's right, it’s about combining two different ways of looking at the audio and text features to get better scores for overall impression and textual alignment.
Taro: It’s trying to fix the problem where those current methods miss fine details like timbre or expressiveness in the music.
Rosa: Right, so they used a dual encoder, one from CLAP and one from MERT, which is trained on a huge amount of music audio data. They fuse them at the score level, not by mixing the raw features early on.
Dev: That late fusion strategy lets each part model its specific perceptual aspect before they come together into a final prediction via a linear head. It sounds like a way to keep things stable during training when you have two very different feature sets.
Taro: I'm curious about that ranking-aware loss function they designed. They use contrastive ranking loss and truncated regression loss to train the model, which should help it learn the relative quality of music segments better than just predicting an exact score.
Rosa: Exactly, so one part learns how things rank against each other, and the other part keeps training steady by ignoring errors that are too small or noisy. It’s a way to make sure the training signal is really focused on what matters for human preference.
Dev: The results on MusicEval were pretty strong then, showing significant improvements in mean squared error and rank correlation metrics over the current methods.
Taro: Those improvements suggest that this fusion approach actually captures those musical characteristics they were missing before, making the AI's output feel more natural to human listeners.
Rosa: So FUSEMOS seems to be a solid step toward getting text-to-music generation evaluated in a way that actually matters for quality and musical nuance.
Dev: It’s definitely worth looking at how this dual encoder and ranking loss setup could apply to other multimodal tasks where you have audio and text inputs together.
Jing Yang, Haoyu Wang, Ningning Pan, Zhao Wang, Jianxuan Yang
School of Electronic Information, Wuhan University · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · MiLM Plus, Xiaomi Inc.
eess.AS
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: Accepted to ICASSP 2026
DOI: 10.1109/ICASSP55912.2026.11461672
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 89/100
The gist: FUSEMOS proposes a dual-encoder fusion framework with ranking-aware loss for TTM perceptual evaluation to better capture both high-level semantic alignment and detailed musical characteristics for…
Key concepts
- Dual-Encoder Architecture
- This architecture uses two separate encoders, CLAP and MERT, to process text and music features independently. The CLAP branch creates embeddings that can be directly compared for semantic alignment. The MERT branch extracts detailed musical characteristics using a query-based attention pooling mechanism.
- Late Fusion Strategy
- Instead of combining raw features early, this strategy waits until both encoders produce their preliminary score estimations. These intermediate predictions are then combined using a lightweight linear fusion head to create the final scores, allowing each encoder to model its specific perceptual aspect before integration.
- Ranking-Aware Composite Loss Functions
- This loss function combines contrastive ranking loss (to model relative quality) and truncated regression loss (to stabilize training against noisy labels). This combination helps the model learn how one music segment is preferred over another while preventing it from overfitting to exact rating values.
- Overall Musical Impression (OMI) & Textual Alignment (TA)
- These are the two main perceptual scores being predicted. OMI assesses the general musical feeling, while TA measures how well the generated music matches the input text. FUSEMOS aims to provide more accurate estimations for both of these critical metrics.
Terminology
Summary
FUSEMOS proposes a dual-encoder fusion framework with ranking-aware loss for TTM perceptual evaluation to better capture both high-level semantic alignment and detailed musical characteristics for more accurate overall musical impression and textual alignment estimation
Dual-Encoder Architecture
The framework employs a dual-encoder fusion architecture that leverages complementary representations from CLAP and MERT, with an innovative ranking-aware composite loss The CLAP branch produces two L2-normalized embeddings, E Text CLAP and E Music CLAP, which reside in a common semantic space enabling direct comparison via dot product For coarse overall musical impression estimation, the music embedding E Music CLAP is used as input to a dedicated MLP The MERT branch extracts hidden states from the final layer of MERT, denoted as E0, and applies a learnable query-based attention pooling mechanism to obtain the final music representation EMERT Since the CLAP text encoder and the MERT model are trained independently and do not share a common representation space for text, they are concatenated to form a joint feature EMC The resulting features are passed through dedicated MLP to produce intermediate estimations of OMI and TA scores
Late Fusion Strategy
The authors adopt a late fusion strategy that combines predictions from both encoders at the score level, rather than fusing raw features early in the network This approach allows each encoder pathway to independently model perceptual aspects aligned with its inductive bias before integrating their outputs into a unified prediction The intermediate predictions, representing each branch’s preliminary assessment of perceptual quality, are concatenated and passed through a lightweight linear fusion head to produce the final scores This fusion head adaptively weights contributions from each branch
Ranking-Aware Composite Loss Functions
Effective training requires an objective function that captures relative preferences in human judgments by combining contrastive ranking loss and truncated regression loss The contrastive ranking loss Lcon explicitly models the relative quality ranking relationships between samples by constraining the consistency between predicted score differences and true score differences of sample pairs The contrastive ranking loss function is calculated as Lcon = 1/N(N − 1) X N i=1 X N j=1 j̸=i max (0, dij) The truncated regression loss Ltrun can prevent the model from overfitting to noisy labels or exact rating values by ignoring errors below a certain threshold τ By integrating contrastive ranking loss with truncated regression loss, the model can learn the relative quality of samples while maintaining the stability of the training process and preventing overfitting The final total loss is defined as Lsum = β · Lcon + γ · Ltrun
Experimental Validation
Experiments on the MusicEval benchmark demonstrate that FUSEMOS outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin For OMI, U-MSE and SMSE are substantially reduced by 53.6% and 87.6%, respectively, with U-SRCC reaching 0. p. 2811 and S-SRCC of 0. p. 2977 For TA, USRCC improves from 0. p. 2495 to 0.644 and S-SRCC from 0. p. 2861 to 0.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Dual-Encoder Fusion Architecture: The system can capture both high level semantic alignment and detailed musical characteristics for more accurate overall musical impression (OMI) and textual alignment (TA) estimation by leveraging CLAP for semantic matching and MERT for fine-grained musical structure.
-
Ranking-Aware Composite Loss Function: By introducing a ranking aware composite loss function that combines a contrastive ranking loss and a truncated regression loss, the system can better capture the ordinal nature of human perceptual judgments, leading to higher correlation with human judgments than single encoder baselines.
-
Late Fusion Strategy: Adopting a late fusion strategy allows each encoder pathway to independently model perceptual aspects aligned with its inductive bias before integrating their outputs into a unified prediction, which preserves modality-specific integrity while enabling effective score level integration.
Sources
- InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generation
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings
- Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models