FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss
summary
The gist
FUSEMOS proposes a dual-encoder fusion framework with ranking-aware loss for TTM perceptual evaluation to better capture both high-level semantic alignment and detailed musical characteristics for…
In short
FUSEMOS is a new method for evaluating text-to-music generation by combining two different AI models, CLAP and MERT, using a dual-encoder fusion architecture. It also introduces a novel ranking-aware loss function to better capture both high-level musical quality and detailed features. This approach significantly improves the accuracy of estimating overall musical impression and textual alignment compared to current state-of-the-art methods.
Key concepts
- Dual-Encoder Architecture
- This architecture uses two separate encoders, CLAP and MERT, to process text and music features independently. The CLAP branch creates embeddings that can be directly compared for semantic alignment. The MERT branch extracts detailed musical characteristics using a query-based attention pooling mechanism.
- Late Fusion Strategy
- Instead of combining raw features early, this strategy waits until both encoders produce their preliminary score estimations. These intermediate predictions are then combined using a lightweight linear fusion head to create the final scores, allowing each encoder to model its specific perceptual aspect before integration.
- Ranking-Aware Composite Loss Functions
- This loss function combines contrastive ranking loss (to model relative quality) and truncated regression loss (to stabilize training against noisy labels). This combination helps the model learn how one music segment is preferred over another while preventing it from overfitting to exact rating values.
- Overall Musical Impression (OMI) & Textual Alignment (TA)
- These are the two main perceptual scores being predicted. OMI assesses the general musical feeling, while TA measures how well the generated music matches the input text. FUSEMOS aims to provide more accurate estimations for both of these critical metrics.
Terminology used across episodes
This episode discusses
- FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss · Paper Radio
- InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generation
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings
- Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
The paper
FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss · Read on arXiv
Jing Yang, Haoyu Wang, Ningning Pan, Zhao Wang, Jianxuan Yang
School of Electronic Information, Wuhan University · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · MiLM Plus, Xiaomi Inc.
DOI: 10.1109/ICASSP55912.2026.11461672
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss".
Dev: Here's the gist of the paper. FUSEMOS proposes a dual-encoder fusion framework with ranking-aware loss for TTM perceptual evaluation to better capture both high-level semantic alignment and detailed musical characteristics for more accurate overall musical impression and textual alignment estimation Dual-Encoder Architecture The framework employs a dual-encoder fusion architecture that leverages complementary representations from CLAP and MERT, with an innovative ranking-aware composite loss The CLAP branch produces two L2-normalized embeddings, E Text CLAP and E Music CLAP, which reside in a common semantic space enabling direct comparison via dot product For coarse overall musical impression estimation, the music embedding E Music CLAP is used as input to a dedicated MLP The MERT branch extracts hidden states from the final layer of MERT, denoted as E0, and applies a learnable query-based attention pooling mechanism to obtain the final music representation EMERT Since the CLAP text encoder and the MERT model are trained independently and do not share a common representation space for text, they are concatenated to form a joint feature EMC The resulting features are passed through dedicated MLP to produce intermediate estimations of OMI and TA scores Late Fusion Strategy The authors adopt a late fusion strategy that combines predictions from both encoders at the score level, rather than fusing raw features early in the network This approach allows each encoder pathway to independently model perceptual aspects aligned with its inductive bias before integrating their outputs into a unified prediction The intermediate predictions, representing each branch’s preliminary assessment of perceptual quality, are concatenated and passed through a lightweight linear fusion head to produce the final scores This fusion head adaptively weights contributions from each branch Ranking-Aware Composite Loss Functions Effective training requires an objective function that captures relative preferences in human judgments by combining contrastive ranking loss and truncated regression loss The contrastive ranking loss Lcon explicitly models the relative quality ranking relationships between samples by constraining the consistency between predicted score differences and true score differences of sample pairs The contrastive ranking loss function is calculated as Lcon = 1/N(N − 1) X N i=1 X N j=1 j̸=i max (0, dij) The truncated regression loss Ltrun can prevent the model from overfitting to noisy labels or exact rating values by ignoring errors below a certain threshold τ By integrating contrastive ranking loss with truncated regression loss, the model can learn the relative quality of samples while maintaining the stability of the training process and preventing overfitting The final total loss is defined as Lsum = β · Lcon + γ · Ltrun Experimental Validation Experiments on the MusicEval benchmark demonstrate that FUSEMOS outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin For OMI, U-MSE and SMSE are substantially reduced by 53.6% and 87.6%, respectively, with U-SRCC reaching 0.811 and S-SRCC of 0.977 For TA, USRCC improves from 0.495 to 0.644 and S-SRCC from 0.861 to 0.
Paper summary: Rosa: So we're looking at this paper called FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss. Basically, they're trying to figure out how to properly evaluate music that an AI creates from text prompts.
Dev: Right, so the core idea here is that existing methods only use one model called CLAP for both audio and text features to predict scores like overall musical impression or textual alignment. But CLAP has a problem because it can't really capture the fine details of music, like timbre or expressiveness, you know?
Taro: That sounds like a limitation when you're testing real-world scenarios where those subtle musical qualities matter for how something sounds.
Rosa: Exactly. So FUSEMOS proposes a dual-encoder fusion framework that combines CLAP with another model called MERT, which is self-supervised and trained on large music audio datasets. The goal is to get better at catching both the big picture semantic alignment and those detailed musical characteristics for more accurate scores.
Dev: The architecture they propose uses two branches, one from CLAP and one from MERT, which are then fused together in a late fusion strategy. They use CLAP to create embeddings that live in a shared space so you can compare the music embedding directly with the text embedding using a dot product for things like textual alignment.
Taro: So they're trying to use two different kinds of learned representations—one focused on text-audio similarity and one focused on broad music characteristics—to get a fuller picture.
Rosa: And then they combine those results by passing them through separate layers and then using a lightweight linear fusion head to weight their contributions for the final overall musical impression or textual alignment predictions.
Dev: That sounds like they are trying to let each part of the model handle its own perceptual aspect independently before merging them into one final score.
Taro: What about how they actually train this thing? Because getting those scores right needs a good training signal, not just a simple regression loss.
Rosa: That's where they introduce a novel ranking-aware composite loss function. They combine a contrastive ranking loss with a truncated regression loss to train the model.
Dev: The contrastive part of that loss is supposed to explicitly model the relative quality rankings between different samples by checking for consistency in predicted score differences versus the actual score differences of pairs, which they call Lcon.
Taro: And then they add this truncated regression loss, Ltrun, which ignores errors below a certain threshold tau to stop the model from overfitting to noisy labels or exact rating values.
Rosa: By combining these two losses into a total loss, Lsum equals beta times the contrastive ranking loss plus gamma times the truncated regression loss, they aim to learn relative quality while keeping training stable and preventing that over-fitting issue.
Dev: The experimental validation on the MusicEval benchmark shows that this approach outperforms existing state-of-the-art methods in both mean squared error and rank correlation metrics by a large margin. For example, for overall musical impression, they see reductions in U-MSE and SMSE of fifty-three point six percent and eighty-seven point six percent, respectively, while the U-SRCC gets to zero point eight one one and the S-SRCC reaches zero point nine seven seven >
Taro: So if we're talking about textual alignment, their USRCC improves from a score of zero point four nine five to zero point six four four, and the S-SRCC goes from zero point eight six one up to zero point eight <ref:2611.07562#pg3>
Conclusion: Rosa: So FUSEMOS is about improving how we judge music made by text to music AI using a dual encoder setup and a new loss function.
Dev: Yeah, that's right, it’s about combining two different ways of looking at the audio and text features to get better scores for overall impression and textual alignment.
Taro: It’s trying to fix the problem where those current methods miss fine details like timbre or expressiveness in the music.
Rosa: Right, so they used a dual encoder, one from CLAP and one from MERT, which is trained on a huge amount of music audio data. They fuse them at the score level, not by mixing the raw features early on.
Dev: That late fusion strategy lets each part model its specific perceptual aspect before they come together into a final prediction via a linear head. It sounds like a way to keep things stable during training when you have two very different feature sets.
Taro: I'm curious about that ranking-aware loss function they designed. They use contrastive ranking loss and truncated regression loss to train the model, which should help it learn the relative quality of music segments better than just predicting an exact score.
Rosa: Exactly, so one part learns how things rank against each other, and the other part keeps training steady by ignoring errors that are too small or noisy. It’s a way to make sure the training signal is really focused on what matters for human preference.
Dev: The results on MusicEval were pretty strong then, showing significant improvements in mean squared error and rank correlation metrics over the current methods.
Taro: Those improvements suggest that this fusion approach actually captures those musical characteristics they were missing before, making the AI's output feel more natural to human listeners.
Rosa: So FUSEMOS seems to be a solid step toward getting text-to-music generation evaluated in a way that actually matters for quality and musical nuance.
Dev: It’s definitely worth looking at how this dual encoder and ranking loss setup could apply to other multimodal tasks where you have audio and text inputs together.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G