Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

The gist: MixtureTT proposes a system for flexible per-stem timbre transfer directly from polyphonic mixtures, jointly transferring all stems to specified instruments through a shared diffusion process.

How it works

MixtureTT decomposes each musical signal into a time-varying content space and a timeinvariant timbre space, recombining them at inference through a joint latent diffusion process over N stems (N=4 for the SATB choral data used in our experiments). The central component is a Joint Stem Diffusion Transformer whose three-stage attention design balances per-stem independence with cross-stem coordination, allowing attention to propagate harmonic structure and timbral coherence directly during denoising.

The pipeline involves several key steps:

1. Audio Codec: A convolutional autoencoder following the RAVE design, pretrained on our dataset and frozen, maps raw waveforms into a compact latent sequence z ∈ R L×Dz (Page 3).

2. Content Extraction: The encoder of HT Demucs is adopted to obtain per-stem content vectors c(i) ∈ R L×Dc per target voice (Page 3).

3. Timbre Encoding and Disentanglement: Timbre references are encoded by the frozen codec and a timbre encoder into global embeddings τ(i) ∈ R Dτ (Page 3).

4. Joint Stem Diffusion Transformer: This denoises N noisy stem latents jointly through three stages (Intra-Stem → Cross-Stem → Refinement), conditioned on content and timbre via decoupled FiLM (Page 3).

Key Technical Components

The Joint Stem Diffusion Transformer utilizes a tokenization scheme where each stem latent zi ∈ R L×Dz is patchified, linearly projected to dimension D, and concatenated into h ∈ R NsL′×D (64 tokens for N=4, L′=16) before processing by a stack of transformer blocks (Page 3). The three-stage attention mechanism partitions the block stack into three stages: Stages A (×NA) and C (×NC) use intrastem attention, while Stage B (×NB) removes the mask to allow every token to attend across all stems, building clean per-voice representations before opening a dedicated channel for cross-stem coordination (Page 3).

Conditioning is achieved via decoupled FiLM pathways: each transformer block is conditioned on the diffusion timestep σ, the per-stem content c(i), and the per-stem timbre τ(i) through three independent FiLM pathways, which share no parameters to ensure that gradients to the timbre pathway remain well-conditioned even under strong content variation (Page 3).

Disentanglement Mechanisms

To enforce disentanglement, MixtureTT employs auxiliary losses:

1. A classifier Cζ: R L×Dc → R Dτ is trained to predict the timbre embedding from the content vector, c(i), minimizing Eq. 1 (Page 3).

2. A cross-stem term drives the four voice timbres toward mutual orthogonality within each mixture, defined by Lcrossdiv (Page 3).

3. A batch-variance term prevents the encoder from collapsing to an input-agnostic constant, defined by Lvardiv (Page 3).

Experimental Findings

MixtureTT outperforms strong single-instrument diffusion baselines on both objective and subjective metrics on the SATB choral dataset, demonstrating that dedicated multi-instrument modeling matches and exceeds separate-then-transfer pipelines even under a strictly harder input condition (Page 5). The controlled ablation against a matched single-stem variant shows that joint denoising is essential for capturing the cross-stem dependencies that mixture-level timbre transfer requires (Page 5). Subjective evaluation confirms this, showing MixtureTT obtains the highest MOS on all four axes, particularly for Inter-stem Coherence (IC), which measures the form of ensemble-level consistency that separate-then-transfer pipelines have no mechanism to produce (Page 5). Furthermore, scaling with pseudo-labeled data indicates that MixtureTT can be trained with limited or no paired stems, opening the door to scaling beyond datasets where ground-truth separation is available (Page 6).

Conclusion

This work confirms that cross-stem modeling is essential for mixture-level timbre transfer as the proposed joint setting consistently exceeds an equivalent single-stem ablation (Page 1). The paper presents MixtureTT, the first system capable of flexibly re-instrumenting individual voices within a mixture without explicit source separation or query audio (Page 6). This research initiates a reflection on treating the polyphonic mixture as the natural object of study in generative music modeling (Page 6).

--- Page 1 ---

Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

Leduo Chen, Junchuan Zhao†, Shengchen Li lec015@ucsd.edu, Junchuan@u.nus.edu, Shengchen.Li@xjtlu.edu.cn

Abstract

Timbre transfer aims to modify the timbral identity of a musical recording while preserving the original melody and rhythm [1]. While single-instrument timbre transfer has made substantial progress, existing approaches to multi-instrument settings rely on separate-then-transfer pipelines that propagate source separation artifacts and produce incoherent synthesized timbres across stems. This paper proposes MixtureTT, to the best of our knowledge the first system for flexible per-stem timbre transfer directly from a polyphonic mixture. Given a mixture and a separate timbre reference for each target voice, MixtureTT jointly transfers all stems to the specified instruments through a shared diffusion process. Modeling the dependencies across the perstem content and cross-stem harmonic, the proposed joint stem diffusion transformer eliminates cascaded separation error, reduces inference cost by a factor equal to the number of stems, and yields more coherent multi-stem outputs. Despite operating under a strictly harder input condition, evaluations on the SATB choral dataset show that MixtureTT outperforms singleinstrument baselines on both objective and subjective metrics demonstrating the necessity of dedicated multi-instrument timbre transfer over the naive separate-then-transfer pipelines. As a result, this work confirms that the cross-stem modeling is essential for mixture-level timbre transfer as the proposed joint setting consistently exceeds an equivalent single-stem ablation.

Audio samples are available at supporting webpage 1

Index Terms: timbre transfer, source separation

--- Page 2 ---

1. Introduction

Timbre transfer is a form of music style transfer in which the perceptual identity of an instrument is recast onto the musical content of another, while pitch, rhythm, and articulation are preserved. Modeling timbre is notoriously difficult: as the attribute that distinguishes instruments playing the same note at the same loudness and duration, it manifests as heterogeneous, sourcedependent patterns across time and frequency [1]. Most prior work on timbre transfer has focused on the single-instrument setting, where both the content source and the timbre reference are isolated monophonic recordings. Early systems based on autoregressive waveform models [2, 3] and adversarial translation [4, 5, 6] were succeeded by VAE-based methods [7, 8, 9] and, most recently, by diffusion-based approaches [10, 11, 12]. Multi-instrument timbre transfer remains comparatively underexplored: existing mixture-level systems either treat the full ensemble as a single timbral unit [14, 15], permitting only fixed ensemble-to-ensemble conversion, or require external persource query audio to enable source-level control [16]. To our knowledge, no existing method extracts per-stem content implicitly from a polyphonic mixture, assigns an independent target timbre to each voice, and performs all transfers jointly in a single pass. This gap matters because musicians rarely work with isolated stems, and a producer typically receives a stereo bounce rather than separated tracks. When the goal is to re-instrument a mixture itself, yet a practitioner aiming for per-stem transfer with today’s tools is forced into a separate-then-transfer pipeline: run a source separator to recover individual stems, then apply a single-instrument transfer model to each in turn. This workaround is a downstream consequence of the field’s single-instrument focus and carries three compounding drawbacks. Separation introduces modeling error that propagates into every transfer step; the diffusion model runs once per stem, scaling both training and inference cost with the number of voices; and because the per-stem runs proceed independently, neither the harmonic relationships binding the voices nor the acoustic consistency of a shared timbral space can be enforced. The ensemble is reconstructed stem by stem rather than as an integrated whole (Fig. 1). To fill this gap, this paper proposes MixtureTT, a system that performs per-stem timbre transfer through joint denoising of all stems within a shared diffusion trajectory. Given a polyphonic mixture and one timbre reference per target voice, MixtureTT generates every re-instrumented stem in parallel: each reverse-diffusion step updates all stems through one network call, amortizing the per-stem cost of separate-then-transfer.

arXiv:2605.09259v1 [cs.SD] 10 May 2026

--- Page 3 ---

3. Method

MixtureTT decompose each musical signal into a time-varying content space (melody, rhythm, articulation) and a timeinvariant timbre space (instrument identity), and recombine them at inference through a joint latent diffusion process over N stems (N=4 for the SATB choral data used in our experiments). Fig. 2 shows the full pipeline.

3.1. Audio Codec

Audio codecs are widely used in audio generation and conversion, as they map raw waveforms into compact latent representations [21, 25, 26, 27, 28]. Following [11], our audio codec is a convolutional autoencoder following the RAVE design [29], featuring the adversarial discriminator of [30].

1. Bold Header: Joint Stem Diffusion Transformer Architecture Improvement

The current three-stage attention design in Figure 2 explicitly separates per-stem independence from cross-stem coordination; improving Stage B to more explicitly enforce cross-stem coordination that ensemble coherence requires could further enhance the harmonic and timbral coherence across voices that independent passes lack.

2. Bold Header: Timbre Disentanglement Loss Refinement

The proposed loss function includes a cross-stem term, but increasing the weight or modifying its structure to better enforce mutual orthogonality within each mixture could lead to more distinct and perceptually accurate timbre transfers for each target voice, addressing the potential for timbre collapse.

3. Bold Header: Conditioning Pathway Robustness Enhancement

The decoupled FiLM pathways utilize three independent MLPs, but investigating a shared parameter mechanism or a gated structure that prevents weaker signals from being bottlenecked could ensure that the critical timbre pathway [is] well-conditioned even under strong content variation.

4. Bold Header: Content Disentanglement Pressure Tuning

The auxiliary classifier's role in enforcing disentanglement needs careful tuning; optimizing the loss function to better guide the content encoder to discard timbre information could prevent undesirable leakage between the per-stem content and timbre representations during training.

5. Bold Header: Scalability via Pseudo-Labeled Data Integration

The system should be optimized to leverage pseudo-stems from HT Demucs more effectively; this would allow MixtureTT to operate successfully in scenarios where paired supervision is strictly necessary is not available, enabling deployment on larger, unpaired datasets like CocoChorales.