Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network
summary
The gist
Estimating piano dynamic from audio is addressed by proposing an efficient multi-task network that jointly predicts dynamic levels, change points, beats…
In short
A multi-task network jointly predicts piano dynamics, change points, beats, and downbeats from audio using a shared latent representation. It uses Bark-scale specific loudness as input to achieve state-of-the-art results in dynamics and change point estimation while remaining highly parameter efficient.
Key concepts
- Bark-scale specific loudness (BSSL)
- This is the input feature used for audio analysis, derived from a psychoacoustic model of human loudness perception across different frequency bands. It is chosen because it helps improve speaker recognition under short time constraints and has shown success in previous piano dynamics research.
- Multi-gate Mixture-of-Experts (MMoE)
- This is a task-aware decoder layer that processes the shared representation. It contains eight lightweight experts, each being a temporal convolutional block, and four task-specific gates that decide how much influence each expert should have for the four different prediction tasks.
- Composite Loss Function (LMTL)
- The training uses a combined loss function summing four parts: LDyn (dynamics), LCPt (change points), LBeat (beats), and LDbt (downbeats). This ensures the model learns to accurately predict all musical structure elements simultaneously, using shift-tolerant binary cross-entropy for targets like beats.
Terminology used across episodes
This episode discusses
- Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network · Paper Radio
The paper
Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network · Read on arXiv
University of Western Australia · University of New South Wales
Estimating piano dynamic from audio recordings is a fundamental challenge in computational music analysis. In this paper, we propose an efficient multi-task network that jointly predicts dynamic levels, change points, beats, and downbeats from a shared latent representation. These four targets form the metrical structure of dynamics in the music score. Inspired by recent vocal dynamic research, we use a multi-scale network as the backbone, which takes Bark-scale specific loudness as the input feature. Compared to log-Mel as input, this reduces model size from 14.7 M to 0.5 M, enabling long sequential input. We use a 60-second audio length in audio segmentation, which doubled the length of beat tracking commonly used. Evaluated on the public MazurkaBL dataset, our model achieves state-of-the-art results across all tasks. This work sets a new benchmark for piano dynamic estimation and delivers a powerful and compact tool, paving the way for large-scale, resource-efficient analysis of musical expression.
DOI: 10.1109/ICASSP55912.2026.11460839
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network".
Title and authors: Rosa: So we're looking at this paper today called "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network." Basically, they’ve built an efficient network that looks at audio and tries to figure out four things all at once: the dynamic levels, where the changes in dynamics happen, the beats, and the downbeats.
Dev: It’s interesting because these four things together form what we consider the metrical structure of dynamics in a music score. Rosa, what caught your eye about this approach?
Rosa: Well, they use something called Bark-scale specific loudness as their input feature instead of the log-Mel spectrogram we often see in other work. They say this choice helps improve speaker recognition under short time constraints and has worked well before for piano dynamics research.
Taro: I wonder if using that specific loudness measure is a solid foundation when you're trying to capture the nuance of piano playing compared to a standard spectrogram input.
Rosa: Yeah, they mention that this choice reduces the model size from fourteen point seven million parameters down to just half a million parameters, which opens up possibilities for processing much longer audio sequences.
Dev: Half a million parameters is pretty compact for something that handles such complex tasks like beat tracking and dynamic estimation. How do they manage that complexity? What's the architecture looking like?
Rosa: They use a multi-scale network as their backbone, but it’s not just one network. It has three parallel branches operating at different temporal resolutions: T, T over s, and T over s squared.
Taro: So they’re looking at the audio structure from different time scales simultaneously to get a richer representation before feeding it into the main prediction part?
Dev: Exactly. And then they use this latent sequence, which is shape T times eight features, and feed that into a Multi-gate Mixture-of-Experts layer. This layer has eight shared experts, each being a lightweight temporal convolutional block.
Rosa: What’s the role of those four gates in that Mixture-of-Experts? That sounds like the part that makes it task aware.
Dev: The four gates are what determine how to weigh the contributions of those eight experts for each of their four tasks, which is dynamic levels, change points, beats, and downbeats. They're essentially steering the model to focus on what it needs to predict at any given moment.
Taro: That task awareness sounds important when dealing with things like beat tracking where the context changes rapidly depending on where you are in the musical phrase.
Rosa: Right. And for training, they use a composite loss function called LMTL, which is just the sum of four separate losses: one for dynamic levels, one for change points, one for beats, and one for downbeats.
Dev: For those binary targets like change points and beats, they are using something called shift-tolerant weighted binary cross-entropy to deal with the fact that the targets are sparse and also account for timing imprecision in the annotations.
Title and authors: Taro: That sounds like a practical way to handle real-world data where perfect alignment isn't always possible.
Rosa: And for the dynamic levels, they use a standard cross-entropy loss but they mask it by using the ground truth beat positions to enforce a data-driven prior on what the dynamics should look like.
Dev: It’s a clever way to guide the dynamic prediction using that underlying metrical information, which is exactly what this paper is trying to achieve by jointly predicting all four targets.
Taro: So they are not just predicting dynamics in isolation; they are tying them directly into the underlying timing of the music structure through these loss functions.
Rosa: And they’ve shown state-of-the-art results across all those tasks on the public MazurkaBL dataset, which is a good benchmark for this kind of piano music analysis work.
Dev: The efficiency gain is also pretty significant, with just half a million parameters compared to four times the size of a single task network that might have had around zero point four million parameters. That parameter efficiency is crucial for making these kinds of systems practical in real-world applications.
Taro: I noticed they mention that the extended sixty-second input context provided a significant advantage for those dynamics-related tasks, which makes sense when you’re trying to capture longer musical phrases.
Rosa: That's an interesting finding from their ablation study. It shows that adding more time—that extra sixty seconds—really helps the model capture better dynamic information and change points.
Dev: Disabling the Mixture-of-Experts layer resulted in a noticeable drop in performance, which proves that this task-aware feature processing module is genuinely important for getting those complex predictions right.
Taro: So they’re telling us that for capturing the full picture of piano expression and timing, you really need that layered approach to features and the gated experts.
Rosa: It sounds like a really practical setup for someone trying to build something robust enough to handle real-world audio inputs without needing massive computational resources.
Dev: Before we wrap up, we’ve seen how this multi-task network handles the complexity of piano dynamics and structure prediction by looking at both short and long time scales simultaneously.
Taro: Yeah, it shows that when you combine a multi-scale backbone with task-aware gating, you can get very good results across all those musical structure elements.
Rosa: Exactly. So, the next step for this kind of work is combining these predictions with score transcription systems to actually produce music scores that have dynamic markings directly from the performance audio.
Dev: That would be a big leap in terms of how we can turn raw audio into structured musical data, and developing good evaluation methods for those outputs is going to be a new challenge for us.
Taro: I think it’s exciting because it moves beyond just analysis and starts creating something tangible, like markings on a score.
Rosa: Indeed. So that's what we have today on the paper "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network."
The paper's summary: Rosa: So, to wrap up what we just talked about, this paper is basically about building one network that can figure out four things from piano audio at the same time: the dynamic levels, where those changes in volume happen, the beats, and where those downbeats fall.
Dev: It’s not just predicting dynamics separately; they’re linking all these elements together through a shared internal representation. That means if it gets one part right—like detecting a beat—that information helps it predict the dynamic level better.
Rosa: Exactly, and the big deal is how they do it. They use this multi-scale network structure, looking at audio at three different time resolutions, and then they use a Mixture-of-Experts layer that has four gates controlling which expert handles which of those four tasks.
Dev: That gating mechanism is what makes it task aware; it allows the model to focus its computational effort on the right thing for each prediction. And they’re training this with these four different loss functions added up together, one for each task.
Rosa: They also used a specific input feature called Bark-scale specific loudness instead of standard log-Mel spectrograms, and that choice helped them get a model down to just half a million parameters while still getting state-of-the-art results on the public MazurkaBL dataset.
Dev: That parameter efficiency is huge, Rosa. Having only five hundred thousand parameters for this kind of complex joint prediction means we can run these kinds of analyses on much longer audio files than before, which is a major win for real-time applications.
Rosa: And the ablation study showed that adding an extended sixty-second context gave the best results for the dynamics, which tells us that capturing a longer musical phrase really helps when you’re trying to figure out how expression changes over time.
Dev: It also showed that without that Mixture-of-Experts layer, the model's performance dropped significantly, meaning those task-specific gates are really important for making sense of the data.
Rosa: So what this means for us is that we’re moving toward AI systems where a single audio input can give us a much richer understanding of both how the music sounds and its underlying rhythmic structure.
Dev: It opens up possibilities for things like automatically adding dynamic markings to existing scores without needing human annotators to do all that tedious work.
Rosa: And that’s just the start, because if we can get this level of accuracy on piano dynamics, imagine what we could do when we combine it with systems that transcribe music into a formal score.
The paper's improvements: Taro: So, we've seen how they get all four predictions—dynamics, change points, beats, and downbeats—working together in one network on the MazurkaBL dataset. Now let’s talk about what they suggest to make it even better.
Rosa: They point out that using that extended sixty-second input context gave them a real advantage for the dynamics part of the prediction, so they recommend keeping that longer sequence for those tasks.
Dev: That makes sense; having more time to look at helps you see how a dynamic shift develops over a longer musical idea, not just in one tiny frame. It's about capturing the temporal context better.
Taro: And they also showed that ditching the Mixture-of-Experts layer hurts performance, so they suggest keeping that module in place because it's really good at handling those different tasks at once.
Rosa: They basically say that this combination of a multi-scale backbone and task-aware gating is what makes the system robust enough to handle both the timing and the expression simultaneously.
Dev: The main takeaway here is parameter efficiency; they managed to keep the model small, only using half a million parameters, while still hitting state-of-the-art performance on dynamics and change points.
Taro: That's pretty impressive when you look at how complex piano playing is; being that efficient means we can deploy this kind of analysis on smaller devices without needing massive computational power.
Rosa: Exactly, and the authors also suggest that combining this joint prediction with score-level transcription systems would be the next logical step.
Dev: That’s where things get really interesting for me because if you can get audio to predict dynamics and structure so accurately, you could create a system that generates music scores directly marked with those dynamic markings.
Conclusion: Rosa: So, to wrap up on this paper, "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network," we’ve seen how this AI model can figure out dynamic levels, change points, beats, and downbeats all at once from piano audio.
Dev: It’s really about creating one unified system instead of four separate models because that shared internal representation is what allows the prediction of one task to inform the others.
Taro: I just think the parameter efficiency they achieved is pretty smart; being that small with such complex joint prediction capability shows a lot of clever design work.
Rosa: They did use Bark-scale specific loudness as input, which helped them keep the model size down to half a million parameters instead of those larger versions we've seen before.
Dev: That small size means it could actually run in real-time loops on less powerful hardware, which is important for deployment when you need low latency and a solid safety case.
Taro: It really shows that autonomy research isn't just about building big models; it’s about making those models compact enough to handle the messy, unscripted world where things go wrong.
Rosa: And they pointed out that the extended sixty-second input context is crucial for getting good dynamic estimation, so we should keep that longer sequence in mind for future work.
Dev: Yeah, it’s a practical limitation; you can't just throw away context when you're dealing with piano music where the dynamics can change slowly over a whole phrase.
Taro: So the main implication is that this type of model could be used to automatically add musical markings to existing scores, which would be huge for how we interact with music data.
Rosa: It’s certainly a possibility; imagine an AI system taking raw performance audio and outputting a score with accurate dynamic markings right on top.
Dev: That moves us from just analyzing the sound to actually creating structured musical assets, which opens up some interesting new avenues for control systems.
Rosa: So, that’s what we covered today on "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network."
Dev: Yeah, it was a solid look at how multi-task learning can be made efficient by using task-aware gating and input features like Bark loudness.
Taro: I’m just interested to see how they handle the next step—integrating this prediction into a full music transcription pipeline.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G