Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network".
Title and authors: Rosa: So we're looking at this paper today called "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network." Basically, they’ve built an efficient network that looks at audio and tries to figure out four things all at once: the dynamic levels, where the changes in dynamics happen, the beats, and the downbeats.
Dev: It’s interesting because these four things together form what we consider the metrical structure of dynamics in a music score. Rosa, what caught your eye about this approach?
Rosa: Well, they use something called Bark-scale specific loudness as their input feature instead of the log-Mel spectrogram we often see in other work. They say this choice helps improve speaker recognition under short time constraints and has worked well before for piano dynamics research.
Taro: I wonder if using that specific loudness measure is a solid foundation when you're trying to capture the nuance of piano playing compared to a standard spectrogram input.
Rosa: Yeah, they mention that this choice reduces the model size from fourteen point seven million parameters down to just half a million parameters, which opens up possibilities for processing much longer audio sequences.
Dev: Half a million parameters is pretty compact for something that handles such complex tasks like beat tracking and dynamic estimation. How do they manage that complexity? What's the architecture looking like?
Rosa: They use a multi-scale network as their backbone, but it’s not just one network. It has three parallel branches operating at different temporal resolutions: T, T over s, and T over s squared.
Taro: So they’re looking at the audio structure from different time scales simultaneously to get a richer representation before feeding it into the main prediction part?
Dev: Exactly. And then they use this latent sequence, which is shape T times eight features, and feed that into a Multi-gate Mixture-of-Experts layer. This layer has eight shared experts, each being a lightweight temporal convolutional block.
Rosa: What’s the role of those four gates in that Mixture-of-Experts? That sounds like the part that makes it task aware.
Dev: The four gates are what determine how to weigh the contributions of those eight experts for each of their four tasks, which is dynamic levels, change points, beats, and downbeats. They're essentially steering the model to focus on what it needs to predict at any given moment.
Taro: That task awareness sounds important when dealing with things like beat tracking where the context changes rapidly depending on where you are in the musical phrase.
Rosa: Right. And for training, they use a composite loss function called LMTL, which is just the sum of four separate losses: one for dynamic levels, one for change points, one for beats, and one for downbeats.
Dev: For those binary targets like change points and beats, they are using something called shift-tolerant weighted binary cross-entropy to deal with the fact that the targets are sparse and also account for timing imprecision in the annotations.
Title and authors: Taro: That sounds like a practical way to handle real-world data where perfect alignment isn't always possible.
Rosa: And for the dynamic levels, they use a standard cross-entropy loss but they mask it by using the ground truth beat positions to enforce a data-driven prior on what the dynamics should look like.
Dev: It’s a clever way to guide the dynamic prediction using that underlying metrical information, which is exactly what this paper is trying to achieve by jointly predicting all four targets.
Taro: So they are not just predicting dynamics in isolation; they are tying them directly into the underlying timing of the music structure through these loss functions.
Rosa: And they’ve shown state-of-the-art results across all those tasks on the public MazurkaBL dataset, which is a good benchmark for this kind of piano music analysis work.
Dev: The efficiency gain is also pretty significant, with just half a million parameters compared to four times the size of a single task network that might have had around zero point four million parameters. That parameter efficiency is crucial for making these kinds of systems practical in real-world applications.
Taro: I noticed they mention that the extended sixty-second input context provided a significant advantage for those dynamics-related tasks, which makes sense when you’re trying to capture longer musical phrases.
Rosa: That's an interesting finding from their ablation study. It shows that adding more time—that extra sixty seconds—really helps the model capture better dynamic information and change points.
Dev: Disabling the Mixture-of-Experts layer resulted in a noticeable drop in performance, which proves that this task-aware feature processing module is genuinely important for getting those complex predictions right.
Taro: So they’re telling us that for capturing the full picture of piano expression and timing, you really need that layered approach to features and the gated experts.
Rosa: It sounds like a really practical setup for someone trying to build something robust enough to handle real-world audio inputs without needing massive computational resources.
Dev: Before we wrap up, we’ve seen how this multi-task network handles the complexity of piano dynamics and structure prediction by looking at both short and long time scales simultaneously.
Taro: Yeah, it shows that when you combine a multi-scale backbone with task-aware gating, you can get very good results across all those musical structure elements.
Rosa: Exactly. So, the next step for this kind of work is combining these predictions with score transcription systems to actually produce music scores that have dynamic markings directly from the performance audio.
Dev: That would be a big leap in terms of how we can turn raw audio into structured musical data, and developing good evaluation methods for those outputs is going to be a new challenge for us.
Taro: I think it’s exciting because it moves beyond just analysis and starts creating something tangible, like markings on a score.
Rosa: Indeed. So that's what we have today on the paper "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network."
The paper's summary: Rosa: So, to wrap up what we just talked about, this paper is basically about building one network that can figure out four things from piano audio at the same time: the dynamic levels, where those changes in volume happen, the beats, and where those downbeats fall.
Dev: It’s not just predicting dynamics separately; they’re linking all these elements together through a shared internal representation. That means if it gets one part right—like detecting a beat—that information helps it predict the dynamic level better.
Rosa: Exactly, and the big deal is how they do it. They use this multi-scale network structure, looking at audio at three different time resolutions, and then they use a Mixture-of-Experts layer that has four gates controlling which expert handles which of those four tasks.
Dev: That gating mechanism is what makes it task aware; it allows the model to focus its computational effort on the right thing for each prediction. And they’re training this with these four different loss functions added up together, one for each task.
Rosa: They also used a specific input feature called Bark-scale specific loudness instead of standard log-Mel spectrograms, and that choice helped them get a model down to just half a million parameters while still getting state-of-the-art results on the public MazurkaBL dataset.
Dev: That parameter efficiency is huge, Rosa. Having only five hundred thousand parameters for this kind of complex joint prediction means we can run these kinds of analyses on much longer audio files than before, which is a major win for real-time applications.
Rosa: And the ablation study showed that adding an extended sixty-second context gave the best results for the dynamics, which tells us that capturing a longer musical phrase really helps when you’re trying to figure out how expression changes over time.
Dev: It also showed that without that Mixture-of-Experts layer, the model's performance dropped significantly, meaning those task-specific gates are really important for making sense of the data.
Rosa: So what this means for us is that we’re moving toward AI systems where a single audio input can give us a much richer understanding of both how the music sounds and its underlying rhythmic structure.
Dev: It opens up possibilities for things like automatically adding dynamic markings to existing scores without needing human annotators to do all that tedious work.
Rosa: And that’s just the start, because if we can get this level of accuracy on piano dynamics, imagine what we could do when we combine it with systems that transcribe music into a formal score.
The paper's improvements: Taro: So, we've seen how they get all four predictions—dynamics, change points, beats, and downbeats—working together in one network on the MazurkaBL dataset. Now let’s talk about what they suggest to make it even better.
Rosa: They point out that using that extended sixty-second input context gave them a real advantage for the dynamics part of the prediction, so they recommend keeping that longer sequence for those tasks.
Dev: That makes sense; having more time to look at helps you see how a dynamic shift develops over a longer musical idea, not just in one tiny frame. It's about capturing the temporal context better.
Taro: And they also showed that ditching the Mixture-of-Experts layer hurts performance, so they suggest keeping that module in place because it's really good at handling those different tasks at once.
Rosa: They basically say that this combination of a multi-scale backbone and task-aware gating is what makes the system robust enough to handle both the timing and the expression simultaneously.
Dev: The main takeaway here is parameter efficiency; they managed to keep the model small, only using half a million parameters, while still hitting state-of-the-art performance on dynamics and change points.
Taro: That's pretty impressive when you look at how complex piano playing is; being that efficient means we can deploy this kind of analysis on smaller devices without needing massive computational power.
Rosa: Exactly, and the authors also suggest that combining this joint prediction with score-level transcription systems would be the next logical step.
Dev: That’s where things get really interesting for me because if you can get audio to predict dynamics and structure so accurately, you could create a system that generates music scores directly marked with those dynamic markings.
Conclusion: Rosa: So, to wrap up on this paper, "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network," we’ve seen how this AI model can figure out dynamic levels, change points, beats, and downbeats all at once from piano audio.
Dev: It’s really about creating one unified system instead of four separate models because that shared internal representation is what allows the prediction of one task to inform the others.
Taro: I just think the parameter efficiency they achieved is pretty smart; being that small with such complex joint prediction capability shows a lot of clever design work.
Rosa: They did use Bark-scale specific loudness as input, which helped them keep the model size down to half a million parameters instead of those larger versions we've seen before.
Dev: That small size means it could actually run in real-time loops on less powerful hardware, which is important for deployment when you need low latency and a solid safety case.
Taro: It really shows that autonomy research isn't just about building big models; it’s about making those models compact enough to handle the messy, unscripted world where things go wrong.
Rosa: And they pointed out that the extended sixty-second input context is crucial for getting good dynamic estimation, so we should keep that longer sequence in mind for future work.
Dev: Yeah, it’s a practical limitation; you can't just throw away context when you're dealing with piano music where the dynamics can change slowly over a whole phrase.
Taro: So the main implication is that this type of model could be used to automatically add musical markings to existing scores, which would be huge for how we interact with music data.
Rosa: It’s certainly a possibility; imagine an AI system taking raw performance audio and outputting a score with accurate dynamic markings right on top.
Dev: That moves us from just analyzing the sound to actually creating structured musical assets, which opens up some interesting new avenues for control systems.
Rosa: So, that’s what we covered today on "Joint Estimation of Piano Dynamics and Metrical Structure with a Multi-task Multi-Scale Network."
Dev: Yeah, it was a solid look at how multi-task learning can be made efficient by using task-aware gating and input features like Bark loudness.
Taro: I’m just interested to see how they handle the next step—integrating this prediction into a full music transcription pipeline.
University of Western Australia · University of New South Wales
eess.AS, cs.LG, cs.SD
Submitted: 2025-10-21
Updated: 2026-02-03
Comments: Accepted to ICASSP2026 conference
Journal ref: Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP 2026), pp. 14607-14611, 2026
DOI: 10.1109/ICASSP55912.2026.11460839
Code: https://github.com/zhanh-he/piano-dynamic-estimation
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 81/100
The gist: Estimating piano dynamic from audio is addressed by proposing an efficient multi-task network that jointly predicts dynamic levels, change points, beats…
Key concepts
- Bark-scale specific loudness (BSSL)
- This is the input feature used for audio analysis, derived from a psychoacoustic model of human loudness perception across different frequency bands. It is chosen because it helps improve speaker recognition under short time constraints and has shown success in previous piano dynamics research.
- Multi-gate Mixture-of-Experts (MMoE)
- This is a task-aware decoder layer that processes the shared representation. It contains eight lightweight experts, each being a temporal convolutional block, and four task-specific gates that decide how much influence each expert should have for the four different prediction tasks.
- Composite Loss Function (LMTL)
- The training uses a combined loss function summing four parts: LDyn (dynamics), LCPt (change points), LBeat (beats), and LDbt (downbeats). This ensures the model learns to accurately predict all musical structure elements simultaneously, using shift-tolerant binary cross-entropy for targets like beats.
Terminology
Summary
Estimating piano dynamic from audio is addressed by proposing an efficient multi-task network that jointly predicts dynamic levels, change points, beats, and downbeats from a shared latent representation The model achieves state-of-the-art results across all tasks on the public MazurkaBL dataset
Input Feature Selection
The paper utilizes Bark-scale specific loudness (BSSL) as the input feature, which is derived from a psychoacoustic model of human loudness perception across critical bands This choice is motivated by research showing that BSSL can improve speaker recognition under short-duration constraints and has shown success in previous piano dynamics research The feature extraction pipeline involves converting audio to mono, peak-normalizing it to −1 dBFS, and resampling the audio to 22.05 kHz before computing a Short-Time Fourier Transform (STFT) The resulting BSSL matrix X ∈ R 22×T is then input to the model
Model Architecture Design
The proposed model architecture adapts a multiscale network as a shared encoder to address divergent acoustic requirements for dynamic estimation versus beat tracking The encoder operates on three parallel branches at different temporal resolutions, corresponding to lengths of T, T /s, and T /s2 A Multi-gate Mixture-of-Experts (MMoE) layer acts as a task-aware decoder to process the shared representation This module contains eight shared experts, each being a lightweight temporal convolutional block, and four task-specific gates that determine how to weigh the contributions of these experts for each of the four tasks
Training Objectives and Loss Function
The training involves a composite loss function, LMTL, which is defined as the sum of four task-specific losses: LDyn + LCPt + LBeat + LDbt For binary targets like change points, beats, and downbeats, shift-tolerant weighted binary cross-entropy is employed to counteract target sparsity and accommodate annotation timing imprecision For the multiclass target of dynamics, a standard crossentropy loss is used but masked by ground-truth beat positions to enforce a data-driven prior
Performance and Efficiency Gains
The proposed multi-task model achieves state-of-the-art performance in dynamics and change point estimation, while performing competitively on the remaining tasks The model is highly parameter-efficient, utilizing only 0.5 M parameters compared to 4× the single-task network's 0.4 M parameters Furthermore, BSSL features are optimal for dynamics, change points, and beats estimation, whereas log-Mel features are preferable for downbeat tracking The extended 60-second input context provides a significant advantage in dynamics-related tasks
Ablation Study Insights
An ablation study revealed that the extended 60-second input context provided a significant advantage in dynamics-related tasks Disabling the MMoE module resulted in a performance drop, indicating its importance for task-aware feature processing The extended 60-second input context proved beneficial for dynamics-related tasks This demonstrates not only the model’s practical utility but also its significant potential for broader applications
Conclusion and Future Work
In conclusion, the paper proposes a compact multi-task, multi-scale network that jointly estimates piano dynamics, change points, beats, and downbeats directly from audio The model achieves state-of-the-art results for dynamics and change point detection while demonstrating competitive performance in beat and downbeat tracking Future work will focus on combining our proposed model with score-level piano transcription systems Such an end-to-end pipeline could produce music scores with dynamic markings from the performance audio The development of appropriate evaluation methods for such comprehensive outputs presents a new challenge.
References
[3] Huan Zhang, Vincent K.M. Cheung, Hayato Nishioka, Simon Dixon, and Shinichi Furuya, “Llaqo: Towards a query-based coach in expressive music performance assessment,” in Proc. p. 4 [3] Huan Zhang, Vincent K. Cheung, Hayato Nishioka, Simon
Dixon, and Shinichi Furuya, “Llaqo: Towards a query-based
coach in expressive music performance assessment,” in Proc. [3] Huan Zhang, Vincent K. Cheung, Hayato Nishioka, Simon
Dixon, and Shinichi Furuya, “Llaqo: Towards a query-based
coach in expressive music performance assessment,” in Proc. Cheung, Hayato Nishioka, Simon
Dixon, and Shinichi Furuya, “Llaqo: Towards a query-based
coach in expressive music performance assessment,” in Proc. ICASSP, 2025, pp.Where the paper says it
Where the paper says it
Where the paper says it
[6] Zhanhong He, David Cooper, Defeng Huang, and Roberto Togneri, “Filling midi velocity using u-net image colorizer,” in Proc. 17th Int. Symp. Computer Music Multidisciplinary Research, 2025, pp. p. 4 [6] Zhanhong He, David Cooper, Defeng Huang, and Roberto
Togneri, “Filling midi velocity using u-net image colorizer,”
in Proc. Computer Music Multidisciplinary Research, 2025, pp.Where the paper says it
[18] Dichucheng Li, Mingjin Che, Wenwu Meng, Yulun Wu, Yi Yu, Fan Xia, and Wei Li, “Frame-level multi-label playing technique detection using multi-scale network and self-attention mechanism,” in Proc. ICASSP, 2023, pp. p. 4 [18] Dichucheng Li, Mingjin Che, Wenwu Meng, Yulun Wu, Yi Yu,
Fan Xia, and Wei Li, “Frame-level multi-label playing technique detection using multi-scale network and self-attention
mechanism,” in Proc. ICASSP, 2023, pp.Where the paper says it
[19] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in Proc. p. 4 [19] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and
Ed H. Chi, “Modeling task relationships in multi-task learning
with multi-gate mixture-of-experts,” in Proc. [19] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and
Ed H. Chi, “Modeling task relationships in multi-task learning
with multi-gate mixture-of-experts,” in Proc. Chi, “Modeling task relationships in multi-task learning
with multi-gate mixture-of-experts,” in Proc. ACM SIGKDD
Int. Chi, “Modeling task relationships in multi-task learning
with multi-gate mixture-of-experts,” in Proc. ACM SIGKDD
Int. for Music Notation and Rep. , 2018, pp. for Music Notation and Rep. , 2018, pp.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[26] Francesco Foscarin, Jan Schluter, and Gerhard Widmer, “Beat this! accurate beat tracking without dbn postprocessing,” in Proc. p. 4 [26] Francesco Foscarin, Jan Schluter, and Gerhard Widmer, “Beat ¨
this! accurate beat tracking without dbn postprocessing,” in
Proc. [26] Francesco Foscarin, Jan Schluter, and Gerhard Widmer, “Beat ¨
this! accurate beat tracking without dbn postprocessing,” in
Proc. [26] Francesco Foscarin, Jan Schluter, and Gerhard Widmer, “Beat ¨
this! accurate beat tracking without dbn postprocessing,” in
Proc. [26] Francesco Foscarin, Jan Schluter, and Gerhard Widmer, “Beat ¨
this! accurate beat tracking without dbn postprocessing,” in
Proc. [14] Hyon Kim and Xavier Serra, “A method for midi velocity estimation for piano performance by a u-net with attention and
film,” in Proc. ISMIR, 2024, pp. [14] Hyon Kim and Xavier Serra, “A method for midi velocity estimation for piano performance by a u-net with attention and
film,” in Proc. ISMIR, 2024, pp. [14] Hyon Kim and Xavier Serra, “A method for midi velocity estimation for piano performance by a u-net with attention and
film,” in Proc. ISMIR, 2024, pp.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[31] Francesco Foscarin, Andrew McLeod, Philippe Rigaux, Florent Jacquemard, and Masahiko Sakai, “ASAP: a dataset of aligned scores and performances for piano transcription,” in Proc. ISMIR, 2020, pp. p. 4 Davies, “Deconstruct, anal- ¨
yse, reconstruct: How to improve tempo, beat, and downbeat
estimation,” in Proc. ISMIR, 2020, pp. Davies, “Deconstruct, anal- ¨
yse, reconstruct: How to improve tempo, beat, and downbeat
estimation,” in Proc. ISMIR, 2020, pp. Davies, “Deconstruct, anal- ¨
yse, reconstruct: How to improve tempo, beat, and downbeat
estimation,” in Proc. ISMIR, 2020, pp.Where the paper says it
Where the paper says it
Where the paper says it
[28] Katerina Kosta, Computational Modelling and Quantitative Analysis of Dynamics in Performed Music, Ph.D. thesis, Centre for Digital Music, Queen Mary University of London, 2017
[30] Curtis Hawthorne et al., “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in Proc. p. 4 [30] Curtis Hawthorne et al. , “Enabling factorized piano music
modeling and generation with the MAESTRO dataset,” in
Proc. [30] Curtis Hawthorne et al. , “Enabling factorized piano music
modeling and generation with the MAESTRO dataset,” in
Proc. [30] Curtis Hawthorne et al. , “Enabling factorized piano music
modeling and generation with the MAESTRO dataset,” in
Proc.Where the paper says it
Where the paper says it
Where the paper says it
References List
[1] Carlos E Cancino-Chacon, Maarten Grachten, Werner Goebl, ´ and Gerhard Widmer, “Computational models of expressive music performance: A comprehensive and critical review,” Frontiers in Digital Humanities, vol. 5, pp. 25, 2018
[2] Jyoti Narang, Marius Miron, Ajay Srinivasamurthy, and Xavier Serra, “Analysis of musical dynamics in vocal performances using loudness measures,” in Proc. Int. Conf. Digital Audio Effects (DAFx), Vienna, Austria, 2022, pp. p. 4 [2] Jyoti Narang, Marius Miron, Ajay Srinivasamurthy, and Xavier
Serra, “Analysis of musical dynamics in vocal performances
using loudness measures,” in Proc. Digital Audio
Effects (DAFx), Vienna, Austria, 2022, pp.Where the paper says it
[4] Eun Ji Park, “Music dynamics visualization for music practice and education,” Multimedia Tools and Applications, vol. 84, pp. 36145–36161, 2025
[5] Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J. Bryan, “Music controlnet: Multiple time-varying controls for music generation,” IEEE/ACM Trans. p. 4 [5] Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and
Nicholas J. Bryan, “Music controlnet: Multiple time-varying
controls for music generation,” IEEE/ACM Trans. Bryan, “Music controlnet: Multiple time-varying
controls for music generation,” IEEE/ACM Trans. Audio,
Speech, Lang.Where the paper says it
Where the paper says it
[7] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew, “Practical implications of dynamic markings in the score: Is piano always piano? p. 4 [7] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew, “Practical implications of dynamic markings in the score: Is piano
always piano? Bandtlow, and Elaine Chew, “Practical implications of dynamic markings in the score: Is piano
always piano? ,” in AES 53rd International Conference: Semantic Audio, 2014, pp.Where the paper says it
Where the paper says it
[8] Gabriel Jones and Anders Friberg, “Probing the underlying principles of dynamics in piano performances using a modelling approach,” Frontiers in Psychology, vol. 14, pp. 1269715, 2023
[9] Katerina Kosta, Rafael Ramirez, Oscar F. Bandtlow, and Elaine Chew, “Mapping between dynamic markings and performed loudness: A machine learning approach,” Journal of Mathematics and Music, vol. p. 4 [9] Katerina Kosta, Rafael Ramirez, Oscar F. Bandtlow, and Elaine
Chew, “Mapping between dynamic markings and performed
loudness: A machine learning approach,” Journal of Mathematics and Music, vol.Where the paper says it
[10] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew, “MazurkaBL: score-aligned loudness, beat, and expressive markings data for 2000 chopin mazurka recordings,” in Proc. p. 4 [10] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew,
“MazurkaBL: score-aligned loudness, beat, and expressive
markings data for 2000 chopin mazurka recordings,” in Proc. Bandtlow, and Elaine Chew,
“MazurkaBL: score-aligned loudness, beat, and expressive
markings data for 2000 chopin mazurka recordings,” in Proc. for Music Notation and Rep.Where the paper says it
Where the paper says it
[11] Sam van Herwaarden, Maarten Grachten, and W. Bas de Haas, “Predicting expressive dynamics in piano performances using neural networks,” in Proc. p. 4 [11] Sam van Herwaarden, Maarten Grachten, and W. Bas de Haas,
“Predicting expressive dynamics in piano performances using
neural networks,” in Proc. Bas de Haas,
“Predicting expressive dynamics in piano performances using
neural networks,” in Proc. ISMIR, 2014, pp.Where the paper says it
Where the paper says it
[12] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Sageev Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck, “Onsets and frames: Dual-objective piano transcription,” in Proc. ISMIR, 2018, pp. p. 4 [12] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon,
Sageev Huang, Sander Dieleman, Erich Elsen, Jesse Engel,
and Douglas Eck, “Onsets and frames: Dual-objective piano
transcription,” in Proc. ISMIR, 2018, pp.Where the paper says it
[13] Hyon Kim, Marius Miron, and Xavier Serra, “Score-informed midi velocity estimation for piano performance by film conditioning,” in Proc. of the Sound and Music Computing Conf. p. 4 [13] Hyon Kim, Marius Miron, and Xavier Serra, “Score-informed
midi velocity estimation for piano performance by film conditioning,” in Proc. of the Sound and Music Computing Conf.Where the paper says it
[14] Hyon Kim and Xavier Serra, “A method for midi velocity estimation for piano performance by a u-net with attention and film,” in Proc. ISMIR, 2024, pp. 304–310
[15] Axel Berndt and Tilo Hahnel, “Modelling musical dynamics,” ¨ in Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound, 2010, pp. 1–8
[21] Eberhard Zwicker and Hugo Fastl, Psychoacoustics: Facts and Models, vol. 22 of Springer Series in Information Sciences, Springer, Berlin, 2nd, updated edition, 1999
[23] Roberto San Millan-Castillo, Eduardo Latorre-Iglesias, Mar- ´ tin Glesser, Salome Wanty, Daniel Jim ´ enez-Caminero, and ´ Jose Mar ´ıa Alvarez-Jimeno, “MOSQITO: an open-source and free toolbox for sound quality metrics in the industry and education,” in INTER-NOISE and NOISE-CON Congress and Conference Proceedings, 2021, pp. 1164–1175
[24] Elias Pampalk, “A matlab toolbox to compute music similarity from audio,” in Proc. ISMIR, 2004, pp. p. 4 [24] Elias Pampalk, “A matlab toolbox to compute music similarity
from audio,” in Proc. ISMIR, 2004, pp.Where the paper says it
[25] Elias Pampalk, Andreas Rauber, and Dieter Merkl, “Contentbased organization and visualization of music archives,” in Proc. ACM Int. Conf. Multimedia, 2002, pp. p. 4 [25] Elias Pampalk, Andreas Rauber, and Dieter Merkl, “Contentbased organization and visualization of music archives,” in
Proc. Multimedia, 2002, pp.Where the paper says it
[27] Sebastian Bock and Matthew E.P. Davies, “Deconstruct, anal- ¨yse, reconstruct: How to improve tempo, beat, and downbeat estimation,” in Proc. p. 4 [27] Sebastian Bock and Matthew E. Davies, “Deconstruct, anal- ¨
yse, reconstruct: How to improve tempo, beat, and downbeat
estimation,” in Proc.Where the paper says it
[33] Tzu-Ching Hung, Jingjing Tang, Kit Armstrong, Yi-Cheng Lin, and Yi-Wen Liu, “Eme33: A dataset of classical piano performances guided by expressive markings with application in music rendering,” in Proc. of IEEE Int. p. 4 [29] Tzu-Ching Hung, Jingjing Tang, Kit Armstrong, Yi-Cheng
Lin, and Yi-Wen Liu, “Eme33: A dataset of classical piano
performances guided by expressive markings with application
in music rendering,” in Proc. of IEEE Int. [29] Tzu-Ching Hung, Jingjing Tang, Kit Armstrong, Yi-Cheng
Lin, and Yi-Wen Liu, “Eme33: A dataset of classical piano
performances guided by expressive markings with application
in music rendering,” in Proc. of IEEE Int. of IEEE Int. on Big Data,
2024, pp.Where the paper says it
Where the paper says it
Where the paper says it
[36] Beat This [26], “Beat this! accurate beat tracking without dbn postprocessing,” in Proc. ISMIR, 2024, pp. 962–969
[37] ASAP: a dataset of aligned scores and performances for piano transcription, “ASAP: a dataset of aligned scores and performances for piano transcription,” in Proc. ISMIR, 2020, pp. 534–541
[38] MAESTRO dataset, Curtis Hawthorne et al., “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in Proc. ICLR, 2019
[3] Huan Zhang, Vincent K.M. Cheung, Hayato Nishioka, Simon Dixon, and Shinichi Furuya, “Llaqo: Towards a query-based coach in expressive music performance assessment,” in Proc. ICASSP, 2025
[6] Zhanhong He, David Cooper, Defeng Huang, and Roberto Togneri, “Filling midi velocity using u-net image colorizer,” in Proc. 17th Int. Symp. Computer Music Multidisciplinary Research, 2025
[18] Dichucheng Li, Mingjin Che, Wenwu Meng, Yulun Wu, Yi Yu, Fan Xia, and Wei Li, “Frame-level multi-label playing technique detection using multi-scale network and self-attention mechanism,” in Proc. ICASSP, 2023
[19] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., 2018
[26] Francesco Foscarin, Jan Schluter, and Gerhard Widmer, “Beat this! accurate beat tracking without dbn postprocessing,” in Proc. ISMIR, 2024
[33] Tzu-Ching Hung, Jingjing Tang, Kit Armstrong, Yi-Cheng Lin, and Yi-Wen Liu, “Eme33: A dataset of classical piano performances guided by expressive markings with application in music rendering,” in Proc. of IEEE Int. Conf. on Big Data, 2024
[36] Beat This [26], “Beat this! accurate beat tracking without dbn postprocessing,” in Proc. ISMIR, 2024
[37] ASAP: a dataset of aligned scores and performances for piano transcription, “ASAP: a dataset of aligned scores and performances for piano transcription,” in Proc. ISMIR, 2020
[38] MAESTRO dataset, Curtis Hawthorne et al., “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in Proc.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multi-task joint prediction of four musical structure elements allows an AI system to enrich music scores that possess reliable beat information but lack dynamic markings by predicting
dynamic levels, change points, beats, and downbeats from a shared latent representation.
-
The model can operate fully end-to-end without a pre-existing music score because the joint prediction of beat and downbeat positions allows it to do so directly from audio.
-
The system can be made highly parameter efficient with only 0.5 million parameters while achieving state-of-the-art results in dynamics and change point estimation, which is crucial for large-scale, resource-efficient analysis of musical expression.
-
Using Bark scale specific loudness as input instead of log Mel spectrograms reduces model size from 14.7 M to 0.5 M, enabling long sequential input suitable for processing longer audio sequences with better temporal information capture.
Abstract
Estimating piano dynamic from audio recordings is a fundamental challenge in computational music analysis. In this paper, we propose an efficient multi-task network that jointly predicts dynamic levels, change points, beats, and downbeats from a shared latent representation. These four targets form the metrical structure of dynamics in the music score. Inspired by recent vocal dynamic research, we use a multi-scale network as the backbone, which takes Bark-scale specific loudness as the input feature. Compared to log-Mel as input, this reduces model size from 14.7 M to 0.5 M, enabling long sequential input. We use a 60-second audio length in audio segmentation, which doubled the length of beat tracking commonly used. Evaluated on the public MazurkaBL dataset, our model achieves state-of-the-art results across all tasks. This work sets a new benchmark for piano dynamic estimation and delivers a powerful and compact tool, paving the way for large-scale, resource-efficient analysis of musical expression.
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models