Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription
summary
The gist
MIDI velocity is crucial for capturing expressive dynamics in human performances <ref:2508.07757#pg2> In this work, we propose a modular…
In short
The work introduces a lightweight, modular Transformer correction module to improve MIDI velocity estimation in Automatic Music Transcription (AMT) when an aligned music score is available. By refining initial velocity guesses using score features, this method achieves state-of-the-art performance on MAESTRO and shows better generalization across other datasets like SMD and MAPS.
Key concepts
- Score-Informed MIDI Velocity Estimation
- This technique uses a reference music score to help correct the velocity estimates generated by an audio transcription system. Instead of guessing velocity purely from sound, the system looks at where notes are in the score to make more accurate predictions about how hard those notes were played.
- Score-Informed Correction Module
- This is a small Transformer network added to existing AMT systems. It takes features derived from the MIDI score (like note and onset information) and uses them to adjust the preliminary velocity estimates, making them more accurate before they are finalized.
- Hybrid Velocity Loss
- The training objective combines two types of error measurement: frame-wise BCE loss for general accuracy and an onset-masked L1 loss. This ensures the model learns to evaluate velocity at the note level rather than just at every small time frame, leading to better performance.
- Generalization across Datasets
- The method is designed to perform well not just on the training data (MAESTRO), but also on different, unseen datasets (SMD and MAPS). The score acts as a stable guide, helping the model estimate velocities reliably even when encountering new types of piano recordings.
Terminology used across episodes
This episode discusses
- Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription · Paper Radio
The paper
Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription · Read on arXiv
University of Western Australia
MIDI velocity is crucial for capturing expressive dynamics in human performances. In practical scenarios, a music score with inaccurate velocities may be available alongside the performance audio (e.g., music education and free online archives), enabling the task of score-informed MIDI velocity estimation. In this work, we propose a modular, lightweight score-informed Transformer correction module that refines the velocity estimates of Automatic Music Transcription (AMT) systems. Leveraging high-quality expressive datasets for rigorous evaluation, we focus on piano transcription, integrating our module into multiple existing systems. Trained exclusively on the MAESTRO dataset, our method consistently reduces velocity errors and improves cross-dataset generalization to SMD and MAPS datasets. Under this training protocol, integrating our score-informed module with HPT (named Score-HPT) establishes a new state-of-the-art, outperforming existing score-informed methods and piano transcription models while adding only 1 M parameters. Furthermore, our framework remains robust to temporal misalignments caused by imperfect audio-score synchronization, a common real-world challenge.
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription".
Paper summary: Rosa: So this paper is about taking automatic music transcription, or AMT, which usually figures out the basic notes and timing of a piano piece from audio, and improving how accurate it gets when trying to figure out the actual velocity of those notes.
Dev: Right. The authors call their approach the Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription. Basically, they are using a modular correction module that takes an existing AMT system and refines its initial velocity guesses by looking at a music score they already have.
Taro: So it’s like you have a rough draft of the piano performance, but you also have the sheet music, and this new module uses the sheet music to polish those rough velocity numbers. That makes sense for capturing how expressive a human performance actually sounds.
Rosa: Exactly. The main claim is that this method consistently reduces velocity estimation errors on the MAESTRO training split and it helps with generalization when they test it on other datasets like SMD and MAPS.
Dev: What’s important here is how they build that correction module. It’s not a whole new model; it’s lightweight, designed to be integrated into existing AMT systems without changing the core velocity estimation branch.
Taro: So the architecture has two parts: you have the original velocity estimation part that stays exactly the same, and then this score-informed correction module is added on top of it. That makes sense for keeping things stable.
Rosa: But how does that correction module actually work? They say they use features extracted from the MIDI score to refine those preliminary velocity estimates, specifically by treating each onset as a token.
Dev: They take each onset coordinate, which is a time and a pitch, and they create an embedding for it by combining embeddings for the pitch and the time with some trainable weight parameters.
Taro: So it’s like giving every important moment in the music score its own little piece of information that tells the correction module how to adjust the velocity estimate at that specific point.
Rosa: And they process these onsets through a lightweight Transformer encoder, which has twelve eight layers and four attention heads, and then they use a bounded update to calculate the final corrected velocity.
Dev: That final calculation takes the preliminary velocity guess and adds some adjustment based on those score-informed tokens, keeping the result strictly between zero and one. They use a hybrid loss function that mixes frame-wise binary cross-entropy with onset masked l1 loss.
Taro: So they are balancing how well it matches every single frame in time with how accurately it predicts the velocity specifically at those onsets, which is a smart way to look at both aspects.
Rosa: The results show that when they integrate this module into HPT, which is one of the baselines they tested, Score-HPT achieves lower mean absolute error and standard deviation than the original HPT across all conditioning variants.
Dev: They specifically found that onsets plus frames give them the best overall performance on the MAESTRO test set, which shows that focusing on both onset and frame information is effective.
Taro: It seems like this score-informed correction really helps concentrate those velocity estimates around where the note actually starts in time.
Rosa: The paper also points out that this framework shows better cross-dataset generalization when moving from the training data, MAESTRO, to out-of-distribution datasets like SMD and MAPS.
Dev: They noticed during their training that if you train for too long only on MAESTRO data, the model starts overfitting to those specific sounds and loses its ability to generalize. The score seems to act as a regularizer against that overfitting.
Taro: So even when the world misbehaves with new audio data, like SMD or MAPS, the structural cues from the score help keep the velocity estimation stable and reliable.
Rosa: Comparing it to other systems, they say Score-HPT and Score-DynEst achieve new state-of-the-art performance on the SMD dataset when compared against existing methods.
Dev: They also outperformance dedicated score informed models that use something like FiLM, which are other deep learning approaches for this task. They even beat systems like hFT Transformer and TransKun v2.
Taro: It’s interesting because they only add about one million parameters to these existing AMT systems to get those improvements. That modularity really seems efficient for real-world deployment where you don't want to design a whole new system from scratch.
Rosa: So, the core idea is that you don't need a completely different model; you can just plug this lightweight correction module into an existing velocity estimation branch and get better results when you have a score available.
Dev: The authors mention they are restricted to piano datasets for now because high-quality MIDI ground truth is mostly found on acoustic pianos with velocity sensors. That’s a limitation they are aware of.
Taro: So the immediate future work they suggest is transferring this whole framework from piano music estimation over to other instruments, like guitars or strings, for MIDI velocity estimation.
Rosa: It sounds like a solid contribution because it focuses on making existing technology better by using an external source of truth—the score—in a modular way.
Dev: The implication is that if you're working on AMT right now and you have access to scores, this framework offers a straightforward way to boost your velocity accuracy without needing massive retraining cycles.
Conclusion: Rosa: So we've been talking about how these researchers took an existing music transcription system and added this little correction module that uses the sheet music to fix the velocity of piano notes, and now we're looking at what it really means for those systems.
Dev: Right. The paper is titled "Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription," and it shows how they built this modular part to refine those initial velocity guesses using features extracted from the score.
Rosa: Basically, the big point here is that you don't need a completely new model architecture if you already have a baseline AMT system; you just plug in this lightweight Transformer correction module to improve accuracy.
Dev: The authors are showing that when they train this module exclusively on the MAESTRO training split, it consistently lowers the error rate on that set and actually helps when they test it on different datasets like SMD and MAPS.
Rosa: It seems like the main takeaway is that having a score available gives you a stable reference point for correcting those messy audio estimations, even when the world throws out new types of data.
Dev: Yeah, it’s about using that external structural information—the score—to regularize the model and make sure it doesn't just memorize the training sounds. It’s about making the velocity estimation more robust across different recordings.
Rosa: And they mention that while this whole approach is focused on piano music right now because of how we get those high-quality ground truth, there’s a clear path to apply this idea to other instruments later on.
Dev: Exactly. The authors flag that limitation, and the next thing we need to look at is how they handle the real-world deployment and latency when running this correction module in a live system.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G