Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription".
Paper summary: Rosa: So this paper is about taking automatic music transcription, or AMT, which usually figures out the basic notes and timing of a piano piece from audio, and improving how accurate it gets when trying to figure out the actual velocity of those notes.
Dev: Right. The authors call their approach the Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription. Basically, they are using a modular correction module that takes an existing AMT system and refines its initial velocity guesses by looking at a music score they already have.
Taro: So it’s like you have a rough draft of the piano performance, but you also have the sheet music, and this new module uses the sheet music to polish those rough velocity numbers. That makes sense for capturing how expressive a human performance actually sounds.
Rosa: Exactly. The main claim is that this method consistently reduces velocity estimation errors on the MAESTRO training split and it helps with generalization when they test it on other datasets like SMD and MAPS.
Dev: What’s important here is how they build that correction module. It’s not a whole new model; it’s lightweight, designed to be integrated into existing AMT systems without changing the core velocity estimation branch.
Taro: So the architecture has two parts: you have the original velocity estimation part that stays exactly the same, and then this score-informed correction module is added on top of it. That makes sense for keeping things stable.
Rosa: But how does that correction module actually work? They say they use features extracted from the MIDI score to refine those preliminary velocity estimates, specifically by treating each onset as a token.
Dev: They take each onset coordinate, which is a time and a pitch, and they create an embedding for it by combining embeddings for the pitch and the time with some trainable weight parameters.
Taro: So it’s like giving every important moment in the music score its own little piece of information that tells the correction module how to adjust the velocity estimate at that specific point.
Rosa: And they process these onsets through a lightweight Transformer encoder, which has twelve eight layers and four attention heads, and then they use a bounded update to calculate the final corrected velocity.
Dev: That final calculation takes the preliminary velocity guess and adds some adjustment based on those score-informed tokens, keeping the result strictly between zero and one. They use a hybrid loss function that mixes frame-wise binary cross-entropy with onset masked l1 loss.
Taro: So they are balancing how well it matches every single frame in time with how accurately it predicts the velocity specifically at those onsets, which is a smart way to look at both aspects.
Rosa: The results show that when they integrate this module into HPT, which is one of the baselines they tested, Score-HPT achieves lower mean absolute error and standard deviation than the original HPT across all conditioning variants.
Dev: They specifically found that onsets plus frames give them the best overall performance on the MAESTRO test set, which shows that focusing on both onset and frame information is effective.
Taro: It seems like this score-informed correction really helps concentrate those velocity estimates around where the note actually starts in time.
Rosa: The paper also points out that this framework shows better cross-dataset generalization when moving from the training data, MAESTRO, to out-of-distribution datasets like SMD and MAPS.
Dev: They noticed during their training that if you train for too long only on MAESTRO data, the model starts overfitting to those specific sounds and loses its ability to generalize. The score seems to act as a regularizer against that overfitting.
Taro: So even when the world misbehaves with new audio data, like SMD or MAPS, the structural cues from the score help keep the velocity estimation stable and reliable.
Rosa: Comparing it to other systems, they say Score-HPT and Score-DynEst achieve new state-of-the-art performance on the SMD dataset when compared against existing methods.
Dev: They also outperformance dedicated score informed models that use something like FiLM, which are other deep learning approaches for this task. They even beat systems like hFT Transformer and TransKun v2.
Taro: It’s interesting because they only add about one million parameters to these existing AMT systems to get those improvements. That modularity really seems efficient for real-world deployment where you don't want to design a whole new system from scratch.
Rosa: So, the core idea is that you don't need a completely different model; you can just plug this lightweight correction module into an existing velocity estimation branch and get better results when you have a score available.
Dev: The authors mention they are restricted to piano datasets for now because high-quality MIDI ground truth is mostly found on acoustic pianos with velocity sensors. That’s a limitation they are aware of.
Taro: So the immediate future work they suggest is transferring this whole framework from piano music estimation over to other instruments, like guitars or strings, for MIDI velocity estimation.
Rosa: It sounds like a solid contribution because it focuses on making existing technology better by using an external source of truth—the score—in a modular way.
Dev: The implication is that if you're working on AMT right now and you have access to scores, this framework offers a straightforward way to boost your velocity accuracy without needing massive retraining cycles.
Conclusion: Rosa: So we've been talking about how these researchers took an existing music transcription system and added this little correction module that uses the sheet music to fix the velocity of piano notes, and now we're looking at what it really means for those systems.
Dev: Right. The paper is titled "Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription," and it shows how they built this modular part to refine those initial velocity guesses using features extracted from the score.
Rosa: Basically, the big point here is that you don't need a completely new model architecture if you already have a baseline AMT system; you just plug in this lightweight Transformer correction module to improve accuracy.
Dev: The authors are showing that when they train this module exclusively on the MAESTRO training split, it consistently lowers the error rate on that set and actually helps when they test it on different datasets like SMD and MAPS.
Rosa: It seems like the main takeaway is that having a score available gives you a stable reference point for correcting those messy audio estimations, even when the world throws out new types of data.
Dev: Yeah, it’s about using that external structural information—the score—to regularize the model and make sure it doesn't just memorize the training sounds. It’s about making the velocity estimation more robust across different recordings.
Rosa: And they mention that while this whole approach is focused on piano music right now because of how we get those high-quality ground truth, there’s a clear path to apply this idea to other instruments later on.
Dev: Exactly. The authors flag that limitation, and the next thing we need to look at is how they handle the real-world deployment and latency when running this correction module in a live system.
University of Western Australia
eess.AS, cs.SD
Submitted: 2025-08-11
Updated: 2026-11-17
Comments: published at SMC2026 Conference, https://github.com/zhanh-he/score-informed-amt/tree/main
Code: https://github.com/zhanh-he/score-informed-amt
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 84/100
The gist: MIDI velocity is crucial for capturing expressive dynamics in human performances <ref:2508.07757#pg2> In this work, we propose a modular…
Key concepts
- Score-Informed MIDI Velocity Estimation
- This technique uses a reference music score to help correct the velocity estimates generated by an audio transcription system. Instead of guessing velocity purely from sound, the system looks at where notes are in the score to make more accurate predictions about how hard those notes were played.
- Score-Informed Correction Module
- This is a small Transformer network added to existing AMT systems. It takes features derived from the MIDI score (like note and onset information) and uses them to adjust the preliminary velocity estimates, making them more accurate before they are finalized.
- Hybrid Velocity Loss
- The training objective combines two types of error measurement: frame-wise BCE loss for general accuracy and an onset-masked L1 loss. This ensures the model learns to evaluate velocity at the note level rather than just at every small time frame, leading to better performance.
- Generalization across Datasets
- The method is designed to perform well not just on the training data (MAESTRO), but also on different, unseen datasets (SMD and MAPS). The score acts as a stable guide, helping the model estimate velocities reliably even when encountering new types of piano recordings.
Terminology
Summary
MIDI velocity is crucial for capturing expressive dynamics in human performances <ref:2508. p. 0 SCORE-INFORMED TRANSFORMER FOR REFINING MIDI VELOCITY IN
AUTOMATIC MUSIC TRANSCRIPTION
Zhanhong HE(zhanh. com)
1
, Roberto TOGNERI1
, and Defeng (David) HUANG1
1Signal Processing and Recognition Lab, University of Western Australia, Perth, WA 6000 Australia
ABSTRACT
MIDI velocity is crucial for capturing expressive dynamics in human performances. , music education and free online archives), enabling the task of score-informed MIDI
velocity estimation. In this work, we propose a modular,
lightweight score-informed Transformer correction module that refines the velocity estimates of Automatic Music
Transcription (AMT) systems. We integrate the proposed
module into multiple AMT systems (HPT, HPPNet, and
DynEst). Trained exclusively on the MAESTRO training
split, our method consistently reduces velocity estimation
errors on MAESTRO and improves cross-dataset generalization to SMD and MAPS datasets. Trained exclusively on the MAESTRO training
split, our method consistently reduces velocity estimation
errors on MAESTRO and improves cross-dataset generalization to SMD and MAPS datasets. Under this training protocol, integrating our score-informed module with
HPT (named Score-HPT) establishes a new state-of-the-art
performance, outperforms existing score-informed methods and velocity-enabled AMT systems while adding only
1 M parameters.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
Automatic music transcription (AMT) aims to capture symbolic music information from audio <ref:2508.07757#pg2>. While traditional AMT systems focus on pitch and note boundaries to construct the basic MIDI score <ref:2508. p. 0 INTRODUCTION
Automatic music transcription (AMT) is a longstanding
topic in Music Information Retrieval (MIR), aiming to capture symbolic music information from audio. While traditional AMT systems focus on pitch and note boundaries
to construct the basic MIDI score [1], recent advancements have expanded to estimating MIDI velocity in piano
performances [2]. INTRODUCTION
Automatic music transcription (AMT) is a longstanding
topic in Music Information Retrieval (MIR), aiming to capture symbolic music information from audio. While traditional AMT systems focus on pitch and note boundaries
to construct the basic MIDI score [1], recent advancements have expanded to estimating MIDI velocity in piano
performances [2]. While traditional AMT systems focus on pitch and note boundaries
to construct the basic MIDI score [1], recent advancements have expanded to estimating MIDI velocity in piano
performances [2]. This expansion has facilitated the creation of large-scale piano datasets [3–7], benefiting downstream MIR tasks such as music education and generation. This expansion has facilitated the creation of large-scale piano datasets [3–7], benefiting downstream MIR tasks such as music education and generation. However, AMT outputs frequently require manual
post-correction. This challenge has driven the audio-toscore alignment techniques to address timing discrepancies
[8–10], and motivated our research to refine the AMT’s estimated MIDI velocity. Leveraging music score to aid velocity estimation from
audio is known as score-informed MIDI velocity estimation. Leveraging music score to aid velocity estimation from
audio is known as score-informed MIDI velocity estimation. This paradigm is a practical scenario where a reference score and performance audio co-exist.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The proposed methodology involves integrating a lightweight Transformer encoder as a score-informed correction module into existing AMT systems <ref:2508.07757#pg2>. The velocity estimation branch of the chosen AMT baseline is kept identical, and the correction module refines the preliminary velocity estimates using features extracted from the MIDI score <ref:2508. p. 1 The system includes a velocity estimation module, which is identical to the velocity branch of the chosen AMT baseline. The correction module then rectifies the preliminary velocity estimates using features extracted from the MIDI score.Where the paper says it
The model architecture consists of two main parts:
-
Velocity Estimation Module, which is identical to the velocity branches of selected AMT baselines including HPT [14], HPPNet [15], and DynEst [16] <ref:2508. p. 2
07757#pg2>.Where the paper says it
Since AMT systems are multi-task architectures that estimate onset, offset, frame, and velocity simultaneously, we isolate their velocity estimation branch and extend it with a score-informed correction module. Velocity Estimation Module: This component is identical to the velocity branches of the selected AMT baselines, including HPT [14], HPPNet [15], and DynEst [16].
-
Score-informed Correction Module, which refines the preliminary velocity Vˆ0 using onset as tokens <ref:2508. p. 2
07757#pg2>. For each onset token i = (ti, pi) ∈ I, an embedding x i is formed where x i = e(p) p i + e(t) t i + Wf f i <ref:2508. p. 2Where the paper says it
The velocity branch outputs a preliminary velocity map Vˆ 0 ∈ [0, 1]𝑇 ×𝑃 . Score-informed Correction Module: The preliminary velocity Vˆ 0 is refined using the onset as tokens.
07757#pg2>. These embeddings are processed by a lightweight Transformer encoder, which uses d=128, 2 layers, and 4 attention heads <ref:2508. p. 2Where the paper says it
For computational efficiency, we tokenize only the score onset coordinates ℐ, resulting in a variable-length sequence of 𝑁 = |ℐ| tokens per segment. For each onset token 𝑖 = (𝑡𝑖 , 𝑝𝑖) ∈ ℐ, we form an embedding x𝑖 = e (𝑝) 𝑝𝑖 + e (𝑡) 𝑡𝑖 + W𝑓 f𝑖 , (3) where e (𝑝) 𝑝𝑖 and e (𝑡) 𝑡𝑖 are learnable pitch/time embeddings.
07757#pg2>. The final refined velocity is calculated using a bounded update: Vˆ corr t i,pi = clip(Vˆ 0,t i,p i + α tanh(delta i), 0, 1) <ref:2508. p. 2Where the paper says it
Stacking all embeddings yields X ∈ R 𝑁×𝑑 , which is processed by the proposed lightweight Transformer encoder (a streamlined variant of the Conformer architecture [37]), as detailed in Fig. We use 𝑑=128, 2 layers, and 4 attention heads, applying a dropout probability of 0.
07757#pg2>.Where the paper says it
Given the encoder outputs, a linear head predicts a scalar residual 𝛿𝑖 for each token, and we apply a bounded update: 𝑉ˆ corr 𝑡𝑖,𝑝𝑖 = clip(︁ 𝑉ˆ 0,𝑡𝑖,𝑝𝑖 + 𝛼 tanh(𝛿𝑖), 0, 1 )︁ , (4) where 𝛼 controls the maximum correction magnitude (empirically set to 𝛼=0. The refined values replace the preliminary estimates in Vˆ 0 at onset positions, while nononset entries remain unchanged.
Where the gains come from
The primary gain stems from the modular design which allows for easy integration into existing systems rather than designing specific model architectures from scratch <ref:2508.07757#pg2>. The training objective uses a hybrid velocity loss combining frame-wise BCE and onset-masked l1: L = 1/J ∑(t,p)∈J lbce(Vt,p, Vˆ tp)+ 1/I ∑(t,p)∈I (V tp − Vˆ tp) <ref:2508. p. 2 We optimize using the hybrid velocity
loss proposed in [12], combining frame-wise BCE and
onset-masked ℓ1:
ℒ =
1
|𝒥 |
∑︁
(𝑡,𝑝)∈𝒥
ℓbce(𝑉𝑡,𝑝, 𝑉ˆ
𝑡,𝑝)+ 1
|ℐ|
∑︁
(𝑡,𝑝)∈ℐ
⃒
⃒
⃒
𝑉𝑡,𝑝 − 𝑉ˆ
𝑡,𝑝
⃒
⃒
⃒ . (5)
where V is the ground-truth velocity target and Vˆ is the
predicted velocity output of either (i) the baseline AMT
velocity branch or (ii) with the proposed score-informed
module applied. 3 Evaluation Metrics
Consistent with [12], we use the mean absolute error
(MAE) and standard deviation of the error (STD) as evaluation metrics, calculated on the standard MIDI velocity scale of [0, 127]. By applying onset-masking, we ensure a note-level evaluation that avoids the temporal biasWhere the paper says it
Where the paper says it
Validation of Score-Informed Module
When integrated into HPT, the proposed Score-HPT consistently achieves lower MAE and STD under all conditioning variants compared to the original HPT <ref:2508.07757#pg2>. Specifically, onset + frameex yields the best overall performance on the MAESTRO test set <ref:2508. p. 3 Compared with the original HPT, Score-HPT consistently achieves lower MAE and STD under all conditioning variants, with only a modest increase in model parameters (from 4. Among these Score-HPT variants, onset + frameex yields the best overall performance,
while onset + frame performs comparably. 4 provides a qualitative visualization of
this improvement. Compared to the baseline, Score-HPT
produces more concentrated velocity estimates around onset positions, clearly illustrating the effectiveness of the
score-guided correction.Where the paper says it
Where the paper says it
Generalization across Datasets
The model demonstrates superior cross-dataset generalization across in-domain (MAESTRO) and out-of-distribution (SMD, MAPS) data <ref:2508.07757#pg2>. A distinct trend observed during training suggests that prolonged training overfits to MAESTRO-specific acoustics and harms generalization <ref:2508. p. 3 A distinct trend emerges during training: while indomain performance steadily improves, out-of-distribution
performance peaks early and then declines. This suggests
that prolonged training overfits to MAESTRO-specific
acoustics and harms generalization; similar sound biases
in AMT systems—particularly for dynamics/velocity estimation under distribution shifts—were also reported in a
recent systematic analysis [41]. This suggests
that prolonged training overfits to MAESTRO-specific
acoustics and harms generalization; similar sound biases
in AMT systems—particularly for dynamics/velocity estimation under distribution shifts—were also reported in a
recent systematic analysis [41]. Nevertheless, Score-HPT
achieves better in-domain results and improved out-ofdistribution performance, highlighting the regularizing effect of the score, which provides stable, acoustics-invariant
structural cues for reliable velocity estimation across diverse environments.Where the paper says it
Where the paper says it
Comparison with Existing Works
The proposed framework achieves new SOTA performance on the SMD dataset when compared against existing methods and representative AMT systems <ref:2508.07757#pg2>. Specifically, Score-HPT and Score-DynEst achieve new SOTA performance, outperforming dedicated score-informed FiLM models [12,13] as well as leading AMT systems like hFT-Transformer [28] and TransKun v2 [30] <ref:2508. p. 3 We use the onset + frame configuration for all proposed variants, given its optimal cross-dataset generalization found in Sec. Consequently, both Score-HPT and
Score-DynEst achieve new SOTA performance, outperforming dedicated score-informed FiLM models [12,13] as
well as leading AMT systems, including hFT-Transformer
[28] and TransKun v2 [30]. This confirms that our score-informed correction paradigm is not
restricted to a specific AMT backbone (i. , not only effective to HPT), but generalizes effectively across different
AMT systems, adding a marginal overhead of only ∼1 M
parameters.Where the paper says it
Where the paper says it
Conclusion and Future Work
The work presents a novel, modular score-informed Transformer correction module for refining MIDI velocity in AMT when an aligned score is available <ref:2508.07757#pg2>. The module can be attached to an existing AMT velocity branch with minimal architectural changes <ref:2508. p. 4 CONCLUSION AND FUTURE WORK
In this work, we presented a novel, modular scoreinformed Transformer correction module for refining
MIDI velocity in AMT when an aligned score is available. By operating on score onset tokens and predicting bounded
residual updates, the proposed module can be attached to
an existing AMT velocity branch with minimal architectural changes. Experiments with MAESTRO-only training demonstrate consistent improvements across multiple
AMT backbones (HPT, HPPNet, and DynEst), stronger
cross-dataset generalization to SMD and MAPS, and stateof-the-art performance on SMD under our training protocol, surpassing both prior score-informed models and competitive velocity-enabled AMT systems. A key limitation of the current study is its restriction to
piano datasets, constrained by the fact that high-fidelity
MIDI velocity ground truth is predominantly captured via
acoustic pianos equipped with velocity sensors. For many
other instruments, velocity annotations are scarce or unavailable, making supervised training and evaluation difficult. An important direction for future work is to transfer
the proposed framework from piano to other instruments
for MIDI velocity estimation, potentially leveraging selfsupervised learning, transfer learning, or continual learning to reduce dependence on explicit velocity labels.Where the paper says it
Where the paper says it
Where the paper says it
Acknowledgments
This research was carried out while the author was in receipt of a University International Fees Scholarship and a University Postgraduate Award at The University of Western Australia <ref:2508.07757#pg2>. The authors acknowledge the use of Kaya, the High-Performance Computing facility at The University of Western Australia, and the technical assistance provided by the UWA IT – HPC team <ref:2508. p. 5 Acknowledgments
This research was carried out while the author was in receipt of a University International Fees Scholarship and a
University Postgraduate Award at The University of Western Australia. The authors acknowledge the use of Kaya,
the High-Performance Computing facility at The University of Western Australia, and the technical assistance provided by the UWA IT – HPC team.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Modular Integration for Existing Systems: The proposed approach is designed to be a
modular, lightweight score-informed Transformer correction module that refines the velocity estimates of Automatic Music Transcription (AMT) systems,
allowing it to beintegrated into multiple AMT systems (HPT, HPPNet, and DynEst)
withminimal architectural changes.
-
State-of-the-Art Performance: Integrating the module with HPT establishes a
new state-of-the-art performance, outperforms existing score-informed methods and velocity-enabled AMT systems while adding only 1 M parameters,
specifically achieving the SOTA in score informed velocity estimation on the MAESTRO dataset. -
Cross-Dataset Generalization: The method demonstrates
superior cross-dataset generalization
to datasets like SMD and MAPS, suggesting that the score providesstable, acoustics-invariant structural cues for reliable velocity estimation across diverse environments.
-
Robust Training Protocol: Training exclusively on the MAESTRO training split and using a hybrid velocity loss combining frame-wise BCE and onset-masked l1 ensures that the model is trained effectively to predict both sustained notes and onsets accurately.
-
Velocity Refinement Mechanism: The correction module refines preliminary velocity estimates V0 by predicting a bounded residual update using
onset as tokens
and applying abounded update: V hat corr t i,p i = clip(Vˆ0,t i,p i + α tanh(δ i), 0, 1),
which replaces preliminary estimates only at onset positions. -
Handling Score Features: The system tokenizes only the
score onset coordinates I
and forms embeddings where x i incorporates theonset-wise velocity Vˆ0,t i,p i and optional score-derived features (F or F ex),
allowing it to utilize both structural information and ground truth context during correction. -
Generalizability Across Architectures: The framework is shown to
generalize effectively across different AMT systems,
confirming its utility not just for HPT but also for architectures like HPPNet and DynEst, despite feature mismatches such as using logMel spectrograms instead of CQT spectrograms or BSSL features.
Abstract
MIDI velocity is crucial for capturing expressive dynamics in human performances. In practical scenarios, a music score with inaccurate velocities may be available alongside the performance audio (e.g., music education and free online archives), enabling the task of score-informed MIDI velocity estimation. In this work, we propose a modular, lightweight score-informed Transformer correction module that refines the velocity estimates of Automatic Music Transcription (AMT) systems. Leveraging high-quality expressive datasets for rigorous evaluation, we focus on piano transcription, integrating our module into multiple existing systems. Trained exclusively on the MAESTRO dataset, our method consistently reduces velocity errors and improves cross-dataset generalization to SMD and MAPS datasets. Under this training protocol, integrating our score-informed module with HPT (named Score-HPT) establishes a new state-of-the-art, outperforming existing score-informed methods and piano transcription models while adding only 1 M parameters. Furthermore, our framework remains robust to temporal misalignments caused by imperfect audio-score synchronization, a common real-world challenge.
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models