Explore This! Beat and Downbeat Tracking From a Learned Tatum Grid

The gist The proposed novel architecture combines deep learning with differentiable theory-based modules to predict a tatum grid, group tatum events into beats and downbeats, and explicitly model multiple options for tempo and time signature in a bar-pointerlike fashion

Introduction

Beat/downbeat tracking is the task of estimating the temporal positions of beats/downbeats in a music recording. Most systems follow some variant of the same strategy: (1) The audio is converted to a spectrogram and fed to a neural network that outputs framewise beat/downbeat activations. (2) The network is trained with a loss that pushes the activation to be high at beats/downbeats and low otherwise, so it can be interpreted as a local probability of beat/downbeat appearance. (3) For inference, activations are postprocessed to materialize beats and downbeats, either by using a Dynamic Bayesian Network (DBN) or by finding large local maxima of the activation function. This strategy has foundational shortcomings because the activation function models probabilities independently and cannot express interdependence between probabilities at the various metrical levels.

Related Work

Multiple proposals follow the strategy of predicting framewise beat/downbeat probabilities and postprocessing those with Böck et al.’s DBN [1]. While the original [1] uses recurrent Long-Short Term Memory (LSTM) units, Böck et al. [11] switched to Temporal Convolutional Networks, Zhao et al. [12] use Transformers with dilated self-attention on demixed spectrograms, and Hung et al. [13] employ the Transformer-based SpecTNT architecture. The idea of finding a pulse, then defining the beat or downbeat on top of it has already been used in early beat trackers such as [6, 9, 18]. A prominent technique for the first step is the Predominant Local Pulse (PLP) [19], which can be derived from a feature sequence through deterministic signal processing and is used for real-time beat tracking in [20]. In contrast to earlier hierarchical approaches [6–10], which derive the pulse from the audio using hand-designed operations, we propose a differentiable approach to predict the local tatum grid and subgrid and integrate it with the Transformer of Beat This [2].

Method

The architecture of our model is summarized in Figure 2. We use Beat This [2] as the basis, as it is open source and is among the highest-performing models not using a DBN. We insert a module that derives a tatum grid from the output of the first three Transformer blocks, and a module that predicts the subgrid after three more Transformer blocks.

Pre-grid

Before the tatum grid is estimated, the audio input (a 30-second mel spectrogram with 128 bins and 1500 frames per second) is processed by a frontend of alternating convolutions, frequency-wise and time-wise Transformer blocks [2, Sec. 3.1.1]. These are projected to K dimensions for the main Transformer. We then pass the sequence through three of the six full Transformer blocks [2, Sec. 3.1.2].

Grid

From its 1500×256 input, the grid block tries to estimate the tatum grid, which is subject to several requirements. The grid should have 100% recall on the beats, as any missed beats will be missed in the model’s final prediction. To make (3) tractable, we choose the grid to be expressed as a tempo and phase, rather than independent framewise probabilities. To allow for local fluctuations, we split the input into 1-second windows with individual predictions. We then define the soft recall as the mean of the sine wave evaluated at ground truth beats (Figure 3), and set the training objective to maximize the soft recall. To help with the convexity of the loss, we use a mixture-of-experts model with multiple candidate waves.

Subgrid

From the 1500×256 tensor that encodes both the audio and the soft tatum grid, the subgrid block attempts to estimate the local metric distribution at every tatum point. We define a subgrid state as a set of four integers: Beat period TB represents how many tatums there are per beat, beat phase φB ∈ [1, TB] represents the index of the current tatum, while downbeat period TD and downbeat phase φD ∈ [1, TD] represent the period and index of downbeats in terms of beats. For every estimated tatum point, we predict a categorical distribution over these 280 bins (Figure 5, Figure 6). We calculate the prediction loss based on this probability and the ground truth annotations snapped to the nearest tatum point. As every subgrid state implies the state at every other tatum in that measure, we can directly measure the internal consistency of the subgrid by comparing the implied and estimated distributions at every tatum, and penalize the KL-divergence between the distributions.

Inference

Our model allows two ways to infer beats and downbeats for an audio excerpt. First, if we interpret the sequence of subgrid states at tatum points as a Markov model with fully known probabilities, we can compute the most probable total path with the Viterbi algorithm, with some uncertainty constant s determining the probability of unexpected metric jumps. Alternatively, we can calculate the total probability of a beat or downbeat at every tatum point as in the subgrid prediction loss, and use it to define a (very sparse) framewise activation function. For simplicity, we apply a threshold of 0.5 on the framewise probabilities, and refer to this as threshold inference.

Experiments

We perform experiments to assess the quality of the tatum grid, to compare our model to the original Beat This for different subsets of the data, and to evaluate it against the state of the art. We use standard beat tracking metrics F1, CMLt and AMLt [39]. Our model performs on par with or slightly below Beat This on beats, but reaches significantly higher continuity metrics on downbeats, especially on pieces with changing time signatures. The Viterbi inference is slightly better than threshold across the board, with the greatest improvements displayed for downbeat continuity metrics.

Conclusion

The presented model performs worse than the best DBN models across the board, but beats other DBN-free models on downbeat continuity metrics by a significant margin. The fact that the threshold inference managed to beat Foscarin et al. [2] on continuity metrics nonetheless shows the regularizing power of theory-informed design.

Acknowledgments

This research was funded in part by the Austrian Science Fund (FWF) [10.

1. Bold header: Differentiable Tatum Grid Prediction

The system can predict a tatum grid by modeling a tempo and time signature expressed in terms of that pulse, the subgrid (a subset of the grid, not a subdivision). This allows for modeling multiple interpretations of tempo and time signature simultaneously rather than relying on a single DBN assumption.

2. Bold header: Improved Continuity Metrics

The proposed architecture improves continuity metrics compared to existing DBN-free systems and is more versatile than the de-facto standard DBN. Specifically, it reaches significantly higher continuity metrics on downbeats, especially on pieces with changing time signatures, addressing a key weakness of prior methods.

3. Bold header: Robust Ambiguity Handling

By modeling a joint distribution with a larger set of allowed time signatures and using a mixture-of-experts model with multiple candidate waves, the system can handle metric ambiguity better than previous approaches by considering 12 non-overlapping tempo ranges and their associated confidence distributions.

4. Bold header: Enhanced Inference Flexibility

The system offers two inference methods: Viterbi inference (using a Markov model) and threshold inference (using framewise probabilities). This flexibility allows users to choose the appropriate method for different needs, with Viterbi being noted as slightly better than threshold across the board.

5. Bold header: Separable Design Potential

The formulation is highly separable, allowing researchers to focus on improving the learning of the tatum grid independently of improvements to the subgrid block, suggesting independent optimization pathways.

State-of-the-art beat tracking is based on neural networks that model beats and downbeats as individual events, often followed by a separate Dynamic Bayesian Network (DBN) enforcing musical consistency. We propose a novel architecture that combines deep learning with differentiable theory-based modules: Specifically, we design a system that first predicts a tatum grid, then groups tatum events to beats and downbeats, explicitly modelling multiple options for the current tempo and time signature in a bar-pointer-like fashion. Our approach improves continuity metrics compared to existing DBN-free systems and is more versatile than the de-facto standard DBN, but still falls behind the best DBN-based methods on average. Given the many design choices and hyperparameters involved, we assume we have not yet explored the full potential of this idea and publish our code and invite others to explore this further.