Harmonic Representations in Symbolic Music Transformers are Localized and Controllable

arXiv:2610.15957 · cs.SD, cs.LG · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Harmonic Representations in Symbolic Music Transformers are Localized and Controllable".

Jane: The gist The internal representations of a symbolic music transformer encode harmonic context in a structured and localized form,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're diving into this paper called "Harmonic Representations in Symbolic Music Transformers are Localized and Controllable." It sounds like they're looking at how these music models actually understand harmony internally and if we can use that knowledge to guide what they generate.

Jane: Right, it’s about taking a look inside the black box of these symbolic music transformers and figuring out what harmonic context the model is holding onto when it hits a cadence, and then seeing if we can steer that context to get different musical outcomes.

Lu: The core idea here is that the internal representations of music are structured in a way that we can control them locally during generation, which suggests harmonic computation isn't something baked into the model from the start but develops as it goes through the layers.

Meng: From a deployment standpoint, if we can manipulate these representations at inference time without retraining, that’s pretty efficient. But what does "controlled" actually look like in practice when you’re running on real traffic?

Lalam: I've reviewed the work; it shows that harmonic decisions emerge progressively from middle layers onward, meaning early layers have little causal harmonic information, but around layer six that effect starts growing and keeps increasing through later layers.

Tom: That progression is interesting. And they use linear probes to show a hierarchy: mode becomes decodable in the same layer range where the patching identifies growing causal effect, while key identity emerges later on page five of this paper.

Jane: So it seems the model first learns whether something is major or minor in those middle layers, before it figures out which specific key signature we're working with. That’s a neat way to think about tonal hierarchy.

Lu: Exactly, and they use these linear probes to actually measure how accessible that information is by training them to predict things like mode and key on page four <ref:2610.15957#pg1>.

Meng: But what’s the practical application? I need a concrete example of what steering those latent directions actually does when you’re generating something complex like a chorale.

Tom: They confirm these representations are actionable by showing that steering via latent directions corresponding to a specific harmonic change reliably produces either a deceptive or minor resolution, while preserving structural coherence and generation quality on page two <ref:2610.15957#pg1,steering via latent directions corresponding to a>.

Title and authors: Jane: So, if we want the model to resolve a chord in its minor mode instead of major, they have a specific direction they can apply to the hidden states during generation and it seems to work consistently.

Lu: They define this steering using a normalized steering vector v-l based on probe weights and then apply an intervention by adding this vector scaled by a factor alpha to the residual stream activations, which is described in equation three on page three <ref:2610.15957#pg1>.

Tom: That’s the mechanism for the control. And they found that steering all token positions, including generated ones, yields the strongest and most consistent shifts across the board on page five.

Jane: That means you can influence almost every part of what it generates by applying these steering vectors derived from those probes. It really puts control directly into our hands at inference time.

Meng: If I were deploying this, I’d be worried about stability. How much does this interference affect the overall quality metrics when we push alpha to a higher value? Does it just introduce noise?

Lalam: The experiments showed that key steering substantially redirects harmonic output, with relative minor steering shifting mass toward the submediant vi and parallel minor steering producing a decisive shift toward the parallel minor tonic i on page seven.

Tom: That specific example with the relative minor steering pushing towards vi is very telling about how localized that control is working. It’s not just random noise, it’s directed manipulation of the harmonic structure.

Jane: And they also found that when you steer between relative keys, the representation space shows those keys are more confusable than parallel keys on page eight. So steering to a relative key resolution might be less decisive than steering to a parallel minor one.

Lu: That asymmetry in the geometry explains why the steering isn't perfectly uniform; it’s based on how close or far apart those tonal relationships are represented internally, as shown by the confusion matrices on page eight.

Title and authors: Tom: So, to summarize what we’ve seen with this paper, harmonic computation develops progressively from middle layers onward, mode comes before key identity in decoding, and we have actionable steering vectors derived from probes that can reliably alter cadence resolution while keeping the music structurally sound.

Jane: It really shifts the perspective on how we think about these models—they aren't just predicting sequences; they are holding structured musical knowledge that we can actively manipulate during output.

Lu: The implication is that this internal geometry mirrors established music theory structures, specifically relative keys being closer together in representation space than parallel keys, which grounds the steering asymmetry in something we already understand about music itself.

Meng: For us shipping models, this means we have a new knob to turn for harmonic flavor without having to retrain the whole thing, which is a huge win for iterative development and rapid prototyping of new styles.

Lalam: I’m pretty sure the main point is that this paper shows how much usable structure exists within these transformer representations, and it proves that we can tap into it to control high-level musical decisions without needing massive amounts of new training data for every tiny tweak.

Tom: So, as a wrap up for "Harmonic Representations in Symbolic Music Transformers are Localized and Controllable," the key findings are the progressive emergence of harmonic computation, the hierarchy where mode is decoded before key identity, and the proof that we can steer these latent directions to reliably produce specific harmonic resolutions while maintaining good generation quality.

Jane: This work suggests that understanding what a model "knows" about harmony at different layers allows us to gain direct control over the musical decisions it makes during inference.

Lu: The geometry they uncovered, where relative keys are more confusable than parallel keys, provides a theoretical underpinning for why the steering effect is not uniform across all tonal shifts.

Meng: For production engineers, this means we can fine-tune stylistic elements like cadence changes with targeted interventions rather than broad conditioning that might break other aspects of the music generation.

Lalam: This paper solidifies the idea that internal representations aren't just opaque blobs; they are structured musical concepts that we can map and manipulate to influence the final output in a predictable way.

The paper's summary: Tom: So, to recap, this paper is looking at how these music transformers internally represent harmony and proving that we can actually control those representations during generation by making targeted adjustments.

Jane: Right, so they’re showing that if you look deep inside a transformer model trained on music—specifically Bach chorales—the harmonic information isn't all mixed up randomly; it’s organized in a way that lets us pinpoint exactly which layers hold the tonal context.

Lu: Exactly, and they found something pretty specific about where this emerges: the causal effect of harmonic decisions starts showing up progressively from the middle layers, not right at the beginning.

Meng: That sounds promising for deployment because if we know *where* to look in a model to change a musical feature, we don't have to retrain the whole thing just to nudge it toward a different style.

Tom: And they use linear probes—which is like training little predictors on the hidden states—to show that mode, like whether something’s major or minor, becomes decodable in those same middle layers where that causal effect grows.

Jane: That’s a key point for me; it suggests the model first establishes the big tonal category before it gets down to resolving the specific key signature.

Lu: Yeah, they show a hierarchy where mode decoding happens earlier than key identity emergence, which is interesting because it means we can target those middle layers for control.

Meng: If I’m shipping a model, knowing that layer six is where the real harmonic decisions start becoming causally influential tells me exactly where my intervention needs to be.

Tom: And they prove this knowledge is actionable; they show you can derive specific steering directions from those probes to modify the hidden states during inference.

Jane: So, we aren't just guessing; we are applying a vector derived from that probe weight to the residual stream activations, scaling it by some alpha factor.

Lu: That steering vector v-l is calculated based on the probe weights and then you add it to the state activations, which reliably produces a deceptive or minor resolution in the output.

Meng: So, if I want a specific harmonic outcome—say, pushing a cadence toward the submediant vi—I can apply that steering vector and I should get that result consistently while keeping the overall music quality high.

Tom: They show that this manipulation preserves structural coherence; it doesn't just make the music sound weird, it keeps the structure intact.

Jane: It’s a big deal because it means we can manipulate cadence resolution directly at inference time without needing massive retraining cycles for every new harmonic nuance.

Lu: And they even looked at how this geometry works; their confusion matrices show that relative keys are more confusable than parallel keys in the representation space, which explains why the steering isn't perfectly uniform.

Meng: That’s a nice piece of context for me; it tells me that if I try to steer between two close relative keys, the intervention might be less decisive than steering between two very distant parallel keys.

Tom: So, overall it’s about showing that these internal representations are structured enough to be controlled, and the geometry reflects music theory itself.

Jane: It shifts how we think about these models—they aren't just generating sequences; they are holding a structured musical knowledge that we can actively manipulate during output.

Lu: And this grounding in music theory structure is what makes the steering mechanism work so well, showing that the internal geometry mirrors real music theory constraints.

Meng: For production engineers, this means we have a new knob to turn for harmonic flavor without having to do broad conditioning that might break other aspects of the music generation.

Tom: So it’s about localized control, where you can target specific layers and specific harmonic changes with derived vectors while maintaining quality on inference time.

Jane: It really shows that understanding what an AI knows about harmony at different layers allows us to gain direct control over the musical decisions it makes during inference.

Lu: The implication is that this paper provides a roadmap for how to probe and manipulate these representations based on the theory of tonal relationships in music.

The paper's improvements: Tom: So, looking at what they found on page one and page two the paper isn't just about describing what’s there; it suggests concrete improvements for how we use these models to generate music.

Jane: Right, they propose making harmonic computation progressive by leveraging layer-specific interventions during generation. It’s like having a dimmer switch for harmony that gets stronger as you go deeper into the network.

Lu: They point out that because mode decoding happens earlier than key identity, we can use those earlier layers precisely for controlling the tonal category part of the music.

Meng: That means if I want to shift the overall "feel" of a passage, I don't have to mess with every single token; I just target the relevant layer where that mode information is being processed.

Tom: And they introduce this specific steering method using a normalized steering vector v-l derived from probe weights to modify those hidden states directly.

Jane: This is actionable because applying that vector scaled by some alpha factor reliably produces a deceptive or minor resolution while preserving the overall structural coherence of the piece.

Lu: They also have a trick involving representation geometry; they show that relative keys are more confusable than parallel keys in the internal space, which means steering between those relative keys requires less intervention to produce a shift.

Meng: That geometric detail is helpful because it tells me how much effort I need to put in—how large alpha needs to be—to get the desired harmonic effect.

Tom: And they found that key steering specifically has a big impact, with relative minor steering pushing mass toward the submediant vi and parallel minor steering shifting it toward the parallel minor tonic i.

Jane: So for someone just listening to this, it changes things because we’re moving beyond just saying "make this sound sad" to saying "force this specific harmonic shift at this exact point in the song."

Lu: It fundamentally changes how we think about music generation; it’s not just pattern matching anymore; it’s about manipulating underlying theory structures inside the model.

Meng: From a deployment standpoint, if we can do targeted harmonic control at inference time without retraining, that makes iteration super fast for new styles or genres.

Tom: They also set up this progressive mechanism where we can intervene layer by layer to sculpt the final sound according to our desired harmonic rules.

Jane: So the next big question is how much of this control we can achieve before it starts breaking the music altogether, and that’s where the real tuning comes in.

Conclusion: Tom: So we’ve looked at how the paper "Harmonic Representations in Symbolic Music Transformers are Localized and Controllable" shows that music harmony has structure we can actually manipulate during generation.

Jane: Right, it boils down to this: these transformer models aren't just guessing notes; they hold a structured internal map of harmony that we can read and use to steer the output.

Lu: The main implication is that this isn't just an observation about music; it’s a method for control, showing that harmonic computation develops progressively through the network layers.

Meng: For production, it means we can fine-tune stylistic elements like cadence changes with targeted interventions rather than having to retrain the whole model every time we want a slight flavor shift.

Tom: They proved that this steering is reliable; you get a deceptive or minor resolution while keeping the overall generation quality high, which is exactly what I look for when I'm building something for shipping.

Jane: It changes how we think about these models—they are not just predicting sequences; they are holding musical knowledge that we can actively manipulate during output.

Lu: The internal geometry they uncovered, where relative keys are more confusable than parallel keys in the representation space, provides a theoretical underpinning for why the steering effect isn't uniform across all tonal shifts.

Meng: That geometric detail is helpful because it tells me how much effort I need to put in—how large alpha needs to be—to get the desired harmonic effect.

Tom: So, overall it’s about localized control, where you can target specific layers and specific harmonic changes with derived vectors while maintaining quality on inference time.

Jane: It really shows that understanding what an AI knows about harmony at different layers allows us to gain direct control over the musical decisions it makes during inference.

Lu: The implication is that this paper provides a roadmap for how to probe and manipulate these representations based on the theory of tonal relationships in music.

Meng: If we can reliably control these features, it opens up new avenues for creating highly specialized AI tools, moving beyond general music generation toward very precise musical composition tasks.

Tom: Yeah, I’m excited about the engineering potential here because it moves us closer to models where we aren't just observing what they do but actually directing them to do something specific.

Jane: It’s a solid piece of work that moves this field from passive generation toward active musical direction.

Lu: It lays a strong foundation for future work in representation learning and causal intervention in music models, which is where the next big push is going to be.

Adi Silberschein, Megan Wei, Tamar Rott Shaham

Weizmann Institute of Science · Brown University · MIT CSAIL

cs.SD, cs.LG

Submitted: 2026-10-11

Updated: 2026-10-11

Comments: 9 pages, 7 figures, 1 table. Accepted at ISMIR 2026 (27th International Society for Music Information Retrieval Conference)

Project page: https://adisilber.github.io/harmonic-interp

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The internal representations of a symbolic music transformer encode harmonic context in a structured and localized form, and that intervening on these representations enables control over

Key concepts

Causal Localization: This concept reveals that the model's ability to make harmonic decisions is not present everywhere at once. Instead, it emerges gradually, starting in the middle layers of the transformer. Early layers have little causal effect on harmonic choices, while later layers become increasingly responsible for these complex structural decisions.
Mode vs. Key Identity: The research showed a hierarchy in harmonic understanding: the model first learns to distinguish between major and minor modes (mode) in middle layers. It only develops the ability to identify the specific tonal center or key identity later in the network, suggesting a progression from broad tonal categories to precise musical details.
Steering via Latent Directions: This is a method used to control music generation. By training probes on specific harmonic attributes, researchers create 'steering vectors' from these probes. Applying these vectors during inference allows them to directly nudge the model's hidden states toward desired harmonic changes, such as forcing a deceptive resolution while keeping the overall structure sound.

Terminology

Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The internal representations of a symbolic music transformer encode harmonic context in a structured and localized form, and that intervening on these representations enables control over harmonic decisions during generation.

How it works

Causal Interventions reveal that harmonic decisions emerge progressively from middle layers onward, suggesting that harmonic computation is not present from the start but develops through the network <ref:2610.15957#pg5>.
Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later <ref:2610.15957#pg5>.
The analysis confirms that representations are actionable because steering via latent directions corresponding to a specific harmonic change reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality <ref:2610.15957#pg2>.

Where Harmonic Information is Encoded

The study utilized Bach chorales as an ideal testbed because they are harmonically rich, governed by well-understood conventions, and admit precise counterfactual construction <ref:2610.15957#pg4>. To isolate harmonic decisions at cadences, researchers constructed minimal contrastive pairs consisting of an original sequence x and a counterfactual variant x' <ref:2610.15957#pg4>. Causal interventions demonstrated that early layers (0–5) carry little causal harmonic information, as patching them has minimal effect <ref:2610.15957#pg5>. Conversely, an increased causal effect emerges around layer 6, and continues to grow through later layers <ref:2610.15957#pg5>.

Characterizing Harmonic Representations

Linear probing was used to assess what harmonic information is represented across layers by training probes to predict attributes such as mode and key <ref:2610.15957#pg4>. The findings indicate a processing hierarchy: mode (major vs. minor) becomes decodable in middle layers, while key identity emerges later <ref:2610.15957#pg5>. This suggests that the model first establishes tonal category before resolving the specific tonal center <ref:2610.15957#pg5>.

Controlling Harmonic Output at Inference Time

The researchers demonstrated that these representations are actionable by deriving steering directions from trained probes to modify hidden states <ref:2610.15957#pg4>. The steering method involves defining a normalized steering vector vl based on probe weights and then applying an intervention by adding this vector, scaled by a factor α, to the residual stream activations <ref:2610.15957#pg4>. Experiments showed that steering all token positions, including generated ones, yields the strongest and most consistent shifts <ref:2610.15957#pg5>. Specifically, key steering substantially redirects harmonic output, with relative minor steering shifting mass toward the submediant (vi) and parallel minor steering producing a decisive shift toward the parallel minor tonic (i) <ref:2610.15957#pg7>. These manipulations were shown to preserve generation quality and that steered generations are more harmonically varied rather than collapsing onto a single chord <ref:2610.15957#pg7>.

Internal Geometry and Tonal Relationships

The internal geometry of the model was found to reflect music theory structures, as relative keys lie closer together than parallel keys in representation space <ref:2610.15957#pg7>. The confusion matrices confirmed this asymmetry: relative keys are more confusable than parallel keys <ref:2610.15957#pg8>. This geometric difference explains the steering asymmetry, where steering between relative keys means pushing in a direction the model already finds ambiguous, producing less decisive shifts <ref:2610.15957#pg8>. The analysis is grounded entirely in Bach chorales, which are more harmonically constrained than broader symbolic music corpora <ref:2610.15957#pg4>.

The paper concludes that harmonic computation emerges progressively from middle layers onward, where mode becomes decodable before key identity, and that steering via latent directions derived from key probes redirects cadence resolution while preserving musical coherence. The geometry of these representations mirrors music theory: relative keys lie closer together than parallel keys in representation space <ref:2610.15957#pg7>.

--- Page 1 ---

HARMONIC REPRESENTATIONS IN SYMBOLIC MUSIC

TRANSFORMERS ARE LOCALIZED AND CONTROLLABLE

Adi Silberschein

Weizmann Institute of Science

adi.silberschein@weizmann.ac.il Megan Wei

Brown University meganwei@brown.edu Tamar Rott Shaham MIT CSAIL tamarott@mit.edu

ABSTRACT

Transformer models trained on symbolic music have become remarkably capable; however, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control <ref:2610.15957#pg2>. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally “know” about its harmonic context, and can that knowledge be used to affect the generation <ref:2610.15957#pg2>. Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward <ref:2610.15957#pg5>. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later <ref:2610.15957#pg5>. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to a specific harmonic change reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality <ref:2610.15957#pg2>. This demonstrates that, at inference time, the model’s internal harmonic knowledge can be directly manipulated to control the harmonic output <ref:2610.15957#pg5>.

--- Page 2 ---

RELATED WORK

Music Generation Models. Music generation models have made rapid progress, producing high-fidelity audio across genres and styles [20–25]. On the symbolic side, early neural approaches modeled Bach chorales [1, 2]. Music Transformer [3] laid the ground for transformerbased symbolic generation, inspiring work on tokenization schemes that better capture musical structure [26,27], controllability via conditioning [28,29], and new architectures for long-range structure [4]. Recent work has pushed symbolic generation further, scaling to larger models, extending to richer musical contexts, and adding new sources of control [11,14–17,30,31]. Understanding the inner mechanisms of these models could unlock more intuitive methods of inference-time control for musicians and researchers <ref:2610.15957#pg3>.

Interpretability in Music Models. Linear probing [32,33] has emerged as a tool for understanding internal representations of pretrained models, and has been applied to music models <ref:2610.15957#pg3>. Castellon et al. [34] show that Jukebox representations transfer to MIR tasks such as genre and key, and self-supervised models such as MERT [35] and MusicFM [36] are likewise evaluated by linear probing of frozen representations <ref:2610.15957#pg3>. Wei et al. [37] introduce SynTheory and probe Jukebox and MusicGen for core musictheoretic concepts, finding they are encoded in deeper layers, while Koo et al. [38] probe MusicGen attention heads across tasks, including key detection, finding that some musical concepts are localized to a small subset of heads <ref:2610.15957#pg3>.

Recent work has taken steps toward causal intervention and steering: [39] uses interchange interventions to study how MusicGen composes instrument and genre concepts; [40] adds probe-derived vectors to MusicGen’s residual stream to intervene on chord representations; [41] steers in a feedback loop to control instrument presence; [42] constructs steering vectors from hidden states to control tempo and timbre; [43] applies sparse autoencoders to MusicGen to discover and steer interpretable musical concepts; [44] derives steering directions via Recursive Feature Machines, controlling music theory concepts; [45] steers composer style in symbolic music models; and TADA [46] extends activation patching and steering to audio diffusion models, controlling instruments, genre, and mood <ref:2610.15957#pg3>.

While prior work has probed for static chord and key attributes [37, 40], none has studied how the model handles dynamic harmonic decisions at structural moments like cadences <ref:2610.15957#pg3>.

Improvements for AI systems

No retraining is required for targeted harmonic control at inference time; This demonstrates that, at inference time, the model’s internal harmonic knowledge can be directly manipulated to control the harmonic output. This allows for real-time steering of generation toward specific properties like deceptive or minor resolution while preserving structural coherence and generation quality.
Implement a progressive harmonic computation mechanism by leveraging layer-specific interventions; Causal effect of harmonic information emerges in middle layers, suggesting that harmonic computation is not present from the start but develops progressively through the network, where early layers show little causal harmonic information compared to around layer 6, and continues to grow through later layers.
Develop a hierarchical decoding strategy by separating mode and key identity processing; linear probes reveal a hierarchy where mode becomes decodable in middle layers, while key identity emerges later, suggesting the model first establishes tonal category before resolving the specific tonal center.
Create actionable steering vectors derived from probe weights to modulate output; We define the normalized steering vector at a layer l as: vl = hl · (wc1 − wc2) / wc1 − wc2. Applying this intervention reliably alter[s] the model’s generated resolutions, inducing deceptive (V → vi) or minormode (V → i) cadences while preserving overall musical coherence.
Optimize steering targeting by exploiting representational geometry; The confusion matrices show that relative keys are more confusable than parallel keys, implying that steering along a direction corresponding to a well-separated parallel key shift requires a smaller intervention (α = 0.1) sufficient to produce dramatic harmonic shifts.

Abstract

Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.