MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

The paper introduces MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout "describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music."

The authors integrate MusicLayout into "a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. This allows the resulting MusicLayout to be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control."

The paper notes that current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. While prompts can describe genre, mood, instrumentation, or tempo, they cannot precisely specify how sections are organized, materials recur or vary, textures evolve, or instruments enter and leave. Existing controllable methods add melody, chords, drums, dynamics, symbolic lead sheets, or semantic representations but are often limited to individual attributes, require additional musical inputs, or remain implicit and do not expose a unified, time-aligned representation for specifying the organization and arrangement of an entire piece.

The paper draws inspiration from image generation, noting that explicit planning representations improve controllability and interpretability in image generation by exposing high-level organization before content (citing PlanGen), and argues that for music, this principle could improve long-range structural organization and controllability though transferring it requires representing organization along time rather than space.

The paper lists three main contributions:

1. MusicLayout representation: "an explicit planning representation for unified audio language models. It represents music as a time-aligned layout of sections, repetitions, variations, textures, and instrument arrangements, providing structured musical plans beyond global textual prompts."

2. Unified autoregressive framework: "a single model first produces an explicit MusicLayout plan and then continues to generate audio tokens conditioned on the preceding layout within a single sequence. By making musical planning explicit, this framework supports long-range structural organization and layout-level control."

3. Evaluation evidence: "Through evaluations of layout-conditioned generation, layout manipulation, and matched-data ablations, we provide evidence that MusicLayout can serve as an interpretable planning interface and improve long-range structural organization in text-to-music generation."

MusicLayout externalizes temporal musical organization as a structured, human-readable sequence that bridges high-level structural intent and audio-token generation. It follows three design principles: temporal alignment, hierarchical organization, and controllability.

The representation combines piece-level structural relationships among musical materials with time-aligned segments, each corresponding to one musical section. At the piece level, it organizes recurring musical materials and their relationships, including repetitions and variations across sections. At the segment level, each segment is associated with a time-aligned span and describes its section identity, texture evolution, and instrument participation.

Key components include:

• Section labels: selected from intro, verse, prechorus, chorus, bridge, breakdown, outro, transition, hook, solo, and build

• Families: a piece-local group of segments that share similar musical material with labels like fam 0, fam 1, where the same label may denote different musical material in another piece

• Variation types: unique, same, light var, or strong var indicating that the segment has no repeated family, closely repeats its family pattern, or departs from that pattern to a smaller or larger degree

• Texture field: a coarse description of how a musical segment is organized with categories including layered, rhythm driven, melodic front, harmonic bed, percussive, build up, sparse pulse, lead front, contrast, and other

• Instrument roles: a block listing its active instruments using the form instrument:register:density:energy with a predefined vocabulary of 25 categories and attributes for register (low/mid/high), density (sparse/med/dense), and energy (low/med/high), encoding pitch range, activity, and intensity

MusicLayout annotations were derived from symbolic representations by extracting temporal sections, repetitions, variations, textures, and instrument arrangements and rendered into audio and aligned each layout with its audio and text prompt to form prompt–layout–audio training triples.

The model is built upon ACE-Step 1.5, extending its LM to generate a MusicLayout before producing layout-conditioned audio tokens. The original audio tokenizer and DiT remain frozen.

Modeling: Given conditioning context c, the LM serializes MusicLayout sequence l before audio-token sequence a within a single autoregressive sequence: y = [c,, l,, a]. The joint distribution factorizes as p(l, a c) = p(l c)p(a c, l) where p(l c) describes layout planning from the conditioning context, while p(a c, l) describes audio-token generation given the context and layout.

Training: The LM is trained on two complementary next-token tasks: layout planning (generating l from c) and layout-conditioned audio generation (predicting a from c and a ground-truth MusicLayout). The training initially optimize[s] layout planning to learn the MusicLayout schema and the schema-specific special tokens then alternate[s] the tasks, applying the loss only to the corresponding layout or audio target span. Only the LM is updated.

Inference and control: "In automatic inference, the LM generates a MusicLayout from the conditioning context and then continues with audio tokens. Alternatively, the human-readable layout can be inspected and adjusted before audio generation to change temporal organization or instrument arrangement."

Datasets: Three datasets were used:

• FreeMIDI: provided the training data and a disjoint in-domain evaluation set (24,474 train, 2,719 eval items), with audio synthesized via FluidSynth and prompts generated by MOSS-Music

• MidiCaps: used to evaluate generalization to out-of-domain MIDI (1,040 items) with existing captions

• MuChin: provided an out-of-domain evaluation on real audio (1,000 items) with translated captions and vocal accompaniment separation

Compared systems: MusicGen, ACE-Step 1.5, ACE-Step XL-Turbo, and Stable Audio 3 Medium, plus a matched-data no-layout control (ACE-Step 1.5-FT) that directly predicts audio from the conditioning context c without MusicLayout tokens.

Evaluation metrics: FAD, PaSST-KL, SSIM, CLAPScore, SCM Energy Distance, and F0.5/F3.0 acoustic-boundary agreement.

The paper states: No single system dominates all aspects of generation. The model with generated layouts records the lowest FAD and PaSST-KL on FreeMIDI and the lowest PaSST-KL on MidiCaps. However, its lower CLAPScore relative to Stable Audio and ACE-Step XL-Turbo... shows that explicit planning may not uniformly improve global text–audio correspondence.

The matched-data no-layout control is weaker than the original ACE-Step 1.5 on nearly all metrics, attributed to the training-data difference: the original model was trained on high-quality real recordings, whereas our finetuning data consists of MIDI-synthesized audio. Relative to this control, the reference-layout condition has more favorable values on all seven metrics for FreeMIDI and MidiCaps and the generated-layout condition does so on six metrics for FreeMIDI and five for MidiCaps and MuChin.

The reference-layout condition records the highest boundary scores shown for FreeMIDI, the highest MidiCaps F3.0, and a tie for the highest MidiCaps F0.5, providing evidence that MusicLayout captures target organization and that conditioning on it can guide corresponding structure in the generated audio.

With model-generated layouts, both boundary scores are numerically above all baselines on FreeMIDI, F3.0 is numerically above all baselines on MidiCaps, and both boundary scores are numerically above all baselines on MuChin, showing that explicit intermediate planning improves the organization of long-range musical structure in end-to-end generation.

The paper presents four separate layout-manipulation cases showing that the matched generation shows recurrence patterns similar to the target reference, while the manipulated generation loses some target-aligned patterns and exhibits patterns closer to the donor reference, illustrating that MusicLayout can be manipulated before audio generation to control the structure of the generated music.

Three direct edits to a single 240-second MusicLayout were tested:

1. Role replacement (0–11s): replacing sparse strings with dense drums and bass, producing energy below 250 Hz becomes dominant and repeated vertical transients appear across the band

2. Reduced density/energy (142–196s): changing every active role to sparse density and medium energy, resulting in visibly reduced broadband activity and its RMS level decreases by 15% (1.4 dB)

3. Form rewrite (91–240s): rewriting a largely repeated chorus sequence into a breakdown, bridge, build, solo, chorus, and outro, producing the intended succession of contrasting regimes

Three matched-data controls form a progressive ablation of MusicLayout:

1. No-layout control: removes MusicLayout entirely

2. Shuffled-layout training: pairs each training item with a layout from another item

3. Shuffled-layout inference: restores correctly paired layouts during training but replaces the conditioning layout with one from another item at inference

Results show shuffled-layout training has favorable values on five of seven metrics relative to the no-layout control for each of the three datasets, suggesting layout-shaped supervision may retain a generic structural benefit even when item-level information is incorrect. The full model has favorable values on five of seven metrics relative to shuffled-layout training for both FreeMIDI and MidiCaps, indicating correct layout–audio correspondence during training provides information beyond the presence of layout-shaped tokens.

Subjective ratings (0–5 scale) from 27 participants (15 with music experience) showed that the reference-layout condition has higher observed scores on all three dimensions compared to the matched-data control, and within the experienced subset, both layout conditions score numerically higher than the control across all dimensions. Notably, the reference-layout condition also has higher observed text consistency than the original ACE-Step 1.5 among experienced listeners (3.183 vs. 3.108).

A follow-up structural similarity rating found the reference-layout condition received a higher observed structural similarity than ACE-Step 1.5, providing subjective evidence that MusicLayout captures reference organization and carries it into generated audio.

The paper acknowledges: MusicLayout adjusts structural plans only before synthesis and cannot edit existing audio or regenerate selected regions. Our reliance on MIDI-synthesized training audio may also limit audio fidelity. Additionally, the categorical instrument representation describes arrangement-level source classes rather than exact timbres, performances, or production effects, and a valid layout is therefore a well-formed structural request, not a guarantee that every specified attribute will be realized perfectly in audio.

The paper concludes: "We presented MusicLayout, an explicit, time-aligned representation of section organization, development, and instrument arrangement for a unified autoregressive audio LM. Generated before audio tokens, it makes structural planning inspectable and adjustable. The same LM plans the layout and predicts audio tokens conditioned on it while the synthesis components remain frozen. Our results suggest that MusicLayout provides an interpretable interface that supports long-range structural organization and layout-level control."

Based on this paper, here are the specific improvements I can make to AI systems and what the improved systems can do:

1. Add an explicit structural planning layer before generation: I can integrate a MusicLayout-style intermediate representation (time-aligned sections, families, variations, textures, instrument roles) into my autoregressive generation pipeline. This means generating a human-readable structural plan first, then producing audio tokens conditioned on that plan within a single sequence.

2. Enable layout-level control and manipulation: I can allow users to inspect, edit, or replace the generated layout before audio synthesis. This includes changing section labels, reordering families, modifying variation types (unique/same/light var/strong var), adjusting instrument roles (register, density, energy), or rewriting entire form structures.

3. Support fine-grained regional edits: I can accept targeted modifications to specific time spans (e.g., replacing sparse strings with dense drums in the first 11 seconds, reducing density/energy in a middle section, or rewriting a chorus into a breakdown-bridge-build-solo-outro sequence) and regenerate only the affected audio regions coherently.

4. Improve long-range structural coherence: By training on layout-audio paired data, I can better maintain section boundaries, repetition patterns, and texture evolution across 3–4 minute pieces, rather than relying solely on global text prompts that leave structure implicit.

5. Provide interpretable failure analysis: When generation quality is suboptimal, I can expose the layout as a diagnostic tool, allowing users to see whether the issue lies in the structural plan or the audio rendering, and correct the plan accordingly.

• Generate music from text prompts with explicit, editable structure: A user can prompt an EDM track with a build-up intro, a drop chorus, and a breakdown in the middle and receive a layout showing exact time-aligned sections, instrument entries/exits, and variation labels—then edit any part before audio is rendered.

• Manipulate musical form after generation planning but before synthesis: Swap a verse and chorus, change a repetition to a strong variation, or remove an instrument from a specific section, all by editing the layout text rather than regenerating from scratch.

• Apply fine-grained regional control: Directly modify a 10-second span to change density from sparse to dense, shift register from low to high, or rewrite a repeated section into a contrasting one, with the audio reflecting those changes in measurable ways (e.g., RMS level, spectral energy, transient patterns).

• Maintain structural fidelity across long pieces: Generate 3–4 minute compositions where section boundaries align with the intended form, repetitions are recognizable, and texture evolution follows the plan—outperforming systems that rely on text-only conditioning for structural organization.

• Serve as an interpretable interface for human-AI collaboration: Musicians can review, annotate, or correct the layout before committing to audio, enabling iterative refinement of structure without re-prompting or regenerating the entire piece.

• Provide a unified framework for both planning and synthesis: The same model handles layout generation and audio token prediction, so no separate modules or external alignment tools are needed—making the system simpler to deploy and modify.