TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction

arXiv:2610.25344 · cs.SD, eess.AS · Submitted 2026-10-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction".

Jane: Here's the gist of the paper. Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control.

Tom: TOMORI Architecture and Workflow TOMORI connects a language model to the REAPER DAW through a code-based mixing agent harness that provides project observations and executes model-generated Lua programs as DAW editing actions.

Jane: The harness utilizes DeepSeek Harness (DSH), a plugin-based agent runtime, which manages model interaction, tool calling, and dynamic plugin loading.

Tom: This system allows the model to read the current project, select operations, and execute code within the DAW.

Paper summary: Tom: The authors of "TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction" are showing us how to create an agent harness that connects a language model directly to a native digital audio workstation like REAPER <ref:2610.25344#pg3>.

Jane: They use custom plugins that give the model access to project inspection, optional audio measurement, and the ability to run generated Lua code as actual editing actions inside the DAW <ref:2610.25344#pg3>.

Lu: The main point is that this harness, which they call DeepSeek Harness, manages all that model interaction and dynamic plugin loading so the model can read the project state and execute code based on what it observes <ref:2610.25344#pg3>.

Meng: So for us in production, the implication is that we could potentially use this to automate complex mixing tasks where we don't want to manually click through every parameter adjustment <ref:2610.25344#pg1>.

Lalam: The paper’s evaluation shows that explicit instructions lead to high reference agreement across three primary models, while abstract requests tend to reduce the agreement on both the project and the audio output <ref:2610.25344#pg1>.

Tom: In simple terms, this system is very good when you tell it exactly what edit you want to make in a track, but it gets trickier when you ask it for a general feeling or sound description <ref:2610.25344#pg1>.

Jane: The work suggests that the future of these agents in creative software lies not just in understanding the request, but in having reliable, coded tools to execute those changes accurately <ref:2610.25344#pg2>.

Conclusion: Tom: So, we’re wrapping up this look at TOMORI, the tool for connecting language models to audio software like REAPER.

Jane: It seems the big idea behind this paper is that you can use an AI to actually *do* mixing in a digital audio workstation by giving it code instructions.

Lu: Basically, they built a harness where the model can look at your current project and then write and run small programs in a language like Lua to make edits.

Meng: From my side, what this means is that instead of clicking through fifty sliders to get a sound you want, you could tell the AI what it wants in plain English, and it executes the precise technical changes on your timeline.

Lalam: And the results show that when you give the AI very specific instructions—explicit ones—it gets very good at matching what you asked for in both the project structure and the actual sound file.

Tom: But they also found that when you ask it to just describe a vibe, like "make it sound warmer," things get much less accurate on both the project and the final audio.

Jane: That’s a key distinction, isn't it? It suggests that for these kinds of creative tasks, precision in the input really dictates how well the output is executed by the AI.

Lu: The paper shows that even when you try to measure how good its work is using audio metrics and project state measurements, explicit commands are consistently better than vague ones.

Meng: So if you’re trying to deploy this, it looks like you gotta be very careful about the prompting; it won't guess what you mean if you’re too abstract.

Tom: It really boils down to this: the system is a powerful execution engine, but its intelligence is totally dependent on how clearly the human tells it what to do.

Jane: So, while the harness itself is cool for letting models interact with software, the real magic seems to be in crafting those detailed, explicit commands.

Lu: And this leads us to thinking about how we train these models—do they need more examples of precise code execution versus just general sound descriptions?

cs.SD, eess.AS

Submitted: 2026-10-18

Updated: 2026-11-06

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI…

Key concepts

TOMORI Architecture
TOMORI is a system that links an AI language model to the REAPER digital audio workstation. It uses a code-based agent harness to allow the model to observe the project, select actions, and run Lua programs directly in REAPER. This enables the model to perform complex mixing tasks by interacting with the DAW through code.
DeepSeek Harness (DSH)
DeepSeek Harness is a plugin-based agent runtime that manages how the language model interacts with tools and plugins. It handles tasks like managing model conversations, calling necessary functions, and dynamically loading plugins. This component is essential for allowing the AI to execute actions within the DAW environment.
Instruction Forms (IF0–IF2)
These are different ways users ask for mixing changes. IF0 represents precise control instructions where the user specifies exactly what needs to be done. IF2 represents more abstract requests, describing the intended sound quality without detailing every edit. The system tests how well TOMORI performs under these varying levels of instruction specificity.
Evaluation Metrics
These are specific measurements used to judge how well TOMORI performs its mixing tasks. Metrics like Graph F1 and Parameter Precision assess agreement between the AI's edits and human reference edits in both the project files and the resulting audio. They help quantify whether explicit instructions lead to better accuracy than abstract ones.

Terminology

Summary

Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control. p. 0

Where the paper says it

com Abstract Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it. We present TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control.

TOMORI Architecture and Workflow

TOMORI connects a language model to the REAPER DAW through a code-based mixing agent harness that provides project observations and executes model-generated Lua programs as DAW editing actions. The harness utilizes DeepSeek Harness (DSH), a plugin-based agent runtime, which manages model interaction, tool calling, and dynamic plugin loading. p. 2

Where the paper says it

1 Plugin Architecture TOMORI utilizes DeepSeek Harness (DSH), a plugin-based agent runtime (DeepSeek-AI, 2026). DSH provides model interaction, tool calling, and dynamic plugin loading.

This system allows the model to read the current project, select operations, and execute code within the DAW. p. 2
Where the paper says it

They also manage rendering and return execution results to the model. Through these interfaces, the model reads the current project, selects operations, and executes code within the DAW.

The workflow involves a loop where the model selects project inspection, audio analysis, or code-based project editing based on returned results. p. 1
Where the paper says it

User Language model Agent harness REAPER loop [choose tools; repeat as needed] alt [Project inspection] [Optional audio analysis] [Project editing / rendering] Mixing request Inspect project Read project state Tracks / FX / parameters Project snapshot FFmpeg / analysis code Audio measurements Run editing code Edit / render Updated project / audio Execution result / error Project + rendered audio Figure 1: Sequence diagram of TOMORI. Within the loop, the model selects project inspection, audio analysis, or code-based project editing (alt), then uses returned results to decide whether to continue (loop).

Data Construction for Benchmarking

TOMORI-BENCH pairs 110 human-reviewed reference edits with three instruction forms (IF0–IF2) and evaluates native DAW execution end to end using project-state and rendered-audio metrics. The reference edits are constructed using Phoenix ScotchMorris from MedleyDB, which provides original multitracks and a finished mix. p. 3

Where the paper says it

request–project correspondence has been audited, and redacted examples are provided in Appendix B. To construct executable references, we use Phoenix_ScotchMorris from MedleyDB, which provides original multitracks and a finished mix (Bittner et al.

These references are derived from Everytime-72, a continuous eight-stem production session containing 72 human requests and corresponding project snapshots. p. 2
Where the paper says it

Along with normal edits, the sequence includes overshoots and subsequent repairs to assess execution under different project states and adjustment directions. 2 Reference Construction Task coverage and instruction styles are informed by Everytime-72, a continuous eight-stem production session containing 72 human requests and corresponding project snapshots.

The instruction forms vary in abstraction, moving from precise control instructions (IF0) to descriptions of the intended sound (IF2). p. 3
Where the paper says it

• IF2: Describes the intended sound of the target object, leaving operation selection and parameter settings to the model. From IF0 to IF2, the forms represent requests from users with decreasing mixing expertise, moving from precise control instructions to descriptions of the intended sound.

Evaluation Metrics and Findings

The end-to-end evaluation assesses reference agreement in both the delivered native project and its rendered audio using metrics like Graph F1, Parameter Precision, Audio Metrics, and Edit Progress. Explicit requests yield high reference agreement across three primary models while abstract requests reduce project and audio agreement. p. 0

Where the paper says it

TOMORI-BENCH pairs 110 human-reviewed reference edits with three instruction forms and evaluates native DAW execution end to end through operation structure, parameter precision, and renderedaudio outcomes. Across 990 cases from three primary models, explicit instructions yield high reference agreement, while abstract requests reduce project agreement and audio improvement.

Paired comparisons show that the full workflow improves explicit execution over a single submission, but prompted audio measurement provides no clear overall improvement in reference agreement on selected abstract cases. p. 0
Where the paper says it

Across three primary models, explicit requests are executed precisely, while abstract requests reduce project and audio agreement. Paired comparisons show that the full workflow improves explicit execution over a single submission, but prompted audio measurement provides no clear overall improvement in reference agreement on selected abstract cases.

Ablation and Limitations

Ablation studies assess the editing workflow and access to audio measurements, showing that the full workflow improves explicit execution over a single submission. However, prompted measurement provides no clear overall benefit over prohibited analysis in certain conditions. p. 5

Where the paper says it

On 28 selected signal-dependent or historical-analysis steps, we compare prohibited analysis (L) with prompted measurement before editing (M). Neither condition shows a clear overall difference in reference-based scores relative to the full workflow, and M shows no clear overall benefit over L.

Limitations include the fact that a perceptual request does not uniquely specify an operation, meaning project metrics measure agreement with one chosen implementation rather than satisfying the request. p. 7
Where the paper says it

Project metrics directly test explicit requirements, but under abstract instructions they measure agreement with one chosen implementation. Neither project nor audio agreement alone establishes whether an alternative satisfies the request.

Furthermore, broader generalization and generation stability require additional songs and repeated runs beyond the single song used in TOMORI-BENCH. p. 7
Where the paper says it

Second, TOMORI-BENCH contains edits from one song, with one final delivery per model– instruction pair. Broader generalization and generation stability require additional songs and repeated runs.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Precise execution under explicit instructions can be achieved by pairing language models with a code-based mixing agent harness that inspects REAPER projects, optionally measures audio, and generates Lua code to execute native DAW edits. This system can perform end-to-end evaluation through operation structure, parameter precision, and rendered audio outcomes across 110 human-reviewed reference edits.

  2. The system can generate Lua code to modify the current project by performing several observation and editing rounds within one request, such as comparing vocal and accompaniment band energies or inspecting routing while adjusting EQ. The final output consists of an editable REAPER project and rendered audio, allowing for iterative refinement based on textual feedback like bring the vocals forward.

  3. For abstract requests where explicit instructions are missing, the system can still attempt to improve execution by using a full workflow that includes project inspection, optional audio measurement, and model-generated Lua code. The paper shows that the full workflow improves explicit execution over a single submission.

Abstract

Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it. We present TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control. Custom plugins provide REAPER project inspection, optional audio measurement, and execution of model-generated Lua code. All observations return as text; delivery includes an editable project and rendered audio. TOMORI-BENCH pairs 110 human-reviewed reference edits with three instruction forms and evaluates native DAW execution end to end through operation structure, parameter precision, and rendered-audio outcomes. Across 990 cases from three primary models, explicit instructions yield high reference agreement, while abstract requests reduce project agreement and audio improvement. Paired ablations show that the full workflow improves explicit execution over a single submission. Prompted audio measurement succeeds in all 28 selected abstract cases but yields no clear overall improvement on the reference-based metrics and increases tool calls. These results expose the gap between executable DAW control and precise realization of perceptual mixing goals.

Sources

Related papers