TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction

summary

The gist

Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI…

In short

TOMORI is a code-based mixing agent harness that connects a language model to REAPER for audio editing. It uses a system called DeepSeek Harness to let the model read project files, analyze audio, and execute Lua code within the DAW. The research tested how well this system handles both precise instructions and vague sound descriptions, showing that explicit requests yield better results.

Key concepts

TOMORI Architecture
TOMORI is a system that links an AI language model to the REAPER digital audio workstation. It uses a code-based agent harness to allow the model to observe the project, select actions, and run Lua programs directly in REAPER. This enables the model to perform complex mixing tasks by interacting with the DAW through code.
DeepSeek Harness (DSH)
DeepSeek Harness is a plugin-based agent runtime that manages how the language model interacts with tools and plugins. It handles tasks like managing model conversations, calling necessary functions, and dynamically loading plugins. This component is essential for allowing the AI to execute actions within the DAW environment.
Instruction Forms (IF0–IF2)
These are different ways users ask for mixing changes. IF0 represents precise control instructions where the user specifies exactly what needs to be done. IF2 represents more abstract requests, describing the intended sound quality without detailing every edit. The system tests how well TOMORI performs under these varying levels of instruction specificity.
Evaluation Metrics
These are specific measurements used to judge how well TOMORI performs its mixing tasks. Metrics like Graph F1 and Parameter Precision assess agreement between the AI's edits and human reference edits in both the project files and the resulting audio. They help quantify whether explicit instructions lead to better accuracy than abstract ones.

Terminology used across episodes

This episode discusses

The paper

TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction · Read on arXiv

Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it. We present TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control. Custom plugins provide REAPER project inspection, optional audio measurement, and execution of model-generated Lua code. All observations return as text; delivery includes an editable project and rendered audio. TOMORI-BENCH pairs 110 human-reviewed reference edits with three instruction forms and evaluates native DAW execution end to end through operation structure, parameter precision, and rendered-audio outcomes. Across 990 cases from three primary models, explicit instructions yield high reference agreement, while abstract requests reduce project agreement and audio improvement. Paired ablations show that the full workflow improves explicit execution over a single submission. Prompted audio measurement succeeds in all 28 selected abstract cases but yields no clear overall improvement on the reference-based metrics and increases tool calls. These results expose the gap between executable DAW control and precise realization of perceptual mixing goals.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction".

Jane: Here's the gist of the paper. Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control.

Tom: TOMORI Architecture and Workflow TOMORI connects a language model to the REAPER DAW through a code-based mixing agent harness that provides project observations and executes model-generated Lua programs as DAW editing actions.

Jane: The harness utilizes DeepSeek Harness (DSH), a plugin-based agent runtime, which manages model interaction, tool calling, and dynamic plugin loading.

Tom: This system allows the model to read the current project, select operations, and execute code within the DAW.

Paper summary: Tom: The authors of "TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction" are showing us how to create an agent harness that connects a language model directly to a native digital audio workstation like REAPER <ref:2610.25344#pg3>.

Jane: They use custom plugins that give the model access to project inspection, optional audio measurement, and the ability to run generated Lua code as actual editing actions inside the DAW <ref:2610.25344#pg3>.

Lu: The main point is that this harness, which they call DeepSeek Harness, manages all that model interaction and dynamic plugin loading so the model can read the project state and execute code based on what it observes <ref:2610.25344#pg3>.

Meng: So for us in production, the implication is that we could potentially use this to automate complex mixing tasks where we don't want to manually click through every parameter adjustment <ref:2610.25344#pg1>.

Lalam: The paper’s evaluation shows that explicit instructions lead to high reference agreement across three primary models, while abstract requests tend to reduce the agreement on both the project and the audio output <ref:2610.25344#pg1>.

Tom: In simple terms, this system is very good when you tell it exactly what edit you want to make in a track, but it gets trickier when you ask it for a general feeling or sound description <ref:2610.25344#pg1>.

Jane: The work suggests that the future of these agents in creative software lies not just in understanding the request, but in having reliable, coded tools to execute those changes accurately <ref:2610.25344#pg2>.

Conclusion: Tom: So, we’re wrapping up this look at TOMORI, the tool for connecting language models to audio software like REAPER.

Jane: It seems the big idea behind this paper is that you can use an AI to actually *do* mixing in a digital audio workstation by giving it code instructions.

Lu: Basically, they built a harness where the model can look at your current project and then write and run small programs in a language like Lua to make edits.

Meng: From my side, what this means is that instead of clicking through fifty sliders to get a sound you want, you could tell the AI what it wants in plain English, and it executes the precise technical changes on your timeline.

Lalam: And the results show that when you give the AI very specific instructions—explicit ones—it gets very good at matching what you asked for in both the project structure and the actual sound file.

Tom: But they also found that when you ask it to just describe a vibe, like "make it sound warmer," things get much less accurate on both the project and the final audio.

Jane: That’s a key distinction, isn't it? It suggests that for these kinds of creative tasks, precision in the input really dictates how well the output is executed by the AI.

Lu: The paper shows that even when you try to measure how good its work is using audio metrics and project state measurements, explicit commands are consistently better than vague ones.

Meng: So if you’re trying to deploy this, it looks like you gotta be very careful about the prompting; it won't guess what you mean if you’re too abstract.

Tom: It really boils down to this: the system is a powerful execution engine, but its intelligence is totally dependent on how clearly the human tells it what to do.

Jane: So, while the harness itself is cool for letting models interact with software, the real magic seems to be in crafting those detailed, explicit commands.

Lu: And this leads us to thinking about how we train these models—do they need more examples of precise code execution versus just general sound descriptions?

More episodes

← Home