TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction".
Jane: Here's the gist of the paper. Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control.
Tom: TOMORI Architecture and Workflow TOMORI connects a language model to the REAPER DAW through a code-based mixing agent harness that provides project observations and executes model-generated Lua programs as DAW editing actions.
Jane: The harness utilizes DeepSeek Harness (DSH), a plugin-based agent runtime, which manages model interaction, tool calling, and dynamic plugin loading.
Tom: This system allows the model to read the current project, select operations, and execute code within the DAW.
Paper summary: Tom: The authors of "TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction" are showing us how to create an agent harness that connects a language model directly to a native digital audio workstation like REAPER <ref:2610.25344#pg3>.
Jane: They use custom plugins that give the model access to project inspection, optional audio measurement, and the ability to run generated Lua code as actual editing actions inside the DAW <ref:2610.25344#pg3>.
Lu: The main point is that this harness, which they call DeepSeek Harness, manages all that model interaction and dynamic plugin loading so the model can read the project state and execute code based on what it observes <ref:2610.25344#pg3>.
Meng: So for us in production, the implication is that we could potentially use this to automate complex mixing tasks where we don't want to manually click through every parameter adjustment <ref:2610.25344#pg1>.
Lalam: The paper’s evaluation shows that explicit instructions lead to high reference agreement across three primary models, while abstract requests tend to reduce the agreement on both the project and the audio output <ref:2610.25344#pg1>.
Tom: In simple terms, this system is very good when you tell it exactly what edit you want to make in a track, but it gets trickier when you ask it for a general feeling or sound description <ref:2610.25344#pg1>.
Jane: The work suggests that the future of these agents in creative software lies not just in understanding the request, but in having reliable, coded tools to execute those changes accurately <ref:2610.25344#pg2>.
Conclusion: Tom: So, we’re wrapping up this look at TOMORI, the tool for connecting language models to audio software like REAPER.
Jane: It seems the big idea behind this paper is that you can use an AI to actually *do* mixing in a digital audio workstation by giving it code instructions.
Lu: Basically, they built a harness where the model can look at your current project and then write and run small programs in a language like Lua to make edits.
Meng: From my side, what this means is that instead of clicking through fifty sliders to get a sound you want, you could tell the AI what it wants in plain English, and it executes the precise technical changes on your timeline.
Lalam: And the results show that when you give the AI very specific instructions—explicit ones—it gets very good at matching what you asked for in both the project structure and the actual sound file.
Tom: But they also found that when you ask it to just describe a vibe, like "make it sound warmer," things get much less accurate on both the project and the final audio.
Jane: That’s a key distinction, isn't it? It suggests that for these kinds of creative tasks, precision in the input really dictates how well the output is executed by the AI.
Lu: The paper shows that even when you try to measure how good its work is using audio metrics and project state measurements, explicit commands are consistently better than vague ones.
Meng: So if you’re trying to deploy this, it looks like you gotta be very careful about the prompting; it won't guess what you mean if you’re too abstract.
Tom: It really boils down to this: the system is a powerful execution engine, but its intelligence is totally dependent on how clearly the human tells it what to do.
Jane: So, while the harness itself is cool for letting models interact with software, the real magic seems to be in crafting those detailed, explicit commands.
Lu: And this leads us to thinking about how we train these models—do they need more examples of precise code execution versus just general sound descriptions?
cs.SD, eess.AS
Submitted: 2026-10-18
Updated: 2026-11-06
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI…
Key concepts
- TOMORI Architecture
- TOMORI is a system that links an AI language model to the REAPER digital audio workstation. It uses a code-based agent harness to allow the model to observe the project, select actions, and run Lua programs directly in REAPER. This enables the model to perform complex mixing tasks by interacting with the DAW through code.
- DeepSeek Harness (DSH)
- DeepSeek Harness is a plugin-based agent runtime that manages how the language model interacts with tools and plugins. It handles tasks like managing model conversations, calling necessary functions, and dynamically loading plugins. This component is essential for allowing the AI to execute actions within the DAW environment.
- Instruction Forms (IF0–IF2)
- These are different ways users ask for mixing changes. IF0 represents precise control instructions where the user specifies exactly what needs to be done. IF2 represents more abstract requests, describing the intended sound quality without detailing every edit. The system tests how well TOMORI performs under these varying levels of instruction specificity.
- Evaluation Metrics
- These are specific measurements used to judge how well TOMORI performs its mixing tasks. Metrics like Graph F1 and Parameter Precision assess agreement between the AI's edits and human reference edits in both the project files and the resulting audio. They help quantify whether explicit instructions lead to better accuracy than abstract ones.
Terminology
Summary
Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control. p. 0 com
Abstract
Code-based agents can operate creative software, but mixing requests often describe an
intended sound without specifying the edits
needed to achieve it. We present TOMORI, a
code-based mixing-agent harness for native digital audio workstation (DAW) control.Where the paper says it
TOMORI Architecture and Workflow
TOMORI connects a language model to the REAPER DAW through a code-based mixing agent harness that provides project observations and executes model-generated Lua programs as DAW editing actions. The harness utilizes DeepSeek Harness (DSH), a plugin-based agent runtime, which manages model interaction, tool calling, and dynamic plugin loading. p. 2 1 Plugin Architecture
TOMORI utilizes DeepSeek Harness (DSH), a
plugin-based agent runtime (DeepSeek-AI, 2026). DSH provides model interaction, tool calling, and
dynamic plugin loading. They also manage rendering and return execution
results to the model. Through these interfaces, the
model reads the current project, selects operations,
and executes code within the DAW. User Language model Agent harness REAPER
loop [choose tools; repeat as needed]
alt [Project inspection]
[Optional audio analysis]
[Project editing / rendering]
Mixing request
Inspect project
Read project state
Tracks / FX / parameters
Project snapshot
FFmpeg / analysis code
Audio measurements
Run editing code
Edit / render
Updated project / audio
Execution result / error
Project + rendered audio
Figure 1: Sequence diagram of TOMORI. Within the loop, the model selects project inspection, audio analysis, or
code-based project editing (alt), then uses returned results to decide whether to continue (loop).Where the paper says it
Where the paper says it
Where the paper says it
Data Construction for Benchmarking
TOMORI-BENCH pairs 110 human-reviewed reference edits with three instruction forms (IF0–IF2) and evaluates native DAW execution end to end using project-state and rendered-audio metrics. The reference edits are constructed using Phoenix ScotchMorris from MedleyDB, which provides original multitracks and a finished mix. p. 3 request–project correspondence has been audited,
and redacted examples are provided in Appendix B. To construct executable references, we use
Phoenix_ScotchMorris from MedleyDB, which
provides original multitracks and a finished mix
(Bittner et al. Along with
normal edits, the sequence includes overshoots and
subsequent repairs to assess execution under different project states and adjustment directions. 2 Reference Construction
Task coverage and instruction styles are informed
by Everytime-72, a continuous eight-stem production session containing 72 human requests and
corresponding project snapshots. • IF2: Describes the intended sound of the target object, leaving operation selection and parameter settings to the model. From IF0 to IF2, the forms represent requests from
users with decreasing mixing expertise, moving
from precise control instructions to descriptions
of the intended sound.Where the paper says it
Where the paper says it
Where the paper says it
Evaluation Metrics and Findings
The end-to-end evaluation assesses reference agreement in both the delivered native project and its rendered audio using metrics like Graph F1, Parameter Precision, Audio Metrics, and Edit Progress. Explicit requests yield high reference agreement across three primary models while abstract requests reduce project and audio agreement. p. 0 TOMORI-BENCH
pairs 110 human-reviewed reference edits with
three instruction forms and evaluates native
DAW execution end to end through operation
structure, parameter precision, and renderedaudio outcomes. Across 990 cases from three
primary models, explicit instructions yield high
reference agreement, while abstract requests
reduce project agreement and audio improvement. Across three primary models, explicit requests
are executed precisely, while abstract requests reduce project and audio agreement. Paired comparisons show that the full workflow improves explicit
execution over a single submission, but prompted
audio measurement provides no clear overall improvement in reference agreement on selected abstract cases.Where the paper says it
Where the paper says it
Ablation and Limitations
Ablation studies assess the editing workflow and access to audio measurements, showing that the full workflow improves explicit execution over a single submission. However, prompted measurement provides no clear overall benefit over prohibited analysis in certain conditions. p. 5 On 28 selected signal-dependent
or historical-analysis steps, we compare prohibited
analysis (L) with prompted measurement before
editing (M). Neither condition shows a clear overall difference in reference-based scores relative to
the full workflow, and M shows no clear overall
benefit over L. Project metrics directly test
explicit requirements, but under abstract instructions they measure agreement with one chosen implementation. Neither project nor audio agreement
alone establishes whether an alternative satisfies
the request. Second, TOMORI-BENCH contains edits from
one song, with one final delivery per model–
instruction pair. Broader generalization and generation stability require additional songs and repeated runs.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Precise execution under explicit instructions can be achieved by pairing language models with a code-based mixing agent harness that inspects REAPER projects, optionally measures audio, and generates Lua code to execute native DAW edits. This system can perform end-to-end evaluation through operation structure, parameter precision, and rendered audio outcomes across 110 human-reviewed reference edits.
-
The system can generate Lua code to modify the current project by performing several observation and editing rounds within one request, such as comparing vocal and accompaniment band energies or inspecting routing while adjusting EQ. The final output consists of an editable REAPER project and rendered audio, allowing for iterative refinement based on textual feedback like
bring the vocals forward.
-
For abstract requests where explicit instructions are missing, the system can still attempt to improve execution by using a full workflow that includes project inspection, optional audio measurement, and model-generated Lua code. The paper shows that
the full workflow improves explicit execution over a single submission.
Abstract
Code-based agents can operate creative software, but mixing requests often describe an intended sound without specifying the edits needed to achieve it. We present TOMORI, a code-based mixing-agent harness for native digital audio workstation (DAW) control. Custom plugins provide REAPER project inspection, optional audio measurement, and execution of model-generated Lua code. All observations return as text; delivery includes an editable project and rendered audio. TOMORI-BENCH pairs 110 human-reviewed reference edits with three instruction forms and evaluates native DAW execution end to end through operation structure, parameter precision, and rendered-audio outcomes. Across 990 cases from three primary models, explicit instructions yield high reference agreement, while abstract requests reduce project agreement and audio improvement. Paired ablations show that the full workflow improves explicit execution over a single submission. Prompted audio measurement succeeds in all 28 selected abstract cases but yields no clear overall improvement on the reference-based metrics and increases tool calls. These results expose the gap between executable DAW control and precise realization of perceptual mixing goals.
Sources
- Text2FX: Harnessing CLAP Embeddings for Text-Guided Audio Effects
- DAWZY: A New Addition to AI powered "Human in the Loop" Music Co-creation
- The BrowserGym Ecosystem for Web Agent Research
- GRAFX: An Open-Source Library for Audio Processing Graphs in PyTorch
- Searching For Music Mixing Graphs: A Pruning Approach
- WavCraft: Audio Editing and Generation with Large Language Models
- RIME: Enabling Large-Scale Agentic Music Post-Production
- ST-ITO: Controlling Audio Effects for Style Transfer with Inference-Time Optimization
- Diff-MST: Differentiable Mixing Style Transfer
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems