ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent

summary

The gist

Connecting hand input in mixed reality (MR) to a nearby physical device typically requires an integration written in advance for that device, its transport and the mapping On a Meta Quest 3…

In short

ControlAnything allows users to connect hand input in mixed reality to physical devices by having an AI agent generate custom software on demand. Users select an object and state a goal, prompting the system to create a connector and either augmented-reality controls or a hand program that maps their hands to control that device. The system aims for on-demand integration without needing pre-written integrations for every device.

Key concepts

Connector
A small Node program written by an AI agent that declares specific actions, like 'cutoff,' and translates those actions into messages the target physical device can understand. It acts as the bridge between the abstract control commands and the actual hardware communication.
Hand Program
A JavaScript function that runs every frame on a headset, mapping tracked hand movements directly to specific device actions defined by a connector. This program is generated dynamically by the AI agent based on what controls are needed for a given task.
Agent Runtime Loop
The process where an AI agent iteratively writes, tests, and refines connectors until they successfully communicate with the live device. It involves low reasoning effort and strict computational limits to ensure fast and efficient generation of the necessary integration code.
Evidence of Effect
The data gathered from a system to prove that its generated controls or hand programs actually cause a physical change in the connected device. Initially, this evidence was limited to transport acknowledgments, but future work aims to use headset sensors like cameras and microphones to observe actual physical results.

Terminology used across episodes

This episode discusses

The paper

ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent · Read on arXiv

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent".

Jane: Here's the gist of the paper. Connecting hand input in mixed reality (MR) to a nearby physical device typically requires an integration written in advance for that device, its transport and the mapping On a Meta Quest 3, the wearer selects an object outlined by SAM 3.1 and states what their hands should control ControlAnything defers this integration to the moment of use and assigns it to a coding agent This system generates augmented-reality (AR) controls, a per-frame hand program, or both The findings point to connector reuse and the observation of effects through the headset’s own sensors as next steps for on-demand device integration in MR.

Tom: The Core Mechanism ControlAnything implements a loop where an agent writes a connector, tests it against the live device, and returns augmented-reality (AR) controls or a hand program The process involves selecting an object using SAM 3.1 and stating a goal, such as “give me knobs for my OP1 filter connected to this Mac” An agent built on Meta’s Muse Spark model writes a connector, a small Node program that declares named actions such as cutoff and translates them into device messages The agent runs the connector, sends test values through it to the live device and reads the acknowledgements It then returns AR controls bound to the connector’s actions, a hand program that the headset runs on every frame to map tracked hands to those actions, or both.

Jane: Design Goals and Implementation The design is guided by four goals: DG1, integration at the time of use; DG2, grounded and checked output; DG3, no model in the control loop; and DG4, interruptible and bounded execution The interaction follows five steps: Select (SAM outlines), Ask (spoken goal), Generate (agent's code card), Annotate, and Worker (AR controls or hand program) The server runs a tool loop with muse-spark-1.3, with streaming function calling, one tool call per turn and low reasoning effort Connectors are Node programs that declare actions and translate them into transport operations A hand program is a JavaScript function wire(s, out, mem) of at most 4,096 characters that the headset calls on every frame.

Tom: Technical Evaluation Results The technical evaluation characterized the loop on real hardware through three questions: EQ1: Does the loop reach a ready state, with controls or a hand program bound to a running connector, for targets that use different transports?

Title and authors: Tom: So the core mechanism is this loop where an agent writes a connector, tests it against the live device using hand-written transports, and then spits out either augmented reality controls or a hand program to run on your headset. It’s this whole generation and testing cycle happening in real time.

Jane: Let's unpack that a little more: you select something outlined by SAM three point one, you give the agent a goal like “give me knobs for my OP-one filter connected to this Mac”, and then an agent built on Meta’s Muse Spark model writes a small Node program, which is this connector <ref:2611.08321#pg2,built on Meta’s Muse Spark model writes a>.

Lu: That connector declares named actions, like saying "cutoff," and translates those into actual device messages. The agent then runs that connector, sends test values through it to the live device, checks for acknowledgements, and only when it’s good does it return the controls or hand program.

Meng: It sounds like a lot of sequential calls here. How long is this wait time you have to deal with before you actually get those controls ready? I’m worried about latency in a real-world scenario where the device communication isn't instant.

Lalam: The technical evaluation showed that nine out of ten requests reached a ready state, and the median time to ready was about thirty two point five seconds. But the paper notes that sequential model calls accounted for about eighty nine point nine percent of that waiting time.

Tom: Eighty-nine point nine percent is huge; it means most of the waiting is just the agent talking to itself or running those model calls, not necessarily waiting for the physical device connection itself.

Jane: The paper also mentions that while they get acknowledgements from the transport layer—like a gimbal bridge logging eighty eight nine rotate answers from an Osmo Mobile four—they don't actually measure any physical effect yet. It’s more about verifying the communication path than seeing a tangible result on the device.

Lu: That is a key distinction they make: they get confirmation that the data went somewhere, but not necessarily that it caused the device to move or change state in a meaningful way.

The paper's summary: Tom: The paper outlines four main design goals for this system. First, integration at the time of use, which we already talked about—no pre-integration needed. Second, grounded and checked output, meaning it has to be verified before you use it.

Jane: And the third goal is no model in the control loop itself. This means that once you have those controls or hand program generated, the headset runs that code locally on every frame instead of needing constant back-and-forth with a big model for every tiny movement.

Lu: The fourth design goal is interruptible and bounded execution, which addresses how these agents can run without just spinning indefinitely. They are aiming for this to be able to stop or handle things if the process gets too long or complex.

Meng: If we’re talking about deployment, that boundedness is crucial. We need systems that respect computational budgets because running a complex agent loop on every single frame sounds expensive at scale.

Lalam: They detail the interaction following five steps: Select, Ask, Generate code card, Annotate, and then the Worker which is either the AR controls or the hand program itself. It’s a very clear workflow for what happens in each step of this generative loop.

Tom: So it’s not just one big generation step; it’s a structured sequence of actions where the agent writes code, tests it, and then hands off the resulting control mechanism to be executed by the headset. It formalizes how that on-demand integration happens.

The paper's improvements: Jane: So to wrap up this paper, ControlAnything treats both the interface to a device and the connection underneath it as artifacts that an AI agent writes on request, on top of hand-written transports. It’s essentially generative UI driven by an AI model for physical interactions.

Tom: The main open problems they point out are shortening that waiting time and closing the gap between just getting acknowledgements from the transport and actually seeing a physical effect happen. That's where the real engineering challenge lies right now.

Lu: I think what’s really important is how they approach those limitations; they suggest things like tags on devices themselves or an operating system sandbox specifically for connectors to make it more reliable in production environments.

Meng: From a deployment view, I think their focus on making the agent loop more robust and predictable under strict computational limits is where the immediate practical value is, even before we solve the physical effect measurement part.

Lalam: And Lalam says that if they can improve those things, like adding tags to devices and building an operating system sandbox for those connectors, it moves this from a cool lab experiment to something you could actually deploy in real mixed reality applications.

Jane: So we’re looking at a system where the interface creation is on demand, driven by an AI agent writing code that connects things together, and now the next step is making sure that connection feels real and works reliably in production.

Tom: That's ControlAnything. It shows how agents can be used not just to talk to other software, but to build the very interfaces we interact with physically in mixed reality. That’s a solid direction for AI interaction design.

Conclusion: Tom: So we just looked at ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent, and the main idea is that an AI agent can write code to connect your hand movements in mixed reality to physical devices on demand.

Jane: It’s about selecting an object you outline with your hand, saying what you want it to do, and having an agent instantly generate the controls or a program for your headset to use right then.

Lu: The core mechanism is this loop where an agent writes a connector that tests itself against the live device using hand-written transports and returns augmented reality controls or a hand program. It’s about generating both the interface and the connection beneath it simultaneously.

Meng: But what we saw in the testing was that it takes quite a while to get that ready state, with about thirty two point five seconds median wait time, and most of that time was just sequential calls between models.

Lalam: And they confirmed that while they got acknowledgements from the device—like seeing rotation answers from a gimbal—they didn't actually measure any physical effect yet. It was more about verifying the communication path than seeing a tangible result on the device.

Tom: That gap between getting an acknowledgement and actually feeling something happen is still pretty big, which is what they call the main open problem: shortening that wait time and closing that gap to physical effect.

Jane: They’re looking at using the headset's own sensors, like its camera and microphone, to observe things like ripples or gimbal motion instead of just relying on the transport acknowledgements.

Lu: That would be a big step in making sure the system is grounded and checked because it moves beyond just confirming data transfer to actually proving physical interaction.

Meng: For deployment at scale, I’m still concerned about how fast that loop can run when you have thousands of users trying to generate these connectors simultaneously. The computational budget needs to be strictly managed.

Lalam: I'm fairly confident in the evidence they gathered from one wearer and one session, but they are aware that those aren't independent trials, so it’s a starting point for what’s possible.

Tom: So ControlAnything shows us how AI can act as a rapid prototyping tool to build interfaces on the fly, but making that interface feel physical and reliable is the next big hurdle.

Jane: Exactly. We have to move past just proving the connection works and start proving that it actually *does* something in the real world.

Lu: It’s interesting because it treats both the interface and what's underneath as generated artifacts, which changes how we think about building dynamic systems for mixed reality.

Meng: Yeah, so if we can build a connector host that reports success back to the agent and restricts those hand programs before they even hit the headset, that would make it much more stable for production.

Lalam: I agree with Meng; adding those checks sounds like a necessary layer of defense before you let an agent write something live.

Tom: It’s a fascinating piece of work on generative UI and on-demand device integration, so keep an eye on how they tackle that physical effect measurement next.

More episodes

← Home