Decentralized LLM-Driven Coordination of Acoustic Robots for Contactless Object Manipulation

arXiv:2605.29378 · cs.RO · Submitted 2026-05-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Decentralized LLM-Driven Coordination of Acoustic Robots for Contactless Object Manipulation".

Dev: The gist: This paper presents a decentralized framework for natural language-driven coordination of acoustic robots for contactless object manipulation.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: The authors are Yingying Wang, Narsimlu Kemsaram, and Sriram Subramanian. They put this paper out because they see a real need for making these multi-robot systems more accessible to non-experts through language.

Dev: Exactly, the title itself points to the core idea: decentralized coordination driven by a large language model for acoustic robots doing contactless manipulation. It’s about taking spoken words and turning them into actual robot tasks across several machines without needing a central brain that dictates every single move.

Taro: So, when you look at the authors and the title, it signals they are focusing on integrating something brand new—the LLM aspect—with an existing field like acoustic manipulation and multi-robot systems. It suggests they’re moving away from those older methods of hard-coding commands into flexible language instructions.

Rosa: Right, so it’s about making the human part much easier by giving them a natural interface for these complex physical tasks. We need to think about what that means for real-world use, right?

Dev: It means less time spent on writing complex control scripts and more time focusing on designing the physical acoustic manipulation part itself, assuming the language parsing actually works reliably.

Taro: And from an autonomy standpoint, it’s interesting because it moves the high-level reasoning—what to do next—out of a pre-programmed sequence and into something that can be interpreted dynamically based on what a user says.

The paper's summary: Rosa: The summary explains that the system takes spoken instructions and converts them into executable task plans by combining speech recognition, language parsing, structured JSON task representation, and distributed scheduling. It’s this pipeline that turns what you say into what the robots do.

Dev: That structured JSON part is key; it’s not just a string of words from the LLM; it encodes things like which robot does what, when they need to synchronize, and where they should be located relative to each other. It translates the messy human language into something machine-readable for coordination.

Taro: That structure directly addresses the coordination challenge I mentioned earlier. It’s not just a list of actions; it specifies temporal dependencies and synchronization constraints, which is crucial when you have multiple acoustic robots working together in the same space.

Rosa: And it supports four different ways of coordinating: single robot sequential execution, multi-robot parallel execution, cross-robot ordered execution, and synchronous collaborative execution. That flexibility is what makes this framework powerful for different kinds of manipulation jobs.

Dev: It’s a big step because existing LLM frameworks rarely handle the distributed coordination mechanisms or the real-time synchronization needed across multiple robots in a dynamic environment. This paper proposes integrating speech recognition and language-based task planning to convert natural-language commands into executable actions for a collaborative robot, which is pretty ambitious.

Taro: The summary also touches on the motivation—that acoustic manipulation, like ultrasonic levitation, is important for contamination-sensitive areas like biomedical labs or pharmaceutical manufacturing where physical contact must be avoided. That grounds the whole system in a very practical application space.

The paper's improvements: Rosa: The authors point out several improvements they made to the original concept. They focused on making the language parsing more robust by using a multi-stage prompting strategy with constrained action vocabularies and parameter validation.

Dev: That’s smart because it makes the system less likely to break when a user uses slightly different words than expected; they start with a low sampling temperature for less creative interpretation and increase it only if the parsing fails, which should keep things stable.

Taro: I also see them improving the coordination logic by introducing a TCPstyle three-way handshake to ensure reliable task acknowledgment and synchronized start signals over ROS two wireless communication. That’s how they get that barrier synchronization logic guarantee for simultaneous execution in synchronous mode.

Rosa: So, they’re adding explicit mechanisms for reliability and synchronization on top of the basic LLM translation. It seems like they are trying to make the coordination modes—sequential, parallel, and synchronous—much more dependable than what was possible before.

Dev: They also refined how the scheduling bridges those parsed tasks to actual execution by defining sequence and sync group fields in that JSON representation, which is fundamental for managing those different coordination modes effectively. It’s about making sure the schedule is rigorous enough for real-time robot control.

Conclusion: Rosa: So, to wrap up on the "Decentralized LLM-Driven Coordination of Acoustic Robots for Contactless Object Manipulation," this framework successfully combines speech recognition, language parsing, distributed scheduling, and robotic control into one unified architecture.

Dev: The results showed that sequential tasks achieved a ninety-six percent execution success rate with a parsing accuracy of ninety-nine percent and a latency of one point two seconds. However, the synchronous collaborative scenario was the hardest part at seventy percent execution success with an eight-eight percent parsing accuracy and a latency of two point five seconds.

Taro: What this means for us is that while sequential tasks are pretty reliable, you can see how the execution reliability drops as the coordination complexity increases, especially in that synchronous mode where spatial alignment and timing synchronization between robots become real issues.

Rosa: The paper shows that LLM-based task interpretation can significantly simplify human–robot interaction while enabling flexible coordination in multi-agent robotic systems. It proves the feasibility of using spoken commands to manage multiple acoustic robots doing contactless manipulation for things like healthcare logistics or lab automation.

Dev: Overall, the integration of LLM-driven task reasoning with distributed acoustic manipulation provides a promising foundation for intuitive human–robot collaboration in future autonomous robotic systems.

Taro: I just want to add that this work points toward a future where we can have more intuitive interaction with swarms of acoustic robots, even when things go wrong, because the framework is designed to handle dynamic environments better than traditional scripts.

Rosa: That’s a good point about robustness in dynamic environments. So, that’s what we have for this paper today on the "Decentralized LLM-Driven Coordination of Acoustic Robots for Contactless Object Manipulation."

University College London · University of Malaya

cs.RO

Submitted: 2026-05-28

Updated: 2026-05-28

Comments: This paper has been accepted for publication in the Proceedings of the 2026 IEEE 22nd International Conference on Automation Science and Engineering (CASE 2026), August 17-21, 2026, Shenyang, China

Journal ref: Proceedings of the 2026 IEEE International Conference on Automation Science and Engineering (CASE 2026), 17-21 August, 2026, Shenyang, China

DOI: 10.1109/CASE69030.2026.11704165

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 73/100

The gist: The gist: This paper presents a decentralized framework for natural language-driven coordination of acoustic robots for contactless object manipulation.

Key concepts

Natural Language-Driven Coordination: This concept involves letting users issue high-level commands using everyday spoken language rather than writing specific code. The system interprets these words to figure out which robot should do what, when, and how to work together on a task.
LLM-Based Semantic Parsing: The Large Language Model (LLM) acts as the brain that understands human language. It takes spoken or written instructions and translates the meaning into a structured format—a machine-readable plan—detailing robot assignments, dependencies, and timing constraints.
Structured JSON Task Representation: The parsed instructions are converted into a strict JSON format. This standardized structure precisely encodes all necessary details for robot execution, including which robots handle which parts of the task, when actions must happen sequentially or in parallel, and any required synchronization points.
Distributed Scheduling and Coordination: This module manages how multiple robots execute tasks simultaneously. It handles different coordination styles like sequential steps, running tasks at the same time (parallel), or working together on a shared object (synchronized), ensuring all robots follow the correct plan.

Terminology

Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes

Summary

The gist: This paper presents a decentralized framework for natural language-driven coordination of acoustic robots for contactless object manipulation.

System Concept

The proposed system introduces a decentralized coordination framework for acoustic mobile robots that enables users to issue high-level natural language commands for contactless object manipulation. Unlike traditional robotic systems that rely on predefined scripts or centralized planners, the proposed framework transforms spoken instructions into executable multi-robot task sequences using large language models (LLMs). The overall objective is to reduce the need for domain-specific programming while enabling flexible multi-robot coordination in acoustophoretic robotic systems

System Architecture

The overall architecture of the proposed framework is shown in Figure 2, comprising six functional modules that transform user input into coordinated robot actions The system comprises (i) speech recognition, (ii) LLM-based language parsing, (iii) structured task representation, (iv) distributed scheduling and coordination, (v) robot control and motion execution, and (vi) acoustic manipulation Natural language interaction begins with an onboard speech interface using the Whisper automatic speech recognition model The language parsing module forms the core of the proposed system by converting natural language instructions into structured machine-readable task representations using an LLM This parsed output is encoded as a structured JSON task plan containing robot assignments, action types, sequence dependencies, and synchronization constraints

Control Flow and Execution Pipeline

The proposed framework adopts a finite-state machine (FSM) to coordinate the full control pipeline from startup to task execution The control flow involves several stages:

Idle State and Wake-Word Detection: The system starts in an idle state and continuously monitors for predefined activation phrases through low-power audio processing
Speech Processing: Upon wake-word activation, the speech subsystem uses voice activity detection to capture the spoken command and then transcribes it using Whisper
Language Parsing: The language processing module converts user commands into structured JSON task representations using a multi-stage parsing pipeline with error recovery This framework uses constrained prompts that define action vocabularies, parameter specifications, and coordination semantics
Distributed Scheduling and Coordination: The scheduler bridges parsed tasks and execution and supports four coordination modes: single-robot sequential execution, multirobot parallel execution, cross-robot ordered execution, and synchronous collaborative execution Coordination is represented using sequence and sync group fields
Robot Motion and Task Execution: The robot controller dispatches low-level motion commands derived from the parsed tasks For synchronous collaborative transport, the contactlessTransport module enables coordinated dual-robot object manipulation
Feedback and State Management: The system provides real-time audio feedback throughout execution to confirm task progress and completion status

Experimental Validation

The framework was evaluated in three task scenarios using two TurtleBot3-based acoustic robots in a tracked indoor environment The three representative task coordination scenarios considered are:

Sequential Task Execution: This scenario evaluates whether a single acoustic robot can reliably execute ordered multi-step transport tasks derived from natural language instructions
Parallel Multi-Robot Execution: This scenario evaluates decentralized task execution in which two robots operate simultaneously on separate object transport tasks
Synchronous Collaborative Manipulation: This scenario evaluates the most challenging coordination mode where two robots cooperatively transport a shared object using synchronized acoustic manipulation

System Performance

Table I summarizes system performance across the three task coordination modes in terms of execution success, parsing accuracy, and latency Sequential tasks achieved the highest execution success rate of 96%, with parsing accuracy of 99% and latency of 1.2 seconds Parallel execution achieved 86% success with 95% parsing accuracy and 1.8 seconds latency The synchronous collaborative scenario remained the most challenging, achieving 70% execution success with 88% parsing accuracy and 2.5 seconds latency Overall, these results indicate that the proposed framework maintains high natural language parsing performance across all scenarios, while execution reliability decreases as coordination complexity increases

Natural Language and Speech Interface Performance

The combination of Whisper-based speech recognition and LLM-based task parsing provided a reliable interface for natural language-driven multi-robot coordination Speech recognition performance was evaluated using three Whisper model configurations, with the Whisper API achieving the highest recognition accuracy (98%) while also providing lower latency than the locally deployed Whisper Tiny (92%) and Whisper Base (94%) variants Overall, the combination of Whisper-based speech recognition and LLM-based task parsing provided a reliable interface for natural language-driven multi-robot coordination

Conclusion

The proposed system combines speech recognition, language parsing, distributed task scheduling, and robotic control into a unified architecture that allows non-expert users to interact with multiple robots using intuitive spoken commands Experimental evaluation across three task scenarios demonstrated the feasibility of the proposed framework Sequential task execution achieved high parsing accuracy and low latency, while parallel multi-robot coordination successfully enabled simultaneous task execution Synchronous collaborative manipulation highlighted the potential of distributed acoustic transport, although it also revealed challenges related to spatial alignment and timing synchronization between robots Overall, the results demonstrate that LLM-based task interpretation can significantly simplify human–robot interaction while enabling flexible coordination in multi-agent robotic systems The proposed framework shows promise for applications in healthcare logistics, laboratory automation, and contamination-sensitive environments where contactless manipulation is desirable This work demonstrates that integrating LLM-driven task reasoning with distributed acoustic manipulation provides a promising foundation for intuitive human–swarm interaction in future autonomous robotic systems

Acknowledgments

This work was supported by the EPSRC Prosperity Partnership Program - Swarm Spatial Sound Modulators (EP/V037846/1), and by the Royal Academy of Engineering through their Chairs in Emerging Technology Program (CIET 17/18)

References

[1] Y. Rizk, M. Awad, and E. W. Tunstel, “Cooperative heterogeneous multi-robot systems: A survey,” ACM Computing Surveys (CSUR), vol. 52, no. 2, pp. 1–31, 2019

[2] J. Ota, “Multi-agent robot systems as distributed autonomous systems,” Advanced engineering informatics, vol. 20, no. 1, pp. 59–70, 2006

[3] Z. Yan, N. Jouandeau, and A. A. Cherif, “A survey and analysis of multi-robot coordination,” International Journal of Advanced Robotic Systems, vol. 10, no. 12, p. 399, 2013

[4] A. Gautam and S. Mohan, “A review of research in multi-robot systems,” in 2012 IEEE 7th international conference on industrial and information systems (ICIIS). IEEE, 2012, pp. 1–5<ref:

Improvements for AI systems

Improved human-robot interaction through natural language commands by implementing a pipeline that combines Whisper-based speech recognition, LLM-driven semantic parsing, structured JSON task representation, and distributed scheduling. This allows non-expert users to issue high-level commands that are automatically translated into executable actions for a collaborative robot.
Enhanced coordination capabilities by introducing a structured task representation model with a JSON-based task schema that encodes execution order, synchronization constraints, and robot assignments, which enables the system to support sequential, parallel, and synchronized execution modes. This structure directly addresses the challenge of needing structured task representations that capture temporal dependencies, synchronization constraints, and spatial relationships.
Robust decentralized coordination by implementing a mechanism where a TCPstyle three-way handshake ensures reliable task acknowledgment and synchronized start signals over ROS 2 wireless communication, which allows for reliable synchronization without centralized control and supports barrier synchronization logic guarantees simultaneous execution in synchronous mode.
Improved reliability during language parsing by using a multi-stage prompting strategy with constrained action vocabularies and parameter validation, which improves robustness against linguistic variability by using a low sampling temperature initially and increasing it upon failure.
Increased operational flexibility by enabling the system to support four coordination modes: single-robot sequential execution, multirobot parallel execution, cross-robot ordered execution, and synchronous collaborative execution, allowing the framework to handle tasks ranging from single-robot operation to cooperative multi-robot manipulation.

Abstract

Natural language interfaces can simplify interaction with multi-robot systems, especially when non-expert users need to issue high-level commands. Acoustic manipulation using ultrasonic phased arrays also enables contactless object handling for applications such as healthcare, laboratory automation, and precision transport. However, combining large language models (LLMs) with distributed acoustic mobile robots remains underexplored. This paper presents a decentralized framework for natural language-driven coordination of acoustic robots for contactless object manipulation. The system converts spoken instructions into executable multi-robot task plans using Whisper-based speech recognition, LLM-based semantic parsing, structured JSON task representation, and distributed scheduling. The JSON schema encodes robot assignments, temporal dependencies, spatial constraints, and synchronization requirements for sequential, parallel, and synchronized execution. The system is implemented on two TurtleBot3-based acoustic robots, each equipped with an ultrasonic phased array for contactless object transport. Experiments were conducted in three scenarios: sequential execution, parallel multi-robot transport, and synchronized cooperative manipulation. The system achieved task success rates of 96 percent for sequential tasks, 86 percent for parallel execution, and 70 percent for synchronized collaborative transport. These results show that natural language commands can be transformed into distributed robot actions for contactless manipulation, highlighting the potential of LLM-driven automation for human-robot interaction in distributed robotic systems.