Representation Handoffs for OpenArm-Based Laboratory Mobile Manipulation

This paper, authored by Yang Shen, Chonghao Cheng, Ziyi Zhao (University of Technology Sydney) and Jialuo Zhu, Zhenyi Yi, Qi Zhao, Jian Yang, Yuhui Shi (Southern University of Science and Technology), presents an OpenArm-based mobile manipulation prototype for laboratory-style embodied tasks. The authors frame the work as a field report centered on the concept of representation handoffs — explicit interfaces between representations that enable reliable alignment from instructions and observations to safe actions.

The abstract states: Open-source robotics and foundation models have lowered the barrier to embodied AI, yet language-guided laboratory automation still requires reliable alignment from instructions and observations to safe actions. The prototype integrates dual OpenArm manipulators with a mobile base, vertical slide, RGB-D sensing, lidar-based mapping, ROS2/MoveIt execution, and profile-defined skill interfaces.

The system is organized around four key representation handoffs:

1. Natural language requests are constrained into registered skill calls — laboratory instructions are represented by validated skill-call sequences

2. Sensor observations are grounded into maps and object poses — lidar and RGB-D observations are represented by maps and object poses

3. Object priors provide role and skill constraints — object priors are represented by skill arguments

4. Runtime bindings compile validated skills into executable motion goals — runtime bindings are represented by executable motion goals

The authors argue that the main integration question is therefore not only how to represent the scene, but which representation is actionable by a real robot. They emphasize that "This workflow cannot be implemented safely by passing an LLM directly to robot control, and a 6D object pose is not enough either: it must be paired with object roles, skill contracts, frames, geometric operation parameters, safety limits, and execution feedback."

The hardware includes dual OpenArm manipulators with grippers, a vertical slide, a mobile base, RGB-D sensing, and lidar. The software stack uses ROS2 middleware, MoveIt/OpenArm arm execution, linear slide, mobile base navigation, FoundationPose model, and AprilTag.

The top-level agent is a deterministic orchestrator; the LLM is used only by a unified planner that must emit registered SkillCall records against the development profile skill bank. If the planner fails validation or cannot cover a requested capability, the system reports unmapped requests.

The pipeline proceeds as follows: "Before grounding the world, the robot may move to a configured scene origin. World grounding then combines usage-profile object priors, system objects, perception, or a FoundationPose response. The resulting WorldObject list, current robot state, planner-facing skill bank, and memory context form the input to a unified planner. The planner returns a SkillPlan containing ordered SkillCall records, explicit unmapped requests, and safety notes. Validation checks the plan against the skill bank, object role requirements, argument types, and step-id format; one validation-repair attempt is allowed before the request is reported as unmapped. Finally, the embodied runtime pre-checks the plan, refreshes the robot and world states before each skill call, resolves object- and target-specific operation parameters, validates safety limits, and executes a sequence of MotionGoal records."

Spatial representation is the first problem: Laser scans are accumulated into a navigation map for mobile base motion, while RGB-D observations are expressed in calibrated camera and robot frames for manipulation. The calibration profile defines coordinate frames used by the system, including the world, robot base, MoveIt base, tool, camera, lidar, slide, and table frames and records static transforms and verification tolerances used to check frame alignment. AprilTag detections serve as table anchors: given detected tag poses and a known tag-to-table layout, the system can estimate or verify the tabletop frame consumed by downstream grounding and manipulation modules.

The object registry separates system objects from day-to-day laboratory objects with usage profile.yaml storing object priors. Each laboratory object has a name, a point cloud or mesh path, aliases, physical dimensions, semantic roles, and exposed skills. For example, a tube can be represented as movable, a liquid container, a reaction vessel, an insertion target, and a sample container; a rack can be a fixture, a surface, and a place target. The authors relate this to affordance theory: the meaning of an object is grounded in possible actions... For our prototype, affordances are deliberately engineered as explicit roles and skill memberships before they are learned. Each skill in the skill bank declares an owner, target role requirements, typed arguments, preconditions, unsupported requests, and internal process notes.

A WorldObject contains a 6D pose, frame id, confidence, roles, skills, and metadata. The FoundationPose service is "a concrete HTTP/RGB-D protocol: the agent sends target queries, an output frame, camera intrinsics, RGB, depth, and optional masks; the GPU-side service returns object names, frames, confidence, skills, roles, poses, and mesh metadata, with a mock mode for software integration before the GPU perception stack and masks are available."

The operation-parameter planner generates approach, grasp, lift, place, insert, pour, dispense, tap, swipe, and twist parameters, stored as perception args, skill args, and target skill args, then merged by the runtime when compiling a skill call. This handoff is an interpretable bridge between perception outputs and skill templates, so that a dry-run trace can explain exactly why a particular pick pose, place pose, insertion depth, or pour clearance was chosen.

A SkillPlan stores registered SkillCall records, explicit unmapped requests, and safety notes. The runtime expands each skill call into MotionGoal records such as move to, open gripper, set do, slide move, or nav to pose, with every goal passing through a safety gate. The system employs a stop-on-failure policy that is deliberately conservative: "It prevents a language planner from papering over missing perception or unsupported skills. It also makes the representation handoffs easy to debug, i.e., a failure can be localized to scene origin preparation, world grounding, planner validation, skill coverage, operation-parameter resolution, runtime precheck, safety validation, executor routing, or hardware execution."

Table I documents the dry-run trace structure for the instruction put source object on destination object, showing stages from scene setup (scene origin MotionGoal), world state (WorldObject list), planning (SkillPlan plus capability gaps), runtime precheck (required objects, held-object state, safety gate), runtime execution (operation parameters and MotionGoal records like pick-above, pick-down, gripper close, lift, place, release, retreat), and feedback (RuntimeFeedback).

The paper reports that In a dry run, the system can execute all representation handoffs and generate a complete executable trace. For a valid plan, the planner emits a registered move object call against a movable source and a destination object satisfying target-role requirements, and the runtime expands it into a pick-place sequence. When the LLM is unavailable or the request falls outside the skill bank, the same interface produces explicit unmapped requests.

Startup checks reveal the current deployment boundary: "The profile files are parseable, and the dry-run execution path is available, but the real-robot visual loop still depends on recording first-time calibration, replacing placeholder camera/lidar/slide transforms, populating object mesh and point-cloud assets, measuring table height, and enabling the FoundationPose service with real RGB-D input and masks or a detector. The paper explicitly states: currently, it does not provide real-world success rates for visual grasping, pouring, or insertion."

Lesson 1: 6D pose is necessary but insufficient. "The outputs of FoundationPose are valuable, but a mobile manipulation system needs pose plus frame, confidence, object identity, geometry, skills, roles, and target-specific operation parameters. The downstream planner should see an actionable object, not a raw perception result."

Lesson 2: Profiles are representations, not merely configurations. "The usage, calibration, and development profiles encode three different forms of state: object priors, geometric deployment facts, and capability contracts. Treating these profiles as first-class representations made startup checks and dry-run traces more informative than ad hoc configuration files."

Lesson 3: Constrained intermediate forms make LLM planning debuggable. "The LLM only proposes registered skill calls under a skill bank contract. It cannot invent arbitrary ROS commands, bypass role requirements, use undeclared arguments, or execute unregistered recovery. This design sacrifices flexibility, but it makes unsupported capabilities visible as unmapped requests rather than hidden model assumptions."

Lesson 4: Deployment blockers are representation blockers. "The remaining real-robot work is not just hardware bringup. It is the completion of missing representations: measured camera/lidar/slide transforms; table frame localization; object meshes; real RGB-D masks or detections; and calibrated workspace bounds."

The stated contributions are: "(1) A system view of the representation handoffs among lab instructions, maps, object poses, object priors, registered skill calls, runtime bindings, and motion goals; (2) An OpenArm-based integration that connects ROS2/MoveIt execution, mobile base navigation, vertical motion, RGB-D grounding, a skill bank, and deterministic skill templates; (3) Dry-run evidence and field lessons showing where missing calibration, object models, perception outputs, and capability contracts block real-robot deployment."

The conclusion emphasizes that "The central artifact is not a new perception model or controller, but an auditable path from laboratory instructions, maps, object poses, object priors, and skill bank contracts to validated skill calls, operation parameters, motion goals, and runtime feedback. The next steps are to replace placeholder calibration and model assets with measured field data, align the FoundationPose service with the current usage-profile object registry, enable strict real-scene grounding, and evaluate task-level failures across pick, place, insert, pour, clean, and screen-interaction tasks." Code is available at https://github.com/yshenfab/xEI.

Based on the paper, here are specific improvements to AI systems and the resulting capabilities:

1. Constrain LLM planning to a registered skill bank.

Instead of allowing a language model to generate arbitrary robot commands, the planner must emit only validated SkillCall records against a predefined skill contract, with explicit reporting of unmapped requests for unsupported capabilities.

• What the improved system can do: Accept natural-language lab instructions and reliably decompose them into executable, role-checked skill sequences; clearly identify capability gaps rather than silently hallucinating actions.

2. Introduce explicit representation handoffs between perception, semantics, and action.

Design formal interfaces that convert sensor data (RGB-D, lidar) into maps and object poses, then into world objects with roles, skills, and operation parameters, and finally into executable MotionGoal records — rather than passing raw perception outputs or LLM text directly to control.

• What the improved system can do: Localize failures to specific handoffs (e.g., world grounding, planner validation, runtime safety) and provide dry-run traces that explain exactly why each pick pose, insertion depth, or pour clearance was chosen.

3. Engineer object affordances explicitly as roles and skill memberships.

Each object registry entry declares semantic roles (movable, reaction vessel, insertion target, surface, fixture) and the skills it supports, so downstream planning sees actionable objects rather than raw 6D poses.

• What the improved system can do: Correctly determine whether an object can be grasped, placed, poured into, or inserted based on role requirements, enabling safe and flexible task execution without retraining.

4. Provide a deterministic orchestrator with a one-shot validation-repair loop.

Use a top-level agent that validates every LLM-proposed plan against the skill bank, object role requirements, argument types, and step-id format; allow exactly one repair attempt before reporting the plan as unmapped.

• What the improved system can do: Reject malformed or unsafe plans at planning time, avoid cascading runtime errors, and maintain a clear audit trail of why a plan was accepted or rejected.

5. Add runtime pre-checks and safety gates before each skill.

Before executing any skill, refresh robot and world state, resolve object- and target-specific operation parameters, check held-object state, and validate safety limits; enforce stop-on-failure after any error.

• What the improved system can do: Prevent a language planner from masking missing perception or unsupported skills; guarantee that a failed step halts execution and can be debugged to the exact module (perception, planner, runtime, hardware).

6. Treat calibration, usage, and development profiles as first-class representations.

Encode geometric deployment facts (frame transforms, table height), object priors (aliases, roles, skills), and capability contracts (skill declarations, preconditions) as structured, verifiable files used by all components.

• What the improved system can do: Run startup checks that detect missing calibration, placeholder transforms, or misaligned object models before real-robot deployment; reuse the same representations across dry-run and real execution.

7. Enable dry-run traces for full-stack verification without hardware.

Simulate the entire pipeline — scene setup, world grounding, planning, runtime precheck, motion goal expansion, and feedback — with the same data structures and validation logic used on the real robot.

• What the improved system can do: Validate new instructions, skill bank updates, and object registry changes in minutes, catching integration errors before wasting physical robot time.

8. Implement an operation-parameter planner that merges perception outputs with skill templates.

Generate approach, grasp, lift, place, insert, pour, dispense, tap, swipe, and twist parameters as explicit perception args, skill args, and target skill args, which the runtime merges into final motion goals.

• What the improved system can do: Make each action geometrically interpretable and auditable, reducing trial-and-error tuning for mobile manipulation tasks and simplifying transfer to new objects or table layouts.

• Take a laboratory request like put the tube into the rack and automatically produce a complete, validated, executable pick-place trace — including object role checks (tube is movable, rack is an insertion target), operation parameters (approach height, grasp width, insertion depth), and motion goals (nav, move, gripper, lift, place) — all before the robot moves.

• Identify and report unmapped requests when the instruction falls outside the skill bank (e.g., centrifuge the sample when no skill exists), allowing developers to extend capabilities deliberately instead of hoping the LLM improvises.

• Run a full end-to-end dry rehearsal on the exact representations used by the physical robot, making it safe to test new instructions, detection services, or object models without hardware risk.

• Execute mobile manipulation tasks (pick, place, insert, pour, dispense, clean, screen interaction) with conservative stop-on-failure, so any failure is localized to a specific module and never silently overridden by the language model.

Open-source robotics and foundation models have lowered the barrier to embodied AI, yet language-guided laboratory automation still requires reliable alignment from instructions and observations to safe actions. This field report presents an OpenArm-based mobile manipulation prototype for laboratory-style tasks, built by integrating dual OpenArm manipulators with a mobile base, vertical slide, RGB-D sensing, lidar-based mapping, ROS2/MoveIt execution, and profile-defined skill interfaces. The system is organized around representation handoffs: natural language requests are constrained into registered skill calls, sensor observations are grounded into maps and object poses, object priors provide role and skill constraints, and runtime bindings compile validated skills into executable motion goals. We use dry-run traces and startup checks to evaluate this integration path, showing how the prototype exposes missing calibration, incomplete object assets, and unfinished real-scene visual grounding as explicit deployment blockers. These intermediate representations serve as practical debugging interfaces for integrating language, perception, planning, and robot safety in embodied systems.