Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback
cs.RO, cs.CV, cs.LG
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: Project Page: https://fly-by-code.github.io/
Code: https://github.com/jax-ml/jax
Project page: https://fly-by-code.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Embodied coding agents construct robot policies from natural-language instructions by generating executable programs and refining them from execution feedback.
Abstract
Embodied coding agents construct robot policies from natural-language instructions by generating executable programs and refining them from execution feedback. This refinement assumes that execution feedback reveals whether the task succeeded and, after a failure, provides sufficient evidence to guide program revision. Aerial manipulation violates this assumption in two ways. Payload constraints restrict onboard camera coverage and leave task-critical outcomes outside the robot's view (partial observability), while strong dynamic coupling and limited control margins make physical interactions fragile (limited interaction robustness) and leave failure causes unresolved. Re-examining the same feedback cannot recover observations that were never collected or separate failure causes that the available evidence does not distinguish. To acquire this missing evidence, we introduce Active Feedback (AF), an active "View and Probe" framework. When execution feedback leaves unresolved questions, the agent synthesizes a separate diagnostic program that runs in simulation to inspect unseen outcomes from informative viewpoints (View) or test hypotheses about failure causes through targeted physical interactions (Probe). Across six simulated aerial manipulation tasks with 60 trials per condition, AF reduces false completion declarations from 25 to 5 and raises final task success from 31.7% to 75% compared with terminal feedback under the same coding-agent configuration and task-execution limit. Using shared robot APIs, task programs refined with AF in simulation run on a physical aerial manipulator without task-program source-code modification and complete multi-stage manipulation tasks in the real world. These results show that executable diagnosis fills gaps in execution feedback for embodied coding agents, improving both completion assessment and program refinement in aerial manipulation.
Sources
- A Few Words Go a Long Way: Language Guided Robot Policy Synthesis
- RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
- PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration
- ASPIRE: Agentic /Skills Discovery for Robotics
- AERMANI-VLM: Structured Prompting and Reasoning for Aerial Manipulation with Vision Language Models
- Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning
- Creating and Repairing Robot Programs in Open-World Domains
- AeroGrab: A Unified Framework for Aerial Grasping in Cluttered Environments
- $\pi$, But Make It Fly: Physics-Guided Transfer of VLA Models to Aerial Manipulation
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies