Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories

arXiv:2509.16176 · cs.RO · Submitted 2025-09-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Agentic Aerial Cinematography".

Dev: Here's the gist of the paper. We present Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories (ACDC), an autonomous drone cinematography system driven by natural language communication between human directors and drones The main limitation of previous drone cinematography workflows is that they require manual selection of waypoints and view angles based on predefined human intent, which is laborintensive and yields inconsistent performance In this paper, we propose employing large language models (LLMs) and vision foundation models (VFMs) to convert free-form natural language prompts directly into executable indoor UAV video tours Specifically, our method comprises a vision–language retrieval pipeline for initial waypoint selection, a preferencebased Bayesian optimization framework that refines poses using aesthetic feedback, and a motion planner that generates safe quadrotor trajectories We validate ACDC through both simulation and hardware-in-the-loop experiments, demonstrating that it robustly produces professional-quality footage across diverse indoor scenes without requiring expertise in robotics or cinematography These results highlight the potential of embodied AI agents to close the loop from open-vocabulary dialogue to real-world autonomous aerial cinematography.

Rosa: System Architecture Overview The system overview of ACDC involves a sequence where a natural-language prompt leads to an exploratory video, a highfidelity 3D environment model, and the retrieval and refinement of poses for trajectory generation The overall process is illustrated by three main stages: Initial Waypoint Selection, Pose Refinement, and Trajectory Generation.

Dev: Initial Waypoint Selection Process The first stage involves retrieving candidate frames from the exploratory scan that are semantically aligned with the prompt and ordering them into an initial waypoint sequence reflecting user intent This pipeline consists of several steps, including Visual-Textual Embedding, Similarity Matching, Intention-based Parsing, Frame Candidates per waypoint, and Description-based Selection.

Rosa: The process generates a set of ordered initial posed waypoints denoted as Tˇ = 0 to j with 0 < Tˇ ≤ Ck.

Paper summary: Rosa: So, to recap where we are, this paper introduces Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories, which claims they can use natural language communication between humans and drones to create cinematic shots automatically. They suggest that instead of manual waypoint selection, you just give the drone a prompt like "show me the kitchen from above," and it figures out the whole shot for you.

Dev: They are using a pipeline where first, they use a vision-language retrieval system to pick initial frames from an exploratory video that match your description. Then, they use something called preference-based Bayesian optimization to tweak those poses until they look aesthetically pleasing and match your language prompt better.

Taro: The refinement part sounds interesting because it suggests the AI gets feedback on what looks good, using a vision-language model to judge the pose adjustments.

Rosa: That’s right. They are optimizing for aesthetics guided by your language prompt, which they treat as the objective of that optimization process. It's not just about finding a frame; it's about finding the *right* frame according to your artistic vision expressed in words.

Dev: And after those refined poses, they have a motion planner that generates a smooth, collision-free trajectory for the quadrotor to follow through those optimized waypoints. They are planning paths that respect the drone's physical limits and avoid crashing into obstacles.

Taro: So, they’re not just generating random points; they’re planning dynamic movement that actually works in three dimensions while following the visual plan. That sounds like a big piece of work for real-world execution.

Rosa: The validation shows they tested this in both simulations and on actual hardware, proving it can produce professional-quality footage across different indoor scenes without needing anyone who’s an expert in either robotics or cinematography to set up the shots.

Dev: And the results are pretty encouraging too; they report mean prompt-to-video alignment ratings around four point zero across three different scenes, and hardware tests showed the quadrotor could follow the simulation path with only centimeter level differences.

Conclusion: Rosa: So, to wrap up this discussion on Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories, it’s about showing how an AI can close the loop from an open-vocabulary dialogue directly into real-world autonomous aerial cinematography.

Dev: The paper essentially shows that by combining vision-language retrieval for initial selection, preference-based optimization for refinement, and a motion planner for safe trajectory generation, you can move away from manual waypoint setting entirely.

Taro: What this means practically is that non-experts in robotics or cinematography can produce high-quality drone videos just by talking to the drone using natural language prompts. It’s about making the complex choreography of a shot accessible through conversation.

Rosa: Exactly. The authors, including Yifan Lin, Sophie Ziyu Liu, Ran Qi, George Z. Xue, Xinping Song, Chao Qin and Hugh H.-T. Liu on page one of this work called Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories are demonstrating a system that bridges abstract human intent with executable UAV trajectories <ref:2509.16176#pg1,Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories>.

Dev: The real implication is that we’re moving toward embodied AI agents that can close the loop from open-vocabulary dialogue to real-world autonomous aerial cinematography. It’s about creating systems where the human director speaks, and the drone executes.

Division of Engineering Science, University of Toronto · Institute for Aerospace Studies, University of Toronto

cs.RO

Submitted: 2025-09-19

Updated: 2026-11-11

Project page: https://acdc-drone.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 94/100

The gist: We present Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories (ACDC), an autonomous drone cinematography system driven by natural language communication between human…

Key concepts

Vision–Language Retrieval Pipeline
This initial stage uses AI to search through exploratory video frames for visuals that match the user's spoken prompt. It connects the words in the prompt to specific images or scenes captured by the drone, creating a starting sequence of potential viewing points. This ensures the drone begins its tour in a direction relevant to what was asked.
Preference-based Bayesian Optimization (BO)
This mechanism fine-tunes each suggested viewpoint. It uses feedback from an AI model to judge which poses look 'better' or more aesthetic according to the original request. The system iteratively adjusts the drone's position and angle, optimizing for a visually pleasing result until it achieves a high-quality cinematic shot.
Time-Optimal Gate-Traversing (TOGT) Planner
This is the final step that creates the actual flight path. It takes all the refined waypoints and calculates a continuous, smooth route between them while respecting the physical limits of the drone. It ensures the path is collision-free and takes as little time as possible to complete, resulting in a dynamically feasible aerial tour.
Embodied AI Agents
This refers to AI systems that can act in a real-world environment, like a drone. ACDC demonstrates how these agents can close the loop: they take abstract human language (dialogue) and translate it directly into physical actions (flight trajectories) without needing manual control inputs from a human expert.

Terminology

Summary

We present Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories (ACDC), an autonomous drone cinematography system driven by natural language communication between human directors and drones The main limitation of previous drone cinematography workflows is that they require manual selection of waypoints and view angles based on predefined human intent, which is laborintensive and yields inconsistent performance In this paper, we propose employing large language models (LLMs) and vision foundation models (VFMs) to convert free-form natural language prompts directly into executable indoor UAV video tours Specifically, our method comprises a vision–language retrieval pipeline for initial waypoint selection, a preferencebased Bayesian optimization framework that refines poses using aesthetic feedback, and a motion planner that generates safe quadrotor trajectories We validate ACDC through both simulation and hardware-in-the-loop experiments, demonstrating that it robustly produces professional-quality footage across diverse indoor scenes without requiring expertise in robotics or cinematography These results highlight the potential of embodied AI agents to close the loop from open-vocabulary dialogue to real-world autonomous aerial cinematography. p. 0

Where the paper says it

Specifically, our method comprises a vision–language retrieval pipeline for initial waypoint selection, a preferencebased Bayesian optimization framework that refines poses using aesthetic feedback, and a motion planner that generates safe quadrotor trajectories. We validate ACDC through both simulation and hardware-in-the-loop experiments, demonstrating that it robustly produces professional-quality footage across diverse indoor scenes without requiring expertise in robotics or cinematography.

System Architecture Overview

The system overview of ACDC involves a sequence where a natural-language prompt leads to an exploratory video, a highfidelity 3D environment model, and the retrieval and refinement of poses for trajectory generation The overall process is illustrated by three main stages: Initial Waypoint Selection, Pose Refinement, and Trajectory Generation.

Initial Waypoint Selection Process

The first stage involves retrieving candidate frames from the exploratory scan that are semantically aligned with the prompt and ordering them into an initial waypoint sequence reflecting user intent This pipeline consists of several steps, including Visual-Textual Embedding, Similarity Matching, Intention-based Parsing, Frame Candidates per waypoint, and Description-based Selection. The process generates a set of ordered initial posed waypoints denoted as Tˇ = 0 to j with 0 < Tˇ ≤ Ck. p. 2

Where the paper says it

4) Select and Sort: A multi-modal VLM is queried with {Dk} n k=1 and frames in {Ck} n k=1 to choose the bestmatching frame for each waypoint and to order waypoints according to the user-intended flow. The result of this stage is an ordered set of initial posedwaypoints Tˇ = {T0, T1, .

Pose Refinement Mechanism

The second stage utilizes a preference-based Bayesian optimization (BO) framework to adjust each waypoint’s position and orientation to yield views that are more aesthetic and better aligned with the prompt This refinement pipeline supports full 6-DoF optimization, but in this aerial-filming setting, rotation is restricted to the one-parameter subgroup about the world z-axis, optimizing in a 4-DoF space [p, θ]. The refinement loop uses a Gaussian Process (GP) prior to model the unknown “aesthetic utility” f(T) of each pose and updates this posterior based on preference feedback from a vision–language model. p. 3

Where the paper says it

1) GP Prior: We assume each pose T has an unknown “aesthetic utility” f(T). To model f(·) smoothly, we place a Gaussian Process (GP) prior f ∼ GP(0, k) and use a covariance that factorizes over rotation and translation.

Trajectory Generation

The final stage synthesizes a smooth, collision-free, dynamically feasible flight path that passes through the refined waypoints Tref = 0 to n with n k=1 This generation involves three steps: Path Extraction, which connects waypoint positions using A⋆ on the free-voxel graph of M to yield a collision-free polyline P, Rectangular-Tunnel Corridor construction, which defines a collision-free corridor C, and Time-Optimal Trajectory Planning, which uses the Time-Optimal Gate-Traversing (TOGT) planner to compute a continuous, piecewisepolynomial position trajectory that respects quadrotor dynamic limits.

Experimental Validation

The experiments investigate language-to-shot quality, cross-scene generalization, and real-world executability Implementation details include using Gemini 2.5 Pro [37] as the vision–language model and AM-RADIO [38] with a SigLIP 2 adaptor [39] as the pretrained visual encoder. p. 5

Where the paper says it

III-B), we use Gemini 2. 5 Pro [37] as the vision–language model and AM-RADIO [38] with a SigLIP 2 adaptor [39] as the pretrained visual encoder.

Qualitative results show that ACDC produces professional-grade footage, with mean prompt–video alignment ratings of approximately 4. p. 6
Where the paper says it

Mean 1–5 Likert scores (higher is better) for prompt–video alignment across six outputs: Cottage (Left/Right), Restaurant (Left/Right), and Luxury (Left/Right), ordered as in Fig. Bars show independent participant ratings from 41 users (author of prompt self-ratings shown for reference), indicating medium-high prompt–video alignment across different scenes.

0 across three scenes. p. 4
Where the paper says it

" 1 2 3 1 2 3 1 2 3 1 2 3 1 1 2 2 3 3 3 3 2 1 2 1 1 2 3 1 2 3 Fig. Qualitative rollouts across three scenes.

Furthermore, hardware-in-the-loop tests confirm that the quadrotor platform is able to complete the flight within centimeter-level difference to simulation. p. 6
Where the paper says it

External pose estimates from a motion-capture system are fused with onboard IMU measurements to produce full 6- DoF odometry. The quadrotor platform is able to complete the flight within centimeter-level difference to simulation.

The user studies also indicate medium-high prompt–video alignment across different scenes. p. 6
Where the paper says it

Mean 1–5 Likert scores (higher is better) for prompt–video alignment across six outputs: Cottage (Left/Right), Restaurant (Left/Right), and Luxury (Left/Right), ordered as in Fig. Bars show independent participant ratings from 41 users (author of prompt self-ratings shown for reference), indicating medium-high prompt–video alignment across different scenes.

Conclusion and Future Work

ACDC is introduced as a language-conditioned autonomous cinematography system that bridges abstract human intent and executable UAV trajectories By combining semantic retrieval, preference-based optimization, and safe, dynamically feasible trajectory generation, ACDC enables non-experts to produce professional-grade indoor drone videos from natural-language prompts Future work aims to demonstrate an end-to-end, fully on-board system that performs 3D reconstruction from exploratory video, real-time localization and collision avoidance to capture cinematic video in the real world.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Agentic Aerial Cinematography (ACDC) can translate free-form natural language prompts directly into executable indoor UAV video tours by employing a vision–language retrieval pipeline for initial waypoint selection, a preferencebased Bayesian optimization framework that refines poses using aesthetic feedback, and a motion planner that generates safe quadrotor trajectories.

  2. The improved system can enable users with no robotics or cinematography expertise to obtain professional-grade indoor footage by closing the loop from open-vocabulary dialogue to real-world autonomous aerial cinematography.

  3. The system can perform language-conditioned Bayesian optimization which refines visibility oriented initial camera poses to meet aesthetic goals, as described in Section III.B of the paper.

  4. The refined pose optimization restricts rotation to a one parameter subgroup about the world z axis, optimizing in a 4-DoF space [p, theta] with theta in S1, thereby reducing the search space while preserving feasibility and composition objectives.

  5. The trajectory generation can synthesize a smooth, collision-free flight by connecting refined waypoints using A star on the free-voxel graph of the traversability map M and then using a Time Optimal Gate Traversing (TOGT) planner to compute a continuous, piecewisepolynomial position trajectory.

  6. The system can perform real-world executability validation through hardware-in-the-loop testing where an NVIDIA Jetson Orin NX running an autopilot stack executes the trajectories with low tracking error compared to simulation.

Abstract

Agents have demonstrated strong capabilities in digital environments, yet their reasoning abilities alone are insufficient to accomplish tasks in the physical world. We present Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories (ACDC), an agentic framework that bridges this gap for autonomous aerial cinematography. ACDC provides an execution harness that grounds an agent's interpretation of cinematic intent and visual judgment in scene geometry and flight constraints. Given a natural-language request and a 3D scene representation, the agent selects relevant viewpoints and guides their refinement through visual feedback, while optimization and motion planning translate these decisions into executable drone trajectories. Experiments across a variety of simulated indoor scenes and hardware-in-the-loop flight tests demonstrate how this framework connects open-ended user intent to prompt-aligned video tours and physical drone execution.

Sources

Related papers