CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents".
Paper summary: Tom: So we were talking about the CinemaTraj paper, and it’s about automatically generating cinematic camera trajectories from natural language descriptions of three dee scenes <ref:2607.26910#pg1>. The core thesis is that you can solve this by treating trajectory planning as a language-grounded spatial reasoning problem >
Jane: Exactly. They claim that existing methods either lack true three dimensional spatial awareness because they rely on two dimensional image priors, or they treat trajectory generation purely as a geometric optimization problem, divorced from cinematographic semantics >
Lu: They argue that their framework uses a structured scene graph as a prior for the LLM agent so it gets accurate geometric and semantic knowledge about the room >
Meng: And then this scene graph feeds into a User Prompt Translator that breaks down the user prompt into atomic camera movements using tools like Anchor Selector A, Object-level trajectories To, and Transitional trajectories Tt >
Lalam: So the LLM agent produces a structured command sequence, like Anchor A1 with sofa, then Orbit To1, then Trans. Traj Tt1 to two > <ref:2607.26910#pg1>
Tom: It’s not just a path; it’s a plan that respects the scene structure and uses a defined set of cinematic moves >
Jane: And they claim this approach produces prompt-faithful trajectories with high cinematographic quality, outperforming previous methods on prompt alignment, trajectory quality, and safety metrics >
Lu: They focus heavily on how the method handles collision avoidance by using the Collision and Occlusion-Free Trajectory Optimizer to refine parameters via gradient descent on a signed distance field >
Meng: So they are saying that pure geometry isn't enough; you need to optimize for style too, which is what this framework does >
Lalam: Right. They minimize costs like Csdf for collision, Creg for regularization, Coccl for occlusion, and Cbnd for boundary cost >
Tom: That means they are using the SDF to provide geometrically faithful gradients to push the trajectory parameters toward a safer and more visually pleasing result >
Jane: And they also have this cross-room routing feature where if you move between rooms, it builds a room connectivity graph and routes transitional arcs through doorways >
Lu: When consecutive target objects are in different rooms, they route those transitional arcs through door or doorway waypoints using a breadth first search on the room graph to find the shortest path >
Meng: That adds complexity if you’re trying to deploy this for something that needs to be robust handling unexpected layouts. What happens when the layout doesn't match what you expected?
Tom: That’s where I get nervous. The paper says they are focused on one-shot trajectory generation in static three dee scenes right now > <ref:2607.26910#pg1>
Conclusion: Jane: So to wrap up, we’ve discussed the CinemaTraj paper. It focuses on how this framework uses an LLM and a structured scene graph to translate language into camera movement plans. The main takeaway is that it creates trajectories that are grounded in scene structure and follow defined cinematic rules >
Tom: Right. The authors are Qianru Li, Xuyang Chen, Erkin Türköz, Lu Liu, Xuqin Wang, Liqiu Meng >
Lu: It’s a way to formalizing the task of language-guided cinematic video generation from three dee scenes and introducing an end-to-end framework > <ref:2607.26910#pg1>
Meng: So for practical applications like real estate advertising or virtual tours that need high quality output where prompt alignment and collision avoidance are crucial >
Lalam: The paper shows it achieves the best performance across all metrics except collision rate, which ranks second to the w/o Anchor ablation variant >
Tom: So it’s a strong pipeline combining grounding, parametric trajectories for structure and SDF-based optimization for physical plausibility >
Jane: It seems like this is a solid way to get high quality results when you need that visual style, even with complex scene descriptions.
Technical University of Munich Research Center
cs.CV
Submitted: 2026-07-29
Updated: 2026-07-29
Comments: 17 pages, including 8-page main paper, references, and supplementary material; project page: https://cinematraj.github.io/
Journal ref: Proceedings of the 34th ACM International Conference on Multimedia (MM '26), Rio de Janeiro, Brazil, 2026
Project page: https://cinematraj.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value…
Key concepts
- Scene Graph Extractor
- This module builds a semantic map of the 3D scene by analyzing RGB-D frames. It identifies objects, their bounding boxes, and crucial spatial relationships like which objects are inside rooms or attached to structures. This structured knowledge acts as a guide for the LLM to understand the environment geometrically.
- Cinematographic Toolset
- This is a predefined set of atomic camera movements (like Anchor Selector A or Object-level To) that the LLM uses to decompose user prompts into actionable steps. It ensures that camera motions are inherently cinematic, allowing the system to generate high-quality, visually pleasing trajectories rather than just random paths.
- SDF Optimization
- This technique refines the planned camera paths by minimizing costs related to collisions and occlusions. It uses a Signed Distance Field (SDF) derived from an oriented bounding box mesh to create smooth gradients. This allows the system to adjust trajectory parameters precisely to ensure the camera never enters solid geometry.
- User Prompt Translator
- This component translates a user's vague request into a structured sequence of atomic camera movements. It uses the scene graph and cinematographic tools to determine which objects are relevant and how they should be viewed, producing a detailed command sequence for the trajectory builder.
Terminology
Summary
Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation The paper presents CinemaTraj, a framework that reframes camera trajectory planning as a language-grounded spatial reasoning problem It achieves this by equipping an LLM agent with a structured 3D scene graph and using a parametric trajectory representation grounded in cinematographic primitives CinemaTraj generates prompt-faithful, collision-free trajectories with high cinematographic quality, outperforming existing approaches on prompt alignment, trajectory quality, and safety metrics. p. 0 com
Abstract
Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a
challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation. Existing
methods either lack true 3D spatial awareness by relying on 2D
image priors, or treat trajectory generation as a geometric path
planning problem divorced from cinematographic semantics.Where the paper says it
System Architecture
The framework consists of four main modules: a Scene Graph Extractor that builds an instance-level semantic graph of the scene, a User Prompt Translator that decomposes the user’s intent into atomic camera movements via an LLM, a Parametric Atomic Trajectory Builder that instantiates these movements into a continuous trajectory, and a Collision-Free Trajectory Optimizer that refines trajectory parameters to avoid collisions and occlusions.
Language Grounding and Decomposition
The LLM agent is equipped with a structured 3D scene graph, which encodes object identities, bounding boxes, room containment, and spatial relations The User Prompt Translator decomposes a user prompt into a sequence of atomic cinematographic movements using a predefined cinematographic toolset. This toolset includes Anchor Selector A, Object-level trajectories To, and Transitional trajectories Tt. p. 2 The Scene Graph Extractor takes RGB-D images as input and builds a hierarchical 3D scene
graph with oriented bounding boxes (OBBs). Given a user prompt and the extracted scene graph, the User Prompt Translator
equips an LLM agent with a three-layer cinematographic toolset — anchor selector (A), object-level trajectories (To), and
transitional trajectories (Tt) — to decompose the prompt into a structured trajectory plan. Given a user prompt and the extracted scene graph, the User Prompt Translator
equips an LLM agent with a three-layer cinematographic toolset — anchor selector (A), object-level trajectories (To), and
transitional trajectories (Tt) — to decompose the prompt into a structured trajectory plan. This plan is then executed by the
Parametric Atomic Trajectory Builder, which selects optimal viewpoint anchors via scoring (𝑆 (pos)) and initializes each atomic
command as a parametric trajectory 𝜏𝑗 (𝑡; 𝜽fix, 𝜽free).Where the paper says it
Where the paper says it
Trajectory Generation and Optimization
The Parametric Atomic Trajectory Builder instantiates each atomic command into a continuous path using a parametric representation that preserves cinematographic motion profiles. This representation includes fixed parameters from anchors and free parameters optimized by the Collision and Occlusion-Free Trajectory Optimizer. p. 2 This plan is then executed by the
Parametric Atomic Trajectory Builder, which selects optimal viewpoint anchors via scoring (𝑆 (pos)) and initializes each atomic
command as a parametric trajectory 𝜏𝑗 (𝑡; 𝜽fix, 𝜽free). Finally, the Collision and Occlusion-Free Trajectory Optimizer refines
𝜽free via two-pass gradient descent on SDF-based collision (Csdf) and occlusion (Coccl) costs, producing optimized collision and
occlusion-free trajectories. To address this, we introduce a differentiable collision and occlusion-free optimizer built on a signed distance field
(SDF) constructed from the scene’s bounding-box mesh. The SDF
provides smooth, geometry-faithful gradients that refine trajectory
parameters while preserving shot style — a key advantage over
prior methods [1, 5, 20] that rely on noisy 3DGS density fields for
collision proxies.Where the paper says it
Where the paper says it
Evaluation and Results
Experiments on ScanNet++ environments demonstrate that CinemaTraj produces trajectories that are collision-free, cinematographically grounded, and faithful to the user’s intent. Quantitative results show that CinemaTraj achieves the best performance across all metrics except collision rate, where it ranks second to the w/o Anchor ablation variant. p. 6 The full pipeline combines all
three strengths. achieves the best performance across all metrics except collision
rate, where we rank second to the w/o Anchor ablation variant
(0. Combined with a parametric trajectory
representation grounded in professional camera idioms and an SDFbased differentiable optimizer for collision and occlusion avoidance,
CinemaTraj produces prompt-faithful, collision-free trajectories
with high cinematographic quality. Experiments on ScanNet++
demonstrate consistent improvements over existing approaches
across motion fidelity, semantic alignment, collision avoidance, and
object coverage, confirmed by a 40-participant user study in which
our method is strongly preferred on all evaluation criteria.Where the paper says it
Where the paper says it
Key Contributions and Limitations
The technical contributions include formalizing the task of language-guided cinematic video generation from 3D scenes and introducing an end-to-end framework. The method exploits 3D scene graphs as structured priors for spatial reasoning, providing the LLM agent with accurate geometric and semantic knowledge. p. 1 • We exploit 3D scene graphs as structured priors for spatial reasoning, providing the LLM agent with accurate geometric and
semantic knowledge for trajectory planning, combined with
a parametric trajectory representation grounded in cinematographic primitives, enabling precise prompt-aligned trajectory
generation. • We propose an occlusion and collision-free shots optimization
method specified for cinematographic shots creation. However, limitations remain. Our framework currently focuses
on one-shot trajectory generation in static 3D scenes. Our framework currently focuses
on one-shot trajectory generation in static 3D scenes. Extending
to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over
time, maintaining proper framing as spatial configurations change.Where the paper says it
Where the paper says it
Where the paper says it
Scene Graph Construction Details
The Scene Graph Extractor builds a hierarchical graph G = (N, E) from RGB-D frames, retaining only room–object containment and object–structure edges. Nodes N = NR∪NO ∪NS each carry a semantic label li and an oriented bounding box B i = (c i, R i, e i). p. 3 Following HOV-SG [33], we build a
hierarchical graph G = (N, E) from RGB-D frames. Nodes N =
N𝑅∪N𝑂 ∪N𝑆 (room, object, structure) each carry a semantic label ℓ𝑖
and an oriented bounding box (OBB) B𝑖 = (c𝑖
, R𝑖
, e𝑖). The edge set E = E𝑅𝑂 ∪ E𝑂𝑆 encodes two
relation types:
• Room–object containment E𝑅𝑂 : each object is assigned to the
room whose bounding box contains its center, establishing which
spatial region the camera must navigate to reach the target. • Object–structure relation E𝑂𝑆 : each object is labeled 𝑛𝑆,𝑖 ∈
{wall-adjacent, ceiling-attached} based on its OBB proximity to wall surfaces or the ceiling plane.Where the paper says it
Where the paper says it
Trajectory Parameterization
Each atomic trajectory type is defined by a small set of interpretable parameters, preserving cinematographic motion profiles while enabling gradient-based collision optimization. Object-level trajectories are parameterized by free parameters such as radius r, height h, and sweep angles for orbits. p. 3 The camera orientation is always R(𝑡) =
look_at(p(𝑡), c𝑖), where c𝑖
is the target object center, with position
parameterized per shot type:
• Orbit: Circular arc at height ℎ, radius 𝑟, sweeping angle 𝜙𝑠 →
𝜙𝑒 . Free parameters: 𝜽free = (𝑟, ℎ, 𝜙𝑠
, 𝜙𝑒, 𝛿pitch) with 𝛿pitch as an
optional pitch offset. Transitional Trajectory. The Arc connects two anchors (p𝑠
, Q𝑠 )
and (p𝑒, Q𝑒 ) with a single free parameter—the arc angle 𝛼—controlling
lateral displacement:
p(𝑡) = (1−𝑡ˆ) p𝑠 + 𝑡ˆ p𝑒 +
𝐿
2
tan(𝛼) sin(𝜋 𝑡ˆ) nˆ, (4)
where 𝐿 = ∥p𝑒 − p𝑠 ∥ and nˆ is perpendicular to the chord in the
horizontal plane.Where the paper says it
Where the paper says it
Collision Optimization Mechanism
The Collision and Occlusion-Free Trajectory Optimizer refines trajectory parameters via two-pass optimization on the SDF, which provides geometrically faithful gradients. The SDF is constructed from the OBB mesh through a four-stage pipeline involving adaptive voxelization, morphological closing, marching cubes surface extraction, and flood-fill signed distance computation. p. 1 However, parametric trajectories initialized from scene-graph
anchors may still collide with cluttered geometry or lose sight of
target objects. To address this, we introduce a differentiable collision and occlusion-free optimizer built on a signed distance field
(SDF) constructed from the scene’s bounding-box mesh. 2 Cost Formulation. For each atomic trajectory 𝜏𝑘 , we uniformly sample positions and minimize:
𝜽
∗
free,𝑘 = arg min
𝜽free,𝑘
Csdf + Creg + Coccl + Cbnd.Where the paper says it
Where the paper says it
Cross-Room Routing
When consecutive target objects in a trajectory plan reside in different rooms, the system builds a room connectivity graph from the scene graph and routes transitional arcs through door or doorway waypoints. This process involves running BFS on the room graph to find the shortest room-hop path and then selecting the best passage based on openness preference. p. 3 After SDF-based collision filtering (§3. 4), the best anchor is selected
by:
𝑆 (pos) =𝑤vis 𝑆vis + 𝑤dist 𝑆dist + 𝑤elev 𝑆elev
+ 𝑤open 𝑆open + 𝑤perp 𝑆perp,
(2)
where 𝑆vis measures bounding-box corner visibility via ray casting,
𝑆dist penalizes deviation from preferred distance, 𝑆elev scores elevation angle, and 𝑆open favors positions near the room center. Transitional Trajectory. The Arc connects two anchors (p𝑠
, Q𝑠 )
and (p𝑒, Q𝑒 ) with a single free parameter—the arc angle 𝛼—controlling
lateral displacement:
p(𝑡) = (1−𝑡ˆ) p𝑠 + 𝑡ˆ p𝑒 +
𝐿
2
tan(𝛼) sin(𝜋 𝑡ˆ) nˆ, (4)
where 𝐿 = ∥p𝑒 − p𝑠 ∥ and nˆ is perpendicular to the chord in the
horizontal plane.Where the paper says it
Where the paper says it
Output Generation
The system optionally generates synchronized voiceover and subtitles aligned with camera motion via a Vision-Language Model (VLM). The rendered video is segmented, and keyframes are fed to a VLM to output subtitles and voiceover blocks that are temporally fitted to the audio track. p. 4 The rendered video is segmented into short intervals; keyframes are extracted and fed to a vision-language model,
which outputs (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. Each
voiceover block is synthesized via TTS, temporally fitted to its
window, and concatenated into a synchronized audio track. The scene graph acts as a structured spatial prior,
grounding the agent’s reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates
synchronized voiceover and subtitles aligned with camera motion,
producing narrated cinematic video outputs. The scene graph acts as a structured spatial prior,
grounding the agent’s reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates
synchronized voiceover and subtitles aligned with camera motion,
producing narrated cinematic video outputs.Where the paper says it
Where the paper says it
Where the paper says it
User Prompt Interpretation Example
The User Prompt Translator demonstrates the agent's reasoning chain, where it identifies cinematographically relevant objects and respects placement constraints derived from the scene graph. For instance, wall-mounted objects are assigned pan movements rather than full orbits. p. 3 , wall-mounted or ceiling-hung). These relations directly
govern movement feasibility — a wall-adjacent object forbids full
orbits; a ceiling object forbids crane shots. ”), the User Prompt
Translator (§3. 2) identifies cinematographically relevant objects
from the scene graph, determines an appropriate visitation order,
and selects per-object camera movements that match the requested
mood — all without explicit object or movement specifications from
the user.Where the paper says it
Where the paper says it
Ground Truth Creation
Ground truth trajectories are created using a custom Blender addon that allows an operator to select target objects and camera movement types for each segment. These generated keyframed trajectories serve as references for the Motion MSE metric. p. 9 Two metric-level differences from the main-paper evaluation are
worth noting. First, Motion MSE is not reported because partiallyspecified and open-ended prompts do not prescribe exact camera
movements, and thus lack ground-truth trajectories for comparison. We evaluate on 50 scenes
from ScanNet++ [37], a real-world indoor dataset with high-fidelity
3D reconstructions. For each scene we generate two prompts at
each of three specificity levels, yielding six prompts per scene:
• Fully-specified prompts designate both target objects and exact camera movements (e. We evaluate on 50 scenes
from ScanNet++ [37], a real-world indoor dataset with high-fidelity
3D reconstructions. For each scene we generate two prompts at
each of three specificity levels, yielding six prompts per scene:
• Fully-specified prompts designate both target objects and exact camera movements (e.Where the paper says it
Where the paper says it
Where the paper says it
Metric Definitions
The evaluation involves five complementary axes: Motion MSE, CLaTr Score, Collision Rate, Occlusion Rate, and Object Coverage. Motion MSE is the mean squared error between the generated trajectory and a hand-crafted ground-truth trajectory created in Blender. p. 5 Other recent methods [7, 8, 21] either do
not operate in 3D scenes or lack publicly available code for fair
reproduction. We evaluate along five complementary axes:
• Motion MSE: For fully-specified prompts, proposed in [24], we
compute the mean squared error between the generated trajectory and a hand-crafted ground-truth trajectory created in
Blender. Both trajectories are uniformly resampled by arc length
to a fixed number of points. • CLaTr Score [7]: Cosine similarity between trajectory and text
embeddings in the shared CLaTr latent space (↑). • CLaTr Score [7]: Cosine similarity between trajectory and text
embeddings in the shared CLaTr latent space (↑). • Collision Rate: Fraction of trajectory samples where the SDF
value falls below zero (↓). • Occlusion Rate: Fraction of trajectory samples where the target
object is blocked (↓). • Object Coverage: Fraction of planned anchor objects successfully visited (↑).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ablation Study Insights
The ablation study confirms that removing the Anchor Selector degrades object coverage in all settings (0.882→0.857→0.840 for fully-specified, partiallyspecified, and open-ended, respectively). p. 9 (1) Removing the Anchor Selector degrades object coverage
in all settings (0. 840 for fully-specified, partiallyspecified, and open-ended, respectively, vs. 000 for the full pipeline),
confirming that CLIP-based anchor selection is unreliable regardless of how prompts are specified. (2) Removing SDF-based optimization causes collision rates to spike dramatically (0. (2) Removing SDF-based optimization causes collision rates to spike dramatically (0. 512,
and 0. • SDF-based Optimization: Falling back to the 3DGS density
field causes the collision rate to spike to 0. 557 — the worst across
all methods — confirming that raw Gaussian densities are a poor
proxy for solid geometry. 841), indicating that this module primarily governs spatial
safety rather than trajectory shape. The full pipeline combines the strengths of all three components:
the Anchor Selector for correct grounding, parametric trajectories
for cinematographic structure, and SDF-based optimization for
physical plausibility.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Runtime Efficiency
The SDF representation achieves an 45× speedup compared to using the raw 3DGS density field as the collision proxy. The full CinemaTraj pipeline completes in under 20 s per scene on a single NIVIDA RTX 5090 GPU. p. 9 185) since unconstrained 6-DoF diffusion poses
can freely orient toward targets, at the expense of cinematographic
structure. In all three settings, the full CinemaTraj pipeline achieves
the best overall balance: perfect object coverage (1. 185) since unconstrained 6-DoF diffusion poses
can freely orient toward targets, at the expense of cinematographic
structure. In all three settings, the full CinemaTraj pipeline achieves
the best overall balance: perfect object coverage (1. The SDF representation achieves
an 45× speedup. This gap arises from two factors: (1) the SDF is
precomputed once as a compact 2563 grid and queried via trilinear
interpolation, whereas density-based optimization requires evaluating 3DGS splatting at every sample point during each iteration;
and (2) the smooth SDF gradients lead to faster convergence (fewer
10Where the paper says it
Where the paper says it
Where the paper says it
User Study Findings
In a perceptual user study with 40 participants, CinemaTraj is preferred across all three criteria—Prompt Alignment, Collision & Occlusion Avoidance, and Cinematic Quality. Participants noted particularly strong gains in cinematographic quality reflecting the benefit of the parametric shot vocabulary. p. 7 Table 2 reports the results. Our method is preferred across all
three criteria, with particularly strong gains in cinematographic
quality—reflecting the benefit of the parametric shot vocabulary—
and collision avoidance. Table 2 reports the results. Our method is preferred across all
three criteria, with particularly strong gains in cinematographic
quality—reflecting the benefit of the parametric shot vocabulary—
and collision avoidance. We conduct a human evaluation in which 40
participants view trajectory-rendered videos from all methods (including ablation variants) across 4 representative scenes. Each video
is rated on a 5-point Likert scale along three criteria: Prompt Alignment, Collision & Occlusion Avoidance, and Cinematographic Quality. We conduct a human evaluation in which 40
participants view trajectory-rendered videos from all methods (including ablation variants) across 4 representative scenes. Each video
is rated on a 5-point Likert scale along three criteria: Prompt Alignment, Collision & Occlusion Avoidance, and Cinematographic Quality.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Future Work Directions
Limitations include extending the method to dynamic scenes with moving actors, which would require reasoning about temporal occlusion and dynamic obstacle avoidance. The parametric trajectory library also covers only foundational movement types, leaving subject-relative shots like tracking or over-the-shoulder compositions to be addressed. p. 7 Extending
to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over
time, maintaining proper framing as spatial configurations change. Our parametric trajectory library also covers only foundational
movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed. Extending
to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over
time, maintaining proper framing as spatial configurations change. Our parametric trajectory library also covers only foundational
movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed. Our parametric trajectory library also covers only foundational
movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed. Finally, generating multi-shot sequences with cuts introduces the
need for editing grammar—enforcing the 180-degree rule, shot–
reverse-shot patterns, and eyeline continuity—bridging the gap
from single-trajectory generation to full cinematic storytelling. Our parametric trajectory library also covers only foundational
movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed. Finally, generating multi-shot sequences with cuts introduces the
need for editing grammar—enforcing the 180-degree rule, shot–
reverse-shot patterns, and eyeline continuity—bridging the gap
from single-trajectory generation to full cinematic storytelling.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Detailed Implementation Parameters
The system prompt includes the full structure used by the User Prompt Translator, specifying a cinematographic toolset, viewing preference instructions, and trajectory pattern definition. The LLM receives a system prompt comprising four components: (1) the cinematographic toolset specification, (2) the viewing preference instructions, (3) the 1-3-1 trajectory pattern definition, and (4) eight in-context examples. p. 2 We build on
HOV-SG but apply 3D scene graphs to a previously unexplored
setting: grounded spatial context for LLM-based cinematographic
trajectory planning. 3 Method
Given scene images (RGB-D) and a natural language prompt describing desired camera behavior, our system generates a collision-free,
cinematographically plausible camera trajectory. We build on
HOV-SG but apply 3D scene graphs to a previously unexplored
setting: grounded spatial context for LLM-based cinematographic
trajectory planning. 3 Method
Given scene images (RGB-D) and a natural language prompt describing desired camera behavior, our system generates a collision-free,
cinematographically plausible camera trajectory. The LLM also specifies viewing preferences per object: elevation 𝑒 ∈ {low, medium, high, overhead} and framing distance
𝑑 ∈ {close, medium, far}, which guide anchor placement. The
object–structure relation further constrains trajectory selection
(e. The LLM also specifies viewing preferences per object: elevation 𝑒 ∈ {low, medium, high, overhead} and framing distance
𝑑 ∈ {close, medium, far}, which guide anchor placement. The
object–structure relation further constrains trajectory selection
(e. The edge set E = E𝑅𝑂 ∪ E𝑂𝑆 encodes two
relation types:
• Room–object containment E𝑅𝑂 : each object is assigned to the
room whose bounding box contains its center, establishing which
spatial region the camera must navigate to reach the target. • Object–structure relation E𝑂𝑆 : each object is labeled 𝑛𝑆,𝑖 ∈
{wall-adjacent, ceiling-attached} based on its OBB proximity to wall surfaces or the ceiling plane. The edge set E = E𝑅𝑂 ∪ E𝑂𝑆 encodes two
relation types:
• Room–object containment E𝑅𝑂 : each object is assigned to the
room whose bounding box contains its center, establishing which
spatial region the camera must navigate to reach the target. • Object–structure relation E𝑂𝑆 : each object is labeled 𝑛𝑆,𝑖 ∈
{wall-adjacent, ceiling-attached} based on its OBB proximity to wall surfaces or the ceiling plane.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
SDF Construction Pipeline
The Watertight SDF Construction OBBs Mesh SDF pipeline involves adaptive voxelization, morphological closing, marching cubes surface extraction, and flood-fill signed distance computation. The resulting SDF provides smooth, differentiable collision gradients for trajectory optimization. p. 1 To address this, we introduce a differentiable collision and occlusion-free optimizer built on a signed distance field
(SDF) constructed from the scene’s bounding-box mesh. The SDF
provides smooth, geometry-faithful gradients that refine trajectory
parameters while preserving shot style — a key advantage over
prior methods [1, 5, 20] that rely on noisy 3DGS density fields for
collision proxies. The SDF representation achieves
an 45× speedup. This gap arises from two factors: (1) the SDF is
precomputed once as a compact 2563 grid and queried via trilinear
interpolation, whereas density-based optimization requires evaluating 3DGS splatting at every sample point during each iteration;
and (2) the smooth SDF gradients lead to faster convergence (fewer
10Where the paper says it
Where the paper says it
Atomic Trajectory Types Visualization
Figure D.1 visualizes all 13 camera movement types in the cinematographic toolset, showing their characteristic trajectory and target object. p. 9 2), where both target objects and camera movements are explicitly given. 1 extends this analysis to the two less constrained
prompt types — partially-specified (objects named, movements left
to the system) and open-ended (abstract requests with no object or
movement specification) — to verify that the contribution of each
component is robust across varying levels of user specificity. 3 shows examples of several shot types supported by our parametric representation, including orbits, crane shots, dolly moves, and pans/tilts. The parametric formulation preserves each shot’s characteristic
motion profile even after the collision-free optimization adjusts
trajectory parameters. , wall-mounted or ceiling-hung). These relations directly
govern movement feasibility — a wall-adjacent object forbids full
orbits; a ceiling object forbids crane shots. 3 Trajectory Combination. Atomic segments are concatenated
sequentially; positional continuity is guaranteed by construction
since each arc’s endpoints match the surrounding anchors. Atomic segments are concatenated
sequentially; positional continuity is guaranteed by construction
since each arc’s endpoints match the surrounding anchors. When no
visitation order is specified, a nearest-neighbor heuristic minimizes
total camera travel.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Voiceover and Subtitle Generation
The system optionally generates subtitles and voiceover to narrate the camera walkthrough. The rendered video is segmented, and keyframes are extracted and fed to a vision-language model to output (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. p. 4 5 Subtitle and Voiceover Generator
We optionally generate subtitles and voiceover to narrate the camera walkthrough. The rendered video is segmented into short intervals; keyframes are extracted and fed to a vision-language model,
which outputs (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. The rendered video is segmented into short intervals; keyframes are extracted and fed to a vision-language model,
which outputs (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. Each
voiceover block is synthesized via TTS, temporally fitted to its
window, and concatenated into a synchronized audio track.Where the paper says it
Where the paper says it
Anchor Determination Scoring
The Anchor Selector generates candidate viewpoints via face-normal-biased sampling on the six faces of the object’s oriented bounding box. Candidates are scored using Equation 2, which weights visibility, distance, elevation, openness, and perpendicularity scores. p. 3 After SDF-based collision filtering (§3. 4), the best anchor is selected
by:
𝑆 (pos) =𝑤vis 𝑆vis + 𝑤dist 𝑆dist + 𝑤elev 𝑆elev
+ 𝑤open 𝑆open + 𝑤perp 𝑆perp,
(2)
where 𝑆vis measures bounding-box corner visibility via ray casting,
𝑆dist penalizes deviation from preferred distance, 𝑆elev scores elevation angle, and 𝑆open favors positions near the room center. Given a user prompt and the extracted scene graph, the User Prompt Translator
equips an LLM agent with a three-layer cinematographic toolset — anchor selector (A), object-level trajectories (To), and
transitional trajectories (Tt) — to decompose the prompt into a structured trajectory plan. This plan is then executed by the
Parametric Atomic Trajectory Builder, which selects optimal viewpoint anchors via scoring (𝑆 (pos)) and initializes each atomic
command as a parametric trajectory 𝜏𝑗 (𝑡; 𝜽fix, 𝜽free).Where the paper says it
Where the paper says it
Trajectory Optimization Costs
The cost formulation for trajectory optimization minimizes Csdf = λsdf P ∑ t j ∈ P ρ (g j) + ρ (g j) 2, where g j = dsafe − Φ(p(t j)). The occlusion cost Coccl is accumulated via transmittance weighting using softplus with β = 20. p. 4 Samples within the target’s OBB are masked to avoid self-occlusion. Occlusion is accumulated via transmittance weighting:
Coccl =
∑︁𝐾
𝑚=1
𝑇𝑚 · ℎ𝑚, 𝑇𝑚 = exp
−
𝑚∑︁−1
𝑗=1
ℎ𝑗
, (7)
where ℎ𝑚 = softplus𝛽
(𝛿occ − Φ(r𝑗 (𝑠𝑚))) and the transmittance 𝑇𝑚
ensures the first occluder dominates the penalty. Pass 1 optimizes object-level segments with the full cost (including occlusion), operating on a pertype subset of 𝜽free to preserve shot style. Pass 2 rebuilds transitional arcs from updated endpoints and optimizes without occlusion
cost; cross-room arcs are split into sub-arcs through door waypoints
from the room connectivity graph.Where the paper says it
Where the paper says it
Object-Centric Planning Capability
The User Prompt Translator demonstrates the agent's ability to select cinematographically relevant objects based on scene context, determine a visitation order, and select per-object camera movements that match the requested mood. The agent respects placement constraints throughout, such as wall-mounted objects being assigned pan movements instead of full orbits. p. 3 , wall-mounted or ceiling-hung). These relations directly
govern movement feasibility — a wall-adjacent object forbids full
orbits; a ceiling object forbids crane shots. ”), the User Prompt
Translator (§3. 2) identifies cinematographically relevant objects
from the scene graph, determines an appropriate visitation order,
and selects per-object camera movements that match the requested
mood — all without explicit object or movement specifications from
the user.Where the paper says it
Where the paper says it
Runtime Analysis Comparison
The SDF representation achieves an 45× speedup compared to using the raw 3DGS density field as the collision proxy. The full CinemaTraj pipeline completes in under 20 s per scene on a single NIVIDA RTX 5090 GPU. This speedup arises because the SDF is precomputed once as a compact 2563 grid and queried via trilinear interpolation. p. 9 The SDF representation achieves
an 45× speedup. This gap arises from two factors: (1) the SDF is
precomputed once as a compact 2563 grid and queried via trilinear
interpolation, whereas density-based optimization requires evaluating 3DGS splatting at every sample point during each iteration;
and (2) the smooth SDF gradients lead to faster convergence (fewer
10Where the paper says it
User Study Preference Summary
In the perceptual user study, CinemaTraj was preferred across all three criteria—Prompt Alignment, Collision & Occlusion Avoidance, and Cinematic Quality. Participants noted particularly strong gains in cinematographic quality reflecting the benefit of the parametric shot vocabulary. The study showed that participants rated each video on a 5-point Likert scale for these three criteria.
Limitations Summary
The identified limitations include extending the method to dynamic scenes with moving actors, which would require reasoning about temporal occlusion and dynamic obstacle avoidance. The parametric trajectory library also covers only foundational movement types, leaving subject-relative shots like tracking or over-the-shoulder compositions to be addressed. Furthermore, generating multi-shot sequences with cuts introduces the need for editing grammar to enforce cinematic rules. The LLM dependence on a commercial LLM also introduces concerns regarding API costs and reproducibility.
Ground Truth Creation Process
The operator uses a custom Blender addon to select target objects and camera movement types for each segment. The tool generates smooth, collision-free keyframed trajectories that serve as references for the Motion MSE metric. p. 9 Two metric-level differences from the main-paper evaluation are
worth noting. First, Motion MSE is not reported because partiallyspecified and open-ended prompts do not prescribe exact camera
movements, and thus lack ground-truth trajectories for comparison.Where the paper says it
Detailed Prompt Structure
The system prompt provides the full structure used by the User Prompt Translator, specifying a cinematographic toolset, viewing preference instructions, and trajectory pattern definition. The LLM receives a system prompt comprising four components: (1) the cinematographic toolset specification, (2) the viewing preference instructions, (3) the 1-3-1 trajectory pattern definition, and (4) eight in-context examples. The LLM response is a structured JSON containing the object sequence, per-object viewing preferences with placement labels, and numbered atomic trajectory steps. The system prompt encodes hard constraints derived from the scene graph’s against wall and attached to ceiling flags.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Grounding LLM reasoning through a structured 3D scene graph allows agents to
reason jointly about scene semantics and cinematographic intent.
This enables better planning by constraining movement selection based on object properties like wall adjacency or ceiling attachment, as demonstrated by the LLM respectingplacement constraints throughout.
-
Implementing a differentiable collision and occlusion free optimizer built on a signed distance field (SDF) constructed from an oriented bounding box mesh allows for trajectory refinement using
smooth, geometry-faithful gradients
instead of noisy 3DGS density fields. This results in trajectories that arecollision-free, cinematographically grounded, and faithful to the user’s intent.
-
The system can generate synchronized voiceover and subtitles aligned with camera motion by feeding keyframes to a vision-language model, enhancing the
vividness and naturalness of the output.
This produces a complete narrated cinematic video output from the initial 3D scene and prompt. -
The framework can handle complex, abstract requests without explicit object or movement specifications through its ability to
identify cinematographically relevant objects from the scene graph
and select appropriate movements, as shown in open-ended prompt results where the agent selects alogical sequence covering major areas.
-
The system can route camera paths across different rooms by building a room connectivity graph and splitting transitional arcs into sub-arcs through door waypoints, ensuring
collision-free passage through each doorway while allowing gentle curves where space permits.
Abstract
Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation. Existing methods either lack true 3D spatial awareness by relying on 2D image priors, or treat trajectory generation as a geometric path planning problem divorced from cinematographic semantics. We present CinemaTraj, a framework that reframes camera trajectory planning as a language-grounded spatial reasoning problem. Given a set of RGB-D images and a user prompt, CinemaTraj equips an LLM agent with a structured 3D scene graph: the agent decomposes the prompt into a sequence of atomic cinematographic movements (dolly, orbit, crane, pan, tilt, zoom, arc). Each movement is instantiated via a novel parametric trajectory representation that is both cinematographically expressive and optimizable for collision avoidance. The scene graph acts as a structured spatial prior, grounding the agent's reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates synchronized voiceover and subtitles aligned with camera motion, producing narrated cinematic video outputs. We evaluate CinemaTraj on real-world ScanNet++ environments, and show that it produces prompt-faithful, collision-free trajectories with high cinematographic quality, outperforming existing approaches on prompt alignment, trajectory quality, and safety metrics.
Sources
- FOCI: Trajectory Optimization on Gaussian Splats
- Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps
- Pulp Motion: Framing-aware multimodal camera and human motion generation
- Camera Trajectory Generation: A Comprehensive Survey of Methods, Metrics, and Future Directions
- LAMP: Language-Assisted Motion Planning for Controllable Video Generation
- Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories
- WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents
- SplaTraj: Camera Trajectory Generation with Semantic Gaussian Splatting
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs