CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents

arXiv:2607.26910 · cs.CV · Submitted 2026-07-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents".

Paper summary: Tom: So we were talking about the CinemaTraj paper, and it’s about automatically generating cinematic camera trajectories from natural language descriptions of three dee scenes <ref:2607.26910#pg1>. The core thesis is that you can solve this by treating trajectory planning as a language-grounded spatial reasoning problem >

Jane: Exactly. They claim that existing methods either lack true three dimensional spatial awareness because they rely on two dimensional image priors, or they treat trajectory generation purely as a geometric optimization problem, divorced from cinematographic semantics >

Lu: They argue that their framework uses a structured scene graph as a prior for the LLM agent so it gets accurate geometric and semantic knowledge about the room >

Meng: And then this scene graph feeds into a User Prompt Translator that breaks down the user prompt into atomic camera movements using tools like Anchor Selector A, Object-level trajectories To, and Transitional trajectories Tt >

Lalam: So the LLM agent produces a structured command sequence, like Anchor A1 with sofa, then Orbit To1, then Trans. Traj Tt1 to two > <ref:2607.26910#pg1>

Tom: It’s not just a path; it’s a plan that respects the scene structure and uses a defined set of cinematic moves >

Jane: And they claim this approach produces prompt-faithful trajectories with high cinematographic quality, outperforming previous methods on prompt alignment, trajectory quality, and safety metrics >

Lu: They focus heavily on how the method handles collision avoidance by using the Collision and Occlusion-Free Trajectory Optimizer to refine parameters via gradient descent on a signed distance field >

Meng: So they are saying that pure geometry isn't enough; you need to optimize for style too, which is what this framework does >

Lalam: Right. They minimize costs like Csdf for collision, Creg for regularization, Coccl for occlusion, and Cbnd for boundary cost >

Tom: That means they are using the SDF to provide geometrically faithful gradients to push the trajectory parameters toward a safer and more visually pleasing result >

Jane: And they also have this cross-room routing feature where if you move between rooms, it builds a room connectivity graph and routes transitional arcs through doorways >

Lu: When consecutive target objects are in different rooms, they route those transitional arcs through door or doorway waypoints using a breadth first search on the room graph to find the shortest path >

Meng: That adds complexity if you’re trying to deploy this for something that needs to be robust handling unexpected layouts. What happens when the layout doesn't match what you expected?

Tom: That’s where I get nervous. The paper says they are focused on one-shot trajectory generation in static three dee scenes right now > <ref:2607.26910#pg1>

Conclusion: Jane: So to wrap up, we’ve discussed the CinemaTraj paper. It focuses on how this framework uses an LLM and a structured scene graph to translate language into camera movement plans. The main takeaway is that it creates trajectories that are grounded in scene structure and follow defined cinematic rules >

Tom: Right. The authors are Qianru Li, Xuyang Chen, Erkin Türköz, Lu Liu, Xuqin Wang, Liqiu Meng >

Lu: It’s a way to formalizing the task of language-guided cinematic video generation from three dee scenes and introducing an end-to-end framework > <ref:2607.26910#pg1>

Meng: So for practical applications like real estate advertising or virtual tours that need high quality output where prompt alignment and collision avoidance are crucial >

Lalam: The paper shows it achieves the best performance across all metrics except collision rate, which ranks second to the w/o Anchor ablation variant >

Tom: So it’s a strong pipeline combining grounding, parametric trajectories for structure and SDF-based optimization for physical plausibility >

Jane: It seems like this is a solid way to get high quality results when you need that visual style, even with complex scene descriptions.

Technical University of Munich Research Center

cs.CV

Submitted: 2026-07-29

Updated: 2026-07-29

Comments: 17 pages, including 8-page main paper, references, and supplementary material; project page: https://cinematraj.github.io/

Journal ref: Proceedings of the 34th ACM International Conference on Multimedia (MM '26), Rio de Janeiro, Brazil, 2026

DOI: 10.1145/3767308.3835366

Project page: https://cinematraj.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value…

Key concepts

Scene Graph Extractor
This module builds a semantic map of the 3D scene by analyzing RGB-D frames. It identifies objects, their bounding boxes, and crucial spatial relationships like which objects are inside rooms or attached to structures. This structured knowledge acts as a guide for the LLM to understand the environment geometrically.
Cinematographic Toolset
This is a predefined set of atomic camera movements (like Anchor Selector A or Object-level To) that the LLM uses to decompose user prompts into actionable steps. It ensures that camera motions are inherently cinematic, allowing the system to generate high-quality, visually pleasing trajectories rather than just random paths.
SDF Optimization
This technique refines the planned camera paths by minimizing costs related to collisions and occlusions. It uses a Signed Distance Field (SDF) derived from an oriented bounding box mesh to create smooth gradients. This allows the system to adjust trajectory parameters precisely to ensure the camera never enters solid geometry.
User Prompt Translator
This component translates a user's vague request into a structured sequence of atomic camera movements. It uses the scene graph and cinematographic tools to determine which objects are relevant and how they should be viewed, producing a detailed command sequence for the trajectory builder.

Terminology

Summary

Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation The paper presents CinemaTraj, a framework that reframes camera trajectory planning as a language-grounded spatial reasoning problem It achieves this by equipping an LLM agent with a structured 3D scene graph and using a parametric trajectory representation grounded in cinematographic primitives CinemaTraj generates prompt-faithful, collision-free trajectories with high cinematographic quality, outperforming existing approaches on prompt alignment, trajectory quality, and safety metrics. p. 0

Where the paper says it

com Abstract Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation. Existing methods either lack true 3D spatial awareness by relying on 2D image priors, or treat trajectory generation as a geometric path planning problem divorced from cinematographic semantics.

System Architecture

The framework consists of four main modules: a Scene Graph Extractor that builds an instance-level semantic graph of the scene, a User Prompt Translator that decomposes the user’s intent into atomic camera movements via an LLM, a Parametric Atomic Trajectory Builder that instantiates these movements into a continuous trajectory, and a Collision-Free Trajectory Optimizer that refines trajectory parameters to avoid collisions and occlusions.

Language Grounding and Decomposition

The LLM agent is equipped with a structured 3D scene graph, which encodes object identities, bounding boxes, room containment, and spatial relations The User Prompt Translator decomposes a user prompt into a sequence of atomic cinematographic movements using a predefined cinematographic toolset. This toolset includes Anchor Selector A, Object-level trajectories To, and Transitional trajectories Tt. p. 2

Where the paper says it

The Scene Graph Extractor takes RGB-D images as input and builds a hierarchical 3D scene graph with oriented bounding boxes (OBBs). Given a user prompt and the extracted scene graph, the User Prompt Translator equips an LLM agent with a three-layer cinematographic toolset — anchor selector (A), object-level trajectories (To), and transitional trajectories (Tt) — to decompose the prompt into a structured trajectory plan.

The LLM produces a structured command sequence such as A1 To,1 → A2 → Tt,1→2 → To,2 → · · ·. p. 2
Where the paper says it

Given a user prompt and the extracted scene graph, the User Prompt Translator equips an LLM agent with a three-layer cinematographic toolset — anchor selector (A), object-level trajectories (To), and transitional trajectories (Tt) — to decompose the prompt into a structured trajectory plan. This plan is then executed by the Parametric Atomic Trajectory Builder, which selects optimal viewpoint anchors via scoring (𝑆 (pos)) and initializes each atomic command as a parametric trajectory 𝜏𝑗 (𝑡; 𝜽fix, 𝜽free).

Trajectory Generation and Optimization

The Parametric Atomic Trajectory Builder instantiates each atomic command into a continuous path using a parametric representation that preserves cinematographic motion profiles. This representation includes fixed parameters from anchors and free parameters optimized by the Collision and Occlusion-Free Trajectory Optimizer. p. 2

Where the paper says it

This plan is then executed by the Parametric Atomic Trajectory Builder, which selects optimal viewpoint anchors via scoring (𝑆 (pos)) and initializes each atomic command as a parametric trajectory 𝜏𝑗 (𝑡; 𝜽fix, 𝜽free). Finally, the Collision and Occlusion-Free Trajectory Optimizer refines 𝜽free via two-pass gradient descent on SDF-based collision (Csdf) and occlusion (Coccl) costs, producing optimized collision and occlusion-free trajectories.

The optimizer refines free parameters via two-pass gradient descent on a signed distance field (SDF) constructed from an Oriented Bounding Box mesh to minimize collision cost Csdf, regularization Creg, occlusion cost Coccl, and boundary cost Cbnd. p. 1
Where the paper says it

To address this, we introduce a differentiable collision and occlusion-free optimizer built on a signed distance field (SDF) constructed from the scene’s bounding-box mesh. The SDF provides smooth, geometry-faithful gradients that refine trajectory parameters while preserving shot style — a key advantage over prior methods [1, 5, 20] that rely on noisy 3DGS density fields for collision proxies.

Evaluation and Results

Experiments on ScanNet++ environments demonstrate that CinemaTraj produces trajectories that are collision-free, cinematographically grounded, and faithful to the user’s intent. Quantitative results show that CinemaTraj achieves the best performance across all metrics except collision rate, where it ranks second to the w/o Anchor ablation variant. p. 6

Where the paper says it

The full pipeline combines all three strengths. achieves the best performance across all metrics except collision rate, where we rank second to the w/o Anchor ablation variant (0.

A user study confirms that CinemaTraj is preferred across all three criteria—Prompt Alignment, Collision & Occlusion Avoidance, and Cinematographic Quality—by 40 participants. p. 7
Where the paper says it

Combined with a parametric trajectory representation grounded in professional camera idioms and an SDFbased differentiable optimizer for collision and occlusion avoidance, CinemaTraj produces prompt-faithful, collision-free trajectories with high cinematographic quality. Experiments on ScanNet++ demonstrate consistent improvements over existing approaches across motion fidelity, semantic alignment, collision avoidance, and object coverage, confirmed by a 40-participant user study in which our method is strongly preferred on all evaluation criteria.

Key Contributions and Limitations

The technical contributions include formalizing the task of language-guided cinematic video generation from 3D scenes and introducing an end-to-end framework. The method exploits 3D scene graphs as structured priors for spatial reasoning, providing the LLM agent with accurate geometric and semantic knowledge. p. 1

Where the paper says it

• We exploit 3D scene graphs as structured priors for spatial reasoning, providing the LLM agent with accurate geometric and semantic knowledge for trajectory planning, combined with a parametric trajectory representation grounded in cinematographic primitives, enabling precise prompt-aligned trajectory generation. • We propose an occlusion and collision-free shots optimization method specified for cinematographic shots creation.

A key limitation is that the framework currently focuses on one-shot trajectory generation in static 3D scenes. p. 7
Where the paper says it

However, limitations remain. Our framework currently focuses on one-shot trajectory generation in static 3D scenes.

Additionally, extending to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over time. p. 7
Where the paper says it

Our framework currently focuses on one-shot trajectory generation in static 3D scenes. Extending to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over time, maintaining proper framing as spatial configurations change.

The LLM dependence on a commercial LLM also introduces concerns regarding API costs and reproducibility.

Scene Graph Construction Details

The Scene Graph Extractor builds a hierarchical graph G = (N, E) from RGB-D frames, retaining only room–object containment and object–structure edges. Nodes N = NR∪NO ∪NS each carry a semantic label li and an oriented bounding box B i = (c i, R i, e i). p. 3

Where the paper says it

Following HOV-SG [33], we build a hierarchical graph G = (N, E) from RGB-D frames. Nodes N = N𝑅∪N𝑂 ∪N𝑆 (room, object, structure) each carry a semantic label ℓ𝑖 and an oriented bounding box (OBB) B𝑖 = (c𝑖 , R𝑖 , e𝑖).

The edge set E encodes two relation types: Room–object containment ERO and Object–structure relation EOS based on OBB proximity to wall surfaces or the ceiling plane. p. 3
Where the paper says it

The edge set E = E𝑅𝑂 ∪ E𝑂𝑆 encodes two relation types: • Room–object containment E𝑅𝑂 : each object is assigned to the room whose bounding box contains its center, establishing which spatial region the camera must navigate to reach the target. • Object–structure relation E𝑂𝑆 : each object is labeled 𝑛𝑆,𝑖 ∈ {wall-adjacent, ceiling-attached} based on its OBB proximity to wall surfaces or the ceiling plane.

Trajectory Parameterization

Each atomic trajectory type is defined by a small set of interpretable parameters, preserving cinematographic motion profiles while enabling gradient-based collision optimization. Object-level trajectories are parameterized by free parameters such as radius r, height h, and sweep angles for orbits. p. 3

Where the paper says it

The camera orientation is always R(𝑡) = look_at(p(𝑡), c𝑖), where c𝑖 is the target object center, with position parameterized per shot type: • Orbit: Circular arc at height ℎ, radius 𝑟, sweeping angle 𝜙𝑠 → 𝜙𝑒 . Free parameters: 𝜽free = (𝑟, ℎ, 𝜙𝑠 , 𝜙𝑒, 𝛿pitch) with 𝛿pitch as an optional pitch offset.

Transitional trajectories use a single free parameter—the arc angle α—controlling lateral displacement. p. 4
Where the paper says it

Transitional Trajectory. The Arc connects two anchors (p𝑠 , Q𝑠 ) and (p𝑒, Q𝑒 ) with a single free parameter—the arc angle 𝛼—controlling lateral displacement: p(𝑡) = (1−𝑡ˆ) p𝑠 + 𝑡ˆ p𝑒 + 𝐿 2 tan(𝛼) sin(𝜋 𝑡ˆ) nˆ, (4) where 𝐿 = ∥p𝑒 − p𝑠 ∥ and nˆ is perpendicular to the chord in the horizontal plane.

Collision Optimization Mechanism

The Collision and Occlusion-Free Trajectory Optimizer refines trajectory parameters via two-pass optimization on the SDF, which provides geometrically faithful gradients. The SDF is constructed from the OBB mesh through a four-stage pipeline involving adaptive voxelization, morphological closing, marching cubes surface extraction, and flood-fill signed distance computation. p. 1

Where the paper says it

However, parametric trajectories initialized from scene-graph anchors may still collide with cluttered geometry or lose sight of target objects. To address this, we introduce a differentiable collision and occlusion-free optimizer built on a signed distance field (SDF) constructed from the scene’s bounding-box mesh.

The cost formulation minimizes Csdf + Creg + Coccl + Cbnd to ensure safety and style preservation. p. 4
Where the paper says it

2 Cost Formulation. For each atomic trajectory 𝜏𝑘 , we uniformly sample positions and minimize: 𝜽 ∗ free,𝑘 = arg min 𝜽free,𝑘 Csdf + Creg + Coccl + Cbnd.

Cross-Room Routing

When consecutive target objects in a trajectory plan reside in different rooms, the system builds a room connectivity graph from the scene graph and routes transitional arcs through door or doorway waypoints. This process involves running BFS on the room graph to find the shortest room-hop path and then selecting the best passage based on openness preference. p. 3

Where the paper says it

After SDF-based collision filtering (§3. 4), the best anchor is selected by: 𝑆 (pos) =𝑤vis 𝑆vis + 𝑤dist 𝑆dist + 𝑤elev 𝑆elev + 𝑤open 𝑆open + 𝑤perp 𝑆perp, (2) where 𝑆vis measures bounding-box corner visibility via ray casting, 𝑆dist penalizes deviation from preferred distance, 𝑆elev scores elevation angle, and 𝑆open favors positions near the room center.

The original single transitional arc is replaced by a sequence of sub-arcs optimized over the arc angle parameter only, ensuring collision-free passage through each doorway. p. 4
Where the paper says it

Transitional Trajectory. The Arc connects two anchors (p𝑠 , Q𝑠 ) and (p𝑒, Q𝑒 ) with a single free parameter—the arc angle 𝛼—controlling lateral displacement: p(𝑡) = (1−𝑡ˆ) p𝑠 + 𝑡ˆ p𝑒 + 𝐿 2 tan(𝛼) sin(𝜋 𝑡ˆ) nˆ, (4) where 𝐿 = ∥p𝑒 − p𝑠 ∥ and nˆ is perpendicular to the chord in the horizontal plane.

Output Generation

The system optionally generates synchronized voiceover and subtitles aligned with camera motion via a Vision-Language Model (VLM). The rendered video is segmented, and keyframes are fed to a VLM to output subtitles and voiceover blocks that are temporally fitted to the audio track. p. 4

Where the paper says it

The rendered video is segmented into short intervals; keyframes are extracted and fed to a vision-language model, which outputs (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. Each voiceover block is synthesized via TTS, temporally fitted to its window, and concatenated into a synchronized audio track.

The final output includes narrated cinematic video outputs. p. 0
Where the paper says it

The scene graph acts as a structured spatial prior, grounding the agent’s reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates synchronized voiceover and subtitles aligned with camera motion, producing narrated cinematic video outputs.

p. 0
Where the paper says it

The scene graph acts as a structured spatial prior, grounding the agent’s reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates synchronized voiceover and subtitles aligned with camera motion, producing narrated cinematic video outputs.

User Prompt Interpretation Example

The User Prompt Translator demonstrates the agent's reasoning chain, where it identifies cinematographically relevant objects and respects placement constraints derived from the scene graph. For instance, wall-mounted objects are assigned pan movements rather than full orbits. p. 3

Where the paper says it

, wall-mounted or ceiling-hung). These relations directly govern movement feasibility — a wall-adjacent object forbids full orbits; a ceiling object forbids crane shots.

This structured scene graph and cinematographic toolset enable the LLM to produce coherent, constraint-aware plans even without explicit object or movement specifications from the user. p. 9
Where the paper says it

”), the User Prompt Translator (§3. 2) identifies cinematographically relevant objects from the scene graph, determines an appropriate visitation order, and selects per-object camera movements that match the requested mood — all without explicit object or movement specifications from the user.

Ground Truth Creation

Ground truth trajectories are created using a custom Blender addon that allows an operator to select target objects and camera movement types for each segment. These generated keyframed trajectories serve as references for the Motion MSE metric. p. 9

Where the paper says it

Two metric-level differences from the main-paper evaluation are worth noting. First, Motion MSE is not reported because partiallyspecified and open-ended prompts do not prescribe exact camera movements, and thus lack ground-truth trajectories for comparison.

This process is performed for each of the 50 ScanNet++ scenes, yielding two fully-specified prompts and corresponding ground-truth trajectories. p. 4
Where the paper says it

We evaluate on 50 scenes from ScanNet++ [37], a real-world indoor dataset with high-fidelity 3D reconstructions. For each scene we generate two prompts at each of three specificity levels, yielding six prompts per scene: • Fully-specified prompts designate both target objects and exact camera movements (e.

p. 4
Where the paper says it

We evaluate on 50 scenes from ScanNet++ [37], a real-world indoor dataset with high-fidelity 3D reconstructions. For each scene we generate two prompts at each of three specificity levels, yielding six prompts per scene: • Fully-specified prompts designate both target objects and exact camera movements (e.

Metric Definitions

The evaluation involves five complementary axes: Motion MSE, CLaTr Score, Collision Rate, Occlusion Rate, and Object Coverage. Motion MSE is the mean squared error between the generated trajectory and a hand-crafted ground-truth trajectory created in Blender. p. 5

Where the paper says it

Other recent methods [7, 8, 21] either do not operate in 3D scenes or lack publicly available code for fair reproduction. We evaluate along five complementary axes: • Motion MSE: For fully-specified prompts, proposed in [24], we compute the mean squared error between the generated trajectory and a hand-crafted ground-truth trajectory created in Blender.

CLaTr Score measures cosine similarity between the trajectory and text embeddings in a shared latent space. p. 5
Where the paper says it

Both trajectories are uniformly resampled by arc length to a fixed number of points. • CLaTr Score [7]: Cosine similarity between trajectory and text embeddings in the shared CLaTr latent space (↑).

Collision Rate is the fraction of trajectory samples where the SDF value falls below zero, indicating penetration into scene geometry. p. 5
Where the paper says it

• CLaTr Score [7]: Cosine similarity between trajectory and text embeddings in the shared CLaTr latent space (↑). • Collision Rate: Fraction of trajectory samples where the SDF value falls below zero (↓).

Occlusion Rate is the fraction of samples where the target object is blocked, and Object Coverage tracks successfully visited anchor objects. p. 5
Where the paper says it

• Occlusion Rate: Fraction of trajectory samples where the target object is blocked (↓). • Object Coverage: Fraction of planned anchor objects successfully visited (↑).

Ablation Study Insights

The ablation study confirms that removing the Anchor Selector degrades object coverage in all settings (0.882→0.857→0.840 for fully-specified, partiallyspecified, and open-ended, respectively). p. 9

Where the paper says it

(1) Removing the Anchor Selector degrades object coverage in all settings (0. 840 for fully-specified, partiallyspecified, and open-ended, respectively, vs.

Removing SDF-based optimization causes collision rates to spike dramatically (0. p. 9
Where the paper says it

000 for the full pipeline), confirming that CLIP-based anchor selection is unreliable regardless of how prompts are specified. (2) Removing SDF-based optimization causes collision rates to spike dramatically (0.

557, 0.512, and 0. p. 9
Where the paper says it

(2) Removing SDF-based optimization causes collision rates to spike dramatically (0. 512, and 0.

459), confirming that raw Gaussian densities are a poor proxy for solid geometry. p. 7
Where the paper says it

• SDF-based Optimization: Falling back to the 3DGS density field causes the collision rate to spike to 0. 557 — the worst across all methods — confirming that raw Gaussian densities are a poor proxy for solid geometry.

The full pipeline combines the strengths of the Anchor Selector for correct grounding, parametric trajectories for cinematographic structure, and SDF-based optimization for physical plausibility. p. 7
Where the paper says it

841), indicating that this module primarily governs spatial safety rather than trajectory shape. The full pipeline combines the strengths of all three components: the Anchor Selector for correct grounding, parametric trajectories for cinematographic structure, and SDF-based optimization for physical plausibility.

Runtime Efficiency

The SDF representation achieves an 45× speedup compared to using the raw 3DGS density field as the collision proxy. The full CinemaTraj pipeline completes in under 20 s per scene on a single NIVIDA RTX 5090 GPU. p. 9

Where the paper says it

185) since unconstrained 6-DoF diffusion poses can freely orient toward targets, at the expense of cinematographic structure. In all three settings, the full CinemaTraj pipeline achieves the best overall balance: perfect object coverage (1.

p. 9
Where the paper says it

185) since unconstrained 6-DoF diffusion poses can freely orient toward targets, at the expense of cinematographic structure. In all three settings, the full CinemaTraj pipeline achieves the best overall balance: perfect object coverage (1.

This efficiency is due to the SDF being precomputed once as a compact 2563 grid and queried via trilinear interpolation. p. 9
Where the paper says it

The SDF representation achieves an 45× speedup. This gap arises from two factors: (1) the SDF is precomputed once as a compact 2563 grid and queried via trilinear interpolation, whereas density-based optimization requires evaluating 3DGS splatting at every sample point during each iteration; and (2) the smooth SDF gradients lead to faster convergence (fewer 10

The system's runtime is also fast, completing in under 20 s per scene on a single NIVIDA RTX 5090 GPU.

User Study Findings

In a perceptual user study with 40 participants, CinemaTraj is preferred across all three criteria—Prompt Alignment, Collision & Occlusion Avoidance, and Cinematic Quality. Participants noted particularly strong gains in cinematographic quality reflecting the benefit of the parametric shot vocabulary. p. 7

Where the paper says it

Table 2 reports the results. Our method is preferred across all three criteria, with particularly strong gains in cinematographic quality—reflecting the benefit of the parametric shot vocabulary— and collision avoidance.

p. 7
Where the paper says it

Table 2 reports the results. Our method is preferred across all three criteria, with particularly strong gains in cinematographic quality—reflecting the benefit of the parametric shot vocabulary— and collision avoidance.

The study showed that participants rated each video on a 5-point Likert scale for these three criteria. p. 7
Where the paper says it

We conduct a human evaluation in which 40 participants view trajectory-rendered videos from all methods (including ablation variants) across 4 representative scenes. Each video is rated on a 5-point Likert scale along three criteria: Prompt Alignment, Collision & Occlusion Avoidance, and Cinematographic Quality.

p. 7
Where the paper says it

We conduct a human evaluation in which 40 participants view trajectory-rendered videos from all methods (including ablation variants) across 4 representative scenes. Each video is rated on a 5-point Likert scale along three criteria: Prompt Alignment, Collision & Occlusion Avoidance, and Cinematographic Quality.

Future Work Directions

Limitations include extending the method to dynamic scenes with moving actors, which would require reasoning about temporal occlusion and dynamic obstacle avoidance. The parametric trajectory library also covers only foundational movement types, leaving subject-relative shots like tracking or over-the-shoulder compositions to be addressed. p. 7

Where the paper says it

Extending to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over time, maintaining proper framing as spatial configurations change. Our parametric trajectory library also covers only foundational movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed.

p. 7
Where the paper says it

Extending to dynamic scenes with moving actors would require the planner to jointly reason about camera and character trajectories over time, maintaining proper framing as spatial configurations change. Our parametric trajectory library also covers only foundational movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed.

Furthermore, generating multi-shot sequences with cuts introduces the need for editing grammar to enforce cinematic rules. p. 7
Where the paper says it

Our parametric trajectory library also covers only foundational movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed. Finally, generating multi-shot sequences with cuts introduces the need for editing grammar—enforcing the 180-degree rule, shot– reverse-shot patterns, and eyeline continuity—bridging the gap from single-trajectory generation to full cinematic storytelling.

p. 7
Where the paper says it

Our parametric trajectory library also covers only foundational movement types; subject-relative shots such as tracking, over-theshoulder, and point-of-view compositions remain to be addressed. Finally, generating multi-shot sequences with cuts introduces the need for editing grammar—enforcing the 180-degree rule, shot– reverse-shot patterns, and eyeline continuity—bridging the gap from single-trajectory generation to full cinematic storytelling.

The LLM dependence on a commercial LLM also introduces concerns regarding API costs and reproducibility.

Detailed Implementation Parameters

The system prompt includes the full structure used by the User Prompt Translator, specifying a cinematographic toolset, viewing preference instructions, and trajectory pattern definition. The LLM receives a system prompt comprising four components: (1) the cinematographic toolset specification, (2) the viewing preference instructions, (3) the 1-3-1 trajectory pattern definition, and (4) eight in-context examples. p. 2

Where the paper says it

We build on HOV-SG but apply 3D scene graphs to a previously unexplored setting: grounded spatial context for LLM-based cinematographic trajectory planning. 3 Method Given scene images (RGB-D) and a natural language prompt describing desired camera behavior, our system generates a collision-free, cinematographically plausible camera trajectory.

p. 2
Where the paper says it

We build on HOV-SG but apply 3D scene graphs to a previously unexplored setting: grounded spatial context for LLM-based cinematographic trajectory planning. 3 Method Given scene images (RGB-D) and a natural language prompt describing desired camera behavior, our system generates a collision-free, cinematographically plausible camera trajectory.

The LLM response is a structured JSON containing the object sequence, per-object viewing preferences with placement labels, and numbered atomic trajectory steps. p. 3
Where the paper says it

The LLM also specifies viewing preferences per object: elevation 𝑒 ∈ {low, medium, high, overhead} and framing distance 𝑑 ∈ {close, medium, far}, which guide anchor placement. The object–structure relation further constrains trajectory selection (e.

p. 3
Where the paper says it

The LLM also specifies viewing preferences per object: elevation 𝑒 ∈ {low, medium, high, overhead} and framing distance 𝑑 ∈ {close, medium, far}, which guide anchor placement. The object–structure relation further constrains trajectory selection (e.

The system prompt encodes hard constraints derived from the scene graph’s against wall and attached to ceiling flags. p. 3
Where the paper says it

The edge set E = E𝑅𝑂 ∪ E𝑂𝑆 encodes two relation types: • Room–object containment E𝑅𝑂 : each object is assigned to the room whose bounding box contains its center, establishing which spatial region the camera must navigate to reach the target. • Object–structure relation E𝑂𝑆 : each object is labeled 𝑛𝑆,𝑖 ∈ {wall-adjacent, ceiling-attached} based on its OBB proximity to wall surfaces or the ceiling plane.

p. 3
Where the paper says it

The edge set E = E𝑅𝑂 ∪ E𝑂𝑆 encodes two relation types: • Room–object containment E𝑅𝑂 : each object is assigned to the room whose bounding box contains its center, establishing which spatial region the camera must navigate to reach the target. • Object–structure relation E𝑂𝑆 : each object is labeled 𝑛𝑆,𝑖 ∈ {wall-adjacent, ceiling-attached} based on its OBB proximity to wall surfaces or the ceiling plane.

SDF Construction Pipeline

The Watertight SDF Construction OBBs Mesh SDF pipeline involves adaptive voxelization, morphological closing, marching cubes surface extraction, and flood-fill signed distance computation. The resulting SDF provides smooth, differentiable collision gradients for trajectory optimization. p. 1

Where the paper says it

To address this, we introduce a differentiable collision and occlusion-free optimizer built on a signed distance field (SDF) constructed from the scene’s bounding-box mesh. The SDF provides smooth, geometry-faithful gradients that refine trajectory parameters while preserving shot style — a key advantage over prior methods [1, 5, 20] that rely on noisy 3DGS density fields for collision proxies.

This OBB-based construction is substantially faster than computing an SDF from the raw 3DGS point cloud. p. 9
Where the paper says it

The SDF representation achieves an 45× speedup. This gap arises from two factors: (1) the SDF is precomputed once as a compact 2563 grid and queried via trilinear interpolation, whereas density-based optimization requires evaluating 3DGS splatting at every sample point during each iteration; and (2) the smooth SDF gradients lead to faster convergence (fewer 10

Atomic Trajectory Types Visualization

Figure D.1 visualizes all 13 camera movement types in the cinematographic toolset, showing their characteristic trajectory and target object. p. 9

Where the paper says it

2), where both target objects and camera movements are explicitly given. 1 extends this analysis to the two less constrained prompt types — partially-specified (objects named, movements left to the system) and open-ended (abstract requests with no object or movement specification) — to verify that the contribution of each component is robust across varying levels of user specificity.

The parametric formulation preserves each shot’s characteristic motion profile even after the collision-free optimization adjusts trajectory parameters. p. 7
Where the paper says it

3 shows examples of several shot types supported by our parametric representation, including orbits, crane shots, dolly moves, and pans/tilts. The parametric formulation preserves each shot’s characteristic motion profile even after the collision-free optimization adjusts trajectory parameters.

Wall placement rules forbid orbit full/half, and ceiling placement forbids the crane movement. p. 3
Where the paper says it

, wall-mounted or ceiling-hung). These relations directly govern movement feasibility — a wall-adjacent object forbids full orbits; a ceiling object forbids crane shots.

The system enforces a 1-3-1 sequential pattern for atomic steps, ensuring positional continuity through construction. p. 4
Where the paper says it

3 Trajectory Combination. Atomic segments are concatenated sequentially; positional continuity is guaranteed by construction since each arc’s endpoints match the surrounding anchors.

The LLM also determines the visitation order, using a nearest-neighbor heuristic when no visitation order is specified. p. 4
Where the paper says it

Atomic segments are concatenated sequentially; positional continuity is guaranteed by construction since each arc’s endpoints match the surrounding anchors. When no visitation order is specified, a nearest-neighbor heuristic minimizes total camera travel.

Voiceover and Subtitle Generation

The system optionally generates subtitles and voiceover to narrate the camera walkthrough. The rendered video is segmented, and keyframes are extracted and fed to a vision-language model to output (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. p. 4

Where the paper says it

5 Subtitle and Voiceover Generator We optionally generate subtitles and voiceover to narrate the camera walkthrough. The rendered video is segmented into short intervals; keyframes are extracted and fed to a vision-language model, which outputs (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments.

Each voiceover block is synthesized via TTS, temporally fitted to its window, and concatenated into a synchronized audio track. p. 4
Where the paper says it

The rendered video is segmented into short intervals; keyframes are extracted and fed to a vision-language model, which outputs (1) subtitles—one caption per segment, and (2) voiceover blocks—longer narration spans covering 2–4 segments. Each voiceover block is synthesized via TTS, temporally fitted to its window, and concatenated into a synchronized audio track.

The final output includes narrated cinematic video outputs.

Anchor Determination Scoring

The Anchor Selector generates candidate viewpoints via face-normal-biased sampling on the six faces of the object’s oriented bounding box. Candidates are scored using Equation 2, which weights visibility, distance, elevation, openness, and perpendicularity scores. p. 3

Where the paper says it

After SDF-based collision filtering (§3. 4), the best anchor is selected by: 𝑆 (pos) =𝑤vis 𝑆vis + 𝑤dist 𝑆dist + 𝑤elev 𝑆elev + 𝑤open 𝑆open + 𝑤perp 𝑆perp, (2) where 𝑆vis measures bounding-box corner visibility via ray casting, 𝑆dist penalizes deviation from preferred distance, 𝑆elev scores elevation angle, and 𝑆open favors positions near the room center.

The scoring function is defined as S(pos) = wvis Svis + wdist Sdist + welev SElev + wopen Sopen + wperp Sperp. p. 2
Where the paper says it

Given a user prompt and the extracted scene graph, the User Prompt Translator equips an LLM agent with a three-layer cinematographic toolset — anchor selector (A), object-level trajectories (To), and transitional trajectories (Tt) — to decompose the prompt into a structured trajectory plan. This plan is then executed by the Parametric Atomic Trajectory Builder, which selects optimal viewpoint anchors via scoring (𝑆 (pos)) and initializes each atomic command as a parametric trajectory 𝜏𝑗 (𝑡; 𝜽fix, 𝜽free).

Trajectory Optimization Costs

The cost formulation for trajectory optimization minimizes Csdf = λsdf P ∑ t j ∈ P ρ (g j) + ρ (g j) 2, where g j = dsafe − Φ(p(t j)). The occlusion cost Coccl is accumulated via transmittance weighting using softplus with β = 20. p. 4

Where the paper says it

Samples within the target’s OBB are masked to avoid self-occlusion. Occlusion is accumulated via transmittance weighting: Coccl = ∑︁𝐾 𝑚=1 𝑇𝑚 · ℎ𝑚, 𝑇𝑚 = exp − 𝑚∑︁−1 𝑗=1 ℎ𝑗 , (7) where ℎ𝑚 = softplus𝛽 (𝛿occ − Φ(r𝑗 (𝑠𝑚))) and the transmittance 𝑇𝑚 ensures the first occluder dominates the penalty.

The two-pass optimizer optimizes object-level segments with all costs including occlusion in Pass 1, and transitional arcs are rebuilt from updated endpoints and optimized over the arc angle parameter only in Pass 2 without occlusion cost. p. 4
Where the paper says it

Pass 1 optimizes object-level segments with the full cost (including occlusion), operating on a pertype subset of 𝜽free to preserve shot style. Pass 2 rebuilds transitional arcs from updated endpoints and optimizes without occlusion cost; cross-room arcs are split into sub-arcs through door waypoints from the room connectivity graph.

Object-Centric Planning Capability

The User Prompt Translator demonstrates the agent's ability to select cinematographically relevant objects based on scene context, determine a visitation order, and select per-object camera movements that match the requested mood. The agent respects placement constraints throughout, such as wall-mounted objects being assigned pan movements instead of full orbits. p. 3

Where the paper says it

, wall-mounted or ceiling-hung). These relations directly govern movement feasibility — a wall-adjacent object forbids full orbits; a ceiling object forbids crane shots.

This capability shows that the structured scene graph and cinematographic toolset enable the LLM to produce coherent, constraint-aware plans even without explicit object or movement specifications from the user. p. 9
Where the paper says it

”), the User Prompt Translator (§3. 2) identifies cinematographically relevant objects from the scene graph, determines an appropriate visitation order, and selects per-object camera movements that match the requested mood — all without explicit object or movement specifications from the user.

Runtime Analysis Comparison

The SDF representation achieves an 45× speedup compared to using the raw 3DGS density field as the collision proxy. The full CinemaTraj pipeline completes in under 20 s per scene on a single NIVIDA RTX 5090 GPU. This speedup arises because the SDF is precomputed once as a compact 2563 grid and queried via trilinear interpolation. p. 9

Where the paper says it

The SDF representation achieves an 45× speedup. This gap arises from two factors: (1) the SDF is precomputed once as a compact 2563 grid and queried via trilinear interpolation, whereas density-based optimization requires evaluating 3DGS splatting at every sample point during each iteration; and (2) the smooth SDF gradients lead to faster convergence (fewer 10

The system's runtime is also fast, completing in under 20 s per scene on a single NIVIDA RTX 5090 GPU.

User Study Preference Summary

In the perceptual user study, CinemaTraj was preferred across all three criteria—Prompt Alignment, Collision & Occlusion Avoidance, and Cinematic Quality. Participants noted particularly strong gains in cinematographic quality reflecting the benefit of the parametric shot vocabulary. The study showed that participants rated each video on a 5-point Likert scale for these three criteria.

Limitations Summary

The identified limitations include extending the method to dynamic scenes with moving actors, which would require reasoning about temporal occlusion and dynamic obstacle avoidance. The parametric trajectory library also covers only foundational movement types, leaving subject-relative shots like tracking or over-the-shoulder compositions to be addressed. Furthermore, generating multi-shot sequences with cuts introduces the need for editing grammar to enforce cinematic rules. The LLM dependence on a commercial LLM also introduces concerns regarding API costs and reproducibility.

Ground Truth Creation Process

The operator uses a custom Blender addon to select target objects and camera movement types for each segment. The tool generates smooth, collision-free keyframed trajectories that serve as references for the Motion MSE metric. p. 9

Where the paper says it

Two metric-level differences from the main-paper evaluation are worth noting. First, Motion MSE is not reported because partiallyspecified and open-ended prompts do not prescribe exact camera movements, and thus lack ground-truth trajectories for comparison.

This process is performed for each of the 50 ScanNet++ scenes, yielding two fully-specified prompts and corresponding ground-truth trajectories.

Detailed Prompt Structure

The system prompt provides the full structure used by the User Prompt Translator, specifying a cinematographic toolset, viewing preference instructions, and trajectory pattern definition. The LLM receives a system prompt comprising four components: (1) the cinematographic toolset specification, (2) the viewing preference instructions, (3) the 1-3-1 trajectory pattern definition, and (4) eight in-context examples. The LLM response is a structured JSON containing the object sequence, per-object viewing preferences with placement labels, and numbered atomic trajectory steps. The system prompt encodes hard constraints derived from the scene graph’s against wall and attached to ceiling flags.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Grounding LLM reasoning through a structured 3D scene graph allows agents to reason jointly about scene semantics and cinematographic intent. This enables better planning by constraining movement selection based on object properties like wall adjacency or ceiling attachment, as demonstrated by the LLM respecting placement constraints throughout.

  2. Implementing a differentiable collision and occlusion free optimizer built on a signed distance field (SDF) constructed from an oriented bounding box mesh allows for trajectory refinement using smooth, geometry-faithful gradients instead of noisy 3DGS density fields. This results in trajectories that are collision-free, cinematographically grounded, and faithful to the user’s intent.

  3. The system can generate synchronized voiceover and subtitles aligned with camera motion by feeding keyframes to a vision-language model, enhancing the vividness and naturalness of the output. This produces a complete narrated cinematic video output from the initial 3D scene and prompt.

  4. The framework can handle complex, abstract requests without explicit object or movement specifications through its ability to identify cinematographically relevant objects from the scene graph and select appropriate movements, as shown in open-ended prompt results where the agent selects a logical sequence covering major areas.

  5. The system can route camera paths across different rooms by building a room connectivity graph and splitting transitional arcs into sub-arcs through door waypoints, ensuring collision-free passage through each doorway while allowing gentle curves where space permits.

Abstract

Automatically generating cinematically expressive camera trajectories through 3D scenes from natural language descriptions is a challenging task of high practical value, with applications ranging from real-estate advertising to virtual tour creation. Existing methods either lack true 3D spatial awareness by relying on 2D image priors, or treat trajectory generation as a geometric path planning problem divorced from cinematographic semantics. We present CinemaTraj, a framework that reframes camera trajectory planning as a language-grounded spatial reasoning problem. Given a set of RGB-D images and a user prompt, CinemaTraj equips an LLM agent with a structured 3D scene graph: the agent decomposes the prompt into a sequence of atomic cinematographic movements (dolly, orbit, crane, pan, tilt, zoom, arc). Each movement is instantiated via a novel parametric trajectory representation that is both cinematographically expressive and optimizable for collision avoidance. The scene graph acts as a structured spatial prior, grounding the agent's reasoning in accurate geometric and semantic knowledge of the environment. CinemaTraj further generates synchronized voiceover and subtitles aligned with camera motion, producing narrated cinematic video outputs. We evaluate CinemaTraj on real-world ScanNet++ environments, and show that it produces prompt-faithful, collision-free trajectories with high cinematographic quality, outperforming existing approaches on prompt alignment, trajectory quality, and safety metrics.

Sources

Related papers