Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion

The gist The method Ctrl-V leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation by controlling object motion using bounding boxes, which is crucial for developing highly accurate predictions for object motions in domains like autonomous driving<ref:2406.05630#pg2>.

How it works

The method Ctrl-V consists of two sequential steps: first, generating bounding box frames using the BBox Generator to create a skeleton video of moving or stationary bounding boxes, and second, generating a video using the Box2Video network where these bounding box frames act as the control signal<ref:2406.05630#pg6>. Both models utilize an SVD backbone adapted to their respective generation tasks<ref:2406.05630#pg8>.

The BBox Generator is a diffusion-based model that predicts object bounding boxes across all video frames using an SVD backbone. Its four inputs are ˆbt, b(0), b(N-1), z(0), where ˆbt is the encoded “video” of bounding boxes with t levels of noise added, b(0) is the encoded initial bounding box frame, b(N-1) is the encoded final bounding box frame, and z(0) is the encoded initial video frame. The model learns to predict the noise added in ˆbt according to the EDM noise scheduler and recovers the original b from its noisy version ˆbt by calculating the noise with UNet outputs and eliminating the noise through scaling functions.

The Box2Video network generates a video using an SVD backbone and an adapted ControlNet module to process the bounding box control signal. The SVD component takes two inputs, z(0) and ˆt, where z(0) is the encoded initial video frame and ˆt is the encoded full video with t levels of noise added to it. The same input is also sent to the ControlNet module through its own UNet adapter layers, and ControlNet also receives the encoded bounding box frames, b, as input through ControlNet adapter layers. Both transformed inputs are then added together before processed by the ControlNet module.

Key Contributions

The paper presents three main contributions to generating higher fidelity videos using diffusion techniques.

1. A Novel Diffusion Approach for Predicting Bounding Box Trajectories, which generates video frames with 2D or 3D bounding box trajectories at the pixel-level based on their initial and final states, and the first frame of RGB video.

2. 2D and 3D Bounding Box Conditioned Video Generation with Diffusion, which allows video generation by conditioning on 2D or 3D bounding box trajectories which allows fine-grained control over the generated videos. This approach enables multi-subject Generation, synthesizing scenes with any number of objects limited only by the number of clearly renderable bounding boxes per frame, and Uninitialized Object Generation, training the model to be sensitive to all bounding boxes whether they are present from the first frame or appear from the middle of the clip.

3. A New Benchmark for a New Problem Formulation, which presents a new benchmark consisting of a particular way of evaluating video generation models using the KITTI, Virtual KITTI 2 (vKITTI), Berkeley Driving Dataset (BDD 100k) and nuScenes datasets.

Experimental Validation

Extensive experiments conducted on the KITTI, Virtual-KITTI 2, BDD100k, and nuScenes datasets validate the effectiveness of our approach in producing realistic and controllable video generation. The model generates videos that adhere tightly to the desired bounding box motion conditioning, accurately depicting desired object movements.

Evaluation Metrics

The model's performance is assessed across four key aspects: overall visual quality of the generated results, the alignment of predicted bounding box trajectories with the ground truth, and the fidelity of the generated objects in the video to the bounding box control signal. For generation quality, metrics include Frechet Video Distance (FVD), Learned Perceptual Image Patch Similarity (LPIPS), Structural Similarity Index Measure (SSIM), and Peak Signal-to-Noise Ratio (PSNR). For bounding box generation quality, metrics such as maskIoU, maskP, and maskR are used to evaluate the generated bounding box sequences against the ground-truth bounding box masks.

Conclusion

Ctrl-V provides future researchers with an efficient way to simulate driving video data with flexible controllability in the form of bounding boxes, and it further defines an improved metric to evaluate bounding box conditioned video generation to account for objects that are not present in the first frame, and those that do not remain until the last frame.

How it works

The BBox Generator achieves the closest alignment with ground-truth in the first and last frames, which is primarily attributed to conditioning on these key frames. The Box2Video consistently generates high-fidelity videos based on either trajectory provided.

Failure Cases and Future Work

Ctrl-V struggles to accurately encode and decode specific visual details, particularly road signs, building lettering, license-plate information and lane markings. Resolution degradation is a pervasive issue in video generation, particularly when the ego vehicle travels at higher speeds or changing directions. Future work can investigate multistage training paradigms, leveraging auxiliary networks to refine and enhance the visual quality of generated frames.

Notable Discoveries in Box2Video Model

Experiments have yielded findings suggesting that our Box2Video model not only understands bounding box locations in scenes but also the encoded track IDs and 3D bounding box orientation information in the rendered bounding box frames. The model can decode its class information and generate a correct instance of that class in the clip when an object is not present in the initial frame but its bounding box label appears in the final frame.

Future Directions

For future work, it is worthwhile to develop a systematic approach to investigate the aforementioned observations, explore alternative methods for generating bounding boxes, and create a systematic method to measure the likelihood of the bounding box frames instead of directly computing their IoU, recall, and precision rate with ground truth.

Evaluation Details

We evaluate our generation across four autonomous-vehicle datasets: KITTI, Virtual KITTI 2 (vKITTI), Berkeley Driving Dataset (BDD) and nuScenes dataset. The results indicate that the generation quality improves as we condition on more ground-truth bounding box frames.

Data Configuration

We use a standardized frame size of 312 × 520 pixels and a clip length of 25 frames at a frame rate of 7 frames per second for all experiments. The training procedures for the bounding box generators on KITTI or vKITTI datasets involve training each model for 5 epochs, with the 2D bounding box generator (1-to-1) trained on BDD100K for 33,000 iterations.

Model Capacity

The parameter counts for the BBox Generator and Box2Video models are substantial, with the BBox Generator having approximately 2.25 billion parameters and Box2Video having about 2.94 billion parameters.

Generation Quality Comparison

Table 7 summarizes generation quality metrics on the BDD100K dataset, showing that generation quality peaks when conditioned on the entire ground-truth bounding box trajectory.

1. Bold bounding box control for video generation: The Ctrl-V method allows for precise control over object motion by conditioning video generation on 2D or 3D bounding boxes and their trajectories, enabling fine-grained control over the generated videos.

2. Multi-subject synthesis capability: The system enables the synthesis of scenes with an arbitrary number of objects, as it goes beyond prior work by allowing synthesis of scenes with any number of objects, limited only by the number of clearly renderable bounding boxes per frame.

3. Handling uninitialized objects: The model is trained to be sensitive to all bounding boxes, addressing a limitation where previous methods typically overlook bounding boxes that appear only after the first frame, allowing for the synthesis of new bounding box instances during video generation.

4. Improved trajectory prediction accuracy: The BBox Generator employs a novel approach that generates video frames with 2D or 3D bounding box trajectories at the pixel-level based on their initial and final states, showing that this approach yields higher quality bounding box trajectory predictions than a recently proposed state of the art method.

5. Enhanced evaluation metrics: The authors introduce an improved metric to evaluate bounding box conditioned video generation to account for objects that are not present in the first frame, and those that do not remain until the last frame, providing a more comprehensive assessment than traditional IoU, recall, and precision rates.

Controllable video generation has attracted significant attention, largely due to advances in video diffusion models. In domains such as autonomous driving, it is essential to develop highly accurate predictions for object motions. This paper tackles a crucial challenge of how to exert precise control over object motion for realistic video synthesis. To accomplish this, we 1) control object movements using bounding boxes and extend this control to the renderings of 2D or 3D boxes in pixel space, 2) employ a distinct, specialized model to forecast the trajectories of object bounding boxes based on their previous and, if desired, future positions, and 3) adapt and enhance a separate video diffusion network to create video content based on these high quality trajectory forecasts. Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation. Extensive experiments conducted on the KITTI, Virtual-KITTI 2, BDD100k, and nuScenes datasets validate the effectiveness of our approach in producing realistic and controllable video generation.