A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion".

Jane: Here's the gist of the paper. The gist The authors introduce DynamicEval, a comprehensive benchmark designed to assess video generation quality under dynamic camera motion by focusing on background scene consistency and foreground object consistency.

Tom: Existing text-to-video (T2V) evaluation benchmarks suffer from limitations because they emphasize subject-centric prompts or static camera scenes, overlooking the role of camera motion, and they typically aggregate video-level scores into a single model-level score for ranking generative models The rapid advancement of foundational text-to-video models has necessitated the development of automatic evaluation metrics that correlate highly with human preferences Recent works such as VBench (Huang et al., 2024a) and EvalCrafter (Liu et al., 2024b) introduced evaluation prompt suites along with automatic metrics that assess several dimensions of video quality, including background consistency, object consistency, text alignment, and color While these benchmarks provide broad coverage, their prompts are predominantly generic and subject-centric, overlooking the role of camera motion Another critical limitation is their exclusive focus on model-level assessment where average win ratios are computed for each model in human evaluation and automatic metrics Video-level assessment can enhance the effective T2V generation quality either by selecting high-quality videos from those generated for a given prompt, or by optimizing models using video-level metrics as reward signals DynamicEval Dataset Construction DynamicEval consists of two key components designed for video-level evaluation: (1) a procedurally generated prompt suite featuring highly detailed descriptions that explicitly specify camera motion, and (2) 45k high-quality human annotations across 3k videos generated by ten T2V models using this prompt suite The dataset construction involves a procedural prompt curation strategy where keywords for background scene, primary object(s), and camera movement are collected from various databases and then fed into GPT-4o to generate descriptive prompts The dataset includes pairwise video preference annotations along two key dimensions of dynamic video quality: (1) background scene consistency and (2) foreground object consistency Videos are generated using ten state-of-the-art T2V models, including open-source and closed-source variants, resulting in 3k videos per prompt, totaling 3k videos A large-scale subjective study involving Amazon Mechanical Turk collected three human annotations per comparison, yielding 45k fine-grained human annotations Proposed Metrics for Video Quality The paper proposes two key metrics to evaluate the two critical dimensions of dynamic video quality: background (BG) consistency and foreground (FG) object consistency For background (BG) consistency, the authors investigate VBench motion smoothness (VB-MS), which relies on the RAFT optical flow model Teed & Deng (2020), and propose a new metric that debiases motion smoothness by isolating the foreground objects and removing occlusion-related background pixels This debiased error map, Ebar i, is computed as E i × (1 − M edg i ∪ M obj i) For foreground (FG) object consistency, the authors propose Track-FG, which tracks multiple points on the foreground objects using CoTracker to monitor their nearest neighbors and analyze the smoothness of distances between these tracked points over time This method is highly effective in capturing subtle deformations of the object throughout the video, ensuring long-term temporal understanding Experimental Results and Findings Extensive experiments demonstrate that the proposed metrics achieve stronger correlations with human preferences at both the video level and the model level, showing an improvement of more than 2% points For background consistency, MS-Debias outperforms VB-BG by 2.2% points on the full dataset and 2.4% points on the full agreement subset For subject consistency, Tracker-FG achieves a higher PLCC of model-level win ratios with human preference compared to VBench subject consistency (VB-SC) The proposed metrics consistently outperform the baselines when dynamic scenes are present, as shown in Table 4 Furthermore, the proposed metrics achieve significantly higher agreement with human evaluations than baseline metrics on both video-level and model-level evaluation DynamicEval enables research on fine-grained evaluation of text-to-video models ensuring human preference alignment on a video level The interpretable design of metrics enables researchers to localize and mitigate quality issues in generated videos Better evaluation tools can lead to higher quality content generation, which can be misused for malicious purposes The authors recommend using these metrics in conjunction with established real/fake detectors when evaluating generated videos for misuse Additionally, they advise mixing multiple metrics for a more holistic evaluation of the generated videos Limitations There are some limitations in our work, namely that our metrics rely heavily on the correct estimates of the optical flow and point tracks that may not always be the case for generated videos The optical flow computation and CoTracker are trained on real videos while in our work they are to be computed on the generated videos which can lead to erroneous computation The paper also notes that there is a potential for misuse if metrics are used for real/fake detection, as they may incorrectly classify high-quality generated videos as real Additionally, there is a potential for misuse in evaluating and selecting generative models based on these metrics, which can be prone to overoptimization References Adobe.

Jane: Firefly.

Tom: https://www.adobe.com/products/firefly/features/ai-video-generator.html, 2025 Mohammad Asim, Christopher Wewer, Thomas Wimmer, Bernt Schiele, and Jan Eric Lenssen.

Title and authors: Tom: So let's talk about the paper "A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion." We’re looking at how they set up this evaluation system and who wrote it.

Jane: Essentially, this paper introduces DynamicEval, which is a comprehensive benchmark designed to test generation quality specifically when the camera is moving around in the scene.

Lu: The authors are addressing two main problems with current evaluation methods: they focus too much on static scenes or just the subject prompt, and they only give you one model score instead of looking at individual videos.

Meng: That means we need to stop just ranking models generally and start picking the best video for a given prompt. How does DynamicEval solve that?

Lalam: DynamicEval is a benchmark made up of two things: first, a prompt suite that explicitly describes dynamic camera motion, and second, forty-five thousand human annotations across three thousand videos generated by ten different AI models.

Tom: That’s quite a dataset. So they used this data to create specific metrics for background and foreground consistency that are designed to handle those moving scenes.

Jane: Right. For the background consistency, they introduce MS-Debias, which uses object error maps to fix failures like occlusion or disocclusion when the camera moves.

Lu: They define this debiased error map as Ebar i equals E i times one minus M edg i union M obj i, which is a way to isolate what is actually changing in the background versus just artifacts from moving objects.

Meng: That sounds like it could help us filter out the noise that comes with just tracking pixels around moving objects, which is something I worry about when deploying models at scale.

Lalam: And for foreground object consistency, they propose Track-FG, which tracks neighboring points within an object instance to check how smoothly the distances between those tracked points change over time.

Tom: So this method is designed to be robust to both camera and object motion, aiming to capture subtle deformations of the object throughout the video.

Jane: The experimental results show that these proposed metrics achieve stronger agreement with human preferences than baseline methods, showing an improvement of more than two percent points across both video-level and model-level evaluations <ref:2510.07441#pg3>.

Lu: The paper also noted that they evaluate the MS-Debias metric at multiple scales using Gaussian pyramid downsampling to create multi-scale videos before feeding them into the interpolation model, which gives a more robust assessment.

Meng: That’s interesting because dealing with multiple resolutions adds computational cost, so we have to consider if that trade-off is worth the improved quality of the evaluation.

Lalam: Ultimately, this paper provides a strong foundation for researchers looking to build better tools that ensure human preference alignment in complex video generation tasks.

Tom: So to wrap up this discussion on "A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion," the main implication is that we need evaluation methods that specifically account for dynamic camera motion, not just static shots.

The paper's summary: Tom: We’ve covered the setup, and now we’re getting into what these authors actually summarized in "A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion."

Jane: The core idea is that current evaluation systems are basically blind to the camera moving around. They only look at what’s in the scene, not how the camera itself is moving while it shoots that scene.

Lu: They argue that existing benchmarks miss two huge things: first, they focus too much on just a single subject description, and second, they combine all those video scores into one average number for a model.

Jane: This means they have enough real-world data to really test how these models handle background consistency, which is super tricky when the camera moves.

Lu: They are testing foreground object consistency too because that’s another major part of what makes a video look good in reality—making sure the object stays consistent even as it gets closer or farther away.

Jane: The results they found show these new metrics correlate much better with what humans actually prefer, showing improvements of more than two percent points on both the video and model levels.

The paper's improvements: Tom: We’re moving on to discussing the specific improvements the authors suggest in "A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion."

Jane: Essentially, they aren't just proposing a new dataset; they are proposing two actual ways to measure consistency that are much smarter for dynamic scenes.

Lu: For background consistency, the paper introduces MS-Debias. It uses something called object error maps to fix two specific problems: occlusion and disocclusion caused by the camera moving around or objects moving in and out of view.

Meng: So instead of just seeing a messy background because the camera moved, this metric tries to isolate what’s truly wrong with the background versus what’s just an artifact of things moving.

Lalam: The math behind it is that they calculate an error map called Ebar i equals E i times one minus M edg i union M obj i, which is a principled way to tell the difference between real background changes and motion noise.

Tom: That’s a clean mechanism for filtering out those noise factors when we’re looking at video quality metrics.

Jane: And then they have Track-FG for foreground object consistency. This metric tracks neighboring points on an object instance to monitor how smoothly the distances between those tracked points change over time.

Lu: That method is designed to capture subtle deformations of the object itself, which helps evaluate how well an AI keeps that object looking correct even as the camera or the scene moves around it.

Meng: So if you have a car in a video, this metric checks if the car’s shape stays consistent while you’re panning across it, which is what we need to worry about when we deploy things.

Lalam: The results of these metrics are pretty solid; they show that MS-Debias and Track-FG beat the old baseline metrics by more than two percent points on both video and model level evaluations.

Tom: That improvement over the baselines is significant, showing that this approach actually helps us get closer to what humans consider high quality in a moving scene.

Jane: So what this means for us is that when we look at these metrics, we can start to filter videos more effectively or guide our AI training toward models that handle camera motion better.

Lu: The implication here is that video-level assessment isn't just a nice idea; it’s a necessary step for truly understanding what makes a text-to-video model perform well in the real world.

Meng: But I still think we need to be careful because these metrics rely heavily on accurate optical flow and point tracks, which can be wrong when applied to videos that are actually generated by AI.

Lalam: The paper does mention that their methods are robust, but they also flag a limitation: the accuracy of the optical flow estimates and point tracks used in the process can sometimes lead to errors in computation.

Conclusion: Tom: We’re wrapping up our discussion on "A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion," and to recap, this paper introduces DynamicEval, a benchmark focused on background and foreground consistency under dynamic camera movement.

Jane: Exactly. They show that by using metrics like MS-Debias and Track-FG, we can get much better agreement with human preference when the scene is actually moving.

Lu: The implication is that video-level assessment—choosing videos or optimizing models—is a really vital next step for understanding how good these AI video models actually are in practice.

Meng: From my side, it means we still have to be cautious because these metrics rely on accurate optical flow and point tracks, which aren't always reliable when you're running things at scale or trying to detect misuse.

Lalam: Ultimately, this paper gives researchers a strong foundation for building better tools that ensure AI video generation aligns with what people actually want in complex scenarios.

Tom: So we have these concrete metrics showing a measurable improvement of over two percent points when dynamic scenes are involved, which is solid evidence.

Jane: It shows that we can move beyond just looking at static shots and start evaluating the actual temporal quality of the AI output.

Lu: This work pushes the field toward more sophisticated representation learning for video, where consistency across time is a primary design goal.

Meng: What I need to see next is how these metrics actually perform when we try to integrate them into our production pipelines without making things too computationally expensive.

Lalam: We’ll be looking at those implementation details in the next paper, but for now, it confirms that measuring motion consistency is a major area where AI video quality needs more focus.