Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality

arXiv:2410.05203 · cs.CV, cs.AI, cs.LG · Submitted 2024-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality".

Jane: The gist The Fréchet Video Distance (FVD) falls short as a standalone metric for video generation evaluation because it relies on critical assumptions, and JEDi, the JEPA Embedding Distance,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Now, to recap what we've just covered about 'Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality', the authors are basically saying that because of those three limitations—non-Gaussianity, lack of temporal sensitivity, and impractical sample sizes—FVD isn't a reliable way to measure generation quality on its own.

Jane: So, the paper argues that a metric needs to be better at capturing temporal consistency and diversity in videos, not just image quality metrics like LPIPS or MSE.

Lu: They are looking at how different features capture spatial and temporal distortions; they show that Ithree dee features can be impacted by salt and pepper noise significantly more than other types of distortions.

Meng: That’s a crucial piece of information for production systems; if a metric is overly sensitive to one type of noise, it might give misleading quality scores when we deploy models facing those specific real-world artifacts.

Lalam: The paper suggests that JEDi, using the JEPA Embedding Distance based on MMD with polynomial kernel in a V-JEPA feature space, offers a superior alternative because it handles these issues better.

Tom: Exactly, and they show that JEDi requires much less data to get a good estimate—specifically saying it needs only seven hundred samples on UCF-one hundred one and eight hundred on SSv2 to converge, while FVD requires more than that.

Jane: It seems the authors are emphasizing that JEDi leverages the robust representations of V-JEPA models which they found to be more aligned with human preferences when compared against FVD.

Lu: The paper highlights that among all the metrics they tested, JEDi has the highest similarity to human preferences, which is a key validation point for any new evaluation tool.

Meng: For us in production, that alignment with human preference is important because it means if JEDi says a video looks good, it's more likely to match what an actual user perceives as high quality.

Lalam: So the paper's thesis is that JEDi provides a more practical and accurate way to evaluate video generation quality than FVD does.

Conclusion: Tom: So, wrapping up 'Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality', the authors are proposing JEDi as the new standard because it fixes the fundamental flaws in older metrics like FVD.

Jane: The title itself says 'Beyond FVD', which suggests they're moving past that old approach and proposing something fundamentally different based on embedding distances.

Lu: What this means is that we are shifting away from metrics built on rigid distributional assumptions, like assuming Gaussian features, toward methods like MMD with polynomial kernels.

Meng: For deployment teams, the implication is that if you use JEDi instead of FVD, you can get more reliable quality scores faster on smaller datasets because it's sample efficient.

Tom: And they’ve shown that for things like training duration, JEDi shows a higher correlation with training time than FVD does, which is an interesting finding when looking at model development speed.

Jane: It really boils down to JEDi being more robust, requiring less data to work well, and aligning better with what humans actually want in terms of video quality evaluation.

Lalam: The paper's conclusion is that JEDi emerges as the most effective and practical metric for guiding current video generation research because it performs better across all these dimensions tested.

Mila - Quebec Artificial Intelligence Institute · Samsung - SAIT AI Lab, Montreal

cs.CV, cs.AI, cs.LG

Submitted: 2024-10-07

Updated: 2024-10-08

Code: https://github.com/oooolga/JEDi

Project page: https://oooolga.github.io/JEDi.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The Fréchet Video Distance (FVD) falls short as a standalone metric for video generation evaluation because it relies on critical assumptions, and JEDi, the JEPA Embedding Distance, proves

Key concepts

Fréchet Video Distance (FVD): FVD is a metric used to compare the quality of generated videos by comparing their feature distributions. It relies on the assumption that features from a specific model (I3D) follow a Gaussian distribution, which limits its reliability.
Inflated 3D Convnet (I3D) Feature Space: This refers to the feature space derived from the I3D model used in FVD. The paper shows this space is non-Gaussian and insensitive to temporal distortions, which are critical flaws that undermine FVD's performance.
JEPA Embedding Distance (JEDi): JEDi is a proposed alternative metric based on features from a Joint Embedding Predictive Architecture (JEPA). It uses Maximum Mean Discrepancy with a polynomial kernel to measure distribution distance, offering better sample efficiency and higher alignment with human perception.

Terminology

Alignment — 1× in this paper · explained in 1 episode
Non-Gaussianity — 1× in this paper · explained in 2 episodes

Summary

The gist The Fréchet Video Distance (FVD) falls short as a standalone metric for video generation evaluation because it relies on critical assumptions, and JEDi, the JEPA Embedding Distance, proves to be a superior alternative by requiring only 16% of the samples to reach its steady value while increasing alignment with human evaluation by 34% on average

Limitations of FVD

The Fréchet Video Distance (FVD) is limited because its effectiveness relies on critical assumptions, specifically (1) the non-Gaussianity of the Inflated 3D Convnet (I3D) feature space (Page 1), (2) the insensitivity of I3D features to temporal distortions (Page 1), and (3) the impractical sample sizes required for reliable estimation (Page 1). Furthermore, FVD is not effective in capturing long-term realism and is more suitable for comparing generation model variants of the same architecture (Page 2). It overlooks motion collapse and is biased towards image quality rather than video quality (Page 2).

Proposed Metric: JEDi

The paper proposes JEDi, the JEPA Embedding Distance, which is based on features derived from a Joint Embedding Predictive Architecture and measured using Maximum Mean Discrepancy with polynomial kernel (Page 1). JEDi employs a Maximum Mean Discrepancy (MMD) metric with a polynomial kernel, eliminating the need for parametric assumptions about the underlying video distribution, unlike FVD which relies on the Gaussianity assumption to make its metric feasible (Page 1). JEDi significantly reduces the number of samples needed to make an accurate estimate by using an MMD metric in a V-JEPA feature space, enabling reliable use in smaller datasets that do not meet the requirement when using FVD (Page 1). JEDi leverages the robust representations of a V-JEPA model, which are found to be more aligned with human evaluations compared to FVD (Page 1).

Feature Space Analysis

The study investigates the normality assumption by showing that the null hypothesis that the I3D features were drawn from a multivariate Gaussian distribution was strongly rejected (p = 0) for each of the datasets and normality tests (Page 4). The notion that I3D video features do not follow multivariate Gaussian distributions was investigated using Mardia’s Skewness, Mardia’s Kurtosis, and Henze-Zirkler normality tests (Page 4). The analysis showed that the I3D clip features deviate from Gaussian distributions within each dataset (Page 4). Moreover, V-JEPASSv2 stands out as the more robust feature space among them (Page 9).

Sample Efficiency and Convergence

The sample efficiency issue is addressed by comparing convergence rates across different feature spaces (Page 6). The key findings include that the sample sizes needed for convergence are similar across datasets, but JEDi requires only 700 samples on UCF-101 and 800 samples on SSv2 to converge, whereas FVD requires more (Page 6). MMDPOLY demonstrates the quickest convergence rate across non-I3D feature spaces, requiring only 1,500 and 700 samples to reach a steady metric value on UCF-101 in V-JEPAPT and V-JEPASSv2 feature spaces (Page 6). Fréchet Distance has the worst sample efficiency among metrics, while I3D features exhibit the worst sample efficiency across feature spaces (Page 6).

Metric Robustness and Human Alignment

The choice of feature space has a greater impact on distance values than the choice of distribution distance metric (Page 7). Metrics in the I3D feature space are impacted by salt and pepper noise significantly more than by other types of distortions, and I3D does not align with human preferences with respect to distortions (Page 7). V-JEPASSv2 stands out as the more robust feature space among them (Page 9). The results of the study suggest that among all the metrics investigated, JEDi has the highest similarity to human preferences (Page 9).

Conclusion

Based on our comprehensive analysis, JEDi emerges as the most effective and practical metric for guiding the current surge in video generation research (Page 9). The results show that while both FVD (FD+I3D) and JEDi (MMDPOLY+V-JEPASSv2) are positively correlated with distortion level, only JEDi is highly negatively correlated to training duration (Page 9). The study concludes that among all the metrics investigated, JEDi has the highest similarity to human preferences (Page 9).

How it works

JEDi employs a Maximum Mean Discrepancy (MMD) metric with a polynomial kernel, eliminating the need for parametric assumptions about the underlying video distribution (Page 1).
JEDi significantly reduces the number of samples needed to make an accurate estimate by using an MMD metric in a V-JEPA feature space (Page 1).
JEDi leverages the robust representations of a V-JEPA model, which are found to be more aligned with human evaluations compared to FVD (Page 1).

Distribution Metric Overview

The study explores several statistical methods for computing probability distribution distances in video feature spaces (Page 15). These include Mixture Wasserstein (MW2), Energy Statistics, and Maximum Mean Discrepancy (MMDs) (Page 15). MMD is a general class of kernel-based sample tests that maximize the mean difference between samples from two distributions by optimizing over all data transformations f within a function space F (Page 15).

Noise and Generation Model Impact

The study investigates metric reliability when presented with videos affected by three noise distortion types (salt and pepper noise, temporal blur, and elastic distortion) and two image-to-video generation models (Page 5). The key findings include that metrics in the I3D feature space are impacted by salt and pepper noise significantly more than by other types of distortions (Page 7). Furthermore, V-JEPASSv2 stands out as the more robust feature space among them (Page 9).

Human Evaluation

A small scale survey was conducted to investigate human alignment on the perception of video quality degradation under various noise distortions (Page 8). The results suggest that among all the metrics investigated, JEDi has the highest similarity to human preferences (Page 9).

How it works

Improvements for AI systems

Improve video generation quality assessment by replacing Fréchet Video Distance (FVD) with JEDi, as it requires only 16% of the samples to reach its steady value and increasing alignment with human evaluation by 34%, on average.
Enhance model robustness against noise distortion by utilizing JEDi, as the study found that JEDi remains much more sample efficient compared to FVD in this condition and demonstrated that JEDi exhibits significantly faster convergence than FVD even with added noise.
Increase human metric alignment by adopting JEDi, as it was found to be superior in Section 5.4, where the results showed that "metrics within a feature space generally perform at the same level, [but] distances calculated in the feature space of V-JEPASSv2 or V-JEPAPT model resoundingly outperform both I3D and VideoMAE-based metrics in terms of alignment with human evaluation."
Improve sample efficiency for distribution distance metrics by adopting JEDi over FVD, as Maximum Mean Discrepancy (MMDPOLY) demonstrates the quickest convergence rate across non-I3D feature spaces and it requires fewer samples to reach a steady metric value.
Mitigate the curse of dimensionality in high-dimensional feature spaces by employing dimension reduction techniques like PCA or autoencoders, as dimension reduction significantly enhances the sample efficiency of the Fréchet Distance and energy statistic metrics in our experiments with Gaussian data.

Abstract

The Fréchet Video Distance (FVD) is a widely adopted metric for evaluating video generation distribution quality. However, its effectiveness relies on critical assumptions. Our analysis reveals three significant limitations: (1) the non-Gaussianity of the Inflated 3D Convnet (I3D) feature space; (2) the insensitivity of I3D features to temporal distortions; (3) the impractical sample sizes required for reliable estimation. These findings undermine FVD's reliability and show that FVD falls short as a standalone metric for video generation evaluation. After extensive analysis of a wide range of metrics and backbone architectures, we propose JEDi, the JEPA Embedding Distance, based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel. Our experiments on multiple open-source datasets show clear evidence that it is a superior alternative to the widely used FVD metric, requiring only 16% of the samples to reach its steady value, while increasing alignment with human evaluation by 34%, on average.