Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection
summary
The gist
AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization…
In short
AI video detection leaderboards are flawed due to uncontrolled confounds. The authors introduce a rigorous, six-control audit protocol to test detectors like WaveRep. This audit shows trivial classifiers perform poorly and identifies robust performers, proving that cross-substrate information from codec motion vectors genuinely improves detection accuracy across different video generation sources.
Key concepts
- Audited Evaluation Protocol (C1–C6)
- A six-step, rigorous testing framework designed to eliminate biases in AI video detector benchmarks. It systematically tests performance under various conditions to ensure reported scores reflect real-world capability rather than dataset quirks or trivial solutions.
- TemporalSpec
- A white-box control method that fuses codec motion vectors with deep feature trajectories (like DINOv2). This fusion verifies that the information from the video's encoding process and its content features genuinely complement each other, revealing a property of the substrates themselves.
- Cross-Substrate Complementarity
- The genuine relationship between different data sources, specifically combining codec motion vectors with deep learning trajectories. The paper demonstrates this is not an artifact of one dataset but a fundamental property of how video generation and feature extraction work across different inputs.
- Audited Metric Tuple
- A recommended way to report detector performance by providing multiple metrics—AUC, above-floor margin, recall, and calibration—instead of just a single AUC score. This comprehensive reporting accurately reflects a detector's deployability in real-world scenarios.
Terminology used across episodes
This episode discusses
The paper
Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection · Read on arXiv
Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban
Luddy School of Informatics, Computing, and Engineering at Indiana University Bloomington · College of Science and Engineering at Hamad Bin Khalifa University in Doha, Qatar
Benchmarks for detecting AI-generated video take their real clips from a few curated collections, so a clip's label is collinear with where it came from. We call a score's dependence on this origin corpus identity, and we measure it on four benchmarks and six real collections. Three probes read a file's declared frame types, the motion vectors its codec stores, and the frames' appearance. Frame-type counts alone separate real from generated at AUC 0.999 on GenVidBench, and re-encoding every clip with one setting at a fixed length still leaves 0.61 to 0.63 on three benchmarks. We introduce a fixed frame-type pattern that removes this leak from every clip of at least 2 s. The same re-encode lowers the ReStraV detector on GenVidBench from 0.85 to 0.60, just above the 0.59 its features reach between two real collections, because its 8 fps repeats frames in real clips. Motion vectors alone separate real collections from one another at AUC 0.51 to 0.98, and only one benchmark ships the two labeled real sources needed to measure this separation within it. In deployment, with each threshold set to flag 5% of its training benchmark's real clips, the motion-vector and appearance probes and linear readouts on ReStraV and WaveRep features each flag 15% or more of another real collection. Choosing the threshold on real videos from the target collection brings the rate back to about 5%, at the cost of catching fewer generated clips. Clearing the real-vs-real level does not guarantee replication: of the six feature sets whose GenVidBench score clears it by 0.15 or more, only WaveRep's two keep their score on AIGVDBench. A benchmark score is therefore best read together with that level, a score on a second benchmark, and false-positive rates on other real collections. The toolkit and the clip lists of every evaluation set are at https://github.com/KurbanIntelligenceLab/vidaudit.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Where the Real Clips Came From".
Jane: Here's the gist of the paper. AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization capabilities.
Tom: The authors introduce a rigorous, audited evaluation protocol incorporating six standard controls (C1–C6) to rigorously assess the performance of AI video detectors.
Jane: This audit debunks misleading high scores by showing that trivial classifiers can achieve near-chance performance (0.529) and that existing baselines are susceptible to dataset identity leakage.
Tom: The protocol certifies robust performers, such as the 2025 forensic detector WaveRep, which achieves a LOGO AUC of 0.996 with chance-level realvs-real coherence.
Paper summary: Tom: So we’ve covered how this paper sets up a rigorous audit protocol using six standard controls to check for confounds in AI video detection leaderboards. We looked at the specific findings about trivial classifiers collapsing and the lift gained from combining motion vectors with deep features >
Jane: And what they really drive home is that moving beyond a single AUC score to reporting an audited metric tuple—including above-floor margin, recall, and calibration—is how we get an accurate picture of deployability >
Lu: The authors are emphasizing that this auditing process isn't just about comparing detectors against each other; it’s about understanding how the detectors are actually working by checking for things like dataset identity leakage >
Meng: For us shipping models, it means we can’t just rely on a high score on one benchmark. We have to know if that performance holds up when we deploy it across different video generation sources and clip pools, which this protocol tries to test >
Lalam: The paper itself is titled Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection, and its implication is that our benchmarks need to stop letting incidental properties of the data dictate the score >
Conclusion: Tom: So, we’re wrapping up this look at "Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection." The main point here is that these video detection leaderboards can be pretty misleading if they aren't audited properly.
Jane: That makes sense. So, if you have a high score on one leaderboard, it might just mean the detector is good at recognizing that specific type of generated video data, not necessarily general generation across different sources.
Tom: Exactly. The paper shows how researchers are doing this audit using six standard controls to see if those high scores are real or just from some kind of data trick. They found that even simple classifiers can do pretty well, and they showed how existing methods can accidentally learn things about the dataset itself, which is a big problem.
Lu: What’s really interesting is their work on this temporal specification thing called TemporalSpec. It uses codec motion vectors and something called XSFF to check if the deep features are actually working together as intended from different video generation channels.
Meng: So you're saying that fusing those different parts of the video data—the motion vectors and the deep features—gives you real information about what’s happening in a genuine way, not just noise from one specific source.
Lalam: Right. The paper argues that this cross-substrate information isn't some fluke of one dataset; it’s a property of the underlying video generation techniques themselves.
Jane: And they suggest that instead of just reporting one score like AUC, we need to report a whole set of metrics—like above-floor margin and recall—to really know if a detector is actually ready for real world use.
Tom: It shifts the focus from just beating another model on a leaderboard to making sure the measurement actually reflects how it performs across different video generation sources. That’s where production models need to be careful.
Lu: It points toward needing this kind of rigorous testing before we trust any benchmark results for deployment. The future work seems to be building tools that can package all these audit checks automatically so everyone can do this properly.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound