Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection

arXiv:2606.31004 · cs.CV · Submitted 2026-06-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Where the Real Clips Came From".

Jane: Here's the gist of the paper. AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization capabilities.

Tom: The authors introduce a rigorous, audited evaluation protocol incorporating six standard controls (C1–C6) to rigorously assess the performance of AI video detectors.

Jane: This audit debunks misleading high scores by showing that trivial classifiers can achieve near-chance performance (0.529) and that existing baselines are susceptible to dataset identity leakage.

Tom: The protocol certifies robust performers, such as the 2025 forensic detector WaveRep, which achieves a LOGO AUC of 0.996 with chance-level realvs-real coherence.

Paper summary: Tom: So we’ve covered how this paper sets up a rigorous audit protocol using six standard controls to check for confounds in AI video detection leaderboards. We looked at the specific findings about trivial classifiers collapsing and the lift gained from combining motion vectors with deep features >

Jane: And what they really drive home is that moving beyond a single AUC score to reporting an audited metric tuple—including above-floor margin, recall, and calibration—is how we get an accurate picture of deployability >

Lu: The authors are emphasizing that this auditing process isn't just about comparing detectors against each other; it’s about understanding how the detectors are actually working by checking for things like dataset identity leakage >

Meng: For us shipping models, it means we can’t just rely on a high score on one benchmark. We have to know if that performance holds up when we deploy it across different video generation sources and clip pools, which this protocol tries to test >

Lalam: The paper itself is titled Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection, and its implication is that our benchmarks need to stop letting incidental properties of the data dictate the score >

Conclusion: Tom: So, we’re wrapping up this look at "Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection." The main point here is that these video detection leaderboards can be pretty misleading if they aren't audited properly.

Jane: That makes sense. So, if you have a high score on one leaderboard, it might just mean the detector is good at recognizing that specific type of generated video data, not necessarily general generation across different sources.

Tom: Exactly. The paper shows how researchers are doing this audit using six standard controls to see if those high scores are real or just from some kind of data trick. They found that even simple classifiers can do pretty well, and they showed how existing methods can accidentally learn things about the dataset itself, which is a big problem.

Lu: What’s really interesting is their work on this temporal specification thing called TemporalSpec. It uses codec motion vectors and something called XSFF to check if the deep features are actually working together as intended from different video generation channels.

Meng: So you're saying that fusing those different parts of the video data—the motion vectors and the deep features—gives you real information about what’s happening in a genuine way, not just noise from one specific source.

Lalam: Right. The paper argues that this cross-substrate information isn't some fluke of one dataset; it’s a property of the underlying video generation techniques themselves.

Jane: And they suggest that instead of just reporting one score like AUC, we need to report a whole set of metrics—like above-floor margin and recall—to really know if a detector is actually ready for real world use.

Tom: It shifts the focus from just beating another model on a leaderboard to making sure the measurement actually reflects how it performs across different video generation sources. That’s where production models need to be careful.

Lu: It points toward needing this kind of rigorous testing before we trust any benchmark results for deployment. The future work seems to be building tools that can package all these audit checks automatically so everyone can do this properly.

Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban

Luddy School of Informatics, Computing, and Engineering at Indiana University Bloomington · College of Science and Engineering at Hamad Bin Khalifa University in Doha, Qatar

cs.CV

Submitted: 2026-06-30

Updated: 2026-11-03

Code: https://github.com/KurbanIntelligenceLab/vidaudit

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization…

Key concepts

Audited Evaluation Protocol (C1–C6)
A six-step, rigorous testing framework designed to eliminate biases in AI video detector benchmarks. It systematically tests performance under various conditions to ensure reported scores reflect real-world capability rather than dataset quirks or trivial solutions.
TemporalSpec
A white-box control method that fuses codec motion vectors with deep feature trajectories (like DINOv2). This fusion verifies that the information from the video's encoding process and its content features genuinely complement each other, revealing a property of the substrates themselves.
Cross-Substrate Complementarity
The genuine relationship between different data sources, specifically combining codec motion vectors with deep learning trajectories. The paper demonstrates this is not an artifact of one dataset but a fundamental property of how video generation and feature extraction work across different inputs.
Audited Metric Tuple
A recommended way to report detector performance by providing multiple metrics—AUC, above-floor margin, recall, and calibration—instead of just a single AUC score. This comprehensive reporting accurately reflects a detector's deployability in real-world scenarios.

Terminology

Summary

AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization capabilities. p. 0

Where the paper says it

Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit Mert Onur Cakiroglu1 Zhihe Lu2 Mehmet Dalkilic1 Hasan Kurban2* 1Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington 2College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar Abstract AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontrolled confounds that can inflate reported generalization. As an existence proof, a three-feature clip-length classifier reaches a leaveone-generator-out (LOGO) AUC of 0.

The authors introduce a rigorous, audited evaluation protocol incorporating six standard controls (C1–C6) to rigorously assess the performance of AI video detectors. p. 1
Where the paper says it

With these controls we can now not only compare performance and gain insight into how the detectors are working, but also provide a path to building better detectors. We combine the six controls into an audited evaluation protocol applying it uniformly to six representative feature sources, with one source selected from each detector family.

This audit debunks misleading high scores by showing that trivial classifiers can achieve near-chance performance (0. p. 0
Where the paper says it

A 20-paper survey finds none applying all six standard controls that would catch this, so we combine them into an audited protocol and apply it to six representative feature sources (three published detectors and three repurposed signal sources), re-running it cross-dataset on AIGVDBench. The audit both debunks and certifies: the trivial classifier collapses to near chance (0.

529) and that existing baselines are susceptible to dataset identity leakage. p. 0
Where the paper says it

The audit both debunks and certifies: the trivial classifier collapses to near chance (0. 529), a CLIP baseline is caught carrying dataset identity, and the 2025 forensic detector WaveRep clears the floor at outof-distribution LOGO AUC 0.

The protocol certifies robust performers, such as the 2025 forensic detector WaveRep, which achieves a LOGO AUC of 0. p. 0
Where the paper says it

The audit both debunks and certifies: the trivial classifier collapses to near chance (0. 529), a CLIP baseline is caught carrying dataset identity, and the 2025 forensic detector WaveRep clears the floor at outof-distribution LOGO AUC 0.

996 with chance-level realvs-real coherence. p. 0
Where the paper says it

529), a CLIP baseline is caught carrying dataset identity, and the 2025 forensic detector WaveRep clears the floor at outof-distribution LOGO AUC 0. 996 with chance-level realvs-real coherence.

A key contribution is TemporalSpec, a white-box positive control based on codec motion vectors and XSFF (cross-substrate feature fusion), which verifies the genuine complementarity between the codec channel and deep feature trajectories. p. 0

Where the paper says it

1%, multiple high-AUC methods fall to single-digit recall and the leaderboard order changes, so we recommend an audited tuple (AUC, above-floor margin, operating-point recall, and calibration) over a single number. As a white-box positive control, we add TemporalSpec (codec motion vectors); via cross-substrate feature fusion (XSFF), a second substrate adds genuine complementarity that survives the audit.

The audit reveals that this cross-substrate complementarity is not an artifact of a single dataset's idiosyncrasies but rather a property of the substrates themselves. Specifically, fusing the codec motion vector (MV) channel with deep features, such as DINOv2 trajectories, provides genuine cross-generator information that single channels do not capture. p. 4
Where the paper says it

Two-channel detection with cross-substrate feature fusion (XSFF) XSFF fuses two motion-evidence channels. The first is the codec-MV branch; the second is the published ReStraV detector [17], which encodes per-frame DINOv2 ViT-S/14 embeddings into a 21-d trajectory vector.

The authors strongly recommend reporting an audited metric tuple—consisting of AUC, above-floor margin, operating-point recall, and calibration—instead of a single AUC score to accurately reflect deployability. p. 1

Where the paper says it

1%, rather than by AUC, changes the order and drops several high-AUC methods to single-digit recall. We therefore report an audited tuple (AUC, margin above the real-vs-real floor, operating-point recall, and calibration) instead of a single metric.

Furthermore, Table S28 demonstrates this complementarity empirically: blending the codec-MV channel with ReStraV’s DINOv2 trajectory channel results in a lift over ReStraV alone (e. p. 4
Where the paper says it

Two-channel detection with cross-substrate feature fusion (XSFF) XSFF fuses two motion-evidence channels. The first is the codec-MV branch; the second is the published ReStraV detector [17], which encodes per-frame DINOv2 ViT-S/14 embeddings into a 21-d trajectory vector.

g., +1.6 AUC on LOGO OOD on AIGVDBench), replicating findings from GenVidBench. p. 7
Where the paper says it

6 AUC points). Detector (LOGO OOD) GenVidBench 27k AIGVDBench ∆ TemporalSpec (13-d, L2-LR) 0.

This suggests that reporting an audited tuple shifts focus from leaderboard rank to whether a result measures what it claims regarding real-world performance across different video generation sources and clip pools. p. 7
Where the paper says it

TemporalSpec and XSFF show the audit is also constructive, with cross-substrate complementarity surviving the controls. Reporting an audited tuple (AUC, margin, recall, calibration) rather than a single AUC shifts attention from leaderboard rank to whether a result measures what it claims.

The final toolkit packages this audit, providing all necessary artifacts for reproducibility, emphasizing that AI-generated video detection benchmarks often fail to measure what they report under unaudited evaluation. p. 7
Where the paper says it

Conclusion AI-generated-video detection benchmarks often do not measure what they report. Under unaudited evaluation, a three-feature clip-length classifier reaches LOGO AUC 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Audited evaluation protocol for video detectors allows comparison based on an audited tuple (AUC, above-floor margin, operating-point recall, and calibration) instead of a single AUC to shift attention from leaderboard rank toward whether a result measures what it claims. This provides a more trustworthy metric for deployment decisions.

  2. Incorporating the six standard controls (C1–C6) into an audited evaluation protocol ensures that reported performance reflects actual detection capability rather than artifacts like clip length or dataset identity, which can inflate reported generalization.

  3. Utilizing the codec-MV detector (TemporalSpec) as a white-box positive control, and fusing it with a second substrate via cross-substrate feature fusion (XSFF), allows for verification that genuine complementarity survives the audit and provides interpretable diagnostic features.

  4. Training detection models using a feature vector derived from both codec motion vectors and deep appearance trajectories, as shown by the MV + ReStraV ensemble, can yield higher LOGO OOD AUC gains compared to single-channel methods, indicating genuine complementary signal.

  5. Reporting operating-point recall at a false positive rate of 0.1% instead of just AUC reveals that high-AUC methods often collapse to single-digit recall at deployable thresholds, guiding practitioners toward metrics relevant for real-world screening systems.

Abstract

Benchmarks for detecting AI-generated video take their real clips from a few curated collections, so a clip's label is collinear with where it came from. We call a score's dependence on this origin corpus identity, and we measure it on four benchmarks and six real collections. Three probes read a file's declared frame types, the motion vectors its codec stores, and the frames' appearance. Frame-type counts alone separate real from generated at AUC 0.999 on GenVidBench, and re-encoding every clip with one setting at a fixed length still leaves 0.61 to 0.63 on three benchmarks. We introduce a fixed frame-type pattern that removes this leak from every clip of at least 2 s. The same re-encode lowers the ReStraV detector on GenVidBench from 0.85 to 0.60, just above the 0.59 its features reach between two real collections, because its 8 fps repeats frames in real clips. Motion vectors alone separate real collections from one another at AUC 0.51 to 0.98, and only one benchmark ships the two labeled real sources needed to measure this separation within it. In deployment, with each threshold set to flag 5% of its training benchmark's real clips, the motion-vector and appearance probes and linear readouts on ReStraV and WaveRep features each flag 15% or more of another real collection. Choosing the threshold on real videos from the target collection brings the rate back to about 5%, at the cost of catching fewer generated clips. Clearing the real-vs-real level does not guarantee replication: of the six feature sets whose GenVidBench score clears it by 0.15 or more, only WaveRep's two keep their score on AIGVDBench. A benchmark score is therefore best read together with that level, a score on a second benchmark, and false-positive rates on other real collections. The toolkit and the clip lists of every evaluation set are at https://github.com/KurbanIntelligenceLab/vidaudit.

Related papers