Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Where the Real Clips Came From".
Jane: Here's the gist of the paper. AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization capabilities.
Tom: The authors introduce a rigorous, audited evaluation protocol incorporating six standard controls (C1–C6) to rigorously assess the performance of AI video detectors.
Jane: This audit debunks misleading high scores by showing that trivial classifiers can achieve near-chance performance (0.529) and that existing baselines are susceptible to dataset identity leakage.
Tom: The protocol certifies robust performers, such as the 2025 forensic detector WaveRep, which achieves a LOGO AUC of 0.996 with chance-level realvs-real coherence.
Paper summary: Tom: So we’ve covered how this paper sets up a rigorous audit protocol using six standard controls to check for confounds in AI video detection leaderboards. We looked at the specific findings about trivial classifiers collapsing and the lift gained from combining motion vectors with deep features >
Jane: And what they really drive home is that moving beyond a single AUC score to reporting an audited metric tuple—including above-floor margin, recall, and calibration—is how we get an accurate picture of deployability >
Lu: The authors are emphasizing that this auditing process isn't just about comparing detectors against each other; it’s about understanding how the detectors are actually working by checking for things like dataset identity leakage >
Meng: For us shipping models, it means we can’t just rely on a high score on one benchmark. We have to know if that performance holds up when we deploy it across different video generation sources and clip pools, which this protocol tries to test >
Lalam: The paper itself is titled Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection, and its implication is that our benchmarks need to stop letting incidental properties of the data dictate the score >
Conclusion: Tom: So, we’re wrapping up this look at "Where the Real Clips Came From: Corpus Identity in AI-Generated Video Detection." The main point here is that these video detection leaderboards can be pretty misleading if they aren't audited properly.
Jane: That makes sense. So, if you have a high score on one leaderboard, it might just mean the detector is good at recognizing that specific type of generated video data, not necessarily general generation across different sources.
Tom: Exactly. The paper shows how researchers are doing this audit using six standard controls to see if those high scores are real or just from some kind of data trick. They found that even simple classifiers can do pretty well, and they showed how existing methods can accidentally learn things about the dataset itself, which is a big problem.
Lu: What’s really interesting is their work on this temporal specification thing called TemporalSpec. It uses codec motion vectors and something called XSFF to check if the deep features are actually working together as intended from different video generation channels.
Meng: So you're saying that fusing those different parts of the video data—the motion vectors and the deep features—gives you real information about what’s happening in a genuine way, not just noise from one specific source.
Lalam: Right. The paper argues that this cross-substrate information isn't some fluke of one dataset; it’s a property of the underlying video generation techniques themselves.
Jane: And they suggest that instead of just reporting one score like AUC, we need to report a whole set of metrics—like above-floor margin and recall—to really know if a detector is actually ready for real world use.
Tom: It shifts the focus from just beating another model on a leaderboard to making sure the measurement actually reflects how it performs across different video generation sources. That’s where production models need to be careful.
Lu: It points toward needing this kind of rigorous testing before we trust any benchmark results for deployment. The future work seems to be building tools that can package all these audit checks automatically so everyone can do this properly.
Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban
Luddy School of Informatics, Computing, and Engineering at Indiana University Bloomington · College of Science and Engineering at Hamad Bin Khalifa University in Doha, Qatar
cs.CV
Submitted: 2026-06-30
Updated: 2026-11-03
Code: https://github.com/KurbanIntelligenceLab/vidaudit
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 93/100
The gist: AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization…
Key concepts
- Audited Evaluation Protocol (C1–C6)
- A six-step, rigorous testing framework designed to eliminate biases in AI video detector benchmarks. It systematically tests performance under various conditions to ensure reported scores reflect real-world capability rather than dataset quirks or trivial solutions.
- TemporalSpec
- A white-box control method that fuses codec motion vectors with deep feature trajectories (like DINOv2). This fusion verifies that the information from the video's encoding process and its content features genuinely complement each other, revealing a property of the substrates themselves.
- Cross-Substrate Complementarity
- The genuine relationship between different data sources, specifically combining codec motion vectors with deep learning trajectories. The paper demonstrates this is not an artifact of one dataset but a fundamental property of how video generation and feature extraction work across different inputs.
- Audited Metric Tuple
- A recommended way to report detector performance by providing multiple metrics—AUC, above-floor margin, recall, and calibration—instead of just a single AUC score. This comprehensive reporting accurately reflects a detector's deployability in real-world scenarios.
Terminology
Summary
AI-generated video detection benchmarks, such as GenVidBench and AIGVDBench, are often leaderboards that suffer from uncontrolled confounds that can artificially inflate reported generalization capabilities. p. 0 Auditing Generalization in AI-Generated Video Detection:
A Six-Control Protocol and the VidAudit Toolkit
Mert Onur Cakiroglu1 Zhihe Lu2 Mehmet Dalkilic1 Hasan Kurban2*
1Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington
2College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
Abstract
AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards,
yet most evaluation protocols leave uncontrolled confounds
that can inflate reported generalization. As an existence
proof, a three-feature clip-length classifier reaches a leaveone-generator-out (LOGO) AUC of 0. With these controls we can now not only compare performance and gain insight into how the detectors are working, but also provide a path to building better detectors. We
combine the six controls into an audited evaluation protocol
applying it uniformly to six representative feature sources,
with one source selected from each detector family. A 20-paper survey finds none applying all six standard controls that would catch this, so we combine them
into an audited protocol and apply it to six representative feature sources (three published detectors and three
repurposed signal sources), re-running it cross-dataset
on AIGVDBench. The audit both debunks and certifies:
the trivial classifier collapses to near chance (0. The audit both debunks and certifies:
the trivial classifier collapses to near chance (0. 529), a
CLIP baseline is caught carrying dataset identity, and the
2025 forensic detector WaveRep clears the floor at outof-distribution LOGO AUC 0. The audit both debunks and certifies:
the trivial classifier collapses to near chance (0. 529), a
CLIP baseline is caught carrying dataset identity, and the
2025 forensic detector WaveRep clears the floor at outof-distribution LOGO AUC 0. 529), a
CLIP baseline is caught carrying dataset identity, and the
2025 forensic detector WaveRep clears the floor at outof-distribution LOGO AUC 0. 996 with chance-level realvs-real coherence.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
A key contribution is TemporalSpec, a white-box positive control based on codec motion vectors and XSFF (cross-substrate feature fusion), which verifies the genuine complementarity between the codec channel and deep feature trajectories. p. 0 1%, multiple high-AUC methods fall to single-digit recall and the
leaderboard order changes, so we recommend an audited
tuple (AUC, above-floor margin, operating-point recall, and
calibration) over a single number. As a white-box positive control, we add TemporalSpec (codec motion vectors);
via cross-substrate feature fusion (XSFF), a second substrate adds genuine complementarity that survives the audit. Two-channel detection with cross-substrate feature fusion (XSFF)
XSFF fuses two motion-evidence channels. The first is
the codec-MV branch; the second is the published ReStraV
detector [17], which encodes per-frame DINOv2 ViT-S/14
embeddings into a 21-d trajectory vector.Where the paper says it
Where the paper says it
The authors strongly recommend reporting an audited metric tuple—consisting of AUC, above-floor margin, operating-point recall, and calibration—instead of a single AUC score to accurately reflect deployability. p. 1 1%,
rather than by AUC, changes the order and drops several
high-AUC methods to single-digit recall. We therefore report an audited tuple (AUC, margin above the real-vs-real
floor, operating-point recall, and calibration) instead of a
single metric. Two-channel detection with cross-substrate feature fusion (XSFF)
XSFF fuses two motion-evidence channels. The first is
the codec-MV branch; the second is the published ReStraV
detector [17], which encodes per-frame DINOv2 ViT-S/14
embeddings into a 21-d trajectory vector. 6 AUC
points). Detector (LOGO OOD) GenVidBench 27k AIGVDBench ∆
TemporalSpec (13-d, L2-LR) 0. TemporalSpec and XSFF show the audit is also constructive, with
cross-substrate complementarity surviving the controls. Reporting an audited tuple (AUC, margin, recall, calibration)
rather than a single AUC shifts attention from leaderboard rank to whether a result measures what it claims. Conclusion
AI-generated-video detection benchmarks often do not
measure what they report. Under unaudited evaluation,
a three-feature clip-length classifier reaches LOGO AUC
0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Audited evaluation protocol for video detectors allows comparison based on an audited tuple (AUC, above-floor margin, operating-point recall, and calibration) instead of a single AUC to shift attention from leaderboard rank toward whether a result measures what it claims. This provides a more trustworthy metric for deployment decisions.
-
Incorporating the six standard controls (C1–C6) into an audited evaluation protocol ensures that reported performance reflects actual detection capability rather than artifacts like clip length or dataset identity, which can inflate reported generalization.
-
Utilizing the codec-MV detector (TemporalSpec) as a white-box positive control, and fusing it with a second substrate via cross-substrate feature fusion (XSFF), allows for verification that genuine complementarity survives the audit and provides interpretable diagnostic features.
-
Training detection models using a feature vector derived from both codec motion vectors and deep appearance trajectories, as shown by the MV + ReStraV ensemble, can yield higher LOGO OOD AUC gains compared to single-channel methods, indicating genuine complementary signal.
-
Reporting operating-point recall at a false positive rate of 0.1% instead of just AUC reveals that high-AUC methods often collapse to single-digit recall at deployable thresholds, guiding practitioners toward metrics relevant for real-world screening systems.
Abstract
Benchmarks for detecting AI-generated video take their real clips from a few curated collections, so a clip's label is collinear with where it came from. We call a score's dependence on this origin corpus identity, and we measure it on four benchmarks and six real collections. Three probes read a file's declared frame types, the motion vectors its codec stores, and the frames' appearance. Frame-type counts alone separate real from generated at AUC 0.999 on GenVidBench, and re-encoding every clip with one setting at a fixed length still leaves 0.61 to 0.63 on three benchmarks. We introduce a fixed frame-type pattern that removes this leak from every clip of at least 2 s. The same re-encode lowers the ReStraV detector on GenVidBench from 0.85 to 0.60, just above the 0.59 its features reach between two real collections, because its 8 fps repeats frames in real clips. Motion vectors alone separate real collections from one another at AUC 0.51 to 0.98, and only one benchmark ships the two labeled real sources needed to measure this separation within it. In deployment, with each threshold set to flag 5% of its training benchmark's real clips, the motion-vector and appearance probes and linear readouts on ReStraV and WaveRep features each flag 15% or more of another real collection. Choosing the threshold on real videos from the target collection brings the rate back to about 5%, at the cost of catching fewer generated clips. Clearing the real-vs-real level does not guarantee replication: of the six feature sets whose GenVidBench score clears it by 0.15 or more, only WaveRep's two keep their score on AIGVDBench. A benchmark score is therefore best read together with that level, a score on a second benchmark, and false-positive rates on other real collections. The toolkit and the clip lists of every evaluation set are at https://github.com/KurbanIntelligenceLab/vidaudit.
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs