GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

The gist The introduction introduces GenState-AI, an AI-generated benchmark centered on controlled state transitions, where each query is paired with a main video, a temporal hard negative that differs only in the decisive end-state, and a semantic hard negative with content substitution <ref:2603.14426#pg2>.

GenState-AI Construction

GenState-AI is an AI-generated textto-video retrieval benchmark explicitly designed around controlled state transitions (the process is shown in Fig. 1) <ref:2603.14426#pg3>. The researchers leverage the open-source Wan2.2-TI2V-5B hybrid text-image-to-video model [33], which can generate 720p videos at 24 fps, to procedurally create a large collection of short clips <ref:2603.14426#pg4>. Each clip is defined by a simple, human-readable prompt describing (1) a small set of tabletop objects, (2) an initial configuration, and (3) a final configuration after a short motion or state change <ref:2603.14426#pg6>. Crucially, they also construct hard negative videos in which the objects, background, and overall visual style remain nearly identical to the main clip, while the final state diverges <ref:2603.14426#pg8>.

Design Principles

GenState-AI is built around three principles <ref:2603.14426#pg9>. These include (1) State-centric semantics, where each video is defined by a clear and observable change in state, such as the actor manipulating objects or the actor’s own positional change constituting an independent form of state transition. (2) Minimalist prompts, which employ concise and unambiguous prompts that focus solely on the target state transition to prevent semantic drift. (3) Hard negatives by state misalignment, where semantic hard negatives modify the target entity, and temporal hard negatives alter the action dynamics themselves, such as keeping an object stationary when movement is required.

Evaluation Protocol

The evaluation protocol involves a standard retrieval setup where for each text query q describing a stateful event, the pool always contains its corresponding state-misaligned hard negatives: a temporal hard negative vtemp (same objects and scene, similar motion pattern but different final state or temporal ordering), a semantic hard negative vsem (same motion structure but different objects or target state), and additional distractor videos. Two representative MLLM-based baselines are evaluated: (i) a likelihood-scoring T2V baseline built upon VideoChat-Flash-Qwen2-7B res448 (denoted as VCF-Lik), and (ii) a Qwen3-VL based baseline. The triplet design allows for the study of relative ordering with respect to the two hard negatives, defined by metrics such as Main-vs-Temporal accuracy (Acctemp) and Main-vs-Semantic accuracy (Accsem).

Experimental Findings

Experiments show consistent failure patterns across both baselines. Both baselines frequently confuse the main video with the temporal hard negative and overprefer temporally plausible but end-state-incorrect clips, indicating insufficient grounding to decisive end-state evidence. The triplet design reveals that temporal hard negatives dominate errors in relational and causal tasks (e.g., “moves from left of the mug to in front of it”), whereas semantic hard negatives more often account for failures in object-substitution cases. Overall, the results support the central motivation that even strong modern baselines can retrieve globally plausible videos, yet still fail to verify the decisive end-state required to distinguish the main video from temporally plausible but incorrect alternatives.

Conclusion

GenState-AI provides a focused and diagnostic testbed for developing future retrieval models that are more sensitive to state dynamics and better aligned with the growing distribution of AI-generated video content. The study concludes that likelihood-based matching can often recognize “off-topic” semantic substitutions, yet still struggles to verify decisive end-state evidence required to distinguish the true video from temporally plausible but end-state-incorrect alternatives. The findings confirm that temporal variants remain the most challenging distractors for likelihood-based scoring, even when semantic mismatches are largely rejected. The study demonstrates that GenState-AI captures diverse failure modes beyond static appearance matching. The research highlights open challenges for state- and time-sensitive text-to-video retrieval. The paper will be released on huggingface.co.

--- Page 1 ---

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

Minghan Li∗† mhli@suda.edu.cn Soochow University China Tongna Chen∗ tnchentnchen@stu.suda.edu.cn Soochow University China Tianrui Lv∗ trlvtrlv@stu.suda.edu.cn Soochow University China Yishuai Zhang yszhang13@stu.suda.edu.cn Soochow University China Suchao An scan@stu.suda.edu.cn Soochow University China Guodong Zhou gdzhou@suda.edu.cn Soochow University China Prompt Triplets (Input) Main: on a shelf there are two apples. A hand adds one more Temporal Negative: two apples. A hand removes one. Semantic Negative: there are two oranges Expert Check (MS/PhD) Wan2.2 AI Video Gen GenState-AI Dataset Diagnostic Metrics: Acc temp Acc sem Figure 1: This demonstrates our entire process, from constructing the prompt to generating videos for the AI, as well as manual annotation, filtering, and evaluation. Abstract Existing text-to-video retrieval benchmarks are dominated by realworld footage where much of the semantics can be inferred from a single frame, leaving temporal reasoning and explicit end-state grounding under-evaluated. We introduce GenState-AI, an AI-generated benchmark centered on controlled state transitions, where each query is paired with a main video, a temporal hard negative that differs only in the decisive end-state, and a semantic hard negative with content substitution. Using Wan2.2-TI2V-5B, we generate short clips whose meaning depends on precise changes in position, quantity, and object relations, providing controllable evaluation conditions for state-aware retrieval. We evaluate two representative MLLM-based baselines, and observe consistent and interpretable failure patterns: both frequently confuse the main video with the temporal hard negative and overprefer temporally plausible but end-state-incorrect clips, indicating insufficient grounding to decisive end-state evidence, while being comparatively less sensitive to semantic substitutions. We further introduce triplet-based diagnostic analyses, including relative-order statistics and breakdowns across transition categories, to make temporal vs. semantic failure sources explicit. GenState-AI provides a focused testbed for state-aware, temporally and semantically sensitive text-to-video retrieval, and will be released on huggingface.co.

--- Page 2 ---

Li et al. However, most existing retrieval benchmarks, and thus most retrieval methods implicitly assume that (1) a single frame conveys most of the semantics, and (2) videos follow natural visual statistics. Consequently, negative samples in real-world datasets often differ from positives in many uncontrolled ways, allowing models to succeed through appearance matching rather than genuine temporal reasoning. Under these conditions, models need not explicitly track how visual states change over time. This limitation raises a fundamental question: How can we reliably evaluate whether retrieval models are genuinely state- and temporal-aware, especially in the context of controllable AI-generated videos. To address this gap, we construct a benchmark dataset where AI-generated hard negatives differ from the main video only in a precisely controlled state. GenState-AI. We introduce GenState-AI, an AI-generated textto-video retrieval benchmark explicitly designed around controlled state transitions (the process is shown in Fig. 1) <ref:

1. A system trained on GenState-AI can be explicitly designed to verify decisive end-state evidence to distinguish between correct and incorrect temporal trajectories, addressing the finding that baselines frequently confuse the main video with the temporal hard negative.

2. An improved retrieval system can utilize triplet-based diagnostic analyses, such as examining relative-order statistics and breakdowns across transition categories, to make temporal vs. semantic failure sources explicit rather than relying solely on global ranking metrics like Recall@K.

3. A model incorporating state-aware reasoning could be engineered to mitigate the observed tendency of models to overprefer temporally plausible but end-state-incorrect clips by focusing evaluation on verifying the specific final configuration described in the query, as suggested by Triplet consistency metrics.