I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

The paper addresses a fundamental gap in video reasoning research. The authors note that real-world video reasoning often involves multimodal, multi-source inputs, whereas existing video reasoning tasks typically assume a simplified video-text setting, limiting identity matching and person-centric reasoning. They observe that most studies still follow the traditional 'video–text' bimodal paradigm, where models only process video content and generate textual outputs, which "overlooks the heterogeneous, multi-source inputs commonly encountered in real applications. For example, users often issue queries conditioned not only on the video itself but also on reference images and specific object descriptions."

To bridge this gap, the authors introduce Identity-conditioned Queries (ICQ), described as "a challenging person-centric video reasoning task. Unlike conventional video reasoning, ICQ requires models to reason over both a complex video and a reference image of a target person. Since the reference image may come from a different source, large appearance gaps in clothing, lighting, makeup, and other factors make identity matching difficult. Models must therefore identify the same person across these variations while handling tasks such as localization, action recognition, temporal grounding, and causal reasoning."

The task presents three key challenges: (i) integrating heterogeneous inputs, (ii) localizing and tracking the target person in complex scenes, (iii) cross-domain identity matching with identity-aware understanding. These abilities are framed as important for applications such as video retrieval, personalized analytics, film production, and intelligent surveillance.

The authors propose ISYV (I Seek You in Videos), "a systematic solution for the ICQ task, comprising three components: ISYV-Bench, an ICQ-specific evaluation benchmark; ISYV-75K, a large-scale training set; and ISYV-Framework, a dedicated model with an ICQ-oriented training strategy."

The training dataset comprises 24,150 unique video clips and 74,578 question-answer pairs, with each video accompanied by a corresponding character patch to guide the model in answering questions. It was built by collecting over 100,000 video clips from authentic film and television productions including films, TV series, and anime.

The construction pipeline is end-to-end automated with multi-stage verification. Specifically: TransNet V2 is used for shot segmentation; Gemini-2.5-Pro generates shot-level descriptions with environmental context and ID-tagged character actions; Qwen3-VL-32B performs quality checks on reference images; Qwen3-Max generates QA pairs from templates; and Qwen3-32B conducts plausibility checks, leakage detection, and CoT rationale generation. For outfit-changed and cross-shot images, the authors use Qwen-Image-Edit for attribute modification and face ReID techniques to retrieve additional appearances of the same character from the same source material, selecting reference images that exhibit maximal differences in pose and background. Identity consistency is verified via cross-checks using Qwen3-VL-32B and Qwen3-32B.

The ICQ task is aligned with the developmental trajectory of human cognitive abilities, comprising six hierarchical task levels with progressively increasing difficulty:

• L1 – Who, what, and which shot? (Basic Perception, BP)

• L2 – Same person, different outfit or new location (Object Permanence, OP)

• L3 – Subtle details (Procedural Observation, PO)

• L4 – Gaze, dialogue, exchange, and social dynamics (Social Cognition, SC)

• L5 – Movement path within and across shots (Spatial Memory, SM)

• L6 – Inferring 'why' and understanding hidden intent (Causal Reasoning, CR)

The evaluation benchmark consists of 1,377 high-quality video clips, 1,377 corresponding character patches, and 1,377 question-answer pairs, constructed via manual curation and verification. Each sample was cross-validated, error-analyzed, and revised by at least three annotators, followed by a second-round spot check.

The authors highlight that unlike existing benchmarks, ISYV features complex transitions, multi-input (video + image), and Human-Aligned Cognition — all marked for thirteen prior benchmarks (TVQA, MVBench, ActivityNet-QA, NExT-QA, LongVideoBench, MMVU, Video-MME, MLVU, Ego-Exo4D, EgoExoLearn, CrossVid), whereas ISYV-Benchmark receives on all three dimensions.

The ICQ Module addresses the problem that directly reusing the original visual encoder and simply concatenating video tokens with image tokens leads to two notable issues: the model often fails to correctly capture the semantic relationship between the reference image and the video, frequently misinterpreting the reference image as the final frame of the video sequence, and the token sequence produced by the image encoder is typically excessively long, which not only introduces irrelevant noise but also incurs unnecessary computational overhead.

The module's mechanism lies in compressing the reference image into a compact representation through learnable tokens and facilitating interaction with image features via attention mechanisms. It first applies self-attention to learnable tokens to model internal dependencies, then uses these tokens as Queries with image tokens as Keys/Values in cross-attention. The learnable token length is ablated, finding 32 tokens optimal.

The training follows a two-stage pipeline: "We first perform supervised fine-tuning as a cold-start phase, enabling the model to acquire foundational knowledge and learn the target output format. Subsequently, we conduct reinforcement fine-tuning using the GRPO algorithm to further enhance model performance. Data is partitioned into SFT and RFT subsets with an intra-class ratio of 6:4," based on quality scores from Qwen3-32B evaluating CoT reasoning quality and label leakage risk.

Four reward functions are used in RFT:

1. Format Reward — decomposes output into four fields: , , , and . It enables the model to learn reference alignment, evidence localization, reasoning, and answer generation, thereby improving task performance while mitigating answer hacking. Partial credit is given with weights wcap=0.2, wcand=0.15, wthink=0.15, wans=0.5.

2. Accuracy Reward — a binary score evaluating consistency between the model answer and ground truth.

3. Caption Semantic Reward — uses an LLM judge to evaluate semantic consistency between generated caption and ground-truth reference image description. This facilitates better understanding of the reference image content related to the question, preventing the model from exploiting shortcuts by analyzing only the video and text while ignoring the reference image.

4. Effective Shot Re-Reasoning (ESR) Reward — the key novel contribution. During GRPO rollout, the model outputs the effective shot sequence, chain-of-thought, and final answer. If the initial answer is correct and the shot sequence parses, the mentioned shots are clipped and concatenated according to timestamp annotations and re-reasoned to obtain a second answer. The reward is defined as resr = 1[A1=Agt] · 1[Parse(S)] · 1[A2=Agt] · (0.5 + rextra), where the extra reward rextra decreases linearly with the number of output shots. This mechanism enables the model to learn how to identify effective shots and achieve localization and acquisition of key information without ground-truth effective shot annotations.

The total reward is: r = thetaacc·racc + thetafmt·rfmt + thetacap·rcap + thetaesr·resr.

Evaluated models include closed-source (GPT-5.2-Global, Gemini-2.5-Pro, Qwen3-VL-235B-A22B), open-source (Qwen2.5-VL-7B/32B, Qwen3-VL-8B, GLM-4.1V-9B-Thinking, Mimo-VL-7B-RL, InternVL3-8B, VideoLLaMA3-7B), and related works (IDA-VLM, PLVM, Video-R1), compared against human performance. All models were evaluated with 32 uniformly sampled frames. Two metrics are reported: O.Avg (correct answers only) and ICQ Q.Avg (answers that are both correct and satisfY the ICQ requirement of containing reference image analysis).

The ICQ task poses significant challenges for existing MLLMs. Current MLLMs exhibit a substantial performance gap compared to humans on ICQ and similar complex tasks. Even the best-performing proprietary model, Gemini-2.5-Pro, falls considerably short of human performance (human 95.13% vs. Gemini 67.10% on O.Avg). Humans demonstrate clear advantages across all difficulty levels, particularly in Level 2 tasks involving disguised character analysis.

Closed-source MLLMs significantly outperform open-source models. All closed-source MLLMs achieve higher overall average accuracy than open-source MLLMs... Furthermore, closed-source models are less inclined to attempt hacking answers. Gemini-2.5-Pro achieves 67.10% overall and 85.93% on Level 1, while open-source models fall below 40% on overall average.

Answer hacking is a serious problem. Some models tend to completely ignore the input reference image and attempt to hack the answer using only the video and question. For instance, VideoLLaMA3-7B achieves a reasonable O.Avg but exhibits catastrophic performance on ICQ Q.Avg (37.25% vs. 3.34%), revealing it fails to comprehend the ICQ task entirely and tends to provide brief responses that appear 'random' or attempt to hack an answer. Qwen2.5-VL-32B shows an 18.01% gap between O.Avg (24.76%) and ICQ Q.Avg (6.75%) in the image-input condition.

The ICQ task is irreplaceable. "The ICQ task cannot be effectively addressed through intuitive alternative methods, such as employing multi-turn dialogue or performing reasoning after substituting images with textual descriptions. Multiple models exhibit significant performance degradation when using text-substituted image inputs."

ISYV-Model performance. In the 7B setting, our proposed model increases the average accuracy to 57%, yielding a significant improvement over the base model and achieving performance that is broadly comparable to closed-source models. Specifically, ISYV-Model-RFT reaches 57.01% O.Avg / 57.01% ICQ Q.Av.g, compared to Qwen2.5-VL-7B-RFT at 54.22% (base model), and outperforms all evaluated 7B–9B open-source models. It excels particularly on Level 6 (Causal Reasoning) at 70.20%, approaching Gemini-2.5-Pro's 82.32%.

Training strategy and rewards (Table 4): SFT alone gives 33.75%; removing ground-truth candidate learning in SFT gives 35.62% (SFT-GTcand); adding RFT with format+accuracy rewards reaches 52.06%; adding caption reward reaches 55.44%; adding ESR reward reaches 57.01%. The results indicate that omitting the learning of... during the SFT stage yields better immediate performance but constrains the model's ability to learn effective shots in the subsequent RFT stage.

ICQ Module token length (Table 5): 32 tokens is optimal (57.01%) versus 16 tokens (46.93%) and 64 tokens (53.31%), since too few tokens fail to adequately represent the reference person image, while too many may introduce noise.

Frame count: 32 frames is best (57.01%); 16 frames degrades to 34.50%; 64 frames gives 56.56%, showing that increasing frames beyond training-time sampling does not help.

For a Level-5 (Spatial Memory) sample, the case study shows: "GLM4.1V-9B and Qwen2.5-VL-32B misinterpreted the reference image, Qwen3-VL-8B incorrectly claimed that the referenced person was absent, and InternVL3-8B failed to follow the required format. Although Qwen2.5-VL-32B selected the correct option, this may be considered a form of hacking behavior because its image description was inaccurate. In contrast, our model accurately described the reference person and answered correctly."

The four main contributions are:

1. We formulate Identity-Conditioned Queries (ICQ), a practical person-centric task requiring joint reasoning over a video, a reference image of the target person, and a text query.

2. "We build a comprehensive data suite for ICQ: ISYV-Bench, a 1,377-sample evaluation benchmark organized into a six-level cognitive hierarchy, and ISYV-75K, a 75K-sample training set spanning diverse scenes, identities, and question types."

3. "We propose ISYV-Framework, featuring an ICQ Module for reference-image abstraction and an ICQ-oriented training strategy with a novel ESR reward that selects identity-relevant evidence without shot-level annotations. With 7B parameters, it rivals closed-source MLLMs."

4. We comprehensively evaluate mainstream MLLMs on ISYV-Bench, revealing key bottlenecks in cross-domain identity matching and long-horizon tracking, with ablations offering insights for future research.

The authors conclude: "Extensive evaluation reveals that the ICQ task poses significant challenges for current MLLMs, with even the best-performing proprietary models exhibiting substantial gaps compared to human-level performance... we uncover fundamental limitations of existing models. We anticipate that the ICQ task will serve as a valuable resource for advancing the boundaries of video reasoning and promoting the development of MLLMs."

1. Add an Identity-Conditioned Query (ICQ) module to video-language models.

Instead of naively concatenating reference-image tokens with video tokens, the model will compress the reference image into 32 learnable query tokens via a two-step attention mechanism: self-attention among learnable tokens, then cross-attention where these tokens act as queries and image features as keys/values. This prevents the model from mistaking the reference image as the last video frame, reduces irrelevant noise, and lowers computational overhead.

2. Train with a two-stage SFT + RFT pipeline where the SFT stage teaches the full output format including `` (effective shot sequence).

The model will be supervised to output four fields: , , , and . Keeping ground-truth candidate learning in SFT is important; although removing it may give a short-term accuracy boost, it constrains later reinforcement learning from learning to localize relevant shots.

3. Use GRPO-based reinforcement fine-tuning with four reward functions.

The model will be optimized with:

• a format reward with partial credits (caption 0.2, candidate 0.15, think 0.15, answer 0.5) to mitigate answer hacking;

• an accuracy reward for exact-answer correctness;

• a caption semantic reward using an LLM judge to check semantic consistency of the generated caption with the reference image, preventing the model from ignoring the image and exploiting video + text shortcuts;

• a novel Effective Shot Re-Reasoning (ESR) reward that, during rollout, clips the predicted effective shots, concatenates them, re-reasons to get a second answer, and rewards consistency along with a linear penalty for too many shots. This teaches evidence localization and key-information acquisition without ground-truth shot annotations.

4. Pre-train or fine-tune on a large-scale, automatically curated multi-stage dataset like ISYV-75K.

The training data will include shot-level descriptions with ID-tagged character actions, reference images with outfit changes and cross-shot appearances, and quality-verified QA pairs with chain-of-thought rationales. Using identity-coherent, real-world heterogeneous inputs will improve cross-domain identity matching.

5. Organize training/evaluation tasks across a six-level cognitive hierarchy.

The model will be trained and evaluated on six difficulty levels mirroring human cognition: Basic Perception, Object Permanence, Procedural Observation, Social Cognition, Spatial Memory, and Causal Reasoning. This ensures the system develops generalized person-centric video reasoning rather than simple perceptual matching.

6. Require the model to explicitly perform reference-image analysis in its output.

During inference, the model will be prompted to describe the reference person, then localize them, then reason, then answer. Evaluation will use both overall accuracy (O.Avg) and ICQ-constrained accuracy (ICQ Q.Avg) that only counts answers containing correct reference-image analysis. This explicitly suppresses answer-hacking behavior.

7. Use a 32-frame sampling strategy at inference.

The model will sample 32 uniformly spaced frames during video encoding. The paper shows 16 frames are insufficient (34.50% vs. 57.01%) while increasing to 64 offers no gain (56.56%), so 32 is the optimal cost-performance point.

8. Integrate identity-verification cross-checks in data generation and evaluation.

For any referenced person, the system will use face ReID and multimodal LLM cross-checks to verify identity consistency across clothing, lighting, makeup, and pose changes. This improves robustness to appearance gaps and prevents hallucinated “person absent” claims.

• Given a video and a single reference image of a target person, the system can answer complex queries that require finding that same person across clothing changes, new locations, subtle details, social interactions, movement paths, and causal/intent reasoning.

• It can output a structured response: a caption of the reference-image person, the specific shots containing the evidence, a chain-of-thought reasoning trace, and a final answer.

• It can explicitly resist the common failure mode of ignoring the reference image and hacking an answer from the video alone; its answers are grounded in both video evidence and identity analysis.

• It can localize and track the target individual across shot boundaries and appearance variations, handling long-horizon visual reasoning.

• It can perform at a level broadly comparable to much larger closed-source models (e.g., 57% accuracy with 7B parameters vs. Gemini-2.5-Pro at 67%) while remaining significantly more robust on identity-matching and causality tasks than current open-source MLLMs.

• It degrades gracefully when inputs are swapped — e.g., it outperforms models that rely on substituting textual descriptions for the reference image, and it maintains high ICQ-constrained accuracy rather than collapsing to near-zero.

• It can be evaluated not just for correct answers, but for whether the reasoning process actually incorporates the identity reference, making its performance more transparent and trustworthy in safety-critical applications such as surveillance, personalized video retrieval, and film-production analysis.