RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement

The paper introduces RAVEN-Eval, a rubric-guided automated evaluation framework for AI video generation models (AIVGMs) based on large multimodal model (LMM) preference judgment. The work addresses two major challenges in evaluating state-of-the-art video generation models: (1) the rising cost and scalability limitations of human preference annotation, and (2) the insufficient discriminative power of conventional evaluation criteria based on predefined dimensions or absolute scoring.

The authors identify two major challenges in current AIVGM evaluation:

Challenge A - Human annotation costs: "Widely used leaderboards such as Artificial Analysis rely heavily on human preference annotations. As the number of AIGVs grows, the associated cost becomes increasingly substantial. Moreover, improvements in AIVGMs make quality differences more subtle, requiring finer-grained perception. Annotators must therefore invest greater expertise, attention, and effort to make reliable decisions, further increasing cost and potentially reducing annotation consistency."

Challenge B - Limited fine-grained discrimination: "Although semantic instruction following, basic physical consistency, and visual fidelity remain essential evaluation criteria, recent SOTA AIVGMs have become increasingly competitive along these dimensions. Consequently, frameworks based on fixed dimensions or handcrafted metrics are less effective at distinguishing advanced models. Meanwhile, commercialization raises user expectations for precise and professional generation, making the remaining capability gaps more evident in fine-grained detail control and general intelligence under task-dependent and context-specific scenarios."

The paper's contributions are summarized as:

1. RAVEN-Eval Benchmark: "We construct the RAVEN-Eval Benchmark for fine-grained detail control and task-dependent model intelligence. Rather than relying on conventional predefined dimensions, we build task-centric evaluation scenarios with context-specific requirements through an LMM-assisted generation and quality-filtering pipeline. The benchmark comprises 250 diverse T2V and I2V tasks and 4,669 high-quality generated videos."

2. Rubric-guided pairwise preference judgment: "We introduce rubric-guided pairwise preference judgement under the LMM-as-a-judge paradigm. Each task is equipped with a customized rubric that guides LMM judges to compare details and determine pairwise preferences. Model capabilities are estimated through tie-aware optimization, while an anchor-based model insertion trick further enables efficient incorporation of new AIVGMs."

3. Two complementary leaderboards: "We establish two complementary RAVEN-Eval Leaderboards: the AIGV Leaderboards for ranking 20 recent open-source and proprietary generation models, and the Judge Leaderboards for benchmarking 13 general-purpose LMM judges against human preferences."

The benchmark comprises three task types: T2V generation (150 tasks), keyframe expansion task (KFT) (50 tasks), and first–last frame completion task (FLT) (50 tasks).

T2V Tasks are constructed along three complementary capability dimensions:

• Scene complexity (S): S0 (few objects, simple layouts), S1 (multiple distinct objects with nontrivial compositional relationships), S2 (groups of objects in complex spatial structures)

• Dynamic interaction complexity (D): D0 (nearly static or limited motion), D1 (coordinated, highly dynamic, complex interaction processes)

• Expert-level knowledge requirement (K): K0 (everyday situations), K1 (specialized phenomena, processes, or constraints including physical experiments, chemical reactions, material transformations, precise text rendering, specialized photographic or aesthetic styles)

Approximately 20% of tasks are designated as reasoning-based tasks. For reasoning tasks, "the prompt contains only clues, without explicitly stating the expected visual outcome. The intended phenomenon or event, which is subsequently included in the task rubric, must be uniquely inferable from the provided clues using general reasoning and relevant domain knowledge."

I2V Tasks are constructed from Creative Commons-licensed videos, with a focus on ASMR content. Such videos typically capture brief yet information-dense processes in specialized settings, such as hydraulic presses crushing materials or red-hot iron balls penetrating different substances. The KFT set contains 31 tasks from ASMR videos and 19 from other sources; the FLT set contains 35 ASMR-derived tasks and 15 other tasks.

A two-stage automatic filtering pipeline is applied:

• Stage 1: Remove all S0 and S1-D0 tasks as low-difficulty combinations... unlikely to reveal meaningful differences among advanced AIVGMs

• Stage 2: Use two solvers with different capability levels—Seedance2 Fast (high-performance proprietary) and LTX 2.3 Fast (smaller, weaker open-weight). Tasks are retained when Seedance2 Fast scores at least 3 and LTX 2.3 Fast scores at most 2 on a five-point scale, indicating clear performance gaps.

Each task receives a two-level task-specific preference evaluation rubric:

• Primary compliance criteria: mainly emphasize how accurately the generated video reproduces the detailed requirements specified in the prompt

• Basic criteria: assess non-primary object consistency, visual fidelity, aesthetic appeal, and naturalness and are applied uniformly across all tasks

Each model generates 3 videos for each task. We use GPT-5.4-mini to rank the candidates based on task-semantic completion and detail control, and retain the highest-ranked video. For I2V tasks, generated videos with reference similarity below 0.98 are excluded.

Three LMM judges are used: GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Flash-Lite. The judgement input content includes 8 uniformly sampled frames from each video, the prompt, the task-specific rubric, and, for I2V tasks, the corresponding visual reference.

An order-robust pairwise judging and confidence-aware processing protocol is adopted: "each pair is evaluated twice with reversed presentation orders. Both outputs are first mapped back to the corresponding model identities. A preference that remains consistent after reversal, or two 'tie' outputs, is treated as a confident judgment; all other cases... are treated as uncertain judgments and downweighted during capability estimation."

Model capabilities are estimated using a modified Davidson tie-aware maximum likelihood estimation objective:

The objective function optimizes latent capability scores s i and a tie parameter γ, where:

• P(i ≻ j) = θ i / (θ i + θ j + η√(θ i θ j))

• P(j ≻ i) = θ j / (θ i + θ j + η√(θ i θ j))

• P(i ∼ j) = η√(θ i θ j) / (θ i + θ j + η√(θ i θ j))

The optimization includes pair-level reliability weights λ p (proportion of confident judgments), outcome weights for confident and uncertain judgments, L2 regularization (α and β coefficients), and the constraint Σs i = 0 to resolve location non-identifiability.

Final capability scores are computed as mean latent score across all tasks in that category minus the standard deviation of these scores, then linearly rescaled to an Elo-style range: Elo i,c = 1000 + (400/ln 10) × s̄ i,c.

The paper evaluates 20 AIVGMs. The top-ranked models for T2V include Seedance 2 (840.7), G-I-V (791.5), Kling 3.0 Pro (764.6), HappyHorse V1.0 (762.6), and Kling 3.0 Omni (733.6). For FLT, Seedance 2 ranks first (890.5), and for KFT, Seedance 2 also ranks first (1094.3).

Human preference experiments are conducted on 50 tasks (30 T2V, 10 KFT, 10 FLT) with 5 expert annotators using TrueSkill-based adaptive scheduling. Inter-annotator agreement (Krippendorff's α) is 0.714, indicating acceptable agreement.

The RAVEN-Eval leaderboards show strong correlation with human rankings and external leaderboards:

• T2V: SRCC of 0.872 with human, 0.835 with AA, 0.810 with Arena

• I2V: SRCC of 0.903 (KFT) and 0.821 (FLT) with human rankings

The rubric-guided pairwise evaluation (RG-PW) outperforms alternatives:

• NR-PW (no rubric): 0.754 (T2V human), 0.852 (KFT human)

• RG-AS (rubric-guided absolute scoring): 0.672 (T2V human), 0.633 (KFT human)

• VideoScore: 0.723 (T2V human), 0.654 (KFT human)

• LOVE: 0.711 (T2V human), 0.625 (KFT human)

• Ours (RG-PW): 0.872 (T2V human), 0.903 (KFT human)

The results demonstrate that "absolute scoring, including both rubric-based (RG-AS) and trained predefined-dimension evaluators, produces substantially weaker ranking consistency than our setting. Moreover, our setting clearly outperforms NR-PW, demonstrating the importance of task-specific rubric guidance."

To reduce evaluation costs for incorporating new models, the paper introduces an anchor-based insertion strategy. Models are divided into five consecutive tiers of three models, and the top-K models with highest reliability scores are selected from each tier as anchors.

Results show that when inserting 5 new models, the anchor-based strategy reduces the pairwise annotation cost by 34.2% for K = 1 and 21.1% for K = 2, while producing rankings close (with SRCC over 0.8) to those inferred from the fully connected comparison graph. The reliability-based anchor selection clearly outperforms random selection.

The paper evaluates 13 LMM judges. Top-performing judges include Claude Opus 4.6 (SRCC 0.853, KRCC 0.686 on T2V), GPT-5.4 (SRCC 0.827, KRCC 0.673), and GPT-5.6 Sol (SRCC 0.837, KRCC 0.660). Larger-size LMMs generally outperform smaller ones, while stronger judges also tend to produce higher proportions of confident judgments.

The paper concludes: "We present RAVEN-Eval, an automated framework for evaluating AIVGMs through rubric-guided LMM preference annotations. The RAVEN-Eval Benchmark comprises 250 T2V and I2V tasks and over 4,500 AIGVs. Based on this, we establish the RAVEN-Eval Leaderboards, which comprehensively record the capability of 20 SOTA AIVGMs and the evaluation capabilities of 13 LMM judges. We further introduce an anchor-based model insertion strategy to control the evaluation cost as the model pool continues to expand. Overall, RAVEN-Eval establishes an efficient and scalable paradigm for automatic AIVGM evaluation."

The authors acknowledge: "Due to the substantial costs of human annotation, proprietary high-performance LMM judges, and closed-source AIVGMs, we are unable to obtain human annotations for all tasks or evaluate the full set of 250 tasks with a broader pool of capable LMM judges. The current AIVGM evaluation is also limited to 20 models. Consequently, the evidence for the large-scale scalability of RAVEN-Eval and its potential for LMM judge model harnessing still need further research."

Based on RAVEN-Eval, here are specific improvements for AI systems:

Improvement: Implement a two-level rubric system (primary compliance + basic criteria) that dynamically generates task-specific evaluation rubrics via LMMs before judging.

Capability: The AI can automatically decompose any generation task into explicit, checkable requirements (e.g., object count must be 3, motion must follow parabolic trajectory) and use these as structured constraints during pairwise comparison, rather than relying on vague holistic scoring.

Improvement: Adopt the modified Davidson model with tie parameter γ and pair-level reliability weights (λ p) that downweight uncertain judgments (e.g., reversed-order inconsistencies).

Capability: The system produces more stable and accurate rankings by explicitly modeling ties and confidence, avoiding false precision when two outputs are genuinely indistinguishable. This is critical for fine-grained model selection where differences are subtle.

Improvement: Implement a tiered anchor selection strategy—pick top-K reliable models per capability tier and only compare new models against anchors, not the full pool.

Capability: The AI can incorporate new models into an existing leaderboard with 21–34% fewer pairwise comparisons while maintaining ranking accuracy (SRCC > 0.8). This enables continuous, cost-effective benchmarking as model counts grow.

Improvement: Build evaluation tasks via an LMM-assisted pipeline that generates context-specific scenarios (e.g., hydraulic press crushing a metal sphere) and filters them using a two-solver gap criterion (strong model ≥3, weak model ≤2 on a 5-point scale).

Capability: The system automatically curates high-discrimination test sets that reveal meaningful capability differences between advanced models, avoiding trivial or saturated tasks that waste evaluation resources.

Improvement: Evaluate each pair twice with reversed presentation order; only consistent preferences or double-ties count as confident, while inconsistent results are downweighted.

Capability: The AI becomes robust to presentation-order bias and position effects in multimodal judging, producing more reliable preference signals—especially important when comparing videos where temporal order can influence perception.

Improvement: Compute final scores as mean latent score − standard deviation across tasks before Elo rescaling.

Capability: The system penalizes models that are inconsistent across tasks, favoring reliable performers over those with high peaks but frequent failures. This yields rankings that better predict real-world deployment reliability.

Improvement: For tasks where prompts contain only clues (not explicit outcomes), automatically generate rubrics that specify the uniquely inferable expected phenomenon, and judge whether the model correctly deduced and rendered it.

Capability: The AI can evaluate higher-order reasoning in generation models—testing whether they can infer implicit physical laws, chemical reactions, or domain-specific constraints from partial information, not just follow explicit instructions.

Improvement: Maintain a separate judge leaderboard that ranks LMM judges by their correlation with human preferences (SRCC/KRCC), and dynamically select the best-performing judge for a given task type.

Capability: The system self-optimizes its evaluation pipeline by choosing judges with proven reliability (e.g., Claude Opus 4.6 over smaller models) for specific domains, reducing annotation error and improving overall ranking fidelity.

AI video generation has advanced rapidly and entered widespread commercial use. As a result, quality differences among videos produced by state-of-the-art AI video generation models (AIVGMs) have become increasingly difficult to discern using conventional evaluation criteria, such as visual fidelity and semantic instruction following. Meanwhile, human evaluation now requires more expertise and sustained attention, substantially increasing annotation costs. This calls for automated evaluation that can reliably distinguish fine-grained differences among advanced AIVGMs with minimal human intervention. To address this challenge, we present RAVEN-Eval, a rubric-guided automated evaluation framework for AIVGMs, built primarily on the LMM-as-a-judge paradigm. Through an automatic task curation and quality-filtering pipeline, RAVEN-Eval curates 150 text-to-video (T2V) tasks and 100 image-to-video (I2V) tasks, and systematically collects more than 4,500 AIGVs. At its core, RAVEN-Eval adopts rubric-guided automated LMM preference judgement, in which LMM judges conduct pairwise comparisons according to task-specific rubrics. It further introduces an anchor-based model insertion approach to reduce the evaluation cost of incorporating new models. Finally, we evaluate 20 high-performance AIVGMs, as well as the judging capabilities of 13 LMM judges, and establish the RAVEN-Eval Leaderboards. Overall, RAVEN-Eval paves a scalable path for automatic and trustworthy evaluation of rapidly evolving AIVGMs.