Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

The paper introduces Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. The authors argue that while video generative models have advanced rapidly in visual fidelity, motion coherence, and controllability, these gains shift the central question from whether a generated video looks plausible to whether it faithfully realizes the events, constraints, and temporal dependencies specified by the prompt. This question is particularly consequential in science-domain settings, where perceptual plausibility can mask fundamental errors in the underlying process. The paper notes that a video may look convincing while violating a conservation law, reversing a causal relation, or depicting an impossible state transition.

The authors identify a gap in existing evaluation: Existing benchmarks provide increasingly broad coverage of perceptual quality, prompt alignment, and compositionality, while reasoning-oriented suites largely focus on generic physical commonsense. Meanwhile, knowledge- and reasoning-intensive evaluation has primarily been studied for video understanding. The closest concurrent benchmark, VideoScience-Bench, evaluates scientific phenomena in physics and chemistry through 200 expert-annotated prompts, but Sci-VBench differs by spanning 60 subjects, providing reusable per-example reference guides and rubrics, and studying evaluation portability across expert, non-expert, and MLLM-based raters.

Sci-VBench is constructed around four core desiderata: "(1) Breadth of domain knowledge: 1,253 examples spanning 60 subjects across four disciplines, from astrophysics to public policy. (2) Depth of expert-level reasoning: every prompt is authored by a domain expert, so generating it correctly demands grounded scientific understanding and multi-step causal reasoning. (3) Completeness of spatiotemporal reasoning: the target phenomenon must unfold over time, so no single well-chosen frame can satisfy the prompt. (4) Reliable and reproducible evaluation: a rubric-based protocol with per-example scoring anchors, released so that other groups can apply the same standard without recruiting domain experts."

The benchmark contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science (45.3%), Healthcare (23.9%), Engineering (22.6%), and Humanities & Social Sciences (8.2%). The distribution includes 567 examples in Natural Science, 300 in Healthcare, 283 in Engineering, and 103 in Humanities & Social Sciences.

Subject selection was informed by a user study with 133 undergraduate and graduate students who curated video prompts requiring expert-level reasoning. The authors selected the 60 subjects across four disciplines whose core concepts are both expert-level and verifiable from video evidence. A total of 61 expert annotators participated, comprising 11 undergraduate students, 45 graduate students, and five of the authors. Each subject was assigned to annotators with matching expertise, and every example is authored by one annotator and independently reviewed by another.

Prompts were annotated through a textbook-guided pipeline where annotators "first select a target concept (or tightly coupled set of concepts) from canonical textbooks and course materials that is representative of the subject's core curriculum and naturally expressed through observable spatiotemporal dynamics. The authors require that selected concepts have mechanism-governed visual realizations... so that videos may look superficially plausible yet still produce clear and systematic deviations when the underlying principles are violated. For each concept, annotators craft a minimal prompt that specifies only the observable initial setup and any explicit intervention or task objective, omitting the expected mechanistic trajectory and the key phenomena to be generated, so that a model must infer the mechanism from the setup rather than reproduce an outcome the prompt already describes."

Each prompt is paired with an evaluation specification consisting of:

1. A high-level reference guide that specifies the target concept(s) and the minimal mechanistic assumptions required for the scenario to be well-defined and summarizes the expected phenomena as a concise phase-based storyline, including key causal transitions and any visibility/viewpoint constraints needed for verification.

2. A detailed evaluation rubric with 1–5 scoring anchors per dimension that specifies what evidence is sufficient for full credit, what constitutes partial correctness, and which violations or omissions warrant low scores.

Every example undergoes a second pass by an independent domain expert who checks that the prompt is clear and fully specified, that the intended mechanism is visually testable in the described scene, and that the prompt, reference guide, and rubric are mutually consistent. An audit of 200 randomly sampled examples found "high agreement on every reasoning dimension (quadratic weighted κ = 0.79 for Prompt Grounding, 0.75 for Scientific and Causal Correctness, and 0.73 for Spatiotemporal Consistency), indicating that a score is determined by the released specification rather than by who wrote it."

Sci-VBench evaluates four dimensions:

1. Low-level Perceptual Fidelity: the perceptual quality of the synthesized video, including temporal coherence, motion dynamics, and the visual quality of individual frames.

2. Prompt Grounding: whether the video faithfully instantiates the explicit conditions stated in the prompt, such as the presence and identity of key entities, their initial states, spatial arrangement, and any required instruments.

3. Scientific and Causal Correctness: whether the generated dynamics are consistent with the domain knowledge and causal mechanisms targeted by the prompt, as specified by the per-example reference guide.

4. Spatiotemporal Consistency: whether the video maintains the coherent temporal evolution and stable spatial relationships needed for the intended mechanism to be interpretable.

Expert human ratings serve as the primary reference labels for each evaluation dimension. For each generated video, an expert annotator in the corresponding discipline is provided with the text prompt, the generated video, and the per-example evaluation specification, and then assigns 1–5 integer scores for all four dimensions. Inter-expert agreement was strong, with Cohen's κ = 0.842 on 300 re-rated samples.

For automated evaluation, Low-level Perceptual Fidelity is assessed via VBench-based Vision Tools (VT) using six metrics from VBench's Video Quality dimension. The other three dimensions are evaluated using a rubric-conditioned MLLM-as-judge pipeline, instantiated with the open-source Qwen3.5-397B-A17B. The judge receives the original text prompt, the generated video, the per-example high-level reference guide, and the 1–5 anchored rubric for that dimension only. Every video–dimension pair is scored in three independent judge runs, with the mean reported.

Providing the evaluation specification to non-experts increases instance-level Pearson correlation in all four dimensions, with improvements from 74.1 to 83.3 for Prompt Grounding, 68.2 to 82.5 for Scientific and Causal Correctness, and 70.3 to 79.6 for Spatiotemporal Consistency. The paper states that with the specification, non-experts agree with experts more closely than any MLLM-as-Judge instantiation we tested, on every dimension.

Among MLLM judges, Qwen3.5-397B-A17B attains the highest correlations overall, and agreement broadly increases with evaluator scale. The rubric-conditioned protocol aligns more closely with expert judgments than prior video scoring methods including VideoScore, VideoScore2, VideoReward, and ETVA.

The paper benchmarks 16 frontier text-to-video models: eight proprietary systems (Sora-2, Veo-3.1-Fast, Veo-3.1, Kling-2.6, Wan-2.6, Seedance-2.0, HappyHorse-1.1, and Gemini-Omni-Flash) and eight open-source models (HunyuanVideo-1.5, LTX-2.0, LTX-2.3, LongCat-Video, Wan2.2-5B, CogVideoX1.5-5B, Cosmos3-Nano, and MiniMax-H3). Proprietary systems were evaluated on the testmini split (150 examples) only, while open-source models were evaluated on both splits.

1. Mechanism, not appearance, separates systems. VT is nearly flat across all 16 models (3.79–4.12) while human Low-level Perceptual Fidelity ratings of the same construct spread wider (2.66–3.76). The reasoning-centric dimensions separate models far more sharply: automatic Scientific and Causal Correctness ranges from 1.24 to 3.34, and human SCC from 1.12 to 3.06.

2. Proprietary–open-source gap is concentrated on reasoning. Gemini-Omni-Flash attains the best automatic (3.38) and human (3.18) averages, followed by HappyHorse-1.1 and Seedance-2.0. Open-source systems fall behind on exactly the reasoning dimensions: MiniMax-H3 leads that group but reaches only 1.63 on SCC, half the proprietary best. However, on Spatiotemporal Consistency they are not behind at all, with Wan2.2-5B attaining the highest automatic score of any model (2.79).

3. No model is uniformly strong across disciplines. "Gemini-Omni-Flash leads three of the four disciplines (3.33 on Engineering, 3.13 on Natural Science, and 2.95 on Humanities & Social Sciences) but Sora-2 takes Healthcare (2.97)."

Errors are classified into three categories:

1. Poor Adherence to Instructions: models misinterpret core concepts and miss the fine-grained details specified in the prompt.

2. Inaccurate Simulation of Scientific Principles: models prioritize visual aesthetics over physical realism, producing factually incorrect dynamics.

3. Deficiencies in Temporal Coherence and Visual Quality: videos suffer from temporal inconsistencies, such as objects changing illogically over time.

Rewriting prompts with Gemini-3-Flash to be more visually concrete produced ordered gains: Scientific and Causal Correctness (SCC) improves most (+23.3% and +51.7%), then Prompt Grounding (+12.4% and +26.7%), while Spatiotemporal Consistency gains least (+7.5% and +16.5%). The paper concludes that explicit wording helps where the prompt left the mechanism implicit, but it cannot supply the mechanistic fidelity the generator lacks.

The paper concludes that "Sci-VBench fills a key gap in evaluating text-to-video models as expert-domain 'world simulators' by providing 1,253 expert-authored prompts spanning 60 subjects across four disciplines, where success hinges on mechanistic reasoning and temporally coherent causal dynamics rather than surface-level realism. Benchmarking reveals a persistent gap between perceptual realism and scientific validity, together with a substantial proprietary/open-source performance gap on the reasoning-centric dimensions. The observed failure modes point to concrete opportunities to improve instruction adherence, mechanistic consistency, and spatiotemporal stability in expert-domain generation."

Improvements to AI Systems:

1. Mechanism-Aware Video Generation: Train video generation models with explicit physics/causal constraints (e.g., conservation laws, state transitions) rather than purely visual objectives. The system should internally simulate the target mechanism before rendering, ensuring dynamics match scientific principles even when prompts omit them.

2. Rubric-Conditioned Self-Evaluation: Integrate the benchmark’s per-example reference guides and 1–5 anchored rubrics into the generation loop as a feedback signal. The model can generate multiple candidate videos, score each against the rubric using its own MLLM judge, and select/refine the highest-scoring output—improving scientific correctness and prompt grounding without human intervention.

3. Domain-Aware Prompt Grounding: Enhance instruction-following by parsing prompts into structured entities, initial states, spatial relations, and interventions (as in Sci-VBench’s annotation pipeline). The system should verify each explicit condition is met in the generated video, flagging and correcting omissions (e.g., missing instruments, wrong entity identity) before final output.

4. Temporal Consistency Enforcement: Add a dedicated temporal coherence module that tracks object identities, positions, and states across frames, preventing illogical changes (e.g., sudden color shifts, disappearing/reappearing objects). This can be achieved via latent-space consistency losses or post-hoc temporal smoothing, targeting the observed failure mode of “objects changing illogically over time.”

5. Discipline-Specific Specialization: Train or fine-tune separate generation heads or adapters for each discipline (Natural Science, Healthcare, Engineering, Humanities) since no single model excels across all. The system should route prompts to the most capable sub-model based on detected subject, improving per-domain accuracy (e.g., Sora-2 for Healthcare, Gemini-Omni-Flash for Engineering).

6. Implicit Mechanism Inference: Improve the model’s ability to infer unstated mechanisms from minimal prompts (e.g., “drop a magnet near a coil” implies electromagnetic induction). This requires training on textbook-guided examples where the prompt omits the outcome, forcing the model to reason from setup to consequence—directly addressing the prompt rewriting finding that explicit wording helps but cannot replace mechanistic fidelity.

7. Multi-Run Consensus Scoring: For any generated video, run the rubric-conditioned MLLM judge three independent times and report the mean, as in Sci-VBench. This reduces evaluation variance and provides a more reliable quality signal for both training and deployment, enabling the system to better rank its own outputs.

What the Improved AI System Can Do:

• Generate scientifically valid videos of expert-level phenomena (e.g., enzyme kinetics, fluid dynamics, policy impacts) that pass human expert review on causal correctness, not just visual plausibility.

• Self-correct its outputs by comparing against per-example rubrics, iteratively refining until it meets anchored criteria for prompt grounding, scientific accuracy, and spatiotemporal consistency.

• Handle minimal prompts by inferring and simulating the underlying mechanism, producing videos that demonstrate the correct causal chain even when the prompt only specifies initial conditions.

• Maintain stable object identities and physical laws across long video sequences, eliminating temporal artifacts that break scientific interpretability.

• Route tasks to discipline-specialized sub-models, achieving near-expert performance in healthcare, engineering, natural science, and humanities without a single monolithic model.

• Provide reliable, reproducible self-assessments of its own video quality, matching expert human judgments more closely than any current automated evaluator, and flagging its own failures for retraining.

We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.