Faithful Where You Look: Anchor-Based Evaluation Hides Text Fabrication in Image-Conditioned Video Generation
cs.CV
Submitted: 2026-10-18
Updated: 2026-10-18
Comments: 18 pages, 2 figures; includes supplementary material (S1-S11). Code and data: https://github.com/geoywang/refanchor-bench
Code: https://github.com/geoywang/refanchor-bench
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Image-conditioned video models are used to show real places, so whether their output stays faithful to the place they were shown is a practical question.
Terminology
Abstract
Image-conditioned video models are used to show real places, so whether their output stays faithful to the place they were shown is a practical question. The text benchmarks we found that check fidelity against a reference do so at an anchor - an annotated region, typically the sign the task is about. We show that this choice hides most of the failure. We conditioned four models from three vendors on 48 street-level references of Tokyo storefronts and measured text fidelity at the annotated sign and in the periphery, the other text the reference covers. At the anchor, all four models are close to a no-model control (anchor text fidelity 0.90-0.94 against 0.996) and are not reliably separated from one another. In the periphery, 36-54% of generated text regions have no counterpart in the reference, against 1.6% for the control, although every prompt forbade new readable text. Two blinded raters agree on which regions are fabricated (kappa = 0.83-0.96), the result held in preregistered rounds on two vendors whose videos the measure had never seen (42% and 56%), and VBench-I2V's reference-based consistency scores show no evidence of tracking it in our sample. Most fabricated text is legible-looking text where the reference shows none or cannot be read - a form that string-matching metrics discard, which is why our own first measure missed it. A preregistered Latin-script replication in London missed its bar in a first round and met it in an independent second round. We release code, annotations, per-video scores, ratings and every preregistration. Code and data: https://github.com/geoywang/refanchor-bench
Sources
- Emerging Properties in Self-Supervised Vision Transformers
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation
- TextDiffuser: Diffusion Models as Text Painters
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing
- WorldScore: A Unified Evaluation Benchmark for World Generation
- GeoT2V-Bench: Benchmarking 3D Consistency in Text-to-Video Models via 3D Reconstruction
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
- X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
- GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structure
- TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution
- VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
- VBench: Comprehensive Benchmark Suite for Video Generative Models
- VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models
- Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinement
- Video-T1: Test-Time Scaling for Video Generation
- Text-Animator: Controllable Visual Text Video Generation
- Are Video Generation Models Geographically Fair? An Attraction-Centric Evaluation of Global Visual Knowledge
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
- Video Text Preservation with Synthetic Text-Rich Videos
- Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs