VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation
cs.CV, cs.AI
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: Code: https://github.com/shim0114/VIF-Bench , Benchmark: https://huggingface.co/datasets/shim0114/VIF-Bench
Code: https://github.com/shim0114/VIF-Bench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such
Terminology
Abstract
Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues. However, existing benchmarks do not evaluate the joint setting in which multiple references must be composed under multiple and heterogeneous visual-instruction images. To address this gap, we introduce VIF-Bench, a benchmark of 1,241 tasks designed to assess the edge of model capabilities in this joint setting by covering: (i) multi-reference generation (up to 7) under multiple heterogeneous visual instructions (up to 6), (ii) cases where reference images can potentially compete with visual instructions (e.g., a strongly posed subject vs. a target pose), and (iii) controlled comparison of visual instructions with text descriptions at different levels of specificity. Using these capabilities, we uncover three findings: (1) models face an adherence-artifact trade-off: once models reach stronger visual instruction adherence, stronger adherence tends to coincide with more instruction artifacts in generated images, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute (e.g., a neon-lit subject under a light-direction instruction), most consistently for light and wind, and (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text; when using text, a moderate level of detail works better than an exhaustive description. VIF-Bench is released as an open benchmark to establish a basis for fair comparison in controllable multi-reference image generation.
Sources
- Qwen3-VL Technical Report
- EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods
- Emerging Properties in Self-Supervised Vision Transformers
- MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation
- Improving Diffusion Models for Authentic Virtual Try-on in the Wild
- Emerging Properties in Unified Multimodal Pretraining
- Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback
- Gemini: A Family of Highly Capable Multimodal Models
- SpotEdit: Evaluating Visually-Guided Image Editing Methods
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- GEMS: Agent-Native Multimodal Generation with Memory and Skills
- Scaling Multi-Reference Image Generation with Dynamic Reward Optimization
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- GLIGEN: Open-Set Grounded Text-to-Image Generation
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
- Boost Your Human Image Generation Model via Direct Preference Optimization
- Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation
- GPT-4 Technical Report
- Learning Transferable Visual Models From Natural Language Supervision
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models