Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing
cs.AI
Submitted: 2026-08-25
Updated: 2026-08-25
Code: https://github.com/blackforest-labs/flux
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult.
Terminology
Abstract
Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult. Fine-grained benchmarks decompose prompts into questions, yet often return them to prompt scores and fixed categories, weakening attribution and ignoring complexity. Related requirements are also scored separately or as one total, obscuring basic versus compositional failure. We present QC-T2I-Bench, a question-centric framework that converts open prompts into attributed atomic questions and organizes their dependencies with Davidsonian Scene Graphs (DSGs). We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails and to prevent simple and complex prompts from receiving the same total weight. We then use the DSG structure to measure joint success within prompts and compare repeated entities across prompts, separating basic realization failures from failures under additional requirements. We evaluate multiple open-source T2I models on English and Chinese prompts. The resulting question-level evidence supports reliable ranking and fine-grained diagnosis: joint completion falls from 80.7% for components with two capabilities to 37.2% for those with seven or more. Finally, we reuse the same records for training-free routing; our cost-aware router matches ERNIE's 89.51-point estimate with 21.3% less GPU-s/MP.
Sources
- Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist
- HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
- HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
- Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models
- FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark
- X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
- EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- OctoT2I: A Self-Evolving Agentic Text-to-Image Router
- GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
- GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
- Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
- Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?
- Cost-Aware Routing for Efficient Text-To-Image Generation
- Evaluating Text-to-Visual Generation with Image-to-Text Generation
- ERNIE-Image Technical Report
- LongCat-Image Technical Report
- PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
- Image-POSER: Reflective RL for Multi-Expert Image Generation and Editing
- WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems