Multimodal Language Models as Text-to-Image Model Evaluators
cs.CV, cs.AI, cs.CL
Submitted: 2025-05-01
Updated: 2026-09-01
Code: https://github.com/LAION-AI/aesthetic-predictor
License: http://creativecommons.org/licenses/by/4.0/
The gist: The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative
Terminology
Abstract
The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE's image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28-105x fewer than the benchmarks' own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE's T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model's own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks.
Sources
- Consistency-diversity-realism Pareto fronts of conditional image generative models
- InstructPix2Pix: Learning to Follow Image Editing Instructions
- Muse: Text-To-Image Generation via Masked Generative Transformers
- All You May Need for VQA are Image Captions
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
- Visual Programming for Text-to-Image Generation and Evaluation
- Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
- A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Visual Programming: Compositional visual reasoning without training
- Improving Model Evaluation using SMART Filtering of Benchmark Datasets
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Imagen Video: High Definition Video Generation with Diffusion Models
- GenAI Arena: An Open Evaluation Platform for Generative Models
- Text encoders bottleneck compositionality in contrastive vision-language models
- Scaling up GANs for Text-to-Image Synthesis
- Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
- ImagenHub: Standardizing the evaluation of conditional image generation models
- VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models