Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models

The paper introduces the Evaluation Agent framework, a human-like, dynamic, and promptable approach for evaluating visual generative models. Traditional evaluation methods, such as VBench and T2I-CompBench, are computationally expensive, requiring hundreds or thousands of samples, and rely on rigid, predefined pipelines that produce numerical scores without explanation. In contrast, the Evaluation Agent mimics human evaluators by quickly forming impressions from a few samples and dynamically adjusting its evaluation strategy.

The framework operates in two stages: a proposal stage and an execution stage. The proposal stage includes a Plan Agent, which decomposes a user's natural-language request into sub-aspects, and a PromptGen Agent, which generates targeted prompts for each sub-aspect. The execution stage samples images or videos from the evaluated model and invokes suitable evaluation tools, such as VBench metrics for text-to-video or T2I-CompBench tools for text-to-image, or a VLM-based VQA tool for open-ended queries. The agent iteratively loops between these stages, updating its plan based on observed evidence until sufficient information is gathered, then produces a detailed, interpretable summary.

The framework offers four key advantages: 1) efficiency, reducing evaluation time to about 10% of traditional methods; 2) promptable evaluation tailored to diverse user needs; 3) explainability beyond single numerical scores; and 4) scalability across various models and tools. Experiments on established T2I/T2V benchmarks show the Evaluation Agent achieves comparable accuracy to full benchmark pipelines while significantly reducing time and sample counts. For example, on VBench, the agent reduces evaluation time from thousands of minutes to 6–24 minutes per query, and on T2I-CompBench, from hundreds of minutes to 5–8 minutes per query.

To reduce dependence on proprietary API-based agents, the authors introduce Open Evaluation Agent (Open-EA). They construct EA-CoT-10K, a dataset of 10,042 history-conditioned step-level instruction-tuning records derived from multi-round evaluation rollouts, and train EA-3B from Qwen2.5-3B-Instruct as a compact local planning backbone. EA-3B preserves the structured reasoning, tool invocation, observation analysis, and summary protocol of the API-based agent. Evaluations on four in-domain and three out-of-domain T2V generator families show partial cross-family transfer, with micro-averaged Exact/Within-1 tier accuracy decreasing from 41.3%/77.3% in-domain to 31.1%/64.9% out-of-domain. Performance varies sharply by evaluation signal, with Overall Consistency and Temporal Style achieving 97.1% and 90.0% Within-1 accuracy, while Object Class, Human Action, and Spatial Relationship reach only 35.7%, 42.9%, and 47.1%, respectively.

The paper also demonstrates the framework's flexibility on a self-constructed open-ended user query dataset of 100 queries. Using a VQA-based evaluation tool, the agent handles queries like How precisely can the user specify object relationships? by progressively probing simple, complex, and non-standard scenarios, providing detailed, user-focused conclusions. The authors release code, data, and models at github.com/Vchitect/Evaluation-Agent.

Improvements to AI systems:

1. Adaptive, resource-efficient evaluation loop: Replace fixed, exhaustive benchmark pipelines with an iterative agent that samples only a few outputs, assesses sufficiency, and stops early—cutting evaluation time by 90% while maintaining accuracy. The system can dynamically switch between lightweight heuristics and heavy metrics (e.g., VBench, T2I-CompBench, VQA) based on observed evidence.

2. Promptable, user-driven evaluation: Enable natural-language queries (e.g., How well does this model handle spatial relationships?) to be decomposed into sub-aspects, with targeted prompts auto-generated per aspect. The system then produces interpretable, multi-dimensional summaries instead of a single opaque score, allowing users to probe specific failure modes or strengths.

3. Open, local planning backbone for evaluation agents: Train a compact 3B-parameter model (EA-3B) on step-level instruction-tuning data (EA-CoT-10K) to replicate the reasoning, tool invocation, and observation-analysis protocol of a proprietary API agent—removing dependence on closed systems. This enables private, low-cost, and customizable evaluation agents that can be fine-tuned for domain-specific needs.

4. Cross-family generalization with signal-aware confidence: The system can flag its own reliability per evaluation dimension (e.g., high confidence on Temporal Style, low on Object Class) and adjust its sampling strategy or tool selection accordingly—e.g., using more samples or specialized detectors for weak signals like spatial relationships.

5. Open-ended, progressive probing for qualitative assessment: For vague queries (e.g., How precise is object control?), the agent iteratively generates increasingly complex or non-standard test cases, building a nuanced, user-focused conclusion rather than a static metric—useful for model debugging, comparative analysis, and user acceptance testing.

What the improved AI system can do:

• Evaluate a new text-to-image or text-to-video model in under 10 minutes with a few dozen samples, producing a detailed, human-readable report (e.g., Strong temporal consistency, but fails on rare object classes and spatial reasoning) instead of a single number.

• Answer arbitrary evaluation questions posed by a developer or end-user, such as Does this model handle multiple objects in a scene? by autonomously designing and running targeted tests, then summarizing findings with evidence.

• Run entirely on-premises with a 3B-parameter model, enabling privacy-preserving evaluation of proprietary models without sending data to external APIs.

• Automatically adapt its evaluation depth based on the signal quality—spending more compute on weak areas (e.g., spatial relationships) and less on strong ones (e.g., style consistency), maximizing information per unit cost.

• Serve as a plug-and-play evaluation layer for model development pipelines, regression testing, or comparative model selection, with the ability to incorporate new tools or metrics as they emerge.