Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist".

Jane: Here's the gist of the paper. The gist The strongest closed-source system we evaluate reaches 0.855 with a 33 pp performance gap across 11 systems, demonstrating substantial discriminative power Faithfulness—how precisely a generated image aligns with its prompt—is increasingly central to the real-world utility of text-to-image (T2I) models Existing faithfulness benchmarks, however, rely on simple atomic instructions, on which top-tier systems already achieve near-perfect scores As T2I models enter creative workflows, users issue multi-faceted requests combining intricate spatial relationships, stylistic constraints, and complex text rendering In this setting, a single binary VLM-judge score no longer captures which specific constraints the model fails to satisfy The Arena-T2I Hard Benchmark Arena-T2I Hard is a 310-prompt stress benchmark drawn from real arena T2I logs, with approximately 30 decomposed yes/no constraints per prompt spanning six categories, including text rendering This benchmark draws prompts from real T2I-arena user votes and selects for compositional difficulty Prompts average ∼430 words and ∼30 decomposed yes/no questions The strongest closed-source system we evaluate reaches 0.855 with a 33 pp performance gap across 11 systems, demonstrating substantial discriminative power High public-arena rankings fail to predict faithfulness, confirming that holistic Bradley-Terry (BT) preference scores prioritize aesthetics over fine-grained prompt adherence Dependency-Aware Checklist Reward We propose a dependency-aware checklist reward that decomposes each prompt into a DAG of yes/no questions and zeroes descendants of failed parents, turning faithfulness into a per-constraint training signal The question graph G maps p to a directed acyclic graph G(p) = (Q(p), E(p)) and only the faithfulness subset Qf (p) = Q(p) where τ (q) = faithfulness is used for the reward If a parent question fails, its descendants are scored NO without a further VLM call, preventing inflated scores that flat checklists produce when an attribute question fires on the wrong object The faithfulness reward is the yes-ratio over Qf (p), sf (x, p) = 1/Qf (p) Σ q∈Qf (p) yq ∈

0, 1: Post-Training with Combined Rewards We propose conducting GRPO with an ensemble of BT rewards and our structured faithfulness score, balanced via Group reward-Decoupled Normalization Policy Optimization (GDPO) This approach resolves the fundamental tension where optimizing for faithfulness alone improves adherence but degrades BT-based aesthetic rewards We demonstrate that combining prompt-specific dependency-aware checklist reward and an aesthetic BT-based reward achieves the best overall performance Specifically, combining Faith + Pick with GDPO improves both evaluation rewards simultaneously, increasing faithfulness by 10.5% and PickScore by 3.5% on FLUX.1-dev We conclude that combining prompt-specific dependency-aware checklist reward and an aesthetic BT-based reward achieves the best overall performance Prompt faithfulness is best served by prompt-specific, dependency-aware checklist rewards, whereas aesthetics is better captured by BT preference rewards trained on human comparisons This combination improves both faithfulness and aesthetics while reducing the trade-off between them How it works 1.

Tom: Arena-T2I Hard is constructed from real arena T2I votes, averaging ∼430 words per prompt and containing ∼30 decomposed yes/no questions 2.

Jane: The dependency-aware checklist decomposes prompts into a DAG of yes/no questions, where parent failures zero out descendants to prevent inflated scores 3.

Tom: We use Group reward-Decoupled Normalization Policy Optimization (GDPO) to combine the faithfulness reward with an aesthetic BT reward, stabilizing training and avoiding reward collapse 4.

Paper summary: Tom: We’re moving on to the Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist paper. To recap, the core thesis is that existing models often fail at complex, multi-faceted prompts because simple checks miss specific failures.

Jane: They introduce Arena-T2I Hard as a three hundred ten-prompt stress benchmark from real arena T2I logs, which has about thirty decomposed yes/no constraints per prompt across six categories <ref:2606.31711#pg1,30 decomposed yes/no constraints per prompt>. This moves beyond simple atomic instructions to test how models handle intricate spatial relationships and text rendering.

Lu: They developed the dependency-aware checklist reward, which uses a DAG of yes/no questions where failing parents zero out their descendants to give a more reliable training signal than flat rubrics.

Meng: The results show that combining this structured faithfulness score with an aesthetic Bradley-Terry reward, balanced by Group Reward-Decoupled Normalization Policy Optimization, leads to better performance on models like SD3 point 5-Medium and FLUX <ref:2606.31711#pg1>.one-dev <ref:2606.31711#pg1>.

Tom: The strongest closed-source system they evaluated reached a score of zero point eight five five with a thirty-three percentage point gap across eleven systems, proving the method has substantial discriminative power in identifying which systems are actually better at following instructions precisely <ref:2606.31711#pg1,The strongest closed-source system>.

Jane: It also confirms their earlier finding that high public arena rankings don't predict faithfulness, meaning the approach they propose prioritizes fine-grained prompt adherence over just looking pretty.

Lalam: So, for us listeners, it means we need to stop assuming a model is good if it looks aesthetically pleasing; we need to check if it actually did what we asked in that long, complicated request.

Tom: Exactly. The paper identifies a reward-task mismatch where standard RLHF often optimizes for aesthetics at the expense of faithfulness, and this new combined approach fixes that by giving faithfulness a structured path to improve.

Lu: It’s about turning prompt adherence into something trainable because they showed how to decompose the prompt into a graph structure that prevents inflated scores from simple checklists.

Meng: I still worry about deploying this at scale, but if the reward signal is this much more precise for faithfulness, maybe we can manage the computational overhead because we aren't relying on massive VLM calls for every single constraint check.

Jane: That’s a fair concern about running costs, but they show that combining these heterogeneous rewards doesn't degrade the overall performance compared to using just one type of reward at a time.

Conclusion: Tom: So wrapping up this discussion on Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist, the main implication is that we need a better way to measure prompt adherence than what we currently have.

Jane: The paper shows that prompt faithfulness is best served by these prompt-specific, dependency-aware checklist rewards because they give us a structured signal tied directly to the actual constraints in the text.

Lu: Conversely, aesthetics are still better captured by those Bradley-Terry preference rewards that rely on human comparisons for their training data.

Meng: So the conclusion is that combining these two—the specific faithfulness reward and the aesthetic BT reward, balanced by GDPO—improves both aspects simultaneously and reduces the trade-off between them.

Tom: It’s a recipe for post-training work that lets us improve both adherence to instructions and visual appeal without one hurting the other. The Arena-T2I Hard benchmark is a powerful tool for figuring out exactly what these models are missing when they try to follow really complex, real-world requests.

Jane: This means we're moving toward training systems that aren't just good at generating pictures, but are good at understanding the detailed requirements of a prompt. That’s where the real utility is right now for T2I models.