Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Where, What, Why, and Importance".
Jane: Here's the gist of the paper. Text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures that require instance-level feedback to diagnose where defects occur, what type they are, why they are defective, and their importance While recent dense-feedback methods move beyond scalar supervision, their heatmapcentric representations still formulate diagnosis as pixel-field regression, making it difficult to localize variable-cardinality defects and bind semantic reasons to individual failures.
Tom: To address this representation bottleneck, the authors propose Structured Defect Grounding (SDG), which casts T2I diagnosis as structured set prediction by modeling each defect as a (location, type, reason, importance) tuple.
Jane: Pruning redundant reasoning tokens The core of the proposed method is Structured Defect Grounding (SDG), which models each defect as a structured (location, type, reason, importance) tuple and formulates diagnosis as predicting a variable-length set of such tuples.
Tom: This formulation unifies two defect types, artifacts and misalignments, within a single instance space.
Title and authors: Tom: So, we're talking about this paper today, "Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback." The title itself tells you a lot about what they're trying to fix.
Jane: It sounds complicated because it’s all about giving the AI feedback that’s specific enough to actually help an editor fix a picture. It moves away from just saying "this area is bad" and tries to give us the whole story.
Lu: They are tackling a problem where text-to-image models still make these really localized, subtle failures, and diagnosing those needs instance level feedback that tells you where the defect is, what it is, why it's defective, and how important that defect is to the whole image quality.
Meng: I mean on our end we see this stuff constantly. We have models that can’t tell the difference between a slightly misaligned hand and just some natural variation in lighting or texture. Giving us feedback like "the arm is too long" versus just a heatmap makes it hard to get specific, actionable fixes.
Lalam: I'm looking at the abstract, and they are proposing Structured Defect Grounding, which basically casts the whole diagnosis thing as a structured set prediction problem. They model each defect as a tuple with four parts: location, type, reason, and importance.
Tom: So it’s trying to unify two different kinds of problems—things that look like artifacts and things that are just misalignments—into one single structure so the AI can handle them uniformly.
The paper's summary: Jane: That structured set prediction idea is key, because traditional heatmap methods still treat diagnosis like pixel field regression, which makes it really hard to pinpoint defects with different kinds of errors and link those errors to specific reasons.
Lu: Exactly. Heatmaps give you a size for a region, but they don't naturally handle variable cardinality defects—meaning one object might have three bad parts while another has one bad part—and they struggle to tie semantic reasons to individual failures.
Tom: The paper is proposing something that lets the AI output a set of these structured tuples, which means we get location, type, reason, and importance for every single issue found. That’s what makes it instance level feedback possible.
Meng: So if we can give the AI that structured output—say "this hand is an artifact because it's fused with the wrist," and that defect is highly important"—that’s much more useful than a generic score on a whole area.
Lalam: They built this idea by creating a dataset called SDG-30K, which has box-grounded annotations across four different text-to-image generators, and they define an evaluation system called SDG-Eval for checking this structured defect set <ref:2606.06113#pg3>.
Jane: It’s not just theory though; they actually constructed a dataset that includes human annotation from eleven annotators and two rounds of quality control using tools like Gemini three Pro enhancement <ref:2606.06113#pg1>. That makes the training signal much more robust than just relying on one set of labels.
The paper's improvements: Tom: They don't stop there with just proposing the structure; they actually trained the detector in two stages to make sure it outputs that format correctly. First, Supervised Fine-Tuning, which teaches it to follow the instruction and emit that required structured format.
Lu: And then they use Group Relative Policy Optimization, or GRPO, which is a technique to further improve localization and description consistency while using a gate to make sure the output actually follows the structure they want.
Meng: I’m interested in how they handle the training signal there. They use coordinate jitter during the first stage of fine-tuning to make it less sensitive to exact coordinate values, which is smart because you don't want your model getting stuck on tiny pixel errors.
Jane: And then in the second stage, GRPO optimizes the structured output using a composite reward that looks at spatial accuracy, description consistency, and importance estimation all at once through a format validity check.
Lalam: The results show that this two-stage training approach—SFT followed by GRPO—leads to stronger performance on structured defect grounding compared to just using scalar supervision or heatmap methods.
Tom: And the downstream application is where it gets really interesting; they have a diagnosis-to-alignment pipeline where BoxFlow, which is another model, takes those structured detections and converts them into box derived spatial rewards for diffusion model alignment.
Conclusion: Jane: So to wrap up on "Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback," the big idea is moving from guessing where things are wrong to precisely describing them with location, type, reason, and importance.
Lu: It sets up a unified interface for diagnosing these failures across artifacts and misalignments in one instance space. This structured feedback isn't just better for diagnosis; it enables more faithful image refinement because you get localized guidance on what needs correcting.
Tom: The experiments show that this method grounds prompt conditioned misalignments that other methods, like ImageDoctor, miss, while avoiding those spurious activations on clean regions that artifact heads sometimes create. They even show the GRPO method achieving a BoxF1 score of zero point two six three over zero shot VLMs.
Meng: From a deployment standpoint, what I see is that this structured feedback can make T2I systems way more interpretable, which is crucial for auditing and making targeted corrections instead of just tweaking parameters blindly across the whole model.
Lalam: And they show that SDG-guided rewards actually improve T2I alignment and support more faithful image refinement, even achieving a relative change of plus two point four percent in one comparison against other methods.
Jane: It seems like this work establishes SDG as a unified, instance level interface for diagnosing, evaluating, and enhancing modern generative models by giving us the specific details we need to make targeted edits.
Tsinghua University · Kolors Team Kuaishou Technology · University of British Columbia Department of Computer Science 4 Vector Institute South China Normal University
cs.CV
Submitted: 2026-06-04
Updated: 2026-11-05
Comments: Accepted at NeurIPS 2026
Code: https://github.com/nianbai006/SDG
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 94/100
The gist: Text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures that require instance-level feedback to diagnose where defects occur, what type they are…
Key concepts
- Structured Defect Grounding (SDG)
- A method that models every image defect not just as a point on a map, but as a structured tuple containing four pieces of information: where it is (location), what kind of error it is (type), why it happened (reason), and how important it is to the final image quality. This turns diagnosis into predicting a set of these detailed descriptions.
- Variable-Cardinality Set Prediction
- The core task SDG performs is not just classifying one defect, but predicting a variable number of defects simultaneously. Instead of outputting a single score, the model outputs a list or set of structured tuples describing all relevant issues in the image, allowing for instance-level feedback.
- BoxFlow-GRPO
- A training technique used to optimize the structured defect predictions. It uses rewards based on spatial accuracy and description consistency to refine localization and importance estimation. This process ensures the model learns to provide high-quality, spatially relevant structural feedback for image alignment.
Terminology
Summary
Text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures that require instance-level feedback to diagnose where defects occur, what type they are, why they are defective, and their importance While recent dense-feedback methods move beyond scalar supervision, their heatmapcentric representations still formulate diagnosis as pixel-field regression, making it difficult to localize variable-cardinality defects and bind semantic reasons to individual failures. p. 0 To address this representation bottleneck, the authors propose Structured Defect Grounding (SDG), which casts T2I diagnosis as structured set prediction by modeling each defect as a (location, type, reason, importance) tuple. p. 0
Pruning redundant reasoning tokens
The core of the proposed method is Structured Defect Grounding (SDG), which models each defect as a structured (location, type, reason, importance) tuple and formulates diagnosis as predicting a variable-length set of such tuples. This formulation unifies two defect types, artifacts and misalignments, within a single instance space. p. 1 The authors construct SDG-30K, a 30K-image dataset with box-grounded annotations across four modern T2I generators and define SDG-Eval for structured defect-set evaluation. p. 1
How the training signal is built
The SDG detector is trained in two stages: Supervised Fine-Tuning (SFT) teaches the model to follow the defect-grounding instruction and emit the required structured format, while Group Relative Policy Optimization (GRPO) further improves localization, description consistency, and importance estimation under a format-validity gate. The SFT stage uses standard teacher-forced negative log-likelihood over the target tokens, incorporating coordinate jitter to reduce sensitivity to exact coordinate values. p. 4 The GRPO stage optimizes the structured output using rewards for spatial accuracy, description consistency, and importance estimation through a composite reward gated by a format-validity check. p. 4
Where the gains come from
The framework includes a diagnosis-to-alignment pipeline where the VLM detector predicts structured defect sets and BoxFlow-GRPO converts them into box-derived, importance-weighted spatial rewards for diffusion model alignment. This approach preserves spatial variation in both the advantage and the likelihood ratio compared to scalar-advantage implementations. p. 5 Extensive experiments show that SDG outperforms leading proprietary VLMs on structured defect grounding, while SDG-guided rewards improve T2I alignment and support localized image refinement. p. 0
The authors demonstrate that the SDG detector outperforms leading proprietary VLMs on structured defect grounding, while GRPO achieves the strongest BoxF1@0.5 (0.263/0.387) and importance accuracy (0. p. 6887/0.893) against zero-shot VLM baselines. p. 6 Furthermore, SDG-guided rewards improve T2I alignment and support more faithful, actionable image refinement. p. 1 The BoxFlow-GRPO method achieves the best average relative change (+2. p. 84%) and the highest P(real) (0. p. 8228, above Base) while maintaining competitive preference and quality metrics. p. 8
SDG consistently grounds prompt-conditioned misalignments that ImageDoctor’s misalignment head misses, while avoiding the spurious face/hand activations that ImageDoctor’s artifact head produces on clean regions. SDG provides localized, instance-level guidance that helps the editor make targeted corrections, such as removing an extra lion cub when the prompt asks for a single cub. p. 1 This structured defect feedback can make T2I systems more interpretable by exposing localized failure modes, supporting dataset auditing, and enabling targeted image refinement. p. 1 The work establishes SDG as a unified, instance-level interface for diagnosing, evaluating, and enhancing modern generative models. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Structured Defect Grounding (SDG) allows diagnosis as
variable-cardinality set prediction over structured (location, type, reason, importance) tuples,
enabling instance-level feedback that answerswhere a defect occurs, what type it is, why the region is defective, and its importance to overall image quality.
-
The SDG detector can be trained in two stages: Supervised Fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO), which improves localization and description consistency under a format validity gate.
-
A diagnosis-to-alignment framework using BoxFlow-GRPO converts SDG predictions into
box-derived, importance-weighted spatial rewards for diffusion model alignment,
allowing for spatially targeted refinement beyond scalar preference optimization. -
The system can perform defect grounding on unseen data by generalizing to a structured format, as shown by SDG achieving
substantially higher misalignment F1 (0.655 vs. 0.250/0.007)
compared to heatmap-based methods on the RichHF-18K test set. -
Defect-guided image refinement can be used to make targeted corrections, such as
removing an extra lion cub when the prompt asks for a single cub,
by providing localized, instance-level guidance via GPT-Image-1.5 editing based on structured feedback.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Training Diffusion Models with Reinforcement Learning
- Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
- HunyuanImage 3.0 Technical Report
- ImageDoctor: Diagnosing Text-to-Image Generation via Grounded Image Reasoning
- Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detection
- Flow-GRPO: Training Flow Matching Models via Online RL
- LongCat-Image Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Towards Pixel-Level VLM Perception via Simple Points Prediction
- Gemini: A Family of Highly Capable Multimodal Models
- MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation
- Detecting Human Artifacts from Text-to-Image Models
- Unified Reward Model for Multimodal Understanding and Generation
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs