Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

arXiv:2610.01670 · cs.CV, cs.LG · Submitted 2026-10-01 · Read on arXiv

cs.CV, cs.LG

Submitted: 2026-10-01

Updated: 2026-10-01

Comments: 30 pages, 9 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 40× in this paper · explained in 2 episodes
• Robustness — 17× in this paper · explained in 5 episodes
• Provenance — 12× in this paper · explained in 3 episodes
• Aggregation — 7× in this paper · explained in 1 episode
• Benchmarking — 6× in this paper · explained in 3 episodes
• Consistency — 6× in this paper · explained in 3 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Calibration — 4× in this paper · explained in 7 episodes
• LLM-as-a-Judge — 4× in this paper · explained in 3 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Verification — 4× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Adversarial Attacks — 1× in this paper · explained in 1 episode
• Agentic Framework — 1× in this paper · explained in 6 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Fidelity — 1× in this paper · explained in 2 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Perceptual Quality — 1× in this paper
• Scaling Laws — 1× in this paper · explained in 4 episodes
• Semantic Consistency — 1× in this paper
• Temporal Reasoning — 1× in this paper · explained in 2 episodes

Sources

• Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling
• ReasonEdit: Towards Interpretable Image Editing Evaluation via Reinforcement Learning
• A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
• Emerging Properties in Unified Multimodal Pretraining
• Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences
• GEditBench v2: A Human-Aligned Benchmark for General Image Editing
• MLLM-as-a-Judge Exhibits Model Preference Bias
• Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
• Step1X-Edit: A Practical Framework for General Image Editing
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
• Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics
• Verbosity Bias in Preference Labeling by Large Language Models
• Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies
• Fine-Grained Human Pose Editing Assessment via Layer-Selective MLLMs
• Unified Reward Model for Multimodal Understanding and Generation
• On the Adversarial Robustness of Multimodal LLM Judges
• MIEScore: Human-Aligned Evaluation for Multi-Source Image Editing
• SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models