When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
cs.CV, cs.AI, cs.CY
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 25 pages including appendix. Code and project page: https://github.com/seochan99/JudgeActs ; data: https://huggingface.co/datasets/seochan99/JudgeActs
Code: https://github.com/seochan99/JudgeActs
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see.
Terminology
Abstract
Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see. Such judges are usually validated by score agreement with human ratings, not by the images they return. We audit VLM judges as decision-makers: on 300 culturally situated prompts, we compare the returned image with human ratings the judge never sees and with random choice from the same candidates, and repeat every decision with the candidates reordered. A 4B-parameter judge barely beats random and falls short of a CLIP similarity baseline. It picks the first image shown in 49% of calls (chance: 28%), and reordering changes its choice on 60% of prompts. For this judge, agreement across orders is informative: decisions that survive reordering are much better than random, whereas agreement with a weaker second judge keeps the wrong ones. An 8B judge shows almost no position bias and outperforms CLIP, yet for it the same filter mostly discards good decisions. Agreement helps only when it targets the judge's failure mode, so filters must be re-audited whenever the judge changes. The 4B judge's slight rise in stereotype ratings is no longer detectable after aggregating across orders or with the larger judge.
Sources
- Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation
- JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
- Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling
- Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation
- SmolVLM: Redefining small and efficient multimodal models
- HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning
- Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models
- Toward Trustworthy Portrait Editing: Evaluation of Demographic Misrepresentation in I2I Models
- Prompt Revision as a Source of Cultural Bias in Text-to-Image Systems
- BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
- Multimodal Reward Hacking in Reinforcement Learning
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models