Visual Distortion Detection in UGC Images Using Large Multimodal Models

The paper proposes VIGIL, an LMM-based framework for precise visual distortion detection in UGC images. The authors identify two major challenges in existing approaches: The limited detection precision caused by textual SFT and the FG-BG indistinguishability in S2A generalization tasks are two major challenges for existing models. To address these, they construct a large-scale training dataset and introduce a novel model architecture.

The paper constructs the VIGIL-140K training set: "From the source pool of over 1000K images, we filter those suitable for distortion injection, perform region-level distortion combination, and finally obtain a large-scale training set with over 200K distortion area labels." The source pool includes COCO-2017-train (110K samples), a subset of CLIP pretraining data (approximately 300K images), short video screenshots from streaming platforms (600K samples), and 10K high-quality aesthetic images from Unsplash. Quality filtering uses 6 no-reference IQA models (TOPIQ-NR, TReS, LIQE, ARNIQA, QualiCLIP, and Q-Align) pretrained on KonIQ-10K, selecting data with scores above 85. The distortion combination process covers 8 major categories (with 31 sub-types): blur, noise, compression, overexposure, contrast strengthen, underexposure, contrast weaken, saturate strengthen, saturate weaken, and oversharpen. Overexposure is merged with contrast strengthen, and underexposure is merged with contrast weaken.

The model architecture leverages the LMM as a powerful encoder: Following DETR, we first encode the image to obtain a sequence of image tokens, and then append N learnable area query tokens. The authors select M layers from the LLM part, each serving as an independent detector with two linear heads for classification (9 classes, including the non-distortion class) and box regression. This multi-detector design is motivated by two considerations: visual distortion detection largely relies on low-level cues, which makes representations from the early stages of the LLM decoder potentially more informative and leveraging multiple detectors for synchronous detection can be viewed as integrating complementary perceptual perspectives. Hungarian Matching is applied separately for each detector.

For the S2A generalization problem, the authors propose a simple technique: "we apply an S2A generalization trick, avoiding discarding predictions assigned to the non-distorted class. Instead, we take the distortion category with the second-highest classification probability and its associated bbox as a valid predicted candidate. This strategy enlarges the candidate set and reduces errors caused by ambiguous FG-BG separation." Post-processing involves class-wise NMS with IoU threshold TNMS and probability filtering with threshold TProb.

The model uses InternVL-3-8B-Instruct (LLM: Qwen2.5-7B) as the base model, with N=3 and M=5 (using the 10th, 15th, 20th, 25th, and final (28th) layers of the LLM). Post-processing thresholds are set to TNMS=0.3, TProb=0.1 for authentic distortion detection, and TNMS=0.7, TProb=0.8 for synthetic distortion localization.

For evaluation, the authors construct a synthetic test set of 4,600 images and an authentic test set of 815 human-labeled UGC images (with 100 reserved for validation). The authentic test set uses an iterative annotation-correction procedure with multiple annotators.

Results show VIGIL-8B achieves state-of-the-art performance. On the synthetic distortion detection task, it achieves mAP of 0.736 and AP50 of 0.879, outperforming all baselines including detection models like YOLO-V11 (mAP 0.708, AP50 0.878) and Grounding-DINO-R50 (mAP 0.687, AP50 0.851). On the authentic distortion detection task, VIGIL-8B achieves mAP of 0.482 and AP50 of 0.502, significantly outperforming the best baseline Grounding-DINO-R50 (mAP 0.362, AP50 0.362).

Ablation studies show that the multi-detector setup significantly improves S2A task accuracy compared to using only the final layer, converges faster during training, and performs better than simply increasing the number of query tokens. The S2A generalization technique enhances the model's generalization ability in real-world scenarios (mAP improves from 0.407 to 0.482 on authentic tasks). The text-generation-based SFT approach performs significantly worse than the proposed detection-based approach (mAP 0.457 vs 0.736 on synthetic, 0.231 vs 0.482 on authentic).

The paper also demonstrates that the model can predict more distortion regions than seen in training (average 4.22 valid predictions vs 1.78 for text-based SFT), and that the training scheme remains compatible with conventional text-generation-based LMM supervised fine-tuning, retaining both detection capability and text-generation ability.

Based on the paper, here are the specific improvements I can make to AI systems:

1. Multi-Layer Perceptual Detection Architecture

• I can implement a detection system that uses multiple intermediate layers of a large language model's decoder as independent detectors, rather than relying only on the final layer. This captures both low-level visual cues (early layers) and high-level semantic understanding (later layers), improving detection of subtle distortions like blur or noise that require fine-grained perceptual analysis.

• The improved system can detect multiple distortion types simultaneously with higher precision, as each detector layer contributes complementary perspectives, reducing missed detections.

2. Second-Highest Probability Candidate Selection for Ambiguous Regions

• I can adopt the S2A generalization trick: when the model is uncertain between a distortion class and a non-distortion class, I take the second-highest probability class and its bounding box as a valid prediction instead of discarding it. This enlarges the candidate set and reduces false negatives in real-world scenarios where foreground-background separation is ambiguous.

• The improved system can better handle UGC images with complex backgrounds, where distortions may overlap with objects, leading to more robust detection in authentic, uncurated content.

3. Region-Level Distortion Combination Training

• I can train on a dataset that combines multiple distortion types at the region level (e.g., blur on one area, noise on another) rather than whole-image distortions. This teaches the model to localize and classify multiple independent distortions within a single image.

• The improved system can output multiple bounding boxes with per-region distortion labels, enabling fine-grained quality assessment for applications like photo editing recommendations or automated content moderation.

4. Class-Wise NMS with Adaptive Thresholds

• I can implement class-wise non-maximum suppression with separate IoU and probability thresholds for synthetic vs. authentic distortions (e.g., stricter thresholds for synthetic, looser for authentic). This balances precision and recall depending on the distortion's nature.

• The improved system can adapt its detection sensitivity based on whether distortions are artificially injected (e.g., in generated content) or naturally occurring (e.g., in user uploads), improving performance across different deployment scenarios.

5. Hybrid Detection-and-Generation Capability

• I can fine-tune the model to retain both detection (via the multi-detector heads) and text-generation abilities, as shown in the paper. This allows the system to not only localize distortions but also generate natural language explanations or recommendations (e.g., The left region has motion blur; suggest increasing shutter speed).

• The improved system can serve as a multimodal assistant for photographers, content reviewers, or automated quality control, providing both structured outputs (boxes/labels) and human-readable feedback.

6. Efficient Convergence via Multi-Detector Supervision

• I can use the multi-detector setup to accelerate training convergence, as each detector provides gradient signals from different abstraction levels. This reduces the number of training epochs needed, saving computational resources.

• The improved system can be trained faster on new distortion types or domains, enabling rapid adaptation to emerging UGC trends (e.g., new compression artifacts from new codecs).

The localized depiction of perceptual quality has long been a crucial, yet underexplored, challenge in image quality assessment (IQA). Existing approaches based on large multimodal models (LMMs) predominantly rely on text-driven supervised fine-tuning (SFT). However, this training paradigm exhibits notable limitations in detection accuracy. Moreover, synthetically distorted images, which are often used as the primary training data source, show a significant generalization gap when deployed in real-world scenarios; thus, the synthetic-to-authentic (S2A) problem represents a critical challenge. Motivated by these issues, we propose VIGIL, which leverages the LMM architecture for precise visual distortion detection. From a candidate pool of over 1000K samples, we construct the VIGIL-140K training set, which consists of over 140K distorted images. These images are obtained through rigorous quality filtering and carefully crafted distortion injection, covering 8 major synthetic distortion categories. Our model leverages different layers of the large language model (LLM) decoder, treating them as multiple detectors that perform synchronous distortion detection using multi-level features. Additionally, we retain distortion cues from predictions assigned to the non-distortion class, which helps mitigate the ambiguous foreground-background (FG-BG) separation commonly encountered in the S2A problem. After post-processing, our model consistently outperforms strong baselines on both in-domain synthetic distortion detection and S2A tasks.