SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation

The gist: The authors introduce artifact prominence as an evaluative target, defined as the fraction of viewers who judge a highlighted region to contain a noticeable artifact, and construct SR-Prominence, a dataset suite containing 3,935 artifact masks annotated with prominence from crowdsourced evaluations.

How it works

The core idea is to move beyond binary defect presence in super-resolution evaluation toward perceptual impact by defining artifact prominence as the fraction of viewers who judge a highlighted region to contain a noticeable artifact The authors design a crowdsourced annotation protocol and construct SR-Prominence, a dataset suite containing 3,935 artifact masks from DeSRA, Open Images, Urban100, and a realistic no-ground-truth Urban100-HR setting This suite includes 593 existing masks from the DeSRA dataset [4] and new masks on Open Images, the standard Urban100 benchmark, and the realistic high-resolution Urban100-HR setting

The annotation protocol involves participants viewing pairs of images labeled “Original” and “Upscaled,” with the artifact region visually highlighted, and then choosing whether it contains distorted objects or textures Every image is assessed by 30 different participants, and prominence is computed as the proportion of votes indicating the artifact is present To ensure integrity, every group of 20 questions contains 4 random control questions, and responses from participants who make mistakes in two or more control questions within an assignment are discarded The authors selected an assessor count of 30 as a practical compromise, giving about ±20% variability at substantially lower cost

Dataset Construction and Components

SR-Prominence is a four-component dataset suite covering different evaluation scenarios It contains 3,935 masks from 15 widely-used SR methods and their variants, each with crowdsourced prominence annotations The components separate four evaluation questions: whether existing binary artifact datasets are viewer-noticeable, how SRs and detectors behave on diverse natural images, how they handle structured urban content, and how the results change in a realistic high-resolution setting

The suite includes Prominence-DeSRA masks from DeSRA [4], Prominence-OpenImages covering diverse natural-image SR artifacts [2605.14847], Prominence-Urban100 targeting structured content where localized artifacts are especially visible [2605.14847], and Prominence-Urban100-HR, a realistic high-resolution setting without HR ground truth [2605.14847>

For the automatic mask collection procedure, heatmaps from existing artifact detection and image-quality assessment methods are thresholded with DeSRA-calibrated thresholds to produce candidate artifact masks The authors applied this protocol to annotate a dataset suite of four complementary components, resulting in 3,935 masks from 15 widely-used SR methods

Evaluation and Benchmarking

The paper audits artifact detectors, image-quality metrics, and SR methods using the constructed suite The analysis shows that binary labels are not enough for SR artifact evaluation, as 48.2% of DeSRA’s in-lab binary artifacts are not noticed by a majority of viewers Full-reference metrics such as SSIM and DISTS provide strong localized prominence signals, whereas no-reference IQA methods and specialized artifact detectors often fail to generalize across datasets and reference settings

The authors propose a threshold-free objective score that measures how well an artifact detection method evaluates artifact prominence on the dataset suite This score is computed as Spearman correlation between the contrast value and the crowdsourced prominence over all masks in a dataset component, denoted as SRCC(insidep50 − outsidep50, GT prominence) This score is designed for rapid benchmarking after the prominence annotations have been collected

Empirical Findings

The empirical findings demonstrate that full-reference structural and perceptual similarity metrics contain the most useful information about viewer-noticeable SR failures Specifically, DISTS and SSIM are the strongest existing metrics by average rank, showing better performance than DeSRA and LDL—purpose-made SR artifact detectors The uncurated benchmark supports the conclusion that no-reference methods are generally worse than those that can use an HR or pseudo-GT reference, with no-reference IQA methods in particular having correlations close to zero or negative on several components

Furthermore, artifact-aware SR training does not guarantee low-prominence artifacts; for instance, LDL-SR is the weakest method across all three components, with the highest mean prominence and many confident masks on both Open Images and Urban100 The authors also characterize where prominent SR artifacts occur by annotating artifact types and semantic context with Qwen 3 VLM They found that hallucinated texture appears in 2377 masks with a mean prominence of 0.423 and 41.5% prominent masks, suggesting viewers are more sensitive to false structured detail than to over-smoothed or waxy surfaces

Conclusion

SR-Prominence is intended to make SR artifact evaluation more perceptually grounded by introducing a viewer-centered target for super-resolution artifact evaluation The released annotations, scoring protocol, pseudo-GT procedure, and reference baseline allow future detectors and SR methods to be evaluated against viewer noticeability without repeating the full crowdsourcing process More broadly, the findings suggest that SR artifact evaluation should move beyond binary defect presence and account for which artifacts viewers actually notice

Limitations and Future Work

The main limitation of SR-Prominence is the lack of precise artifact masks, as delineating exact artifact boundaries is ambiguous even for human annotators The pseudo-GT procedure is also only an approximation, and it can produce false positives when the lightweight SR model used as pseudo-GT fails to reconstruct fine textures Future work could extend prominence modeling to video super-resolution, and researchers should consider semantic artifacts such as object replacement which may need adjustments to annotation and evaluation The authors also observed that SR models show much better artifact robustness than in their main comparison in Section 6, likely because these datasets are commonly used for SR training and evaluation The baseline falls one confident artifact short of DISTS, but otherwise shows competitive results

Acknowledgments

The research was carried out using the MSU-270 supercomputer of Lomonosov Moscow State University The authors would like to thank Valeriy Gorbachev for conducting the VLM annotation for Section 6.4

References

[1] F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He, Y. Qiao, and C. Dong “Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 25669–25680 June 2024

[3] J. Liang, H. Zeng, and L. Zhang “Details or artifacts: A locally discriminative learning approach to realistic image super-resolution” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5657–5666 2022

[4] L. Xie, X. Wang, X. Chen, G. Li, Y. Shan, J. Zhou, and C. Dong “DeSRA: detect and delete the artifacts of gan-based real-world super-resolution models” arXiv preprint arXiv:2307.

1. Improve SR model audit by moving beyond binary defect presence to perceptual impact using artifact prominence as a graded target, allowing for an objective scoring protocol that allows new metrics to be benchmarked on our suite without further crowdsourcing.

2. Develop artifact detection methods that correlate better with human perception by utilizing the threshold-free objective score, which computes Spearman correlation between this contrast value and the crowdsourced prominence over all masks in a dataset component, thereby rewarding detectors that respond more strongly for artifacts noticed by more viewers.

3. Enhance full-reference metric application in no-HR settings by employing a pseudo-GT procedure where Real-time SR methods such as SPAN [21] and RLFN [22] are natural candidates for this role, ensuring that the evaluation of metrics like SSIM remains relevant without requiring high-resolution ground truth.

4. Train artifact detection models to be more robust against false positives by incorporating a feature fusion module using an MLP that minimizes the loss L = L2(MeanInside, GT Prominence) + L2(MeanOutside, 0), training the model to predict the ground-truth prominence value inside the binary mask and 0 (no artifact) outside it.

5. Develop artifact-aware SR training strategies that specifically target low-prominence artifacts by analyzing failure modes, as the paper shows that artifact-aware SR training does not guarantee low-prominence artifacts, suggesting a need to adjust training for specific failure types like hallucinated texture or plastic texture.

6. Create semantic and context-aware evaluation capabilities by utilizing Qwen 3 VLM to assign auxiliary artifact-type labels, enabling researchers to understand that artifacts in art and text images are the most prominent, guiding model refinement toward preserving artifacts in these high-context domains.

Modern image super-resolution methods generate detailed, visually appealing results, but they often introduce visual artifacts: unnatural patterns and texture distortions that degrade perceived quality. These defects vary widely in perceptual impact--some are barely noticeable, while others are highly disturbing--yet existing detection methods treat them equally. We propose artifact prominence as an evaluative target, defined as the fraction of viewers who judge a highlighted region to contain a noticeable artifact. We design a crowdsourced annotation protocol and construct SR-Prominence, a dataset suite containing 3,935 artifact masks from DeSRA, Open Images, Urban100, and a realistic no-ground-truth Urban100-HR setting, annotated with prominence. Re-annotating DeSRA reveals that 48.2% of its in-lab binary artifacts are not noticed by a majority of viewers. Across the suite, we audit SR artifact detectors, image-quality metrics, and SR methods. We find that classical full-reference metrics, especially SSIM and DISTS, provide surprisingly strong localized prominence signals, whereas no-reference IQA methods and specialized artifact detectors often fail to generalize across datasets and reference settings. SR-Prominence is released with an objective scoring protocol that allows new metrics to be benchmarked on our suite without further crowdsourcing. Together, the data and protocols enable SR artifact evaluation to move from binary defect presence toward perceptual impact. SR-Prominence is available at https://huggingface.co/datasets/imolodetskikh/sr-artifact-prominence.