When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents

arXiv:2403.06406 · cs.CV · Submitted 2024-03-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents".

Jane: Here's the gist of the paper. Contemporary no-reference image quality assessment (NR-IQA) models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.

Tom: This work shows that NR-IQA models can serve as natural image priors for real-world image enhancement by performing gradient ascent in the diffusion latent space and provides a new computational means of comparing different NR-IQA models through their resulting enhanced outputs >.

Jane: Framework Introduction The core innovation is the introduction of the diffusion latent MAP estimation framework, which allows "imperfect" NR-IQA models to serve as natural image priors for perceptual optimization.

Tom: This is achieved by augmenting an NR-IQA model with a differentiable and bijective diffusion transform, equipping it with generative capabilities in the diffusion latent space.

Paper summary: Tom: So, summarizing this "When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents" paper, they show that you can use no-reference image quality assessment models for perceptual optimization within the maximum a posteriori estimation framework by doing gradient ascent in the diffusion latent space >

Jane: The thesis is that by augmenting an NR-IQA model with a differentiable and bijective diffusion transform, you equip it with generative capabilities to better represent natural images in the diffusion latent space >

Lu: This setup allows different NR-IQA models to serve as natural image priors for enhancement because they are optimized within this framework, which provides a new way to compare them based on their resulting enhanced outputs >

Meng: It's interesting how they move the optimization away from the raw pixel domain into the diffusion latent space, which is where they leverage the pretrained differentiable and bijective diffusion process >

Lalam: The comparison part involves taking a large set of images and plugging in each NR-IQA model to see what kind of enhanced image it produces, and then using a specific method called maximum likelihood estimation under Thurstone Case V assumptions to rank them >

Tom: So, the big claim is that this method provides a new computational means for comparing different NR-IQA models through their resulting enhanced outputs >

Jane: And they show empirically that contemporary no-reference image quality assessment models can be plugged into the MAP estimation framework for image enhancement and that it works >

Lu: They also mention how they improve an existing model, suggesting a new model called multi-scale LIQE, which integrates a CLIP backbone from LIQE and uses the multi-scale representation of MUSIQ >

Meng: What does this mean for deployment? If we are looking at real traffic, how scalable is this diffusion latent MAP estimation framework compared to just running a standard NR-IQA model inference >

Lalam: The paper also points out that the gains in performance come from several factors, including the choice of backbone networks and the multi-scale image representation which helps perceive variations between objects at different distances >

Conclusion: Tom: So, to wrap up this "When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents" paper, it shows that you can use those NR-IQA models for perceptual optimization within the MAP estimation framework by performing gradient ascent in the diffusion latent space >

Jane: The authors show that this method allows different NR-IQA models to serve as natural image priors for enhancement because they are optimized within this framework and it provides a new way to compare them based on their resulting enhanced outputs >

Lu: They also introduce an improvement, the multi-scale LIQE model, which they show produces more refined and natural-looking results compared to other top performers like LIQE or MUSIQ >

Meng: The paper suggests that future work should focus on designing generative NR-IQA models that estimate the probability of a test image given the quality score rather than predicting the quality score from the image >

Lalam: And they also flag a challenge, which is mitigating shortcut learning where NR-IQA models might just output higher-quality predictions without actually inducing any perceptually meaningful enhancement >

Tom: So, the implication is that we have a new way to rigorously compare these no-reference models based on their ability to produce good enhanced images using this diffusion latent MAP estimation framework >

Weixia Zhang, Dingquan Li, Guangtao Zhai, Xiaokang Yang, Kede Ma

Shanghai Jiao Tong University · Pengcheng Laboratory of Artificial Intelligence Institute of Shanghai Jiao Tong University Department of Networked Intelligence Pengcheng Laboratory Shenzhen City University of Hong Kong Department of Computer Science and Shenzhen Research Institute

cs.CV

Submitted: 2024-03-11

Updated: 2026-11-13

Comments: Accepted by IEEE TIP

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: Contemporary no-reference image quality assessment (NR-IQA) models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.

Key concepts

Diffusion Latent MAP Estimation
This is the core framework where an NR-IQA model is augmented with a diffusion transform to create a generative capability in the latent space. It uses optimization techniques, like gradient ascent, to find an enhanced image that aligns with the quality score predicted by the NR-IQA model.
NR-IQA Model as Prior
Instead of using perfect models, imperfect NR-IQA models are used as 'priors'—starting points for optimization. They guide the enhancement process in a way that respects perceptual quality, allowing diffusion to generate more natural and high-quality enhanced images.
Model Comparison by Synthesis
This methodology compares different NR-IQA models by plugging each one into the main framework. By generating enhanced images from various models on a large dataset and then systematically selecting pairs that best distinguish between them, a ranking score is derived for each model.

Terminology

Summary

Contemporary no-reference image quality assessment (NR-IQA) models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. p. 0

Where the paper says it

Yet, limited efforts have been devoted to treating NR-IQA models as natural image priors for real-world image enhancement, and consequently comparing them from a perceptual optimization standpoint. In this work, we show—for the first time—that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.

This work shows that NR-IQA models can serve as natural image priors for real-world image enhancement by performing gradient ascent in the diffusion latent space and provides a new computational means of comparing different NR-IQA models through their resulting enhanced outputs >. p. 0
Where the paper says it

This is achieved by performing gradient ascent in the diffusion latent space rather than in the raw pixel domain, leveraging a pretrained differentiable and bijective diffusion process. Likely, different NR-IQA models lead to different enhanced outputs, which in turn provides a new computational means of comparing them.

Framework Introduction

The core innovation is the introduction of the diffusion latent MAP estimation framework, which allows imperfect NR-IQA models to serve as natural image priors for perceptual optimization. This is achieved by augmenting an NR-IQA model with a differentiable and bijective diffusion transform, equipping it with generative capabilities in the diffusion latent space. p. 1

Where the paper says it

Inspired by [12], we augment an NR-IQA method with a differentiable and bijective diffusion transform. This enhancement equips the NR-IQA model with generative capabilities to better represent the rich diversity of natural images by operating in the diffusion latent space1 .

The process involves solving the optimization: x⋆T = arg min xT D h(xT), x init − λqw(h(xT)), which can be solved using gradient-based optimization. p. 4
Where the paper says it

This augmentation imbues NR-IQA models with generative capabilities, enabling MAP estimation in the diffusion latent space: x ⋆ T = arg min xT D h(xT ), x init − λqw(h(xT )), (17) where we omit the coupled noise vector yT for notation simplicity. (17) can be conveniently solved using gradientbased optimization, as outlined in Algorithm 1.

Model Comparison Methodology

The paper formalizes NR-IQA model comparison through analysis by synthesis by plugging various NR-IQA models into the diffusion latent MAP estimation framework. This involves compiling a large-scale dataset of photographic images and then plugging each of N NR-IQA models into the framework to produce enhanced images. p. 4

Where the paper says it

Specifically, we first compile a large-scale dataset of photographic images with realistic complex distortions, denoted as X = {x (i)} |X | i=1. Next, we plug each of N NR-IQA models {qj (·)} N j=1 into the diffusion latent MAP estimation framework, producing a total of |X | × N enhanced images, {{rj x (i) } |X | i=1} N j=1.

A systematic comparison is performed by automatically selecting a subset of K images that best differentiate between two models by maximizing their enhancement discrepancy. p. 4
Where the paper says it

For any pair of NR-IQA models qi(·) and qj (·), corresponding to their respective enhancers ri(·) and rj (·), we automatically select a subset of K images (where K ≪ |X |) that best differentiate between the models by maximizing their enhancement discrepancy: x˜ (k+1) = arg max x∈X /S(k) D1 (ri(x), rj (x)) + γD2(x, S (k) ), (18) where S (k) = {x˜ (k ′ )} k k′=1 represents the set of k images already selected by iteratively solving the above equation. D1(·, ·) : RM × RM 7→ R represents the perceptual distance between two images.

The final ranking scores of the NR-IQA models are obtained via maximum likelihood estimation under the Thurstone Case V assumption. p. 4
Where the paper says it

Under the Thurstone Case V assumption [88], the global ranking scores of NR-IQA models µ = [µ1, µ2, . , µN ] ⊺ can be obtained via maximum likelihood estimation for multiple options [89]: ℓµ(C) = X ij CijΦ µi − µj √ 2σ , s.

Model Improvement Strategy

The authors aim to improve the best-performing NR-IQA model by incorporating the advantages of other top-performing methods. This improvement is achieved by designing a new NR-IQA model called multi-scale LIQE (MS-LIQE), which integrates the CLIP backbone from LIQE, preserves the mixed-dataset training strategy, and incorporates the multi-scale image representation of MUSIQ. p. 7

Where the paper says it

IV-B, we advance further by designing a new NR-IQA model that integrates the strengths of the top-performing models. Specifically, we seek to: (1) adopt the CLIP backbone from LIQE, which benefits from a large model capacity with weights pretrained from web-scale data; (2) preserve the mixed-dataset training strategy employed by both LIQE and UNIQUE; and (3) incorporate the multi-scale image representation of MUSIQ.

Empirical Findings

Empirically, the optimized image pair (x⋆, y⋆) exhibits near-identical content and appearance, aligning with observations in [12]. Quantitative results show that LIQE [11] and UNIQUE [40] statistically significantly outperform most other methods, highlighting the effectiveness of ranking image quality across multiple datasets. p. 6

Where the paper says it

First, LIQE [11] and UNIQUE [40] statistically significantly outperform most other methods, with the exception of MUSIQ [19]. This highlights the effectiveness of ranking image quality across multiple datasets.

The MS-LIQE model produced the most refined and natural-looking result compared to LIQE, MUSIQ, and UNIQUE. p. 9
Where the paper says it

While there may be some contrast improvements by MUSIQ, the overall natural appearance is somewhat compromised, due to the introduction of noticeable artifacts, especially in the lower section of the road in Fig. In comparison to UNIQUE, MS-LIQE produces the most refined and natural-looking result, with improved sharpness and clarity, especially in the hair textures.

Future Directions

In the future, the authors aim to develop a general and robust MAP solution for real-world image enhancement. A significant challenge identified is mitigating the phenomenon of shortcut learning, where NR-IQA models might just output higher-quality predictions without inducing any perceptually meaningful enhancement. A promising direction for future work is to design generative NR-IQA models that estimate the conditional probability of a test image given the quality score, p(xq), instead of predicting the quality score from the image, p(qx). p. 7

Where the paper says it

Improved Model Design The original LIQE corresponds an input image x to multiple textual templates, yielding a joint probability distribution over predictions across three tasks: NR-IQA, scene classification, and distortion type identification. In this work, we focus solely on the NR-IQA task, working with a Likert-scale of five quality levels: v ∈ V = {1, 2, 3, 4, 5} = {“bad”, “poor”, “fair”, “good”, “perfect”} and relating discrete v to continuous quality estimate qw(x) by qw(x) = X V v=1 pw(v|x) × v, (22) where pw(v|x) is the estimated probability of image x at quality level v and V = 5.

The authors also suggest reducing computational complexity across the entire (reverse) diffusion process as another practical avenue for advancement. p. 4
Where the paper says it

Next, we plug each of N NR-IQA models {qj (·)} N j=1 into the diffusion latent MAP estimation framework, producing a total of |X | × N enhanced images, {{rj x (i) } |X | i=1} N j=1. Here, rj (·) : RM 7→ RM denotes the entire enhancement process, achieved by solving diffusion latent MAP estimation with the j-th NR-IQA model, as described in Algorithm 1.

Pruning redundant reasoning tokens

The paper investigates how NR-IQA models can be used for perceptual optimization within the MAP estimation framework by augmenting them with a differentiable and bijective diffusion model. This approach leverages the generative capabilities of diffusion models to better represent natural images in the diffusion latent space. p. 1

Where the paper says it

Inspired by [12], we augment an NR-IQA method with a differentiable and bijective diffusion transform. This enhancement equips the NR-IQA model with generative capabilities to better represent the rich diversity of natural images by operating in the diffusion latent space1 .

How the training signal is built

The improved MS-LIQE model is trained by adopting a Likert-scale of five quality levels and relating discrete quality estimates to a continuous quality estimate qw(x) using the formula qw(x) = X V v=1 pw(vx) × v. The training involves pairwise learning-to-rank for model parameter estimation across multiple IQA datasets using the loss function lw(x, y) = 1 − p p(x, y)pw(x, y) − p (1 − p(x, y))(1 − pw(x, y)). p. 8

Where the paper says it

(22) to obtain the quality estimate qw(x). Similar to LIQE and UNIQUE, we employ pairwise learning-to-rank for model parameter estimation across multiple IQA datasets.

Where the gains come from

The gains in performance are attributed to several factors: (1) the choice of backbone networks and pretraining data volume, as seen with LIQE and UNIQUE. (2) the multi-scale image representation in MS-LIQE enhances its ability to perceive variations between objects at varying distances. p. 9

Where the paper says it

In contrast, LIQE blends the midground and the background. These improvements can be attributed to the multi-scale image representation in MS-LIQE, which enhances its ability to perceive variations between objects at varying distances.

(3) the ControlNet-augmented variant of Stable Diffusion 2. p. 8
Where the paper says it

Visual comparison of MAP-enhanced images using vanilla Stable Diffusion 2. 0 and its ControlNet-augmented variant.

0 introduces an implicit fidelity constraint that complements the explicit fidelity term in Eq. p. 7
Where the paper says it

This issue is rooted in the inherent bias and randomness of diffusion models in image generation [104]. In contrast, the ControlNet-augmented variant leverages input images as conditioning, which introduces an implicit fidelity constraint that complements the explicit fidelity term in Eq.

(17). The resulting method demonstrates better color fidelity to the initial image compared to direct diffusion latent optimization. p. 7
Where the paper says it

As shown in Figs. 8(b) and (c), while both methods successfully improve the visual quality of the input image, our diffusion latent MAP estimation demonstrates better color fidelity to the initial, making it particularly suitable for practical and robust enhancement of photographic images.

The improved MS-LIQE model is trained by adopting a Likert-scale of five quality levels and relating discrete quality estimates to a continuous quality estimate qw(x) using the formula qw(x) = X V v=1 pw(vx) × v. The training involves pairwise learning-to-rank for model parameter estimation across multiple IQA datasets using the loss function lw(x, y) = 1 − p p(x, y)pw(x, y)− p (1 − p(x, y))(1 − pw(x, y)). p. 8

Where the paper says it

(22) to obtain the quality estimate qw(x). Similar to LIQE and UNIQUE, we employ pairwise learning-to-rank for model parameter estimation across multiple IQA datasets.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Diffusion latent MAP estimation framework enables existing imperfect NR-IQA models to serve as natural image priors for perceptual optimization of real-world image enhancement methods. This allows for a new computational means of comparing different NR-IQA models by generating enhanced outputs and subjecting them to debiased psychophysical testing.

  2. The system can systematically compare eight contemporary NR-IQA models by plugging each into the diffusion latent MAP estimation framework, producing a total of |X| x N enhanced images. This comparison yields insights into the respective strengths and weaknesses of the competing NR-IQA models in perceptual optimization scenarios.

  3. A state-of-the-art NR-IQA model can be improved by integrating aspects of other top performing methods, resulting in noticeably better results in enhancing real world images afflicted by unknown and complex distortions, all preserving a high degree of image fidelity. This improvement involves adopting the CLIP backbone from LIQE, preserving the mixed dataset training strategy, and incorporating multi-scale image representation.

  4. The improved model (MS-LIQE) can exhibit superior handling of depth of focus compared to LIQE by perceiving variations between objects at varying distances, which is attributed to its multi-scale image representation. This allows for better discrimination between foreground flowers, midground stems, and background trees in enhanced images.

  5. The framework can serve as a post-enhancement step to further refine outputs from other real-world image enhancers by enhancing the perceived quality of their results while ensuring content fidelity through distortion removal and texture enrichment.

Abstract

Contemporary no-reference image quality assessment (NR-IQA) models can effectively quantify perceived image quality, often achieving strong correlations with human perceptual scores on standard IQA benchmarks. Yet, limited efforts have been devoted to treating NR-IQA models as natural image priors for real-world image enhancement, and consequently comparing them from a perceptual optimization standpoint. In this work, we show -- for the first time -- that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. This is achieved by performing gradient ascent in the diffusion latent space rather than in the raw pixel domain, leveraging a pretrained differentiable and bijective diffusion process. Likely, different NR-IQA models lead to different enhanced outputs, which in turn provides a new computational means of comparing them. Unlike conventional correlation-based measures, our comparison method offers complementary insights into the respective strengths and weaknesses of the competing NR-IQA models in perceptual optimization scenarios. Additionally, we aim to improve the best-performing NR-IQA model in diffusion latent MAP estimation by incorporating the advantages of other top-performing methods. The resulting model delivers noticeably better results in enhancing real-world images afflicted by unknown and complex distortions, all preserving a high degree of image fidelity.

Sources

Related papers