When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents".
Jane: Here's the gist of the paper. Contemporary no-reference image quality assessment (NR-IQA) models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.
Tom: This work shows that NR-IQA models can serve as natural image priors for real-world image enhancement by performing gradient ascent in the diffusion latent space and provides a new computational means of comparing different NR-IQA models through their resulting enhanced outputs >.
Jane: Framework Introduction The core innovation is the introduction of the diffusion latent MAP estimation framework, which allows "imperfect" NR-IQA models to serve as natural image priors for perceptual optimization.
Tom: This is achieved by augmenting an NR-IQA model with a differentiable and bijective diffusion transform, equipping it with generative capabilities in the diffusion latent space.
Paper summary: Tom: So, summarizing this "When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents" paper, they show that you can use no-reference image quality assessment models for perceptual optimization within the maximum a posteriori estimation framework by doing gradient ascent in the diffusion latent space >
Jane: The thesis is that by augmenting an NR-IQA model with a differentiable and bijective diffusion transform, you equip it with generative capabilities to better represent natural images in the diffusion latent space >
Lu: This setup allows different NR-IQA models to serve as natural image priors for enhancement because they are optimized within this framework, which provides a new way to compare them based on their resulting enhanced outputs >
Meng: It's interesting how they move the optimization away from the raw pixel domain into the diffusion latent space, which is where they leverage the pretrained differentiable and bijective diffusion process >
Lalam: The comparison part involves taking a large set of images and plugging in each NR-IQA model to see what kind of enhanced image it produces, and then using a specific method called maximum likelihood estimation under Thurstone Case V assumptions to rank them >
Tom: So, the big claim is that this method provides a new computational means for comparing different NR-IQA models through their resulting enhanced outputs >
Jane: And they show empirically that contemporary no-reference image quality assessment models can be plugged into the MAP estimation framework for image enhancement and that it works >
Lu: They also mention how they improve an existing model, suggesting a new model called multi-scale LIQE, which integrates a CLIP backbone from LIQE and uses the multi-scale representation of MUSIQ >
Meng: What does this mean for deployment? If we are looking at real traffic, how scalable is this diffusion latent MAP estimation framework compared to just running a standard NR-IQA model inference >
Lalam: The paper also points out that the gains in performance come from several factors, including the choice of backbone networks and the multi-scale image representation which helps perceive variations between objects at different distances >
Conclusion: Tom: So, to wrap up this "When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents" paper, it shows that you can use those NR-IQA models for perceptual optimization within the MAP estimation framework by performing gradient ascent in the diffusion latent space >
Jane: The authors show that this method allows different NR-IQA models to serve as natural image priors for enhancement because they are optimized within this framework and it provides a new way to compare them based on their resulting enhanced outputs >
Lu: They also introduce an improvement, the multi-scale LIQE model, which they show produces more refined and natural-looking results compared to other top performers like LIQE or MUSIQ >
Meng: The paper suggests that future work should focus on designing generative NR-IQA models that estimate the probability of a test image given the quality score rather than predicting the quality score from the image >
Lalam: And they also flag a challenge, which is mitigating shortcut learning where NR-IQA models might just output higher-quality predictions without actually inducing any perceptually meaningful enhancement >
Tom: So, the implication is that we have a new way to rigorously compare these no-reference models based on their ability to produce good enhanced images using this diffusion latent MAP estimation framework >
Weixia Zhang, Dingquan Li, Guangtao Zhai, Xiaokang Yang, Kede Ma
Shanghai Jiao Tong University · Pengcheng Laboratory of Artificial Intelligence Institute of Shanghai Jiao Tong University Department of Networked Intelligence Pengcheng Laboratory Shenzhen City University of Hong Kong Department of Computer Science and Shenzhen Research Institute
cs.CV
Submitted: 2024-03-11
Updated: 2026-11-13
Comments: Accepted by IEEE TIP
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: Contemporary no-reference image quality assessment (NR-IQA) models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.
Key concepts
- Diffusion Latent MAP Estimation
- This is the core framework where an NR-IQA model is augmented with a diffusion transform to create a generative capability in the latent space. It uses optimization techniques, like gradient ascent, to find an enhanced image that aligns with the quality score predicted by the NR-IQA model.
- NR-IQA Model as Prior
- Instead of using perfect models, imperfect NR-IQA models are used as 'priors'—starting points for optimization. They guide the enhancement process in a way that respects perceptual quality, allowing diffusion to generate more natural and high-quality enhanced images.
- Model Comparison by Synthesis
- This methodology compares different NR-IQA models by plugging each one into the main framework. By generating enhanced images from various models on a large dataset and then systematically selecting pairs that best distinguish between them, a ranking score is derived for each model.
Terminology
Summary
Contemporary no-reference image quality assessment (NR-IQA) models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. p. 0 Yet, limited efforts have
been devoted to treating NR-IQA models as natural image priors
for real-world image enhancement, and consequently comparing
them from a perceptual optimization standpoint. In this work, we
show—for the first time—that NR-IQA models can be plugged
into the maximum a posteriori (MAP) estimation framework for
image enhancement. This is achieved by performing gradient
ascent in the diffusion latent space rather than in the raw pixel
domain, leveraging a pretrained differentiable and bijective diffusion process. Likely, different NR-IQA models lead to different
enhanced outputs, which in turn provides a new computational
means of comparing them.Where the paper says it
Where the paper says it
Framework Introduction
The core innovation is the introduction of the diffusion latent MAP estimation framework, which allows Inspired
by [12], we augment an NR-IQA method with a differentiable
and bijective diffusion transform. This enhancement equips
the NR-IQA model with generative capabilities to better
represent the rich diversity of natural images by operating
in the diffusion latent space1
. This augmentation imbues
NR-IQA models with generative capabilities, enabling MAP
estimation in the diffusion latent space:
x
⋆
T = arg min
xT
D
h(xT ), x
init
− λqw(h(xT )), (17)
where we omit the coupled noise vector yT for notation
simplicity. (17) can be conveniently solved using gradientbased optimization, as outlined in Algorithm 1.imperfect
NR-IQA models to serve as natural image priors for perceptual optimization. This is achieved by augmenting an NR-IQA model with a differentiable and bijective diffusion transform, equipping it with generative capabilities in the diffusion latent space. p. 1Where the paper says it
Where the paper says it
Model Comparison Methodology
The paper formalizes NR-IQA model comparison through analysis by synthesis by plugging various NR-IQA models into the diffusion latent MAP estimation framework. This involves compiling a large-scale dataset of photographic images and then plugging each of N NR-IQA models into the framework to produce enhanced images. p. 4 Specifically, we first compile a large-scale dataset
of photographic images with realistic complex distortions,
denoted as X = {x
(i)}
|X |
i=1. Next, we plug each of N
NR-IQA models {qj (·)}
N
j=1 into the diffusion latent MAP
estimation framework, producing a total of |X | × N enhanced
images, {{rj
x
(i)
}
|X |
i=1}
N
j=1. For any pair of NR-IQA models qi(·) and qj (·),
corresponding to their respective enhancers ri(·) and rj (·), we
automatically select a subset of K images (where K ≪ |X |)
that best differentiate between the models by maximizing their
enhancement discrepancy:
x˜
(k+1) = arg max
x∈X /S(k)
D1 (ri(x), rj (x)) + γD2(x, S
(k)
), (18)
where S
(k) = {x˜
(k
′
)}
k
k′=1 represents the set of k images
already selected by iteratively solving the above equation. D1(·, ·) : RM × RM 7→ R represents the perceptual distance
between two images. Under the Thurstone Case
V assumption [88], the global ranking scores of NR-IQA
models µ = [µ1, µ2, . , µN ]
⊺
can be obtained via maximum
likelihood estimation for multiple options [89]:
ℓµ(C) = X
ij
CijΦ
µi − µj
√
2σ
, s.Where the paper says it
Where the paper says it
Where the paper says it
Model Improvement Strategy
The authors aim to improve the best-performing NR-IQA model by incorporating the advantages of other top-performing methods. This improvement is achieved by designing a new NR-IQA model called multi-scale LIQE (MS-LIQE), which integrates the CLIP backbone from LIQE, preserves the mixed-dataset training strategy, and incorporates the multi-scale image representation of MUSIQ. p. 7 IV-B, we
advance further by designing a new NR-IQA model that integrates the strengths of the top-performing models. Specifically,
we seek to: (1) adopt the CLIP backbone from LIQE, which
benefits from a large model capacity with weights pretrained
from web-scale data; (2) preserve the mixed-dataset training
strategy employed by both LIQE and UNIQUE; and (3)
incorporate the multi-scale image representation of MUSIQ.Where the paper says it
Empirical Findings
Empirically, the optimized image pair (x⋆, y⋆) exhibits near-identical content and appearance, aligning with observations in [12]. Quantitative results show that LIQE [11] and UNIQUE [40] statistically significantly outperform most other methods, highlighting the effectiveness of ranking image quality across multiple datasets. p. 6 First, LIQE [11] and UNIQUE [40] statistically
significantly outperform most other methods, with the exception of MUSIQ [19]. This highlights the effectiveness of ranking image quality across multiple datasets. While there may be some contrast
improvements by MUSIQ, the overall natural appearance is
somewhat compromised, due to the introduction of noticeable
artifacts, especially in the lower section of the road in Fig. In comparison to UNIQUE, MS-LIQE produces the most
refined and natural-looking result, with improved sharpness
and clarity, especially in the hair textures.Where the paper says it
Where the paper says it
Future Directions
In the future, the authors aim to develop a general and robust MAP solution for real-world image enhancement. A significant challenge identified is mitigating the phenomenon of shortcut learning, where NR-IQA models might just output higher-quality predictions without inducing any perceptually meaningful enhancement. A promising direction for future work is to design generative NR-IQA models that estimate the conditional probability of a test image given the quality score, p(xq), instead of predicting the quality score from the image, p(qx). p. 7 Improved Model Design
The original LIQE corresponds an input image x to multiple
textual templates, yielding a joint probability distribution over
predictions across three tasks: NR-IQA, scene classification,
and distortion type identification. In this work, we focus solely
on the NR-IQA task, working with a Likert-scale of five
quality levels: v ∈ V = {1, 2, 3, 4, 5} = {“bad”, “poor”,
“fair”, “good”, “perfect”} and relating discrete v to continuous
quality estimate qw(x) by
qw(x) = X
V
v=1
pw(v|x) × v, (22)
where pw(v|x) is the estimated probability of image x at
quality level v and V = 5. Next, we plug each of N
NR-IQA models {qj (·)}
N
j=1 into the diffusion latent MAP
estimation framework, producing a total of |X | × N enhanced
images, {{rj
x
(i)
}
|X |
i=1}
N
j=1. Here, rj (·) : RM 7→ RM
denotes the entire enhancement process, achieved by solving
diffusion latent MAP estimation with the j-th NR-IQA model,
as described in Algorithm 1.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper investigates how NR-IQA models can be used for perceptual optimization within the MAP estimation framework by augmenting them with a differentiable and bijective diffusion model. This approach leverages the generative capabilities of diffusion models to better represent natural images in the diffusion latent space. p. 1 Inspired
by [12], we augment an NR-IQA method with a differentiable
and bijective diffusion transform. This enhancement equips
the NR-IQA model with generative capabilities to better
represent the rich diversity of natural images by operating
in the diffusion latent space1
.Where the paper says it
How the training signal is built
The improved MS-LIQE model is trained by adopting a Likert-scale of five quality levels and relating discrete quality estimates to a continuous quality estimate qw(x) using the formula qw(x) = X V v=1 pw(vx) × v. The training involves pairwise learning-to-rank for model parameter estimation across multiple IQA datasets using the loss function lw(x, y) = 1 − p p(x, y)pw(x, y) − p (1 − p(x, y))(1 − pw(x, y)). p. 8 (22) to obtain
the quality estimate qw(x). Similar to LIQE and UNIQUE,
we employ pairwise learning-to-rank for model parameter
estimation across multiple IQA datasets.Where the paper says it
Where the gains come from
The gains in performance are attributed to several factors: (1) the choice of backbone networks and pretraining data volume, as seen with LIQE and UNIQUE. (2) the multi-scale image representation in MS-LIQE enhances its ability to perceive variations between objects at varying distances. p. 9 In contrast, LIQE blends the midground and the background. These improvements can be attributed to the multi-scale image representation in MS-LIQE, which enhances its ability
to perceive variations between objects at varying distances. Visual comparison of MAP-enhanced images using vanilla Stable Diffusion 2. 0 and its ControlNet-augmented variant. This issue is rooted in
the inherent bias and randomness of diffusion models in image
generation [104]. In contrast, the ControlNet-augmented variant leverages input images as conditioning, which introduces
an implicit fidelity constraint that complements the explicit
fidelity term in Eq. As shown in
Figs. 8(b) and (c), while both methods successfully improve
the visual quality of the input image, our diffusion latent
MAP estimation demonstrates better color fidelity to the
initial, making it particularly suitable for practical and robust
enhancement of photographic images.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The improved MS-LIQE model is trained by adopting a Likert-scale of five quality levels and relating discrete quality estimates to a continuous quality estimate qw(x) using the formula qw(x) = X V v=1 pw(vx) × v. The training involves pairwise learning-to-rank for model parameter estimation across multiple IQA datasets using the loss function lw(x, y) = 1 − p p(x, y)pw(x, y)− p (1 − p(x, y))(1 − pw(x, y)). p. 8 (22) to obtain
the quality estimate qw(x). Similar to LIQE and UNIQUE,
we employ pairwise learning-to-rank for model parameter
estimation across multiple IQA datasets.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Diffusion latent MAP estimation framework enables existing imperfect NR-IQA models to serve as natural image priors for perceptual optimization of real-world image enhancement methods. This allows for a new computational means of comparing different NR-IQA models by generating enhanced outputs and subjecting them to debiased psychophysical testing.
-
The system can systematically compare eight contemporary NR-IQA models by plugging each into the diffusion latent MAP estimation framework, producing a total of |X| x N enhanced images. This comparison yields insights into the
respective strengths and weaknesses of the competing NR-IQA models in perceptual optimization scenarios.
-
A state-of-the-art NR-IQA model can be improved by integrating aspects of other top performing methods, resulting in
noticeably better results in enhancing real world images afflicted by unknown and complex distortions, all preserving a high degree of image fidelity.
This improvement involves adopting the CLIP backbone from LIQE, preserving the mixed dataset training strategy, and incorporating multi-scale image representation. -
The improved model (MS-LIQE) can exhibit superior handling of depth of focus compared to LIQE by perceiving variations between objects at varying distances, which is attributed to its multi-scale image representation. This allows for better discrimination between foreground flowers, midground stems, and background trees in enhanced images.
-
The framework can serve as a post-enhancement step to further refine outputs from other real-world image enhancers by enhancing the perceived quality of their results while ensuring content fidelity through distortion removal and texture enrichment.
Abstract
Contemporary no-reference image quality assessment (NR-IQA) models can effectively quantify perceived image quality, often achieving strong correlations with human perceptual scores on standard IQA benchmarks. Yet, limited efforts have been devoted to treating NR-IQA models as natural image priors for real-world image enhancement, and consequently comparing them from a perceptual optimization standpoint. In this work, we show -- for the first time -- that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. This is achieved by performing gradient ascent in the diffusion latent space rather than in the raw pixel domain, leveraging a pretrained differentiable and bijective diffusion process. Likely, different NR-IQA models lead to different enhanced outputs, which in turn provides a new computational means of comparing them. Unlike conventional correlation-based measures, our comparison method offers complementary insights into the respective strengths and weaknesses of the competing NR-IQA models in perceptual optimization scenarios. Additionally, we aim to improve the best-performing NR-IQA model in diffusion latent MAP estimation by incorporating the advantages of other top-performing methods. The resulting model delivers noticeably better results in enhancing real-world images afflicted by unknown and complex distortions, all preserving a high degree of image fidelity.
Sources
- Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
- A practical guide and software for analysing pairwise comparison experiments
- Understanding Hallucinations in Diffusion Models through Mode Interpolation
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs