Diffusion models as plug-and-play priors
cs.LG, cs.CV
Submitted: 2022-06-17
Updated: 2026-08-28
Comments: NeurIPS 2022; code: https://github.com/AlexGraikos/diffusion_priors
DOI: 10.52202/068431-1070
Code: https://github.com/AlexGraikos/diffusion_priors
License: http://creativecommons.org/licenses/by/4.0/
The gist: We consider the problem of inferring high-dimensional data x in a model that consists of a prior p(x) and an auxiliary differentiable constraint c(x, y) on x given some additional information y.
Terminology
Abstract
We consider the problem of inferring high-dimensional data x in a model that consists of a prior p(x) and an auxiliary differentiable constraint c(x, y) on x given some additional information y. In this paper, the prior is an independently trained denoising diffusion generative model. The auxiliary constraint is expected to have a differentiable form, but can come from diverse sources. The possibility of such inference turns diffusion models into plug-and-play modules, thereby allowing a range of potential applications in adapting models to new domains and tasks, such as conditional generation or image segmentation. The structure of diffusion models allows us to perform approximate inference by iterating differentiation through the fixed denoising network enriched with different amounts of noise at each step. Considering many noised versions of x in evaluation of its fitness is a novel search mechanism that may lead to new algorithms for solving combinatorial optimization problems.
Sources
- SegDiff: Image Segmentation with Diffusion Probabilistic Models
- The Transformer Network for the Traveling Salesman Problem
- Equivariant Diffusion for Molecule Generation in 3D
- An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem
- Diffusion Models for Adversarial Purification
- Resolving label uncertainty with implicit posterior models
- Image Super-Resolution via Iterative Refinement
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks