Learning a Maximum Entropy Model for Visual Textures using Diffusion

arXiv:2606.17342 · cs.CV · Submitted 2026-06-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Learning a Maximum Entropy Model for Visual Textures using Diffusion".

Jane: Here's the gist of the paper. Visual textures are ubiquitous in visual scenes and provide important cues for recognizing and analyzing materials and objects <ref:2606.17342#pg2> Here, we develop the first principled method for unsupervised learning of a set of statistics that are used to constrain a maximum entropy probability model <ref:2606.17342#pg3>.

Tom: Despite the compactness of our trained model (512 statistics), it generates texture images whose quality is as good as or better than the current state-of-the-art model (177k statistics) <ref:2606.17342#pg6>.

Jane: A more direct comparison of the two models, obtained by synthesizing images that are indistinguishable for one model but maximally different for the other, reveals their relative strengths and weaknesses <ref:2606.17342#pg9>.

Tom: Finally, we show that unlike previous statistical texture models, a straight trajectory in the representation space of our model generates homogeneous texture samples that interpolate smoothly between the features of the two end points <ref:2606.17342#pg10>.

Paper summary: Tom: So, we're talking about this paper here, "Learning a Maximum Entropy Model for Visual Textures using Diffusion." Basically, they’re looking at those visual textures—the stuff like grass or tree bark that are everywhere—and they want to build a way to learn the statistics that describe them without needing someone to hand-design those statistics manually.

Jane: Right. So the big idea here is developing a principled method for unsupervised learning of these statistics, which then constrains a maximum entropy probability model

Jaynes, one thousand nine hundred fifty-seven: <ref:2606.17342#pg2,method for unsupervised learning of>. What they claim is that they can learn these statistics from texture images themselves and use them to define the texture class density.

Lu: They leverage methods developed for generative diffusion models to derive both the training procedures and the sampling procedures, which is pretty novel because it connects this statistical modeling idea with something like a diffusion process

Rombach et al., two thousand twenty-two: <ref:2606.17342#pg2>.

Meng: From where I sit, I’m curious about how they actually learn these statistics. The abstract mentions they use a UNet architecture and treat the channel activations as the network output for training

Rombach et al., two thousand twenty-two: <ref:2606.17342#pg2>. That sounds like a lot of setup to get those statistics.

Tom: Exactly, and then they use two identical networks with independently trained parameters, feeding them noisy images to compute inner products between their outputs to produce one scalar value per channel. And for the parameterization of the density, they introduce a ConvNeXt-T model with dependency on sigma via a sinusoidal embedding and FiLM modulation

Perez et al., two thousand eighteen: <ref:2606.17342#pg2>.

Jane: So, if I’m getting this right, they are using two networks to get these statistics, and then they use that structure to parameterize the conditional density of a texture class. But what are the actual numbers they're working with?

Lalam: They set the number of statistics to d equals five hundred twelve and they say increasing d to one thousand twenty-four doesn't noticeably increase performance

Rombach et al., two thousand twenty-two: <ref:2606.17342#pg2>. And they say that this compactness of the model is actually quite good, generating texture images whose quality is as good as or better than the current state-of-the-art model which uses about 177k statistics

Rombach et al., <ref:2606.17342#pg6>: .

Tom: That’s a big number difference, going from five hundred twelve to one hundred seventy-six thousand six hundred forty statistics for the old Gatys model. It shows that their method can be much more efficient in terms of parameters needed to describe the texture distribution

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Lu: The real contribution seems to be the comparison they make. They synthesize images that are indistinguishable for one model but maximally different for the other, which reveals their relative strengths and weaknesses

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Jane: That sounds like a really solid way to benchmark these types of texture models against each other. So, what’s the main point they want us to get from this specific paper?

Paper summary: Lalam: Their primary contributions are developing a novel method for learning statistics that can parameterize a maximum entropy density model for training data

Rombach et al:two thousand six hundred six point one seven three four two#pg9: , training this on a large ensemble of homogeneous image patches from ImageNet21K and showing the conditional samples closely resemble the texture images they are conditioned on

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Tom: And they also compare their diffusion sampling procedure to the statistics matching procedure used in previous texture models

Rombach et al:two thousand six hundred six point one seven three four two#pg10: . They found that statistics matching generates better quality images than diffusion sampling, but their model with statistics matching still produces visual quality similar to or better than the Gatys model

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Jane: So, it seems they've got a few results here. One is that learning statistics directly leads to better models with fewer parameters, which is appealing for deployment

Rombach et al:two thousand six hundred six point one seven three four two#pg9: , and another is the comparison between their sampling methods.

Lu: What’s really interesting for representation learning is how they examine the structure of the representation space by sampling from the model conditioned on parameter vectors interpolated between those of two different texture images

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Tom: They found that unlike previous statistical texture models, a straight trajectory in the representation space of their model generates homogeneous texture samples that interpolate smoothly between the features of the two end points

Rombach et al:two thousand six hundred six point one seven three four two#pg10: . That suggests the representation space has convexity properties, which is desirable for things like style transfer.

Jane: That interpolation property is key, because it means you can smoothly transition between textures by just moving along a straight line in this learned space

Rombach et al:two thousand six hundred six point one seven three four two#pg10: . What does that mean for someone who just wants to generate some kind of texture?

Meng: For deployment, if the representation space is smooth and convex like they say, it might make manipulating these textures much easier at scale

Rombach et al:two thousand six hundred six point one seven three four two#pg9: . But I’m still worried about running this model on real traffic; how does all that training signal translate to something that won't break when we need to deploy it?

Lalam: The paper is general, and they say their method relies on using networks with inductive biases well-matched to the target density, which means it can be applied to any data modality described by a maximum entropy model

Rombach et al:two thousand six hundred six point one seven three four two#pg1: . Learning statistics directly could improve models for other modalities too.

Tom: And they did a competitive comparison where they let their model compete against another generative texture model using a method similar to MAD competition

Wang and Simoncelli, two thousand eight: , and they found their model shows its blind spot in row d, where it allows for localized oriented structures for most test textures

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Paper summary: Jane: So we have a few different ways to generate these textures now—statistics matching and diffusion sampling—and they show that statistics matching is better in terms of visual quality, while their model with statistics matching is efficient because it only uses five hundred twelve statistics instead of the one hundred seventy-six thousand six hundred forty the old Gatys model used

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Lu: And they also noted that the relationship between two representations, lambda(x, sigma) and f(x), is approximately equivalent, but a straight line in one space is likely to be curved in the other

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Meng: That's an interesting mathematical property for representation learning. But let’s talk about evaluation. They use FID scores to quantitatively evaluate quality and diversity

Heusel et al., two thousand seventeen: , but they flag that FID isn't well suited for evaluating texture models because the network used to train it is usually trained on object recognition tasks, and the test images are out-of-distribution for their texture model

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Tom: So, we’ve got a few ways to look at this. The paper focuses on learning statistics from data to constrain a maximum entropy model, and they show that diffusion-based sampling and statistics matching are the two main ways to generate these textures

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Jane: And the conclusion is that their model trained on a large dataset of textures is visually superior to the state-of-the-art Gatys model, even though it uses far fewer parameters, specifically five hundred twelve versus one hundred seventy-six thousand six hundred forty

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Lalam: So they found that images generated by a model trained on a large dataset of textures are visually superior to the state-of-the-art Gatys model, while relying on a far smaller parameter set

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Tom: The best performing model they found is their own with either statistics matching or diffusion sampling, and it's important to remember that the paper shows that a straight trajectory between the representations of two textures generates a visually smooth sequence of interpolated textures

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Jane: This interpolation property is what suggests the representation space has convexity properties that are desirable for texture manipulation, which is pretty useful information for anyone working on texture generation

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Lu: And they also point out that the relationship between two representations, lambda(x, sigma) and f(x), is approximately equivalent, but a straight line in one space is likely to be curved in the other

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Meng: So we have a few ways to look at this. The paper focuses on learning statistics from data to constrain a maximum entropy model, and they show that diffusion-based sampling and statistics matching are the two main ways to generate these textures

Rombach et al:two thousand six hundred six point one seven three four two#pg10: .

Lalam: And ultimately, they found that images generated by a model trained on a large dataset of textures are visually superior to the state-of-the-art Gatys model, while relying on a far smaller parameter set

Rombach et al:two thousand six hundred six point one seven three four two#pg9: .

Conclusion: Tom: So we're wrapping up this look at "Learning a Maximum Entropy Model for Visual Textures using Diffusion." It seems like they've taken these complex statistics and put them into a framework that works with diffusion models to generate textures.

Jane: What they did is learn the statistical rules for textures directly from data, instead of just using pre-made mathematical formulas. It’s about building a model that describes what a texture *is* based on what it looks like in the training set.

Lu: The core idea is using a maximum entropy formulation, which basically means they're finding the most likely probability distribution that fits all the observed texture data, given some constraints they set up.

Meng: So if I understand this right, they’re not just making pretty pictures; they’re learning the underlying statistical structure of textures so an AI can generate new ones that look real. That’s what matters for deployment.

Tom: Exactly. They trained this on a huge set of patches from ImageNet21K and showed that what their model generates actually looks like the textures it was trained on when you sample from it. The quality is comparable to, or better than, models that use way more parameters.

Jane: And they found something really neat about how you move between different textures in this space. They showed that a straight line connecting two texture representations creates a smooth transition between them, which suggests the model learns a structure where moving linearly makes sense.

Lu: That’s the interpolation property they highlighted. It means their representation space has some kind of smoothness or convexity, which is important for things like style transfer later on.

Meng: For me, that smoothness is good because if you can move smoothly between textures, you don't have to jump around in weird visual artifacts when you try to mix them. But what about the actual running cost?

Tom: They did compare their diffusion sampling method against a traditional statistics matching method and found that statistics matching actually produced better quality images for them. The key is that their statistical model uses only five hundred twelve statistics, which is much less than the hundreds of thousands used in older models.

Jane: So even though it’s a different approach, this new statistical learning method combined with diffusion sampling seems to be the winner here for generating high-quality textures efficiently.

Lu: Their main contribution is showing that this direct learning of statistics can constrain a maximum entropy model effectively, and they also proved how their representation space behaves when you interpolate between examples.

Tom: So, the big picture is that we can build better texture generators by learning their fundamental statistics from data rather than having someone manually define every single statistical parameter.

Jane: And this opens up possibilities for applying this idea to other things too, because the paper suggests this approach could be used for any data type that fits a maximum entropy model.

Meng: I’m still thinking about the evaluation metrics they used, though FID scores aren't perfect for textures. It’s not a guarantee that what looks good on paper will run smoothly in a production environment with real-world noise.

Tom: That's fair. And looking ahead, they suggest we need better ways to evaluate these generative models across different types of data distributions, which is something I think is really important for the next steps.

New York University · Flatiron Institute

cs.CV

Submitted: 2026-06-15

Updated: 2026-11-08

Journal ref: 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 95/100

The gist: Visual textures are ubiquitous in visual scenes and provide important cues for recognizing and analyzing materials and objects <ref:2606.17342#pg2> Here…

Key concepts

Maximum Entropy Formulation
This is a mathematical framework used to find the most probable probability distribution for a set of data, given only some constraints. It helps create a model that is as non-committal as possible while still fitting the observed data statistics.
Statistics Matching
This sampling method generates new textures by first measuring the statistics of a reference texture and then solving an optimization problem to find a new texture that matches those measured statistics exactly. It is used alongside the learned statistical model.
Diffusion Sampling
This method generates new textures by starting with random noise and gradually refining it over time according to a schedule. The process uses the learned statistical parameters to guide this refinement, similar to how diffusion models work in other generative AI applications.

Terminology

Summary

Visual textures are ubiquitous in visual scenes and provide important cues for recognizing and analyzing materials and objects <ref:2606. p. 0

Where the paper says it

edu Abstract Visual textures – spatially homogeneous image regions containing repeated elements (e. a field of grass, the bark of a tree) – are ubiquitous in visual scenes and provide important cues for recognizing and analyzing materials and objects.

17342#pg2> Here, we develop the first principled method for unsupervised learning of a set of statistics that are used to constrain a maximum entropy probability model <ref:2606. p. 0
Where the paper says it

, object recognition). Here, we develop the first principled method for unsupervised learning of a set of statistics that are used to constrain a maximum entropy probability model.

17342#pg3>. Despite the compactness of our trained model (512 statistics), it generates texture images whose quality is as good as or better than the current state-of-the-art model (177k statistics) <ref:2606. p. 0
Where the paper says it

We leverage methods developed for generative diffusion models to derive training and sampling procedures, and compare these to the traditional method of sampling via matching the statistics. Despite the compactness of our trained model (512 statistics), it generates texture images whose quality is as good as or better than the current state-of-the-art model (~177k statistics).

17342#pg6>. A more direct comparison of the two models, obtained by synthesizing images that are indistinguishable for one model but maximally different for the other, reveals their relative strengths and weaknesses <ref:2606. p. 0
Where the paper says it

Despite the compactness of our trained model (512 statistics), it generates texture images whose quality is as good as or better than the current state-of-the-art model (~177k statistics). A more direct comparison of the two models, obtained by synthesizing images that are indistinguishable for one model but maximally different for the other, reveals their relative strengths and weaknesses.

17342#pg9>. Finally, we show that unlike previous statistical texture models, a straight trajectory in the representation space of our model generates homogeneous texture samples that interpolate smoothly between the features of the two end points <ref:2606. p. 0
Where the paper says it

A more direct comparison of the two models, obtained by synthesizing images that are indistinguishable for one model but maximally different for the other, reveals their relative strengths and weaknesses. Finally, we show that unlike previous statistical texture models, a straight trajectory in the representation space of our model generates homogeneous texture samples that interpolate smoothly between the features of the two end points.

17342#pg10>.

Learning the Statistical Model

We aim to learn a parametric family of probability densities over x ∈ R n (for an n pixel image), where each density corresponds to a texture class, and individual texture images are samples <ref:2606.17342#pg2>. We assume a maximum entropy formulation [Jaynes, 1957], on which several existing texture models are based <ref:2606. p. 2

Where the paper says it

1 Maximum entropy density model We aim to learn a parametric family of probability densities over x ∈ R n (for an n pixel image), where each density corresponds to a texture class, and individual texture images are samples. We assume a maximum entropy formulation [Jaynes, 1957], on which several existing texture models are based [Zhu et al.

17342#pg9>. The optimal solution is pλ(x) = 1/Z(λ) exp-X d k=1 λkfk(x)! p. 2
Where the paper says it

E[fk(x)] = µk, k = 1, 2, · · · , d, (1) where f : R n → R d is a function that computes a set of d statistics. The optimal solution is (see Appendix A for the derivation): pλ(x) = 1 Z(λ) exp − X d k=1 λkfk(x) !

, (2) where Z(λ) is a normalizing factor and λ = [λ1, λ2, · · ·, λd] ∈ R d is chosen such that the density satisfies the constraints Epλ [fk(x)] = µk, k = 1, 2, · · ·, d <ref:2606. p. 2
Where the paper says it

The optimal solution is (see Appendix A for the derivation): pλ(x) = 1 Z(λ) exp − X d k=1 λkfk(x) ! , (2) where Z(λ) is a normalizing factor and λ = [λ1, λ2, · · · , λd] ∈ R d is chosen such that the density satisfies the constraints Epλ [fk(x)] = µk.

17342#pg9>. The two d-dimensional vectors λ and µ are uniquely determined by each other (Appendix A), and either can be used as a parametric specification of the conditional density of a texture class <ref:2606. p. 2
Where the paper says it

, (2) where Z(λ) is a normalizing factor and λ = [λ1, λ2, · · · , λd] ∈ R d is chosen such that the density satisfies the constraints Epλ [fk(x)] = µk. The two d-dimensional vectors λ and µ are uniquely determined by each other (Appendix A), and either can be used as a parametric specification of the conditional density of a texture class.

17342#pg9>.

How the training signal is built

To learn fθ(y), we take the first half of a UNet architecture [Rombach et al., 2022] and treat the channel activation at the end of each scale (those that would enter the second half of the UNet as skip connections) as the network output. p. 3

Where the paper says it

4 Network architecture and training To construct fθ(y), we take the first half of a UNet architecture [Rombach et al. , 2022] and treat the channel activation at the end of each scale (those that would enter the second half of the UNet as skip 4

We use two such networks of identical architecture, but whose parameters are initialized and trained independently. p. 4
Where the paper says it

connections) as the network output. We use two such networks of identical architecture, but whose parameters are initialized and trained independently.

We feed them with the same noisy image y and compute inner products between their corresponding output channels, producing one scalar value per channel. p. 4
Where the paper says it

We use two such networks of identical architecture, but whose parameters are initialized and trained independently. We feed them with the same noisy image y and compute inner products between their corresponding output channels, producing one scalar value per channel.

To construct λϕ(x, σ), we use a ConvNeXt-T model [Liu et al. p. 4
Where the paper says it

The network output, fθ(y), corresponds to the vector containing these scalars. To construct λϕ(x, σ), we use a ConvNeXt-T model [Liu et al.

, 2022] and introduce dependency on σ via a sinusoidal embedding and per-channel FiLM modulation [Perez et al. p. 4
Where the paper says it

To construct λϕ(x, σ), we use a ConvNeXt-T model [Liu et al. , 2022] and introduce dependency on σ via a sinusoidal embedding and per-channel FiLM modulation [Perez et al.

, 2018] at the end of each convolution block. p. 4
Where the paper says it

, 2022] and introduce dependency on σ via a sinusoidal embedding and per-channel FiLM modulation [Perez et al. , 2018] at the end of each convolution block.

We set the number of statistics to d = 512 (increasing d to 1024 does not noticeably increase the performance). p. 4
Where the paper says it

See Figure 1 for a diagram. We set the number of statistics to d = 512 (increasing d to 1024 does not noticeably increase the performance).

Where the gains come from

The primary contributions of the paper are:

  1. We develop a novel method for learning a set of statistics that can parameterize a maximum entropy density model for the training data <ref:2606. p. 1

    Where the paper says it

    But unlike previous methods, the architectural and algorithmic nature of conditioning is precisely dictated by the mathematical form of the maximum entropy model that we assume. The primary contributions of the paper are: • We develop a novel method for learning a set of statistics that can parameterize a maximum entropy density model for the training data.

    17342#pg9>.

  2. We train this model on a large ensemble of homogeneous image patches taken from ImageNet21K [Deng et al. p. 1

    Where the paper says it

    To our knowledge, this is the first of its kind. • We train this model on a large ensemble of homogeneous image patches taken from ImageNet21K [Deng et al.

    , 2009], and show on a test set that the conditional samples from the model closely resemble the texture images on which they are conditioned. p. 1
    Where the paper says it

    • We train this model on a large ensemble of homogeneous image patches taken from ImageNet21K [Deng et al. , 2009], and show on a test set that the conditional samples from the model closely resemble the texture images on which they are conditioned.

  3. We compare the conditional diffusion sampling procedure to the statistics matching procedure used to synthesize images in many previous texture models <ref:2606. p. 1

    Where the paper says it

    They also achieve a better FID score for 8 of 9 texture classes tested. • We compare the conditional diffusion sampling procedure to the statistics matching procedure used to synthesize images in many previous texture models.

    17342#pg10>.

  4. We examine the structure of the representation space by sampling from the model conditioned on parameter vectors interpolated between those of two different texture images. p. 1

    Where the paper says it

    Although these are mathematically closely related, they yield different results, with statistics-matching samples generally of higher quality. • We examine the structure of the representation space by sampling from the model conditioned on parameter vectors interpolated between those of two different texture images.

Comparison with Existing Models

We find that statistics matching generates better quality images than diffusion sampling. For comparison, we find that our model (with statistics matching) generates texture images whose visual quality is similar to or better than those generated with the Gatys model. p. 5

Where the paper says it

Over a set of test images 1 , we find that statistics matching generates better quality images than diffusion sampling (Figure 2; additional examples in Figure 8). For comparison, we find that our model (with statistics matching) generates texture images whose visual quality is similar to or better than those generated with the Gatys model.

Also note that our model uses significantly fewer parameters than the Gatys model (512 vs. p. 5
Where the paper says it

For comparison, we find that our model (with statistics matching) generates texture images whose visual quality is similar to or better than those generated with the Gatys model. Also note that our model uses significantly fewer parameters than the Gatys model (512 vs.

176,640, which exceeds the number of pixels in the 128x128 training images). p. 5
Where the paper says it

Also note that our model uses significantly fewer parameters than the Gatys model (512 vs. 176,640, which exceeds the number of pixels in the 128x128 training images).

Interpolation Properties

We examine the structure and generalization capabilities of our model’s representation space by generating texture samples whose parameters lie between those of two existing textures xa and xb. Consistent with the observations in [Portilla and Simoncelli, 2000], we find that the model of [Gatys et al. p. 1

Where the paper says it

• We examine the structure of the representation space by sampling from the model conditioned on parameter vectors interpolated between those of two different texture images. Consistent with the observations in [Portilla and Simoncelli, 2000], we find that the model of [Gatys et al.

, 2015] generally produces images consisting of a patchwise spatial mixture of the two textures. p. 1
Where the paper says it

Consistent with the observations in [Portilla and Simoncelli, 2000], we find that the model of [Gatys et al. , 2015] generally produces images consisting of a patchwise spatial mixture of the two textures.

In contrast, our model generates homogeneous texture samples whose features lie between the two textures. p. 1
Where the paper says it

, 2015] generally produces images consisting of a patchwise spatial mixture of the two textures. In contrast, our model generates homogeneous texture samples whose features lie between the two textures.

We also show that a straight trajectory between the representations (µ or λ) of two given textures generates a visually smooth sequence of interpolated textures, suggesting that the representation space of our model has convexity properties that are desirable for texture manipulation (e. p. 8
Where the paper says it

(d) Diffusion sampling for our model. We also show that a straight trajectory between the representations (µ or λ) of two given textures generates a visually smooth sequence of interpolated textures, suggesting that the representation space of our model has convexity properties that are desirable for texture manipulation (e.

g., style transfer). p. 0
Where the paper says it

, 2020, De Bortoli et al. , 2021], and have served as a basis for style transfer [Gatys et al.

General Applicability

Finally, our method is general. As with generative diffusion models, it relies on the use of networks with inductive biases that are well-matched to the target density, and can be applied to any data modality that can be appropriately described by a maximum entropy model. p. 8

Where the paper says it

Finally, our method is general. As with generative diffusion models, it relies on the use of networks with inductive biases that are well-matched to the target density, and can be applied to any data modality that can be appropriately described by a maximum entropy model, such as homogeneous temporal segments of sounds [McDermott and Simoncelli, 2011], spatiotemporal patches of videos [Xie et al.

Learning statistics directly from the data could lead to improved models for these modalities, as it has for visual texture. p. 8
Where the paper says it

, 2006, Mayzel and Schneidman, 2024]. Learning statistics directly from the data could lead to improved models for these modalities, as it has for visual texture.

Competitive Comparison

In order to more directly compare two generative texture models based on statistical measurements f and g, we let them directly compete to expose each other’s flaws using a method similar to MAD competition [Wang and Simoncelli, 2008]. We find that our model shows its blind spot in row (d), where it allows for a particular type of artifact made of localized oriented structures for most test textures. p. 6

Where the paper says it

7), which prevents the high frequency artifacts, it instead allows the appearance of localized gratings in some textures, and changes in local luminance for mosaic-like textures (row (c)). Our model shows its blind spot in row (d), where it allows for a particular type of artifact made of localized oriented structures for most test textures.

Evaluation Caveats

We quantitatively evaluated the quality and diversity of the generated textures using the FID score [Heusel et al., 2017]. A caveat is that FID is not well suited for evaluation of texture models. p. 7

Where the paper says it

The quality and diversity of generated textures is quantitatively corroborated by the FID scores [Heusel et al. A caveat is that FID is not well suited for evaluation of texture models.

There are many alternative metrics [Binkowski et al. p. 7
Where the paper says it

A caveat is that FID is not well suited for evaluation of texture models. There are many alternative metrics [Binkowski et al.

, 2018, Kynkäänniemi et al. p. 7
Where the paper says it

There are many alternative metrics [Binkowski et al. , 2018, Kynkäänniemi et al.

, 2019, Jayasumana et al. p. 7
Where the paper says it

, 2018, Kynkäänniemi et al. , 2019, Jayasumana et al.

, 2024], but they are also based on a network trained on a specific task (in most cases, object recognition), and on corresponding images that are out-of-distribution for our texture model. p. 7
Where the paper says it

, 2019, Jayasumana et al. , 2024], but ´ they are also based on a network trained on a specific task (in most cases, object recognition), and on corresponding images that are out-of-distribution for our texture model.

Future Directions

An important future direction is the development of a general method for evaluating generative models for any data distribution. We are also interested in the development of interpretable or semantically meaningful directions in the representation space (as in Hudson et al. p. 8

Where the paper says it

, style transfer). We are also interested in the development of interpretable or semantically meaningful directions in the representation space (as in Hudson et al.

[2024]), either with unsupervised dimensionality reduction methods, or using labeled datasets such as the Describable Texture Dataset [Cimpoi et al. p. 8
Where the paper says it

We are also interested in the development of interpretable or semantically meaningful directions in the representation space (as in Hudson et al. [2024]), either with unsupervised dimensionality reduction methods, or using labeled datasets such as the Describable Texture Dataset [Cimpoi et al.

, 2014]. Our texture model can serve as a scientific tool for generating stimuli for neuroscience and psychology experiments [Victor et al. p. 8
Where the paper says it

[2024]), either with unsupervised dimensionality reduction methods, or using labeled datasets such as the Describable Texture Dataset [Cimpoi et al. Our texture model can serve as a scientific tool for generating stimuli for neuroscience and psychology experiments [Victor et al.

, 2017].

Mathematical Derivations

The derivation of the exponential family distribution involves combining the objective and constraints using the method of Lagrange multipliers. We arrive at an exponential family distribution (2): p(x) = 1/Z(λ) exp-X d k=1 λkfk(x)! p. 2

Where the paper says it

E[fk(x)] = µk, k = 1, 2, · · · , d, (1) where f : R n → R d is a function that computes a set of d statistics. The optimal solution is (see Appendix A for the derivation): pλ(x) = 1 Z(λ) exp − X d k=1 λkfk(x) !

, (20). The relationship between the two representations λ(x, σ) and f(x) is approximately equivalent. p. 4
Where the paper says it

Therefore, we can assume the approximation µ ≈ f(x). Furthermore, since µ and λ are uniquely determined by each other in a maximum entropy formalism (Appendix A), the two representations λ(x, σ) and f(x) are approximately equivalent.

We also show that a straight line in one space is likely to be curved in the other. p. 6
Where the paper says it

Next, we examine the structure and generalization capabilities of our model’s representation space by generating texture samples whose parameters lie between those of two existing textures xa and xb. Interpolation can be performed in µ or λ space, using statistics matching or diffusion sampling, respectively: µm = M − m M f(xa)+ m M f(xb), or λm(σ) = M − m M λ(xa, σ)+ m M λ(xb, σ), m = 0, · · · , M (14) Note that interpolation in these two spaces is not expected to be equivalent: λ and µ are related by a highly nonlinear bijection, so a straight line in one space is likely to be curved in the other.

Sampling Procedures

We can generate new texture images from either of these two representations, by using one of the following methods:

  1. Statistics matching Given a reference texture x0, we measure its statistics µ = f(x0) and solve the following optimization problem. p. 4

    Where the paper says it

    We can generate new texture images from either of these two representations, by using one of the following methods. Statistics matching Given a reference texture x0, we measure its statistics µ = f(x0) and solve the following optimization problem: min x ∥f(x) − µ∥ 2 (10) We initialize x with i.

  2. Diffusion sampling Given a schedule of σt and the corresponding λt = λ(x0, σt) for a reference texture x0, we perform the standard DDPM reverse diffusion procedure [Ho et al. p. 4

    Where the paper says it

    [2015], against which we will compare. Diffusion sampling Given a schedule of σt and the corresponding λt = λ(x0, σt) for a reference texture x0, we perform the standard DDPM reverse diffusion procedure [Ho et al.

    , 2020].

Relationship Between Sampling Methods

A more fundamental difference is that the vectorJacobian product with Jf (x) is done with f(x)-f(x0) in statistics matching, and with λ in diffusion sampling. The former is adaptive through optimization, while the latter is predetermined for each step. p. 4

Where the paper says it

5 Image representation and generative sampling Both λ(x, σ) and f(x) can be considered as the “representation” of a texture image x. Note that given x, the former still explicitly depends on σ.

Summary of Findings

We find that images generated by a model trained on a large dataset of textures are visually superior to the state-of-the-art Gatys model, while relying on a far smaller parameter set (512 vs. 176,640). The quality and diversity of generated textures is quantitatively corroborated by the FID scores [Heusel et al. p. 7

Where the paper says it

We find that images generated by a model trained on a large dataset of textures are visually superior to the state-of-the-art Gatys model, while relying on a far smaller parameter set (512 vs. The quality and diversity of generated textures is quantitatively corroborated by the FID scores [Heusel et al.

, 2017]. The best-performing model is ours with either statistics matching or diffusion sampling. p. 5
Where the paper says it

The FID score is then calculated against the original 10,000 crops (Table 1). Except for one texture, the best-performing model is ours with either statistics matching or diffusion sampling.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Data-driven texture modeling for generative AI can be improved by replacing hand-designed statistics with unsupervised learning from large datasets, as this develop[s] a novel method for unsupervised learning of a set of such statistics from a dataset of texture images. This allows AI systems to generate high-quality visual textures using only learned statistics rather than relying on pre-trained networks or manual design.

  2. Enhanced image synthesis quality can be achieved by employing the learned model's conditional diffusion sampling procedure, which is shown to yield texture images whose quality is as good as or better than the current state-of-the-art model while using a far smaller parameter set (512 vs. 176,640). This results in faster and more efficient generation of visually convincing texture samples.

  3. Improved representation space manipulation can be realized by utilizing the learned statistics to interpolate between different textures, where the paper shows that our model generates homogeneous texture samples whose features lie between the two textures, unlike previous methods which produce patchwise spatial mixture[s]. This capability allows for smooth, physically plausible transitions between visual styles or patterns.

  4. A more robust and general generative framework can be established by leveraging the diffusion model structure, as it can be applied to any data modality that can be appropriately described by a maximum entropy model, extending texture generation to spatiotemporal patches of videos or sound textures.

Abstract

Visual textures -- spatially homogeneous image regions containing repeated elements (e.g. a field of grass, the bark of a tree) -- are ubiquitous in visual scenes and provide important cues for recognizing and analyzing materials, surface geometry and objects. A number of existing texture models extract essential statistics from a single texture image, and can then generate high-quality samples that are visually similar to the original by matching these statistics. However, their statistics are either hand-designed or based on a network pretrained for another purpose (e.g., object recognition). Here, we develop the first principled method for unsupervised learning of a set of statistics that are used to constrain a maximum entropy probability model for texture. We leverage methods developed for generative diffusion models to derive training and sampling procedures, and compare these to the traditional method of sampling via matching the statistics. Despite the compactness of our trained model (512 statistics), it generates texture images whose quality is as good as or better than the current state-of-the-art model (177k statistics), both visually and by quantitative perceptual scoring using DreamSim. Finally, we show that images generated from parameters lying along a line between the parameter vectors corresponding to two different reference textures are homogeneous textures with features that smoothly transition between those of the two references.

Sources

Related papers