Sobolev Regularized MMD Gradient Flow

arXiv:2605.11884 · cs.LG · Submitted 2026-05-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Sobolev Regularized MMD Gradient Flow".

Jane: Here's the gist of the paper. Sobolev-regularized Maximum Mean Discrepancy (SrMMD) gradient flow is a regularized variant of MMD gradient flow based on a gradient penalty on the witness function that mitigates non-convexity and yields provable global convergence guarantees in both continuous and discrete time Proposed Method and Scope The authors propose the SrMMD flow, a novel regularized variant of MMD/KSD flow based on a gradient norm penalty on the witness function The proposed framework applies both to generative modelling and to sampling from unnormalized targets, the latter through the use of Stein kernels They prove that this gradient regularization yields global convergence guarantees in both continuous- and discrete-time settings, under a source condition on the trajectory Theoretical Foundations The SrMMD flow is defined as a trajectory satisfying the continuity equation ∂tµt − ∇ · (µt∇fµt,π) = 0, or equivalently, admitting a characteristic representation dXt = −∇fµt,π(Xt)dt The witness function fµ,π attains the supremum in Definition 3.1 The resulting gradient field ∇fµ,π is then used in SrMMD flow to transport particles Convergence Guarantees The SrMMD flow enjoys provably fast convergence in both sampling and generative modelling For sampling, its convergence rate is independent of isoperimetric characteristics of the target distribution π, denoted by β such as log-Sobolev or Poincaré constants The analysis relies on a regularity condition on the difference between kernel mean embeddings Theorem 4.1 shows that the SrMMD flow converges exponentially fast to the target distribution π, as measured either by the MMD (or KSD with a Stein kernel kπ) up to a residual barrier term Practical Implementation and Analysis The SrMMD descent scheme is implemented by a system of interacting particles where x(i)s+1 = x(i)s − γ∇fµˆs,π(x(i)s), ∀s ∈

S: The iteration complexity of SrMMD descent is constrained by 0 1 because of the saturation effect The paper also introduces the Hybrid-regularized MMD (HrMMD) flow, which interpolates between L2 and gradient norm regularizations Empirical Validation The effectiveness of the proposed flow is empirically verified on a broad range of tasks in both generative modelling and sampling Experiments show that SrMMD flow substantially outperforms the vanilla MMD flow, consistent with the theory that SrMMD enjoys exponential convergence The performance of SrMMD flow is comparable to that of DrMMD flow, suggesting that either L2 or gradient regularization improves convergence The hybrid HrMMD flow achieves the best performance on Gaussian mixtures Sampling Performance In sampling from an unnormalized target π, most of the existing convergence rates are established under various tail conditions on π, such as Poincaré (sub-exponential tail) or logSobolev (sub-Gaussian tail) By contrast, Theorem 4.1 relies only on conditions on π that are implicitly encoded through the Stein kernel kπ in Assumptions 1 and 2 The SrMMD flow enjoys an explicit decay rate exp(−2T) independent of the tail conditions of π The particle implementation of SrMMD descent is well-posed under Assumptions 1 and 2, leading to an iteration complexity of O(δ −1/α) The final KSD and W2 values decrease as the particle number N increases, indicating that our particle implementation benefits from a more accurate empirical approximation of the population descent scheme We propose SrMMD flow, a new flow with a gradient penalty to the MMD/KSD witness function for globally convergent continuous- and discrete-time schemes, applicable for both generative modelling and sampling A primary limitation of SrMMD flow is its computational cost: each iteration requires inverting an N d × N d matrix, leading to a cost of O(N3d 3) A next step is therefore to assess at larger-scale tasks where SVGD has proved effective, e.g., earthquake modelling References

AGS05: Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré.

Tom: Gradient Flows: In Metric Spaces and in the Space of Probability Measures.

Jane: Springer Science & Business Media, 2005

AHS23: Fabian Altekrüger, Johannes Hertrich, and Gabriele Steidl.

Tom: Neural Wasserstein gradient flows for discrepancies with Riesz kernels.

Title and authors: Tom: So we're looking at this paper called "Sobolev Regularized Maximum Mean Discrepancy Gradient Flow." It sounds like they've taken the standard maximum mean discrepancy gradient flow and added a regularization term to make it more stable.

Jane: Right, so instead of just relying on the MMD objective, they introduce a gradient penalty on this witness function. What does that actually do for someone trying to train a generative model?

Lu: It mitigates the non-convexity of the MMD objective, which is a big deal because standard flows can get stuck in bad spots. Plus, it gives them provable global convergence guarantees in both continuous and discrete time settings, which is pretty strong.

Meng: Provable global convergence sounds nice on paper, but I need to know how this actually plays out when we're running things at scale. Does this mean we can just throw a massive dataset at it and expect it to work reliably?

Tom: The key part here is that their analysis doesn't even rely on isoperimetric assumptions about the target distribution, which is a surprise because usually, those constants are what make convergence hard in multi-modal scenarios.

Jane: That makes sense; if we don't need those specific tail conditions like log-Sobolev or Poincaré constants to guarantee convergence, it opens up the possibilities for sampling from distributions that have really complex shapes.

Lu: Exactly. The paper states their analysis is based on a regularity condition on the difference between kernel mean embeddings, which is a more general requirement than what we usually need for things like log-Sobolev or Poincaré constants <ref:2605.11884#pg1>.

Meng: Okay, so what's the tangible benefit for deployment? Is it just theoretically cleaner, or does it actually make our models faster to train in practice?

Tom: They show that this gradient regularization yields provable global convergence guarantees for both generative modeling and sampling from unnormalized targets using Stein kernels <ref:2605.11884#pg1>. That means we can use one framework for two different things instead of needing separate methods.

Jane: I'm curious about the specific mechanism they use when they talk about sampling, since that's a tricky area to get right with these flows.

Lu: For sampling from an unnormalized target distribution pi, they apply this flow using Stein kernels <ref:2605.11884#pg1>. This is something previous works haven't done; they show it works for both generative modeling and sampling at the same time.

Meng: If it can handle both, that’s great for our pipeline. But what about the computational cost? I'm worried this regularization adds a huge overhead to every single iteration we run in production.

Tom: They describe the SrMMD descent scheme as a system of interacting particles where each particle moves according to x sub plus one equals x sub minus gamma times the gradient of the witness function evaluated at the current particle position <ref:2605.11884#pg3>.

Title and authors: Jane: So, they are simulating this using an interacting particle system, which is essentially what a Lagrangian formulation amounts to in this case <ref:2605.11884#pg3>.

Lu: The paper mentions that their particle implementation can be efficiently computed by using a closed-form expression for the witness function, even though it involves matrix inversions of operators derived from kernel derivatives <ref:2605.11884#pg3>.

Meng: Matrix inversions at scale? That sounds expensive. What's the actual iteration complexity they claim? I need to know if this is going to be feasible for a real application, not just a proof of concept.

Tom: They state the iteration complexity of the SrMMD descent is constrained by zero one because of a saturation effect <ref:2605.11884#pg3>. It's O delta minus one over alpha <ref:2605.11884#pg2>, where alpha characterizes the regularity of the trajectory.

Jane: That O delta minus one over alpha complexity, compared to the Unadjusted Langevin Algorithm which needs O delta minus beta iterations, suggests a significant speedup when dealing with multi-modal targets where those isoperimetric constants beta could scale exponentially <ref:2605.11884#pg2>.

Lu: And for generative modeling specifically, even though convergence in MMD isn't the main practical metric, they provide what they claim is the first theoretical justification that gradient regularization can accelerate convergence compared to standard flows <ref:2605.11884#pg2>.

Meng: We see that comparison often ends up being with things like FID scores

HRU+seventeen: , so this paper suggests that combining L2 and gradient regularization, as seen in the Hybrid-regularized MMD flow, might actually improve empirical performance in GANs <ref:2605.11884#pg2>.

Tom: They introduce the Hybrid-regularized MMD flow, which interpolates between L2 norm and gradient norm regularizations <ref:2605.11884#pg3>. This is a way to see if mixing the two regularization schemes yields better results than just one of them alone.

Jane: That sounds like a smart compromise for practitioners, trying to get the stability from L2 while still getting the directional guidance from the gradient penalty. What about how this works on real data?

Lu: Empirically, they show that SrMMD flow substantially outperforms vanilla MMD flow in both generative modeling and sampling <ref:2605.11884#pg1>. The performance is comparable to DrMMD flow, which uses L2 regularization on the witness or critic function <ref:2605.11884#pg2>.

Meng: If it's better than DrMMD in practice, that’s a strong signal for us. But what about the sampling performance specifically? I need to know how much faster we can get samples from a complex distribution.

Tom: For sampling, they achieve an explicit decay rate of exp of minus two T which is independent of the tail conditions on pi <ref:2605.11884#pg2>. That means it performs well regardless of whether the target distribution has a log-Sobolev or Poincaré tail <ref:2605.11884#pg2>.

Title and authors: Jane: That’s a big deal because most existing convergence rates rely on those specific tail conditions which can be very restrictive in practice, especially for multi-modal targets where those constants might scale exponentially with the separation between modes <ref:2605.11884#pg2>.

Lu: And the particle implementation is well-posed under Assumptions one and two, leading to that complexity we mentioned earlier, O delta minus one over alpha <ref:2605.11884#pg3>. It’s a clean theoretical structure.

Meng: So, if I have to deploy this for something like earthquake modeling where the data is really messy and multi-modal, does this theory hold up when we move away from simple test distributions?

Tom: They admit that their guarantee is stated in KSD, which is a weaker metric than the KL or chi two divergence commonly used to analyze ULA <ref:2605.11884#pg2>. That's a limitation they openly state.

Jane: So, while it’s robust against the distribution tail conditions, we still have that metric limitation where KSD is weaker than the KL divergence for measuring how close we actually are to the target <ref:2605.11884#pg2>.

Lu: The hybrid flow, HrMMD, is also mentioned as achieving the best performance on Gaussian mixtures <ref:2605.11884#pg3>. It interpolates between L2 and gradient norm regularizations in a way that seems empirically beneficial for image generation tasks <ref:2605.11884#pg2>.

Meng: For deployment, the particle implementation is described as having a closed-form expression for the witness function, which is good because it means we don't have to learn a score function at every step <ref:2605.11884#pg3>. That reduces our training overhead significantly.

Tom: So we're looking at a method that combines regularization, works for both generation and sampling, has strong convergence guarantees independent of distribution shape assumptions, and shows better empirical results than standard MMD flows <ref:2605.11884#pg1>.

Jane: It sounds like a really versatile tool because it bridges the gap between theory that handles complex distributions and practical needs for stable training in generative AI <ref:2605.11884#pg3>.

Lu: The overall implication is that we can use gradient regularization to accelerate convergence, especially when dealing with the complexities of multi-modal targets or when we need strong guarantees for sampling <ref:2605.11884#pg2>.

Meng: And the trade-off is that the particle implementation has a cubic complexity in terms of dimensions, O N cubed times d cubed, which is what they admitted at the end <ref:2605.11884#pg3>. That's a significant cost for deployment on very high-dimensional data.

Tom: Yeah, so the next step is figuring out how to make that particle computation cheaper, maybe by using those closed-form expressions they mentioned <ref:2605.11884#pg3>.

Jane: So we've seen how this Sobolev Regularized Maximum Mean Discrepancy Gradient Flow works, and it seems like a solid framework for tackling both generative modeling and sampling problems with better theoretical guarantees than before <ref:2605.11884#pg1>.

The paper's summary: Tom: So, to wrap up what we just covered about this paper, they're proposing something called the Sobolev Regularized Maximum Mean Discrepancy Gradient Flow, which is basically taking the standard MMD gradient flow and adding a special penalty term on that witness function.

Jane: Right. It’s all about making that flow more stable, less prone to getting stuck in bad spots when you’re training a generative model or trying to sample from something new. They introduce this gradient penalty specifically to fight the non-convexity of the MMD objective, which is a huge deal for training AI models.

Lu: And what makes it interesting is that they've managed to prove global convergence guarantees in both continuous time and discrete time settings, which shows this isn't just some theoretical trick; it’s mathematically sound.

Meng: I want to know what that actually means for deployment, because usually when you add regularization, you end up with a training process that takes forever or breaks at scale.

Tom: Well, the authors state that their analysis is based on a regularity condition on the difference between kernel mean embeddings rather than relying on assumptions about the target distribution itself—like needing log-Sobolev constants for sampling. That’s a big win because it means it works for more complex distributions.

Jane: That’s what I was thinking. So, this method can handle distributions that are really messy, like those multi-modal targets, without needing to know the specific tail conditions of that data.

Lu: Exactly. And they show that this regularization actually helps speed things up in practice compared to the vanilla MMD flow when it comes to convergence rates for both generative modeling and sampling from unnormalized targets using Stein kernels.

Tom: They’re showing better empirical results too, even compared to other methods like DrMMD flow, which uses a different type of L2 regularization.

Jane: So, if we look at the sampling side specifically, they get an explicit decay rate that is independent of those tail conditions we talked about earlier—it decays at an exponential rate related to time T.

Meng: That independence from the tail conditions is important because it means you don't have to worry about whether your target distribution has a log-Sobolev or Poincaré tail, which can be very restrictive in real-world scenarios.

Lu: And for sampling from unnormalized targets, they show that this flow can use Stein kernels and achieve a convergence rate of delta to the minus one over alpha iterations, which is faster than the Unadjusted Langevin Algorithm iteration count if you are dealing with something multi-modal.

Jane: That speedup is significant because those traditional algorithms often need a correction factor related to those tail constants that can scale exponentially with how separated your modes are.

Tom: The authors also introduced the Hybrid-regularized MMD flow, which mixes the L2 norm regularization and the gradient norm regularization together in a way they claim improves empirical performance on Gaussian mixtures.

Meng: If it actually performs better on Gaussian mixtures, that’s something I can take seriously for image generation tasks, even if we have to deal with the computational cost of running these particle systems.

Lu: The authors do acknowledge a limitation, though; the final convergence analysis relies on KSD, which is a weaker metric than KL divergence or chi two divergence that we usually use to measure how close you actually are to your target distribution.

Jane: So, while the method is robust against complex distribution shapes and has faster theoretical rates for sampling, we still have that measurement limitation where KSD doesn't perfectly capture the quality of the approximation.

Tom: It’s a trade-off, right? Stronger convergence guarantees but with a slightly less accurate measure of success for the final output quality.

Meng: I just want to know how much it costs to run this particle implementation, because if each iteration requires inverting an N by N matrix, that sounds like a serious computational burden at scale.

Jane: The paper does describe the particle implementation as being tractable because they use a closed-form expression for the witness function, even though it involves those matrix inversions.

Lu: They say that this tractability allows for practical implementation in high dimensional settings, like in student teacher networks, without needing to learn a score function at every single step.

Tom: So we’re looking at a method that’s theoretically very solid across continuous and discrete time, handles both generation and sampling with better convergence guarantees than before, but we still have the cost of that particle system to worry about.

Jane: It seems like this is a really versatile tool because it bridges the gap between theory that handles complex distributions and practical needs for stable training in generative AI.

Meng: So the next step then is figuring out how to make those particle computations cheaper, maybe by using those closed-form expressions they mentioned, because right now, if we're shipping models to production, I need a clearer picture of the runtime before I commit resources.

The paper's improvements: Tom: So, we’ve talked about what the paper actually proposes—the Sobolev Regularized Maximum Mean Discrepancy Gradient Flow—and now we’re going to talk about how they are improving on previous work.

Jane: They suggest a hybrid approach called the Hybrid-regularized MMD flow, which interpolates between two different types of regularization: an L2 norm and that gradient norm we talked about earlier.

Lu: That interpolation is interesting because it seems to give you the stability from the L2 side while still getting the directional guidance from the gradient penalty.

Tom: I’m looking at this idea because it suggests that mixing different regularization schemes can actually lead to better results than just sticking with one of them alone in generative adversarial networks.

Jane: It sounds like a pragmatic solution for practitioners trying to get better empirical performance on image generation tasks without completely abandoning the flow structure they built.

Meng: Pragmatic is good, but I need to know if this actually translates to anything useful when we’re thinking about deployment costs and what breaks at scale.

Lu: The authors claim that this hybrid flow achieves the best empirical performance on Gaussian mixtures, which is a specific scenario where it really shines.

Tom: That’s a concrete result. They are showing that this combination of regularization schemes outperforms both the standard MMD flow and even something called DrMMD in some benchmarks.

Jane: So, what does that mean for someone just listening to the show? It means if you're training a GAN for something like generating images, trying this hybrid approach might actually give you better visual results than just using the vanilla MMD gradient flow.

Meng: If it improves performance, that’s one thing. But we also need to think about the computational overhead of this hybrid scheme. Does it make the training process significantly slower than a simpler L2-only regularization?

Lu: The paper implies that while there's an interpolation happening, they are still achieving better convergence characteristics, which is what matters for the long run.

Tom: They’re saying that even with this hybrid setup, you get a tighter constraint on the trajectory because of both regularization types working together.

Jane: It seems like a way to combine different theoretical strengths to get a more robust system for modeling data discrepancies, which is what they are aiming for with this new flow.

Meng: So it’s about getting the best convergence rate without adding completely unmanageable complexity that would crash our training pipeline?

Lu: That’s the ambition there. They are trying to find a sweet spot where the theoretical gains from combined regularization outweigh any practical slowdown in the optimization process.

Conclusion: Tom: So we've covered the Sobolev Regularized Maximum Mean Discrepancy Gradient Flow, which is basically taking that standard MMD flow and adding a special penalty term to stabilize it for both generative modeling and sampling.

Jane: Right, so this work is really about finding a way to make these complex flows more stable, especially when dealing with distributions that are messy or multi-modal.

Lu: The big implication is that we can now have provable global convergence guarantees in both continuous and discrete time settings without needing super restrictive assumptions about the target distribution itself.

Meng: I’m still focused on the practical side—if this flow is so much more stable theoretically, does it mean we can actually deploy it reliably in a production setting without it exploding?

Tom: The authors show that they can achieve faster convergence rates for sampling from unnormalized targets using Stein kernels, and that rate is independent of those tail conditions we discussed.

Jane: That’s a big deal because most current methods rely on specific assumptions about the data's tail shape, like log-Sobolev or Poincaré constants, which can be really hard to guarantee in the real world.

Lu: And they also introduced the Hybrid-regularized MMD flow to interpolate between L2 and gradient norm regularization, claiming better empirical results on Gaussian mixtures.

Tom: So we’re seeing a combination of ideas here—better regularization techniques leading to faster convergence rates for sampling and potentially better image quality in generative tasks.

Meng: I just want to know about the cost again. The particle implementation requires matrix inversions, so what is the actual runtime like when we scale up this system?

Jane: The authors do mention that while there is a computational cost associated with each iteration, they found that it’s still more efficient than some other methods for achieving accuracy delta.

Lu: And they point out that because the particle implementation uses a closed-form expression for the witness function, you don't have to spend time learning a score function at every single step.

Tom: So we’re looking at a method that is theoretically very solid across continuous and discrete time, handles both generation and sampling with better convergence guarantees than before, but we still have the cost of that particle system to worry about.

Jane: It seems like this is a really versatile tool because it bridges the gap between theory that handles complex distributions and practical needs for stable training in generative AI.

Lu: They’ve shown that this Sobolev Regularized MMD Gradient Flow provides a strong framework for tackling these convergence issues, even if it stops short on the final measurement metric they use.

Meng: If we can find a way to make those particle computations cheaper, maybe by using those closed-form expressions they mentioned, that would be the next big hurdle for us at scale.

Tom: Exactly. So we’ve seen how this Sobolev Regularized Maximum Mean Discrepancy Gradient Flow works, and it seems like a solid framework for tackling both generative modeling and sampling problems with better theoretical guarantees than before.

Jane: It's a lot of theory, but the results are compelling enough to warrant looking at how to apply these concepts more broadly.

Lu: I think the direction for future work should be focusing on making that particle implementation more computationally efficient, as that’s where the real bottleneck is right now.

Meng: Agreed. We need methods that scale better than cubic complexity if we want to move this from a research paper to something we can actually ship reliably to users.

Chenyang Tian, †Bharath K. Sriperumbudur, ‡Arthur Gretton, ‡Zonghao Chen

Tsinghua University · Pennsylvania State University · University College London · Google DeepMind

cs.LG

Submitted: 2026-05-12

Updated: 2026-11-06

Journal ref: 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

Code: https://github.com/FerdTian/SrMMD

Project page: https://chewisinho.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Sobolev-regularized Maximum Mean Discrepancy (SrMMD) gradient flow is a regularized variant of MMD gradient flow based on a gradient penalty on the witness function that mitigates non-convexity and…

Key concepts

SrMMD Flow
A trajectory defined by a continuity equation that uses the gradient of a witness function (fµ,π) to transport particles. The 'Sobolev regularization' adds a penalty term to this flow, which helps prevent non-convexity and guarantees convergence in both continuous and discrete time settings.
Witness Function
A mathematical function central to the flow that attains its supremum in the definition. The gradient of this function ($ abla f_{\mu,\pi}$) is used as a gradient field to guide the movement of particles, effectively steering them towards the target distribution's characteristics.
Stein Kernels
Special kernels used when sampling from unnormalized targets. They allow the SrMMD flow to achieve explicit convergence rates ($\exp(-2T)$) that are independent of difficult tail conditions like log-Sobolev or Poincaré constants, making sampling faster and more robust.
Hybrid-regularized MMD (HrMMD)
A flow that interpolates between two different regularization methods: the standard L2 norm regularization and the gradient norm penalty. This hybrid approach is empirically shown to achieve better performance in generative modeling tasks compared to using either regularization scheme alone.

Terminology

Summary

Sobolev-regularized Maximum Mean Discrepancy (SrMMD) gradient flow is a regularized variant of MMD gradient flow based on a gradient penalty on the witness function that mitigates non-convexity and yields provable global convergence guarantees in both continuous and discrete time

Proposed Method and Scope

The authors propose the SrMMD flow, a novel regularized variant of MMD/KSD flow based on a gradient norm penalty on the witness function The proposed framework applies both to generative modelling and to sampling from unnormalized targets, the latter through the use of Stein kernels They prove that this gradient regularization yields global convergence guarantees in both continuous- and discrete-time settings, under a source condition on the trajectory

Theoretical Foundations

The SrMMD flow is defined as a trajectory satisfying the continuity equation ∂tµt − ∇ · (µt∇fµt,π) = 0, or equivalently, admitting a characteristic representation dXt = −∇fµt,π(Xt)dt The witness function fµ,π attains the supremum in Definition 3.1 The resulting gradient field ∇fµ,π is then used in SrMMD flow to transport particles

Convergence Guarantees

The SrMMD flow enjoys provably fast convergence in both sampling and generative modelling For sampling, its convergence rate is independent of isoperimetric characteristics of the target distribution π, denoted by β such as log-Sobolev or Poincaré constants The analysis relies on a regularity condition on the difference between kernel mean embeddings Theorem 4.1 shows that the SrMMD flow converges exponentially fast to the target distribution π, as measured either by the MMD (or KSD with a Stein kernel kπ) up to a residual barrier term

Practical Implementation and Analysis

The SrMMD descent scheme is implemented by a system of interacting particles where x(i)s+1 = x(i)s − γ∇fµˆs,π(x(i)s), ∀s ∈ [S] The iteration complexity of SrMMD descent is constrained by 0 1 because of the saturation effect The paper also introduces the Hybrid-regularized MMD (HrMMD) flow, which interpolates between L2 and gradient norm regularizations

Empirical Validation

The effectiveness of the proposed flow is empirically verified on a broad range of tasks in both generative modelling and sampling Experiments show that SrMMD flow substantially outperforms the vanilla MMD flow, consistent with the theory that SrMMD enjoys exponential convergence The performance of SrMMD flow is comparable to that of DrMMD flow, suggesting that either L2 or gradient regularization improves convergence The hybrid HrMMD flow achieves the best performance on Gaussian mixtures

Sampling Performance

In sampling from an unnormalized target π, most of the existing convergence rates are established under various tail conditions on π, such as Poincaré (sub-exponential tail) or logSobolev (sub-Gaussian tail) By contrast, Theorem 4.1 relies only on conditions on π that are implicitly encoded through the Stein kernel kπ in Assumptions 1 and 2 The SrMMD flow enjoys an explicit decay rate exp(−2T) independent of the tail conditions of π The particle implementation of SrMMD descent is well-posed under Assumptions 1 and 2, leading to an iteration complexity of O(δ −1/α) The final KSD and W2 values decrease as the particle number N increases, indicating that our particle implementation benefits from a more accurate empirical approximation of the population descent scheme

Conclusion

We propose SrMMD flow, a new flow with a gradient penalty to the MMD/KSD witness function for globally convergent continuous- and discrete-time schemes, applicable for both generative modelling and sampling A primary limitation of SrMMD flow is its computational cost: each iteration requires inverting an N d × N d matrix, leading to a cost of O(N3d 3) A next step is therefore to assess at larger-scale tasks where SVGD has proved effective, e.g., earthquake modelling

References

[AGS05] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows: In Metric Spaces and in the Space of Probability Measures. p. 2

Where the paper says it

(3) involves both a forward Euler discretization in time and a finite-particle discretization in space. MMD and KSD gradient flow Maximum mean discrepancy (MMD) [GBR+12] between two probability measures is defined as the RKHS norm of the difference between their corresponding kernel mean embeddings [MFSS17], i.

Springer Science & Business Media, 2005

[AHS23] Fabian Altekrüger, Johannes Hertrich, and Gabriele Steidl. Neural Wasserstein gradient flows for discrepancies with Riesz kernels. p. 3

Where the paper says it

These conditions exclude, however, the negative distance kernel k(x, y) = −∥x − y∥ [HWAH24, HHA+24], Riesz kernels k(x, y) = −∥x − y∥ r for r ∈ (0, 2) [AHS23, HGBS24] and the Coulomb kernels k(x, y) = (d−2)∥x−y∥ −(d−2) [BV25] due to the singular behaviour at the origin. Although the MMD gradient flows of these kernels exhibit strong empirical performances in practice, establishing rigorous well-posedness remains an open problem; existing results are essentially restricted to the torus [BV25, CCCFR26] or the real line R 1 [DSB+26].

In International Conference on Machine Learning, pages 664–690. p. 1
Where the paper says it

In particular, the convergence rate measured in MMD/KSD is independent of isoperimetric properties of the target distribution. We show that SrMMD flow admits a tractable finite-sample implementation which, in contrast to diffusion-based approaches [SE19, HJA20, HDH+24], does not require learning a score function at each iteration.

PMLR, 2023

[AKSG19] Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow. p. 0

Where the paper says it

†Equal contribution. May 13, 2026 Abstract We propose Sobolev-regularized Maximum Mean Discrepancy (SrMMD) gradient flow, a regularized variant of maximum mean discrepancy (MMD) gradient flow based on a gradient penalty on the witness function.

In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett editors Advances in Neural Information Processing Systems volume 32 Curran Associates Inc. p. 0
Where the paper says it

The effectiveness of the proposed flow is empirically verified on a broad range of tasks in both generative modelling and sampling. 1 Introduction Sampling and generative modelling, while often studied as distinct tasks, can be cast within a common framework of optimization over the space of probability measures P when only partial information about the target distribution π is available.

, 2019

[ASBG18] Michael Arbel, Danica J Sutherland, Mikołaj Bińkowski, and Arthur Gretton. On gradient regularizers for MMD GANs. p. 0

Where the paper says it

Another example is provided by Wasserstein gradient flows, such as the maximum mean discrepancy (MMD) flow [AKSG19] and the kernel Stein discrepancy (KSD) flow [KAFMA21]: the former is studied in generative modelling, while the latter is primarily used for sampling. In generative adversarial networks (GANs), a class of generative models that has been largely superseded by diffusion models, gradient regularization of the witness/critic function has proved crucial in practice for generating high-quality images [RLNH17, GAA+17, ASBG18].

In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R Garnett editors Advances in Neural Information Processing Systems volume 31 Curran Associates Inc. p. 0
Where the paper says it

The effectiveness of the proposed flow is empirically verified on a broad range of tasks in both generative modelling and sampling. 1 Introduction Sampling and generative modelling, while often studied as distinct tasks, can be cast within a common framework of optimization over the space of probability measures P when only partial information about the target distribution π is available.

, 2018

[BBG25] Sayan Banerjee, Krishna Balasubramanian, and Promit Ghosal. Improved finiteparticle convergence rates for Stein variational gradient descent. p. 6

Where the paper says it

To reach final error MMD2 (µS, π) ≤ δ (or equivalently KSD2 (µS, π) ≤ δ), it suffices to pick a step size γ = O(δ 1/α) and iteration number S = O(δ −1/α log(δ −1 )). Among existing results, the one closest in spirit to ours is Stein variational gradient descent (SVGD) [LW16], which has recently 7

In The Thirteenth International Conference on Learning Representations, 2025

[BFH+18] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas Skye WandermanMilne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs 2018

[BGL14] Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffusion Operators volume 103 Springer, 2014

[BPR07] Frank Bauer, Sergei Pereverzev, and Lorenzo Rosasco. On regularization algorithms in learning theory Journal of Complexity 23(1):52–72, 2007

[BF21] Benoît Bonnet and Hélène Frankowska. Differential inclusions in Wasserstein spaces: the Cauchy-Lipschitz framework Journal of Differential Equations, 271:594–637, 2021

[BFH+18] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke Jake VanderPlas Skye WandermanMilne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs 2018

[BGL14] Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffusion Operators volume 103 Springer, 2014

[BPR07] Frank Bauer, Sergei Pereverzev, and Lorenzo Rosasco. On regularization algorithms in learning theory Journal of Complexity 23(1):52–72, 2007

[BSAG18] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. p. 0

Where the paper says it

Sobolev Regularized MMD Gradient Flow Chenyang Tian1,† , Bharath K. Sriperumbudur2 , Arthur Gretton3,4 , Zonghao Chen3,† 1Tsinghua University, 2Pennsylvania State University, 3University College London, 4 Google Deepmind.

Demystifying MMD GANs. p. 0
Where the paper says it

Another example is provided by Wasserstein gradient flows, such as the maximum mean discrepancy (MMD) flow [AKSG19] and the kernel Stein discrepancy (KSD) flow [KAFMA21]: the former is studied in generative modelling, while the latter is primarily used for sampling. In generative adversarial networks (GANs), a class of generative models that has been largely superseded by diffusion models, gradient regularization of the witness/critic function has proved crucial in practice for generating high-quality images [RLNH17, GAA+17, ASBG18].

In International Conference on Learning Representations, 2018

[BV25] Siwan Boufadène and François-Xavier Vialard. On the global convergence of Wasserstein gradient flow of the Coulomb discrepancy SIAM Journal on Mathematical Analysis, 57(4):4556–4587, 2025

[CBD+25] Miguel Corrales, Sean Berti, Bertrand Denel, Paul Williamson, Mattia Aleardi, and Matteo Ravasi. Annealed Stein variational gradient descent for improved uncertainty estimation in full-waveform inversion Geophysical Journal International, 241(2):1088–1113, 2025

[CCCFR26] Lénaïc Chizat, Maria Colombo, Roberto Colombo, and Xavier Fernández-Real. Quantitative convergence of Wasserstein gradient flows of kernel mean discrepancies arXiv preprint arXiv:2603. p. 0

Where the paper says it

For instance, diffusion models, which have proved successful in generative modelling [SSDK+21], have recently motivated the development of new diffusion-based sampling methods [HDH+24, VCK25]. Another example is provided by Wasserstein gradient flows, such as the maximum mean discrepancy (MMD) flow [AKSG19] and the kernel Stein discrepancy (KSD) flow [KAFMA21]: the former is studied in generative modelling, while the latter is primarily used for sampling.

01977, 2026

[CEL+24] Sinho Chewi, Murat A. Erdogdu, Mufan Li, Ruoqi Shen, and Matthew S. Zhang. Analysis of Langevin Monte Carlo from Poincaré to log-Sobolev Foundations of Computational Mathematics pages 1–51, 2024

[Che24] Sinho Chewi. Log-concave sampling https://chewisinho. p. 3

Where the paper says it

, geodesically convex [V+09]. However, this property typically holds only for f-divergences and a restricted class of log-concave target distributions π [Wib18, Che24].

github.io/, 2024 Book draft

[CKK+25] Zonghao Chen, Toni Karvonen, Heishiro Kanagawa, François-Xavier Briol, and Chris Oates https://arxiv.org/abs/2505.20754, 2025

[CLGL+20] Sinho Chewi, Thibaut Le Gouic, Chen Lu, Tyler Maunu, and Philippe Rigollet. SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence In H. p. 1

Where the paper says it

Wasserstein Gradient Flow Let F : P2(R d ) → R be a functional with a unique minimizer π. In many cases, F is of the form F(·) = D(·∥π), where D is some probability divergence.

Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H Lin editors Advances in Neural Information Processing Systems volume 33 pages 2098–2109 Curran Associates Inc. p. 0
Where the paper says it

The effectiveness of the proposed flow is empirically verified on a broad range of tasks in both generative modelling and sampling. 1 Introduction Sampling and generative modelling, while often studied as distinct tasks, can be cast within a common framework of optimization over the space of probability measures P when only partial information about the target distribution π is available.

, 2020

[CMG+25] Zonghao Chen, Aratrika Mustafi, Pierre Glaser, Anna Korba, Arthur Gretton, and Bharath K Sriperumbudur Journal of Machine Learning Research 26(235):1–77, 2025

[CNB25] Zonghao Chen, Masha Naslidnyk, and Francois-Xavier Briol Nested expectations with kernel quadrature In Forty-second International Conference on Machine Learning, 2025

[CSG16] Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit In International conference on machine learning pages 2606–2615 PMLR, 2016

[CWS10] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence pages 109–116, 2010

[CZ07] Felipe Cucker and Ding Xuan Zhou. Learning Theory: An Approximation Theory Viewpoint volume 24 Cambridge University Press, 2007

[DNS23] Andrew Duncan, Nikolas Nüsken, and Lukasz Szpruch. On the geometry of Stein variational gradient descent Journal of Machine Learning Research 24(56):1–39, 2023

[Dou66] Ronald G. Douglas. On majorization, factorization, and range inclusion of operators on Hilbert space Proceedings of the American Mathematical Society, 17(2):413–415, 1966

[DRSS25] Richard Duong, Nicolaj Rux, Viktor Stein, and Gabriele Steidl. Wasserstein gradient flows of maximum mean discrepancy functionals with distance kernels under Sobolev regularization Philosophical Transactions of the Royal Society A, 383(2298):20240243, 2025

[DSB+26] Richard Duong, Viktor Stein, Robert Beinert, Johannes Hertrich, and Gabriele Steidl. Wasserstein gradient flows of MMD functionals with distance kernel and Cauchy problems on quantile functions ESAIM: Control Optimisation and Calculus of Variations 32:10, 2026

[EG15] Lawrence Evans and Ronald Gariepy. Measure Theory and Fine Properties of Functions Revised Edition CRC Press, 2015

[EHN96] Heinz Werner Engl, Martin Hanke, and Andreas Neubauer. Regularization of Inverse Problems volume 375 Springer Science & Business Media, 1996

[RLNH17] Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization Advances in neural information processing systems volume 30, 2017

[RR07] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines Advances in neural information processing systems volume 20, 2007

[San17] Filippo Santambrogio. Euclidean, metric, and Wasserstein gradient flows: An overview Bulletin of Mathematical Sciences, 7:87–154, 2017

[SC08] Ingo Steinwart and Andreas Christmann. Support Vector Machines Springer Science & Business Media, 2008

[SE19] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution Advances in Neural Information Processing Systems volume 32, 2019

[SNRS25] Viktor Stein, Sebastian Neumayer, Nicolaj Rux, and Gabriele Steidl Wasserstein gradient flows for Moreau envelopes of f-divergences in reproducing kernel Hilbert spaces Analysis and Applications pages 1–45, 2025

[SSDK+21] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole Score-based generative modeling through stochastic differential equations In International Conference on Learning Representations volume 30 2021

[Tes12] Gerald Teschl. Ordinary differential equations and dynamical systems volume 140 American Mathematical Soc. p. 3

Where the paper says it

As a consequence, beyond the globally bounded kernels commonly used in the Wasserstein gradient flow literature—such as Gaussian and Matérn kernels—our Assumptions 1 and 2 also accommodate certain unbounded kernels, including the polynomial kernel k(x, y) = (x ⊤y + 1)2 , and the Stein kernel kπ under some conditions on π [KAFMA21]; see the next section for details. Extension to kernels with even faster growth would be challenging; indeed, classical theory of differential equations shows that linear-growth conditions on the vector field are crucial in ruling out finite-time blow-up [Tes12, Problem 2.

, 2012

[V+09] Cédric Villani et al. Optimal Transport: Old and New volume 338 Springer, 2009

[VCK25] Adrien Vacher, Omar Chehab, and Anna Korba Sampling from multi-modal distributions with polynomial query complexity in fixed dimension via reverse diffusion In The Thirty-ninth Annual Conference on Neural Information Processing Systems volume 33 pages 6840–6851 Curran Associates Inc., 2025

[VW19] Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted Langevin algorithm: Isoperimetry suffices In H. p. 1

Where the paper says it

distribution π, denoted by β, such as log-Sobolev or Poincaré constants [BGL14]. In particular, the Unadjusted Langevin Algorithm (ULA) requires O(δ −1β) iterations to achieve error δ > 0 [CEL+24, VW19], whereas SrMMD flow achieves the same accuracy in O(δ −1/α) iterations, where α characterizes the regularity of the trajectory.

Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R Garnett editors Advances in Neural Information Processing Systems volume 32 Curran Associates Inc. p. 0
Where the paper says it

The effectiveness of the proposed flow is empirically verified on a broad range of tasks in both generative modelling and sampling. 1 Introduction Sampling and generative modelling, while often studied as distinct tasks, can be cast within a common framework of optimization over the space of probability measures P when only partial information about the target distribution π is available.

, 2019

[Wen04] Holger Wendland. Scattered Data Approximation volume 17 Cambridge university press, 2004

[Wib18] Andre Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet editors Conference on Learning Theory pages 2093–3027 PMLR, 2018

[WS00] Christopher Williams and Matthias Seeger.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Sobolev regularized Maximum Mean Discrepancy gradient flow (SrMMD flow) can be used for both generative modeling and sampling from unnormalized targets using Stein kernels, unlike previous works where a gradient flow was suitable for only one task. This allows AI systems to perform both generative tasks and data sampling efficiently within a single optimization framework.

  2. The SrMMD flow provides provable global convergence guarantees in both continuous and discrete time settings, with convergence rates independent of the isoperimetric characteristics of the target distribution pi. This means the system will converge reliably even for multi-modal targets where those constants might scale exponentially with mode separation.

  3. The flow admits a tractable finite-sample implementation that does not require learning a score function at each iteration, unlike diffusion-based approaches. This reduces computational overhead during training and deployment of generative models and sampling algorithms.

  4. For sampling from an unnormalized target distribution, the flow can utilize Stein kernels to achieve convergence rates in O(delta-1/alpha) iterations, which is faster than the O(delta-1/beta) iterations required by traditional Unadjusted Langevin Algorithm (ULA). This significantly speeds up the process of generating samples from complex targets.

  5. The Hybrid-regularized Maximum Mean Discrepancy (HrMMD) flow, which interpolates between L2 norm and gradient norm regularization, can be used in generative modeling to improve empirical performance compared to standard MMD flow or DrMMD flow in GANs. This suggests that combining different regularization schemes can lead to better results for image generation tasks.

  6. The SrMMD descent scheme has an iteration complexity of O(delta-1/alpha log(delta-1)) for achieving accuracy delta, which is generally faster than the O(delta-1/beta) complexity of ULA and comparable to other methods like Stein variational gradient descent (SVGD). This provides a theoretical justification for why combining L2 and gradient regularization improves empirical performance in MMD GANs.

  7. The SrMMD flow's convergence analysis relies on a regularity condition on the difference between kernel mean embeddings rather than isoperimetric assumptions, making it applicable to a broader range of target distributions. This removes the restriction that convergence requires specific tail conditions like log-Sobolev or Poincaré constants for sampling, which are exponential in mode separation.

  8. The particle implementation of SrMMD descent can be efficiently computed by using a closed-form expression for the witness function, which involves matrix inversions of operators derived from kernel derivatives. This tractability allows for practical implementation in high-dimensional settings like student-teacher networks and color transfer tasks without needing complex score function learning at every step.

Abstract

We analyze Sobolev descent, a regularized variant of maximum mean discrepancy (MMD) gradient flow based on a gradient penalty on the MMD witness function. This regularization mitigates the non-convexity of the MMD objective, and yields provable global convergence guarantees in MMD, in both continuous and discrete time. A surprising appeal is that our convergence analysis does not rely on isoperimetric assumptions on the target distribution. Instead, it is based on a regularity condition on the difference between kernel mean embeddings. We extend Sobolev descent such that it is applicable in both sampling (from an unnormalized target distribution)---using Stein kernels---and generative modeling settings, unlike previous works, where a gradient flow is suitable for only generative modeling or sampling but not both. We empirically validate the method on a broad range of tasks across both settings.

Sources

Related papers