Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors

arXiv:2606.06179 · stat.ML, cs.LG · Submitted 2026-06-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Diffusion Models Observe Only Gradients".

Jane: Here's the gist of the paper. Score-based diffusion models are typically trained by minimizing the L2 score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the learned and target distributions The L2 score error is not the right intrinsic measure of marginal distributional quality because a learned diffusion model can incur arbitrarily large L2 score error while perfectly matching the target distribution By decomposing score errors into a gradient and a solenoidal component (a Helmholtz-Hodge decomposition), only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible The authors prove that no monotone function of the L2 score error can uniformly lower-bound any meaningful distributional divergence between pˆ and p⋆ They derive a new upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error, tightening the standard Girsanov bound for generic score networks They propose a tractable estimator of the gradient component of the score error via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L2 error Geometric Decomposition of Score Errors The geometric reason behind observation (1) is that only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible to the marginal dynamics Decomposing the error vector field into gradient and solenoidal components reveals that this weighted divergence filters out the solenoidal component of the error field, making it structurally invisible to the marginals The Helmholtz-Hodge decomposition is used to express the error as es = ΠGs es + ΠG⊥s es, where ΠGs (resp.

Tom: ΠG⊥s) denotes the orthogonal projection operator on Gs (resp.

Jane: on G⊥s) The component ΠG⊥s is naturally divergence-free and thus structurally invisible at the reference density since ∇·(p⋆es) = ∇·p⋆sΠGs es Derivation of the Improved Bound The authors derive a new upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error, denoted as ΠGs es This new bound is strictly tighter than the standard one in (Full L2) whenever the solenoidal component of errors is nonzero, motivating a better diagnostic of learning quality based on the geometry of diffusion The improved upper bound is given by KL(p⋆t0∥pˆt0) ≤ 1/2 Z T t0 σ 2 s ΠGs es 2 L2(p⋆s)ds This bound identifies the component of the score error that is intrinsic to the learned marginal dynamics by projecting onto gradient fields Practical Estimation Procedure The authors introduce a practical procedure to estimate the gradient component of score estimation errors via an H−1 dual variational identity, which avoids explicit projections and is computationally tractable The norm of the gradient component is characterized by∥ΠGs es 2 L2(p⋆s) = sup φ∈C∞c(Rd) - 2 Z Rd φ ∇· (p⋆es) dx - Z Rd ∇φ 2 L2(p⋆s) dx This leads to an estimator Lbobs:= sup ψ Jb(ψ), which is used as a diagnostic rather than a training objective Empirical Validation and Interpretation Empirically, Figure 2 shows that the gradient component Egrad:= Es xs ΠGs es 2 L2(p⋆s) is uniformly more correlated with sample quality than the full error Efull:= Es xs es 2 L2(p⋆s) The new bound (16) is expected to significantly improve on the standard one (15) when the gradient components of the errors have small norms compared to the solenoidal components In contrast, Figure 3 illustrates that after an initial phase in which the observable gradient component is reduced, the remaining score error is dominated by the solenoidal component, creating a persistent gap between the classical full-error bound and our bound Connection to Optimal Transport Geometry The variational characterization of∥ΠGs es 2 L2(p⋆s) has an interpretation in the dynamic formulation of optimal transport where ΠGs es is the component of the score error that lies in the tangent space at p⋆s of the Otto manifold This tangent norm is equivalent to the H−1(p⋆s) norm ∇ · (p⋆es)H−1(p⋆s) of (19), which is also the Otto metric tensor The solenoidal component of es is invisible because it changes the velocity representation without changing the tangent vector ζs, i.e., without changing the infinitesimal motion of probability mass Girsanov-Based Recovery The upper bound in Theorem 3.2 can also be established using Girsanov’s Theorem by applying the Helmholtz-Hodge decomposition to the stochastic integral of errors and its quadratic variation This leads to KL(p⋆t0∥pˆt0) ≤ 1/2 Z T t0 σ 2 s ΠGs es 2 L2(p⋆s)ds This confirms that the standard bound’s looseness is the cost of operating on path-space rather than marginal-space dynamics Experimental Findings The gradient component of the score error decreases by more than three orders of magnitude during training while the solenoidal component remains almost flat and dominates the total error throughout The Spearman rank correlation between feature-FID and ΠGe 2 L2 is in

0.93, 0.99: , whereas the full-error correlation ranges as low as 0.65 This indicates that ΠGe 2 L2 is not only a better but also a substantially more reliable proxy for sample quality than the full L2 score error The inobservability gap is faithful to theoretical predictions independently of K The decomposition shows that purely solenoidal errors may have arbitrarily large L2 norm while leaving all marginals unchanged, and that no distributional discrepancy can be uniformly lower-bounded by the full score error alone The new bound (16) is expected to significantly improve on the standard one (15) when the gradient components of the errors have small norms compared to the solenoidal components Future work suggests extending this diagnostic into a training objective or making it guide the score network towards observable directions that translate into better learning and better capacity The decomposition admits a natural interpretation in Wasserstein-2 geometry where ΠGs es is the component of the score error that lies in the tangent space at p⋆s of the Otto manifold The estimator Lbobs can be extended into a training objective or to make the critic guide the score network towards observable directions that translate into better learning and better capacity The solenoidal component of es is invisible because it changes the velocity representation without changing the tangent vector ζs, i.e., without changing the infinitesimal motion of probability mass References

1: Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré.

Tom: Gradient Flows in Metric Spaces and in the Space of Probability Measures.

Title and authors: Tom: So, we're looking at this paper, "Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors". It tackles how we measure if a diffusion model is actually learning the right distribution.

Jane: Right, it's about how the standard way we calculate the score matching error, that L2 score error, isn't actually telling us everything we need to know about the quality of what the AI has learned.

Lu: Exactly. They start by pointing out that a model can have a really big L2 score error but still perfectly match the target distribution they are supposed to be modeling.

Meng: But what does that mean for us when we're actually shipping these models? If a huge L2 error doesn't mean bad results, how do we know what to look at?

Tom: Well, this paper breaks down that score error into two parts: a gradient component and a solenoidal component, using something called the Helmholtz-Hodge decomposition.

Jane: That decomposition is key because it shows that only the gradient part actually influences the way the probability distribution changes over time in our marginal Fokker-Planck dynamics.

Lu: Yeah, which means that this solenoidal part of error is structurally invisible to those marginal dynamics. It doesn't affect how we see the final distribution on average.

Meng: Invisible sounds like it doesn't matter when we're just checking if the model looks right on a test set, but what about when we deploy it and things get messy?

Tom: That’s where they get interesting. They prove that no monotone function of the total L2 score error can uniformly lower-bound any meaningful difference between the learned and target distributions.

Jane: So, even if you minimize that big L2 number, you can't guarantee it’s getting better at what actually matters for sampling from the distribution.

Lu: That impossibility result sets up the need for a new way to look at things geometrically. They use this insight to derive a new upper bound on the Kullback-Leibler divergence, which is how we measure distribution distance.

Meng: A new bound that depends only on what's actually observable, then? That sounds like something we could actually monitor during training instead of just looking at the final loss number.

Tom: Precisely. They derive a new upper bound on the Kullback-Leibler divergence between the learned and target models that only depends on the observable gradient component of the error, which they call PiGs es.

Jane: The paper says this is strictly tighter than the standard Girsanov bound whenever that solenoidal component of error isn't zero, which is a big deal because it gives us a better diagnostic tool.

Lu: They then propose a way to estimate that gradient component, PiGs es, using something called an H minus one dual variational identity. It’s designed to be tractable and avoids those explicit projections they talked about earlier.

Title and authors: Meng: A tractable estimator is good because we don't want a complicated calculation running during inference when we have millions of samples. What does this estimator actually measure?

Tom: The norm of that gradient component, PiGs es squared, is characterized by the supremum over a function space C infinity c Rd of minus two Z Rd times the divergence between the target density and the measurement record, plus a term involving the gradient of that function.

Jane: So, it’s basically an estimator for how much those observable gradients are pushing us away from what we want, rather than just counting total noise.

Lu: And empirically, they show that this observable gradient component is uniformly more correlated with actual sample quality metrics like Fréchet Inception Distance than the full L2 error does.

Meng: That's a huge win for deployment because if we can use this observable quantity to check quality, we don't have to wait for the final L2 loss to tell us if the model is good.

Tom: And they show that during training, this gradient component drops by more than three orders of magnitude while the solenoidal part stays flat and dominates the total error throughout.

Jane: So it means we can use this method to track when the model is actually getting better at what matters for generating realistic samples, not just minimizing a number that's mathematically convenient.

Lu: The connection to optimal transport geometry is also interesting; they say that PiGs es corresponds to the component of the score error that lies in the tangent space at your target distribution, which relates to the Otto metric tensor.

Meng: That means we are measuring something about how close our learned model is in terms of movement on a curved space, rather than just a flat L2 distance.

Tom: And they connect this to Girsanov’s Theorem by showing that the standard bound's looseness comes from operating on path space instead of marginal space dynamics.

Jane: It clarifies why the standard bound is loose; it’s because it ignores the geometry of how probability moves, focusing too much on the path taken rather than where we end up.

Lu: The practical procedure they give for estimating that gradient component, Lbobs, is used as a diagnostic tool rather than a training objective itself. It's what you use to check the geometry.

Meng: I’m interested in what happens if we actually try to use this estimator to guide the model during training instead of just checking it at the end. What would that look like?

Tom: That’s where future work suggests extending this diagnostic into a training objective or making it guide the score network towards observable directions that translate into better learning and better capacity.

Title and authors: Jane: It sounds like a path forward where we use this geometric understanding to steer the AI toward what actually improves the final output quality.

Lu: The decomposition itself is powerful because it gives us a way to separate what's purely noise from what’s actually steering the distribution in a meaningful way.

Meng: I just want something that scales well, but if this gradient component is so much more informative than the full L2 error, then maybe we can afford to spend a little more computation tracking it.

Tom: Yeah, it seems like using PiGs es as our primary metric for learning quality could really change how we set up our loss functions in the long run.

Jane: It shifts our focus from just minimizing a score matching number to understanding the actual geometry of the learned distribution.

Lu: And that geometric interpretation, linking it to the tangent space at p star s, gives us a deep reason why this decomposition is so geometrically sound.

Meng: Okay, so we have a way to check quality better using this gradient component, and it's tied into geometry and optimal transport. That’s pretty concrete for production considerations.

Tom: It really is; the empirical validation shows that the correlation between PiGs es squared and sample quality is in the range of zero point nine three to zero point nine nine, which is way higher than the full error's correlation of about zero point six five.

Jane: So, if you're building a system, you should probably be looking at that observable gradient component first before worrying about the total L2 score error.

Lu: And remember the core idea: only the gradient component enters our marginal dynamics, which is why ignoring it in favor of the full L2 error is like ignoring how a car actually steers.

Meng: It's interesting because we are seeing that as training progresses, this observable gradient component shrinks dramatically while the solenoidal part stays stubbornly large.

Tom: That suggests that during initial training, the model might be learning some noisy directions first, and then it settles onto the actual gradient directions that lead to better results.

Jane: So we can watch how PiGs es evolves over time as a direct indicator of whether the model is successfully aligning itself with the true data distribution.

Lu: That’s what they are setting up for us; a way to diagnose learning quality based on this geometric decomposition of score errors.

Meng: It gives us a concrete quantity to monitor that correlates strongly with real-world performance, which is exactly what we need when dealing with scaling and deployment issues.

Tom: And ultimately, this paper on "Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors" shows us that the geometry of the error field is more informative than just its total magnitude.

The paper's summary: Tom: So we're diving into this paper called "Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors". Basically, they're saying that the standard way we measure how good a diffusion model is—using the L2 score error—isn't actually telling us everything about its quality.

Jane: That makes sense. They look at the math and say that when you break down that total score error into two pieces, a gradient part and a solenoidal part, only the gradient part actually influences how the probability distribution changes over time in our core dynamics.

Lu: Exactly. The solenoidal component is structurally invisible to those marginal dynamics; it doesn't affect the final average distribution we care about.

Meng: So even if you have a massive L2 score error number, but that error is mostly just that invisible solenoidal stuff, the model might still be doing fine in practice?

Tom: Right. The paper proves that no simple measure of the total L2 score error can always give you a guaranteed upper limit on how much worse your learned model is compared to the target distribution.

Jane: They derive a new upper bound on how far off your learned model is, and this new bound only depends on that observable gradient component we talked about earlier.

Lu: The math shows that this new bound is strictly tighter than the standard one when there's any solenoidal error present, which means it’s a much better diagnostic tool for learning quality.

Tom: And they give us a practical way to estimate that gradient component using something called an H minus one dual variational identity. It’s designed to be computationally simple and avoids those tricky projections the authors mentioned earlier.

Jane: So this gives you an actual quantity you can watch during training, instead of just looking at the final loss number after everything is done.

Lu: And empirically, they show that this observable gradient component correlates way better with real sample quality metrics like Fréchet Inception Distance than the full L2 error does.

Meng: That’s what matters for production. If we can use this estimate to check quality, we don't have to wait for the final score matching number to tell us if the model is actually learning something useful.

Tom: It shows that during training, this observable gradient component shrinks by more than three orders of magnitude while that invisible solenoidal part stays stubbornly large and dominates the total error.

Jane: That tells us something about how models learn; they seem to get good at the main directional movements first, and then they focus on those cleaner gradient directions later.

Lu: The connection to optimal transport geometry is also interesting; this observable gradient component is linked to the tangent space at your target distribution, which relates directly to the Otto metric tensor.

Meng: So it’s not just a number; it’s telling us about the actual geometry of how well the model's learned movement aligns with the true data manifold.

Tom: It really puts a name on why standard bounds are loose; they're operating on path space dynamics instead of what actually matters for marginal distribution changes.

Jane: This geometric view gives us a much clearer way to think about what’s driving the model toward a good result versus just random noise in the error field.

The paper's improvements: Tom: So we've seen how they broke down that score error into gradient and solenoidal parts, but now they’re showing us what to actually do with that information. The paper suggests a new way to use this geometric understanding for better training objectives and diagnostics.

Jane: That's right. Instead of just accepting the total L2 error as the only metric, they propose using that observable gradient component to guide the model during training itself.

Lu: They suggest extending that estimator we talked about earlier into a training objective, which means you can make the AI actively steer toward directions that translate into better sample quality instead of just passively minimizing a loss number.

Meng: If we can use this to guide it, then we could potentially stop relying on purely empirical benchmarks and start using something that actually reflects where the model is learning correctly on real data.

Tom: And they show that this gradient component, PiGs es squared, is empirically much more reliable for predicting things like Fréchet Inception Distance than the full L2 error does. It’s a much better proxy for quality.

Jane: That means when you're debugging a model that’s failing on real samples, you can look at this gradient norm and get a clearer idea of what's going wrong geometrically.

Lu: The connection to optimal transport geometry is powerful here; it frames the error component as being in the tangent space at your target distribution, which links the score error directly to how probability moves on that curved space.

Meng: So we’re not just measuring distance in a flat space anymore; we’re looking at how close our learned movement is to where it should be on that specific geometric surface.

Tom: They also showed this method works really well even when the solenoidal component is large, because their new bound only depends on the gradient part, which makes it robust.

Jane: That robustness is important for deployment because it means we don't have to worry as much about a huge amount of noise in the error field messing up our diagnostic signal.

Lu: The authors mention that this framework has a natural interpretation in Wasserstein two geometry, confirming that the gradient component is precisely what drives the dynamics toward convergence on the target manifold.

Meng: What does this mean for running things at scale? If we can use this geometric insight to guide training, we might be able to build systems that are more stable and less prone to catastrophic failure when deployed in a real environment.

Tom: Exactly. It shifts our focus from just minimizing a score matching number to understanding the actual geometry of the learned distribution, which is what matters for reliable AI.

Conclusion: Tom: So to wrap up, "Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors" shows that we should stop looking at just the total L2 score error when evaluating diffusion models.

Jane: That’s right. The main implication is that by focusing on the observable gradient component of the error, we get a much tighter, more reliable bound on how far off our learned model is from the true target distribution.

Lu: They’ve provided a solid diagnostic tool using an H minus one dual variational identity to estimate that gradient component without needing those complicated explicit projections.

Meng: For us in production, this means we can build better monitoring systems that track learning quality based on something empirically correlated with real sample performance, not just some abstract number.

Tom: It’s about using the geometry of the error field—the gradient versus the solenoidal part—to separate what actually steers the distribution from what’s just noise in the calculation.

Jane: And they proved this decomposition is geometrically sound, linking that observable gradient to a specific tangent space at your target distribution.

Lu: It ties directly into optimal transport geometry, showing that this component is equivalent to the Otto metric tensor, which gives a deep reason why we should focus on it.

Meng: That level of geometric insight could change how we design our training schedules and loss functions if we can actually integrate this gradient norm into the objective function.

Tom: It’s a significant shift from just minimizing a single scalar error to understanding the underlying dynamics of how probability mass moves through space.

Jane: And they explicitly state that purely solenoidal errors can have huge norms while leaving all marginal distributions completely unchanged, which explains why the standard L2 error fails in those cases.

Lu: It suggests we need to be more careful about what we are optimizing for when we only look at the total L2 error across different model architectures.

Meng: So, for anyone shipping these models, keep an eye out for this gradient norm as a way to check if the model is actually learning directions that matter.

Tom: That’s it for this paper on "Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors". It really highlights how geometry matters in diffusion modeling.

Jane: Next up, we’re looking at something totally different—a paper called LCS-SLAM that deals with three dee Gaussian SLAM and how visibility affects visual mapping.

Na­l B. Khelifa, Richard E. Turner, Ramji Venkataramanan

University of Cambridge

stat.ML, cs.LG

Submitted: 2026-06-04

Updated: 2026-11-04

Comments: Accepted at NeurIPS 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Score-based diffusion models are typically trained by minimizing the L2 score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the…

Key concepts

Helmholtz-Hodge Decomposition
This mathematical tool splits any vector field (like the score error) into two parts: a gradient part and a solenoidal part. The key insight is that only the gradient component influences how the probability distribution changes over time, while the solenoidal component is structurally invisible to these marginal dynamics.
Gradient Component ($\Pi G_s e_s$)
This is the observable part of a score error vector field. It represents directions in the space that actually affect how the probability distribution evolves. Because it's observable, it can be used as a reliable proxy to measure how well the model is learning and matching the target distribution.
Solenoidal Component ($\Pi G_{\perp s} e_s$)
This is the part of the error that is divergence-free. Geometrically, this component changes how probability mass moves (velocity representation) but does not change where it is located. Therefore, it is structurally invisible to the marginal Fokker-Planck dynamics and can have a large L2 norm without affecting distribution quality.

Terminology

Summary

Score-based diffusion models are typically trained by minimizing the L2 score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the learned and target distributions The L2 score error is not the right intrinsic measure of marginal distributional quality because a learned diffusion model can incur arbitrarily large L2 score error while perfectly matching the target distribution By decomposing score errors into a gradient and a solenoidal component (a Helmholtz-Hodge decomposition), only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible The authors prove that no monotone function of the L2 score error can uniformly lower-bound any meaningful distributional divergence between pˆ and p⋆ They derive a new upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error, tightening the standard Girsanov bound for generic score networks They propose a tractable estimator of the gradient component of the score error via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L2 error

Geometric Decomposition of Score Errors

The geometric reason behind observation (1) is that only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible to the marginal dynamics Decomposing the error vector field into gradient and solenoidal components reveals that this weighted divergence filters out the solenoidal component of the error field, making it structurally invisible to the marginals The Helmholtz-Hodge decomposition is used to express the error as es = ΠGs es + ΠG⊥s es, where ΠGs (resp. ΠG⊥s) denotes the orthogonal projection operator on Gs (resp. p. 4

Where the paper says it

Expressing the error using this orthogonal decomposition yields the weighted Helmholtz–Hodge decomposition [8]: es = ΠGs es + ΠG⊥s es ∈ L 2 (p ⋆ s ; R d ) = Gs ⊕ G⊥ s (12) where ΠGs (resp. ΠG⊥s ) denotes the orthogonal projection operator on Gs (resp.

on G⊥s) The component ΠG⊥s is naturally divergence-free and thus structurally invisible at the reference density since ∇·(p⋆es) = ∇·p⋆sΠGs es

Derivation of the Improved Bound

The authors derive a new upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error, denoted as ΠGs es This new bound is strictly tighter than the standard one in (Full L2) whenever the solenoidal component of errors is nonzero, motivating a better diagnostic of learning quality based on the geometry of diffusion The improved upper bound is given by KL(p⋆t0∥pˆt0) ≤ 1/2 Z T t0 σ 2 s ΠGs es 2 L2(p⋆s)ds This bound identifies the component of the score error that is intrinsic to the learned marginal dynamics by projecting onto gradient fields

Practical Estimation Procedure

The authors introduce a practical procedure to estimate the gradient component of score estimation errors via an H−1 dual variational identity, which avoids explicit projections and is computationally tractable The norm of the gradient component is characterized by∥ΠGs es 2 L2(p⋆s) = sup φ∈C∞c(Rd) - 2 Z Rd φ ∇· (p⋆es) dx - Z Rd ∇φ 2 L2(p⋆s) dx This leads to an estimator Lbobs:= sup ψ Jb(ψ), which is used as a diagnostic rather than a training objective

Empirical Validation and Interpretation

Empirically, Figure 2 shows that the gradient component Egrad:= Es xs ΠGs es 2 L2(p⋆s) is uniformly more correlated with sample quality than the full error Efull:= Es xs es 2 L2(p⋆s) The new bound (16) is expected to significantly improve on the standard one (15) when the gradient components of the errors have small norms compared to the solenoidal components In contrast, Figure 3 illustrates that after an initial phase in which the observable gradient component is reduced, the remaining score error is dominated by the solenoidal component, creating a persistent gap between the classical full-error bound and our bound

Connection to Optimal Transport Geometry

The variational characterization of∥ΠGs es 2 L2(p⋆s) has an interpretation in the dynamic formulation of optimal transport where ΠGs es is the component of the score error that lies in the tangent space at p⋆s of the Otto manifold This tangent norm is equivalent to the H−1(p⋆s) norm ∇ · (p⋆es)H−1(p⋆s) of (19), which is also the Otto metric tensor The solenoidal component of es is invisible because it changes the velocity representation without changing the tangent vector ζs, i.e., without changing the infinitesimal motion of probability mass

Girsanov-Based Recovery

The upper bound in Theorem 3.2 can also be established using Girsanov’s Theorem by applying the Helmholtz-Hodge decomposition to the stochastic integral of errors and its quadratic variation This leads to KL(p⋆t0∥pˆt0) ≤ 1/2 Z T t0 σ 2 s ΠGs es 2 L2(p⋆s)ds This confirms that the standard bound’s looseness is the cost of operating on path-space rather than marginal-space dynamics

Experimental Findings

The gradient component of the score error decreases by more than three orders of magnitude during training while the solenoidal component remains almost flat and dominates the total error throughout The Spearman rank correlation between feature-FID and ΠGe 2 L2 is in [0.93, 0.99], whereas the full-error correlation ranges as low as 0. p. 5

Where the paper says it

01 tiny small full Model capacity 0. 0 S p e a r m a n ½ ½(FID; Efull) ½(FID; Egrad) Figure 2: Spearman correlation between feature-FID [29] and score-error norms (FashionMNIST [76]).

65 This indicates that ΠGe 2 L2 is not only a better but also a substantially more reliable proxy for sample quality than the full L2 score error The inobservability gap is faithful to theoretical predictions independently of K

Conclusion and Future Directions

The decomposition shows that purely solenoidal errors may have arbitrarily large L2 norm while leaving all marginals unchanged, and that no distributional discrepancy can be uniformly lower-bounded by the full score error alone The new bound (16) is expected to significantly improve on the standard one (15) when the gradient components of the errors have small norms compared to the solenoidal components Future work suggests extending this diagnostic into a training objective or making it guide the score network towards observable directions that translate into better learning and better capacity The decomposition admits a natural interpretation in Wasserstein-2 geometry where ΠGs es is the component of the score error that lies in the tangent space at p⋆s of the Otto manifold The estimator Lbobs can be extended into a training objective or to make the critic guide the score network towards observable directions that translate into better learning and better capacity The solenoidal component of es is invisible because it changes the velocity representation without changing the tangent vector ζs, i.e., without changing the infinitesimal motion of probability mass

References

[1] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. p. 2

Where the paper says it

P(R d ) (resp. P2(R d )) denotes the space of Borel probability measures on R d (resp.

Lectures in Mathematics ETH Zürich. Birkhäuser, 2008

[2] Brian DO Anderson. Reverse-time diffusion equation models. p. 3

Where the paper says it

The reverse-time SDE, running backwards from T down to t0, is written in the standard backward-time convention s : T ↓ t0 (so that ds < 0): dYs = [fs (Ys) − σ 2 s ∇x log p ⋆ s (Ys)] ds + σs dB¯ s, YT ∼ p ⋆ T , s: T ↓ t0, (4) where B¯ s is a Brownian motion under the reverse filtration. The marginals of this reverse SDE (4) exactly match those of the forward so that Law(Ys) = p ⋆ s for all s, and they satisfy the backward Fokker-Planck equation (once again, backward in time i.

Stochastic Processes and their Applications, 12(3):313–326, 1982

[7] Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffusion models via stochastic localization. p. 0

Where the paper says it

the difference between the estimated score ˆss and the true score ∇ log p ⋆ s . Inequality (Full L 2 ) underpins much of the current theory of diffusion models [63], from convergence analyses [17, 46, 15, 7], to minimax optimality results [52, 77, 21, 47], and interpretations of training objectives in terms of sample quality [38, 39].

In The Twelfth International Conference on Learning Representations, 2024

[8] Harsh Bhatia, Gregory Norgard, Valerio Pascucci, and Peer-Timo Bremer. The HelmholtzHodge decomposition—a survey. p. 0

Where the paper says it

We show the L 2 score error is not the right intrinsic measure of marginal distributional quality: a learned diffusion model can incur arbitrarily large L 2 score error while perfectly matching the target distribution. By decomposing score errors into a gradient and a solenoidal component (a Helmholtz-Hodge decomposition), we identify the geometric reason behind this: only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible.

IEEE Transactions on Visualization and Computer Graphics, 19(8):1386–1404, 2013

[9] Vladimir I. Bogachev, Nicolai V. Krylov, Michael Röckner, and Stanislav V. Shaposhnikov. Fokker–Planck–Kolmogorov Equations, volume 207 of Mathematical Surveys and Monographs. p. 0

Where the paper says it

We show the L 2 score error is not the right intrinsic measure of marginal distributional quality: a learned diffusion model can incur arbitrarily large L 2 score error while perfectly matching the target distribution. By decomposing score errors into a gradient and a solenoidal component (a Helmholtz-Hodge decomposition), we identify the geometric reason behind this: only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible.

American Mathematical Society, 2015

[10] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. p. 0

Where the paper says it

Third, we give a tractable estimator of the gradient component via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L 2 error. 1 Introduction Denoising Score Matching (DSM) and its variants [33, 34, 72, 65] have driven the success of scorebased diffusion models [62, 30, 64, 66] by providing a scalable score-estimation objective.

Transactions on Machine Learning Research, 2022

[13] Patrick Cattiaux, Giovanni Conforti, Ivan Gentil, and Christian Léonard. Time reversal of diffusion processes under a finite entropy condition. p. 3

Where the paper says it

Reverse-time diffusion. Under regularity conditions on the drift and diffusion coefficients (assumed throughout), the forward process admits a time reversal [28, 2, 13, 23].

Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 59(4):1844 – 1881, 2023

[15] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. p. 3

Where the paper says it

In practice, the true score ∇x log p ⋆ s is unknown, and is estimated via score matching [33, 34] or denoising score matching (DSM) [72]. In modern diffusion and score-based generative modeling, DSM is used to learn a time-dependent score field sθ : R d × [t0, T] → R d (parametrized by θ) of noise-perturbed marginals, by regressing toward the conditional score [64, 66, 65], LDSM(θ) = Es∼U[t0,T] h λ(s)Ex0 Exs|x0 sθ(xs, s) − ∇xs log p ⋆ s (xs | x0) 2 2 i , (6) where λ : [0, T] → R>0 is a positive time weighting.

In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023

[16] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. p. 2

Where the paper says it

Recent studies have found that standard denoising score matching can be inefficient in various settings. When the data concentrate near a low-dimensional manifold, score matching can waste capacity by forcing the score network to fit large directions normal to the data manifold [56, 10].

40th International Conference on Machine Learning, ICML, 2023

[17] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. p. 1

Where the paper says it

Decomposing errors into gradient and solenoidal components (a Helmholtz–Hodge decomposition [8]) is standard in scientific machine learning [57, 27, 24, 69, 48]. In the context of diffusion models, Horvat and Pfister [32] applied the same orthogonal decomposition to the score field to study exact sampling and density estimation; we instead apply it to the score error and use it to obtain a guarantee on sampling quality.

International Conference on Learning Representations, 2023

[18] Sinho Chewi, Jonathan Niles-Weed, and Philippe Rigollet. Statistical optimal transport. p. 7

Where the paper says it

(20) From (18)-(19), we have ∥ΠGs es∥ 2 L2(p⋆ s ) = sup φ∈C∞c (Rd) −2 Z Rd φ ∇ · (p ⋆ ses) dx − Z Rd ∥∇φ∥ 2 2 p ⋆ s dx . (21) These H−1 -norms have a natural geometric interpretation in optimal transport theory (discussed in Appendix C).

Springer, 2025

[19] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. p. 0

Where the paper says it

the difference between the estimated score ˆss and the true score ∇ log p ⋆ s . Inequality (Full L 2 ) underpins much of the current theory of diffusion models [63], from convergence analyses [17, 46, 15, 7], to minimax optimality results [52, 77, 21, 47], and interpretations of training objectives in terms of sample quality [38, 39].

In International Conference on Computer Vision, 2022

[20] C. Doléans-Dade. Quelques applications de la formule de changement de variables pour les semimartingales. p. 2

Where the paper says it

Bold symbols (e. f, Xt, Yt) denote R d -valued random variables or processes.

Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 16(3):181–194, 1970

[25] Igor V Girsanov. On transforming a certain class of stochastic processes by absolutely continuous substitution of measures. p. 2

Where the paper says it

with finite second moment). For µ, ν ∈ P2(R d ), the Kullback-Leibler divergence between these two measures is defined as KL(µ ∥ ν) = R Rd log( µ ν )µdx if µ is absolutely continuous with respect to ν and +∞ otherwise.

Theory of Probability & Its Applications, 5(3):285–301, 1960

[36] Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus. p. 3

Where the paper says it

2 Background Forward diffusion. A forward diffusion on [0, T] (T > 0) is defined by the stochastic differential equation (SDE): dXt = ft (Xt) dt + σt dBt, X0 ∼ p ⋆ , t: 0 → T, (2) where Bt is a standard d-dimensional Brownian motion, ft : R d → R d is the drift, and σt > 0 is the scalar diffusion coefficient.

Springer, 2014

[54] Grigorios A. Pavliotis. Stochastic Processes and Applications: Diffusion Processes, the FokkerPlanck and Langevin Equations. p. 2

Where the paper says it

Bold symbols (e. f, Xt, Yt) denote R d -valued random variables or processes.

Springer, New York, NY, 2014

[62] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. p. 4

Where the paper says it

We can decompose L 2 (p ⋆ s ; R d ) into a gradient subspace Gs and its orthogonal complement G ⊥ s . Expressing the error using this orthogonal decomposition yields the weighted Helmholtz–Hodge decomposition [8]: es = ΠGs es + ΠG⊥s es ∈ L 2 (p ⋆ s ; R d ) = Gs ⊕ G⊥ s (12) where ΠGs (resp.

In Proceedings of the 32nd International Conference on Machine Learning, 2015

[63] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. p. 0

Where the paper says it

uk Ramji Venkataramanan University of Cambridge rv285@cam. uk Abstract Score-based diffusion models are typically trained by minimizing the L 2 score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the learned and target distributions.

In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021

[64] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. p. 3

Where the paper says it

In practice, the true score ∇x log p ⋆ s is unknown, and is estimated via score matching [33, 34] or denoising score matching (DSM) [72]. In modern diffusion and score-based generative modeling, DSM is used to learn a time-dependent score field sθ : R d × [t0, T] → R d (parametrized by θ) of noise-perturbed marginals, by regressing toward the conditional score [64, 66, 65], LDSM(θ) = Es∼U[t0,T] h λ(s)Ex0 Exs|x0 sθ(xs, s) − ∇xs log p ⋆ s (xs | x0) 2 2 i , (6) where λ : [0, T] → R>0 is a positive time weighting.

Advances in Neural Information Processing Systems, 32, 2019

[65] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. p. 0

Where the paper says it

Third, we give a tractable estimator of the gradient component via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L 2 error. 1 Introduction Denoising Score Matching (DSM) and its variants [33, 34, 72, 65] have driven the success of scorebased diffusion models [62, 30, 64, 66] by providing a scalable score-estimation objective.

In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, Proceedings of Machine Learning Research, 2020

[66] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. p. 3

Where the paper says it

In practice, the true score ∇x log p ⋆ s is unknown, and is estimated via score matching [33, 34] or denoising score matching (DSM) [72]. In modern diffusion and score-based generative modeling, DSM is used to learn a time-dependent score field sθ : R d × [t0, T] → R d (parametrized by θ) of noise-perturbed marginals, by regressing toward the conditional score [64, 66, 65], LDSM(θ) = Es∼U[t0,T] h λ(s)Ex0 Exs|x0 sθ(xs, s) − ∇xs log p ⋆ s (xs | x0) 2 2 i , (6) where λ : [0, T] → R>0 is a positive time weighting.

In International Conference on Learning Representations, 2021

[70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. p. 2

Where the paper says it

Our results, as well as those in [32, 73], clarify why this debate produced contradictory findings: the solenoidal component these constraints and penalties act on is, by Theorem 3. 1, structurally invisible to the marginal Fokker–Planck dynamics, so penalizing it need not improve—and removing it cannot harm—the generated marginals.

In Advances in neural information processing systems, 2017

[72] Pascal Vincent. A connection between score matching and denoising autoencoders. p. 0

Where the paper says it

Third, we give a tractable estimator of the gradient component via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L 2 error. 1 Introduction Denoising Score Matching (DSM) and its variants [33, 34, 72, 65] have driven the success of scorebased diffusion models [62, 30, 64, 66] by providing a scalable score-estimation objective.

Neural Computation, 23(7):1661–1674, 2011

[73] An Vuong, Michael Thompson McCann, Javier E. Santos, and Yen Ting Lin. Are we really learning the score function? p. 8

Where the paper says it

2 Estimation of the Observable Error The variational characterization of ∥ΠGs es∥ 2 L2(p⋆ s ) in (21) suggests a clear training procedure to compute this norm: aim to solve the optimization by learning a scalar-valued critic network φψ with parameters ψ, via gradient ascent on the objective function in (21). In the remainder of this section, we adopt the standard framework of variance-preserving forward diffusion [66] (i.

Reinterpreting diffusion models through Wasserstein gradient flow matching. p. 0
Where the paper says it

Third, we give a tractable estimator of the gradient component via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full L 2 error. 1 Introduction Denoising Score Matching (DSM) and its variants [33, 34, 72, 65] have driven the success of scorebased diffusion models [62, 30, 64, 66] by providing a scalable score-estimation objective.

Transactions on Machine Learning Research, 2025

[74] Binxu Wang and Cengiz Pehlevan. An analytical theory of spectral bias in the learning dynamics of diffusion models. p. 0

Where the paper says it

the difference between the estimated score ˆss and the true score ∇ log p ⋆ s . Inequality (Full L 2 ) underpins much of the current theory of diffusion models [63], from convergence analyses [17, 46, 15, 7], to minimax optimality results [52, 77, 21, 47], and interpretations of training objectives in terms of sample quality [38, 39].

In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

[75] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical Bayes smoothing. p. 2

Where the paper says it

These studies reflect the empirical fact that not all score errors are equally relevant for sample quality. However, all these approaches treat the full score estimation error as the central quantity to analyze, without examining its geometric structure.

In The Thirty Seventh Annual Conference on Learning Theory, pages 4958–4991, 2024

[76] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms, 2017. p. 9

Where the paper says it

Algorithm 1 Observable-error critic estimation Require: Frozen score network ˆsθ; critic network φψ; training dataset D ∼ p ⋆ ; time sampler ν over [t0, T]; noising schedules α(s), σ(s); batch size B; critic steps K; learning rate η. Ensure: Estimate Lbobs of Es∼ν∥ΠGs es∥ 2 L2(p⋆ s ) .

arXiv:1708. p. 0
Where the paper says it

Inequality (Full L 2 ) underpins much of the current theory of diffusion models [63], from convergence analyses [17, 46, 15, 7], to minimax optimality results [52, 77, 21, 47], and interpretations of training objectives in terms of sample quality [38, 39]. arXiv:2606.

07747

[78] Pengze Zhang, Hubery Yin, Chen Li, and Xiaohua Xie. Tackling the singularities at the endpoints of time intervals in diffusion models. p. 5

Where the paper says it

1 identifies the gradient projection of the error, ΠGs es, as the component entering the Fokker-Planck dynamics at time s. Empirically, Figure 2 shows that ∥ΠGe∥ 2 is a better proxy of sampling quality than the full error ∥e∥ 2 as it correlates better with the Fréchet Inception Distance (FID) [29], widely used to measure sampling quality of generative models for images.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Observable Error Diagnostic for Training

The AI system can utilize an observable error component, denoted as PiGs es, to track learning quality rather than relying solely on the full L2 score error. This is achieved by training a critic network to estimate this gradient component via the objective function Jb(psi), which empirically correlates substantially better with sample quality than the full L2 error.

  1. Tighter Upper Bound on Distributional Divergence

The system can derive a new upper bound on the Kullback-Leibler divergence between the learned and target models, Theorem 3.2, using only the observable gradient component of errors: KL(p⋆t0∥pˆt0) <= 1/2 Z T t0 σ squared s PiGs es L2(p⋆s) ds. This bound is strictly tighter than the standard Girsanov bound when solenoidal components are non-zero.

  1. Improved Sampling Quality Proxy

The AI can use the estimated norm of the gradient component of errors, PiGs es L2(p⋆s), as a diagnostic that correlates substantially better with sample quality metrics like Fréchet Inception Distance (FID) than the full L2 error. This allows for a more reliable proxy for assessing how well the learned model matches the target distribution.

  1. Geometric Understanding of Error Components

The system can leverage the Helmholtz-Hodge decomposition to distinguish between gradient and solenoidal error components, understanding that only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible. This geometric insight helps in diagnosing why standard methods penalize errors that do not affect marginals.

  1. Path-Space Dynamics Analysis

The AI can analyze how score errors accumulate along a path to estimate endpoint marginals, as shown by Theorem 3.2, which integrates the gradient component of the error along the path to bound KL divergence at time t0: KL(p⋆t0∥pˆt0) <= 1/2 Z T t0 σ squared s PiGs es L2(p⋆s) ds.

Abstract

Score-based diffusion models are typically trained by minimizing the L squared score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the learned and target distributions. Recent work has shown that the score admits an orthogonal decomposition into gradient and solenoidal components in which only the gradient part matters for exact sampling. We apply this decomposition to the score error of trained models and study its quantitative consequences. Since only the gradient component of the error enters the marginal Fokker-Planck dynamics, a model can incur arbitrarily large L squared score error while perfectly matching the target. Starting from this observation, we present three results. First, we derive an upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error. It tightens the standard Girsanov bound for generic score networks and identifies that bound's looseness as the cost of working with path-space rather than marginal-space dynamics. Second, we prove that no non-trivial function of the L squared score error can uniformly lower bound any divergence between the learned and target distributions. Third, we give an estimator of the gradient component, computable from denoising score matching residuals via a dual Sobolev identity, which empirically correlates substantially better with sample quality than the full L squared error.

Sources

Related papers