Distance Marching for Generative Modeling

summary

Video file (mp4)

The gist

The gist: DISTANCE MARCHING introduces a time-unconditional approach to generative modeling by reformulating generation via distance-field modeling, which yields denoising directions better directed

In short

DISTANCE MARCHING introduces a time-unconditional generative model by using distance field modeling. It reformulates generation to learn scalar fields that represent the distance to data targets, rather than relying on time conditioning. This approach yields denoising directions that are more accurately aimed toward the desired data manifold, improving generation quality.

Key concepts

Distance Field Modeling
This concept involves learning a scalar field $u_ heta(x)$ that acts like an unsigned distance to a target set. At any point $x$, this field provides two crucial pieces of information: the magnitude $u_ heta(x)$, showing how far $x$ is from the target, and the negative gradient $- abla u_ heta(x)$, which indicates the direction to move toward that target set.
Time-Unconditional Generation
Traditionally, generative models learn time-dependent denoising vectors. DISTANCE MARCHING aims for a time-independent approach. The challenge is that without explicit time conditioning, the same noisy input can map to different noise levels and different denoising directions, which confuses the supervision signal.
One-Step Loss (OSL)
This loss function penalizes the difference between a predicted one-step projection and the actual target. It is designed to focus on closer data points, effectively reducing ambiguity about where to project in a single step during training, leading to more effective early denoising.
Directional Eikonal Loss (DEL)
This second constraint fixes the offset by selecting a specific positive branch and setting $C=c_0$. It works alongside the OSL to further constrain the learned field, offering an adjustable balance between focusing on immediate targets (OSL) and standard flow matching behavior.

Terminology used across episodes

This episode discusses

The paper

Distance Marching for Generative Modeling · Read on arXiv

Zimo Wang, Ishit Mehta, Haolin Lu, Chung-En Sun, Ge Yan, Tsui-Wei Weng, Tzu-Mao Li

Modern generative models based on diffusion or flow matching transport noise toward the data manifold, without representing the data manifold explicitly. We propose Distance Marching that explores an alternative design by explicitly modeling a distance field of the data manifold. A distance field offers a geometric view of generation as a descent process toward the data manifold using only the current position. The negative gradient of the distance field provides the direction, and the distance value can optionally determine the step size. However, sparse training data in high-dimensional space make it difficult to identify the closest points on the data manifold needed to train the distance field. To address this challenge, we design new loss functions that emphasize denoising targets from closer data points. Our analysis shows that, without time input, emphasizing closer targets reduces denoising ambiguity and the bias toward the data mean. This distance-field perspective naturally motivates two principled deterministic samplers, showing fast convergence in inference. As a result, our model is the only one without time input that remains competitive with time-conditioned models across datasets and architectures. Moreover, our distance prediction is also helpful for early stopping during sampling and for out-of-distribution detection. We hope distance-field modeling can open a promising direction for exploring generative modeling from a geometric perspective.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Distance Marching for Generative Modeling".

Tom: The gist: DISTANCE MARCHING introduces a time-unconditional approach to generative modeling by reformulating generation via distance-field modeling, which yields denoising directions better directed toward the data manifold.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we’re looking at this paper called "Distance Marching for Generative Modeling." It tackles time-unconditional generation by completely rethinking it through a distance field modeling approach. This means they aren't relying on explicit time conditioning anymore, which is a big shift.

Jane: That sounds complex, Tom. So what’s the core idea here? What exactly are they proposing with this distance field thing?

Tom: Well, the thesis is that instead of learning a vector field that depends on time, they propose learning a scalar field. This scalar field acts like a distance to the target data manifold. And they claim this gives them denoising directions that are much better pointed toward where the real data lives.

Lu: I find it fascinating how they move away from explicit time conditioning and instead focus on this geometric intuition of distance fields, which is usually used in other areas like computer graphics, to model generation. It decouples the generation process from having to track time explicitly.

Meng: From an engineering standpoint, decoupling from explicit time conditioning sounds cleaner for training stability. But how do you make sure that this distance field actually works well when you're just trying to generate images or data?

Lalam: Well, they design specific losses to focus on closer targets. This means the training process is steered toward points that are already close to the actual data distribution, which should help solve the issue of getting multiple noise levels from a single noisy input.

Tom: Exactly. They introduce two principled inference methods here based on this distance field view. One lets you use standard gradient descent, and another one uses the distance value for sphere-tracing style updates. It’s like using the distance to guide your steps instead of just relying on a fixed direction vector.

Jane: That sphere-tracing update sounds really interesting, Tom. So, if you have this scalar field u theta(x), what information does it give you at any given spot x ?

Paper summary: Tom: At any location x where the gradient of that field exists, it gives you two things. First is the magnitude of the scalar value itself, which tells you how far that point x is from the target set. Second is a direction, specifically negative gradient of that field, which tells you exactly where to move to get closer to that target.

Lu: That dual information—distance and direction—is what makes it powerful for both the inference rules and for generation itself. It’s like having a map that tells you not just where you are relative to the destination, but also the best way to walk toward it, which is a key part of how distance field models work.

Meng: And when they talk about those losses they design—the one-step loss and the directional eikonal loss—what’s the practical advantage of that focusing on closer targets compared to just using standard flow matching?

Jane: The paper shows that this loss design explicitly prioritizes closer targets, which it says reduces target ambiguity. This means in the early stages of denoising, you are more focused on getting things right where they need to be, rather than wandering around trying to guess the whole structure at once.

Tom: It’s about reducing that non-uniqueness of the offset C and sign, which is a problem with time-unconditional models. By concentrating learning on closer data points, the learned denoising direction aligns better with the actual data manifold.

Lu: And they show this works across different architectures and scales, consistently improving FID by fifteen percent on CIFAR-ten and ImageNet compared to recent time-unconditional baselines <ref:2602.02928#pg1,on CIFAR-10 and ImageNet>. It seems robust regardless of whether you use class conditioning or not.

Meng: That's a solid result if it holds up under real-world conditions. But they also mention that this distance predictor can help with early stopping during sampling, which is a useful practical feature for engineers working on large models.

Lalam: From an AI perspective, the vision here is that this method offers an adaptive gradient descent sampler that stops once the distance value has increased for ten consecutive steps. This should significantly reduce the average number of sampling steps needed when generating things like images from ImageNet, showing real efficiency gains.

Paper summary: Jane: That efficiency gain is something every engineer wants to hear about. So, we’ve talked about how this paper reformulates generation and the specific advantages of their loss design and inference methods. Now let’s look at what the authors are calling this work.

Tom: Right, let's wrap up with a look at the title and what these implications are for the field as a whole. This paper is titled "Distance Marching for Generative Modeling." It really presents this distance field view as a way to get around some of the issues inherent in time-unconditional generative modeling.

Lu: The implication is that we can gain a new, principled way to model generation that doesn't require us to explicitly condition on time when we want time-independent results. It suggests that geometric structure—the distance to the data manifold—is a more fundamental guide for generation than just following an evolving vector field over time.

Meng: For me, the practical implication is how this could be used in real-time systems or complex simulations where you need fast, stable sampling paths. If it can stop early based on this distance metric, that’s a tangible improvement for deployment speed.

Lalam: Culturally, I see this as a way to make the learning process itself more structured and less chaotic by focusing the optimization effort on local proximity to known data points. It suggests an AI culture that values geometric guidance over purely temporal evolution in its core generative mechanisms.

Jane: So, to summarize, "Distance Marching for Generative Modeling" introduces a distance field view of time-unconditional generation using two principled inference methods and losses focused on closer targets. The authors show this leads to better denoising directions and superior FID scores over baselines like flow matching.

Tom: It’s a method that gives you an adaptive sampler that stops early, which is great for speed, and it has been shown to be quite robust across different architectures and tasks. It’s a solid piece of work in making sense of how we generate data without needing time as an explicit input.

Conclusion: Tom: So we’re wrapping up this deep dive into "Distance Marching for Generative Modeling." Basically, we’ve seen how they take that weird distance field idea and apply it to generating images without needing explicit time inputs.

Jane: It really boils down to them learning a scalar field that maps the space toward the actual data manifold rather than just a vector field that moves over time.

Lu: What makes this interesting is how they design two different ways to use that distance information—standard gradient descent and some kind of sphere tracing style update.

Meng: The results show it improves FID by about thirteen point five percent on CIFAR-ten and ImageNet compared to other time-unconditional models we've seen recently.

Lalam: And they have this feature where the distance predictor can actually tell the sampler when to stop, which cuts down the number of steps needed for generation by almost thirty percent on ImageNet.

Tom: It’s about that efficiency and how they managed to get better denoising directions just by focusing their losses on closer targets.

Jane: So, what does this mean for us who are just listening? It suggests that geometric structure—the distance to the data—is a stronger guide for generating images than just following an evolving vector field.

Lu: The authors claim this approach yields inference rules that are naturally compatible with standard gradient descent, which is pretty neat because it keeps things grounded in familiar optimization methods.

Meng: From an engineering standpoint, if a model can stop early based on how far it thinks it is from the target set, that's a tangible improvement for deployment speed and resource management.

Lalam: Culturally, this points to an AI culture that values using geometric guidance over just tracking temporal evolution when you’re trying to build something complex.

Tom: So we've seen the mechanics and the numbers on how they do it, but now we need to think about what this distance field view means for future generative modeling architectures.

More episodes

← Home