Discretizing Continuous Time Series for Imputation with Masked Diffusion Training
summary
The gist
Time series imputation is addressed by proposing the Masked Diffusion Time-series Imputation Model (MDTIM)…
In short
The Masked Diffusion Time-series Imputation Model (MDTIM) adapts masked diffusion for continuous time series by using Stochastic Discretization and Ordinal-Aware Soft Labeling. This bridges discrete language modeling with continuous data, allowing the model to directly predict original values rather than noise, leading to superior reconstruction accuracy and robustness across various missing patterns.
Key concepts
- Stochastic Discretization
- This technique maps the continuous numerical values in a time series onto discrete tokens suitable for language models. It is designed to preserve the continuous dynamics of the signal while allowing the model to operate within a discrete token space, effectively bridging continuous and discrete modeling paradigms.
- Ordinal-Aware Soft Labeling
- This method captures the inherent order or relationship between different tokens in a sequence. By introducing this labeling, the model learns to respect how values relate to each other in terms of magnitude and order, which is crucial for accurately reconstructing continuous time series data.
- Spectral Consistency Loss
- This regularization term enforces global temporal coherence by comparing the Fourier representations of the reconstructed signal against the ground truth. It ensures that the imputed sequence maintains consistent periodic structures across different frequencies, enhancing the model's ability to capture long-range dependencies.
Terminology used across episodes
This episode discusses
- Discretizing Continuous Time Series for Imputation with Masked Diffusion Training · Paper Radio
- Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models
- TSI-Bench: Benchmarking Time Series Imputation
- Classifier-Free Diffusion Guidance
- DiffWave: A Versatile Diffusion Model for Audio Synthesis
- Score-Based Generative Modeling through Stochastic Differential Equations
The paper
Discretizing Continuous Time Series for Imputation with Masked Diffusion Training · Read on arXiv
Seoul National University
Time series imputation is a crucial area for reliable time series analysis, yet it remains challenging due to the complex temporal dynamics and noise of real-world data. Existing approaches, however, exhibit two limitations: missing and observed values are embedded within the same representation space without explicit structural separation, and continuous diffusion-based methods recover the missing values only through many iterative denoising steps rather than by predicting them directly. To address these, we propose the Masked Diffusion Time-series Imputation Model (MDTIM), which leverages the training paradigm of masked diffusion model for imputation tasks. The [MASK] token is structurally orthogonal to valid observations, and because the corrupted state is the partially observed series itself, the model is trained to predict the missing values directly from the observed ones, naturally aligning both the representation and the learning objective with the imputation task. To bridge the gap between discrete masked diffusion and the continuous, ordinal nature of time series, we further introduce Stochastic Discretization, which maps continuous values to ordinal-aware tokens while preserving continuous dynamics. Our experiments on diverse benchmarks confirm that MDTIM achieves superior robustness and scalability, consistently outperforming state-of-the-art deterministic and generative baselines across various missing scenarios. Code and project page are available at https://dongbeank.github.io/MDTIM/.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Discretizing Continuous Time Series for Imputation with Masked Diffusion Training".
Jane: Here's the gist of the paper. Time series imputation is addressed by proposing the Masked Diffusion Time-series Imputation Model (MDTIM), which bridges discrete masked diffusion and continuous time series modeling The MDTIM framework leverages the training paradigm of masked diffusion model for imputation tasks Addressing Representation Limitations Existing approaches suffer from two main limitations: missing and observed values are embedded within the same representation space without explicit structural separation Furthermore, continuous diffusion-based methods are trained to predict added noise rather than the original signal The
MASK: token is structurally orthogonal to valid observations, which provides a clearer separation between missing and observed states in the representation This structural orthogonality allows the model to directly predict the original values rather than the noise, more closely matching the imputation goal.
Tom: Bridging Discrete and Continuous Spaces The fundamental challenge is bridging discrete masked diffusion and continuous time series modeling because time series data are inherently continuous and ordinal, whereas language tokens are discrete and categorical To solve this, the authors introduce Stochastic Discretization to map continuous values to ordinal-aware tokens while preserving continuous dynamics They also introduce Ordinal-Aware Soft Labeling to capture the ordinal relationships between tokens.
Jane: Training Signal Construction The training objective is formulated as a dual-domain objective combining an Ordinal-Aware Soft Cross-Entropy and a Spectral Consistency Loss The Ordinal-Aware Masked Diffusion loss decomposes into (Equation 6) L = (1 − ϵ)LNELBO + ϵ DKL(u∥pθ) + ϵ H(u) const, which encourages the predicted distribution to align with the ordinal neighborhood of y.
Tom: Additionally, a Spectral Consistency Loss is incorporated to enforce global temporal coherence by minimizing the L1 distance between Fourier representations of the reconstructed and ground-truth signals.
Title and authors: Tom: So we're looking at this paper called "Discretizing Continuous Time Series for Imputation with Masked Diffusion Training," and the authors are Dongbin Kim, Seungyun Lee, Geonwoo Shin, and Jaewook Lee from Seoul National University.
Jane: It sounds like they are trying to solve a really tough problem in time series analysis where you have continuous data but you're using tools that are built for discrete things.
Lu: Basically, the main idea is to take the training method from masked diffusion models and adapt it for continuous time series imputation, which is pretty novel because those two worlds don't usually mix well.
Tom: So what's the core problem they are tackling here, Jane? What makes this approach different from just using a standard diffusion model on time series data?
Jane: Well, the paper points out that existing methods have two big flaws: first, missing and observed values are in the same representation without any clear separation.
Tom: And second, continuous diffusion models usually try to predict some added noise instead of actually figuring out what the original signal was.
Lu: Exactly. They propose a model called the Masked Diffusion Time-series Imputation Model or MDTIM, which they say leverages that masked diffusion training paradigm specifically for imputation tasks.
Jane: And they make a key structural argument about how their approach fixes the representation issue by using a special token called the mask token which is structurally orthogonal to any valid observation.
Meng: Orthogonal to observations—that sounds cleaner than just mixing everything together—but what does that actually mean for the prediction part?
Tom: Because if it's truly orthogonal, can the model directly predict the original values instead of predicting noise?
Lu: Yes, that's the goal. They say this structural orthogonality allows the model to directly predict those original values rather than just noise, which is what we want for imputation. It’s more closely aligned with what imputation actually requires.
Jane: But how do you handle the fact that time series data are continuous and ordinal, while language tokens, which diffusion models usually use, are discrete?
Tom: That's where they introduce Stochastic Discretization to map those continuous values into ordinal-aware tokens while still keeping the continuous dynamics intact.
Jane: And they also add something called Ordinal-Aware Soft Labeling to capture those important relationships between the tokens themselves.
Lu: The training objective is quite specific here, it's a dual-domain objective combining an Ordinal-Aware Soft Cross Entropy and a Spectral Consistency Loss.
Tom: That spectral consistency loss is interesting because they use it to enforce global temporal coherence by minimizing the L1 distance between Fourier representations of the reconstructed signal and the ground truth signal.
Title and authors: Jane: It sounds like they are trying to ensure that when you reconstruct a sequence, not only does it look right locally, but it also maintains a consistent pattern across its entire length in time.
Tom: And they do that by using this dual objective, which encourages the predicted distribution to align with the ordinal neighborhood of the target values.
Meng: From my side as someone who has to worry about deployment, does this complexity translate into a manageable training process?
Lu: They state that the Ordinal-Aware Masked Diffusion loss decomposes into one part involving L NELBO plus some terms with epsilon, which encourages the predicted distribution to align with the ordinal neighborhood of the target values.
Jane: And they say this decomposition shows that for epsilon greater than zero, that additional term penalizes any probability mass placed outside of the ordinal neighborhood of the true value.
Tom: That’s a concrete way they are enforcing those ordinal relationships we talked about earlier.
Lu: They also incorporate a Spectral Consistency Loss to enforce global temporal coherence by minimizing the L1 distance between Fourier representations of the reconstructed and ground truth signals, which is what they call L FFT.
Jane: So, in short, they are taking a masked diffusion model and adding tools to handle continuous ordinal data so it can accurately reconstruct those time series.
Tom: And I see a lot of promise there because if you can get that structural separation right between missing and observed states, you bypass some of the typical noise prediction issues inherent in these models.
Jane: But let's look at how they actually train this thing, because the training setup seems quite involved.
Lu: The training objective is formulated as a dual-domain objective combining an Ordinal-Aware Soft Cross Entropy and a Spectral Consistency Loss, which they call L total equals L diff plus lambda L FFT.
Tom: And they set the spectral loss weight lambda to one point zero for their main experiments on the datasets like Energy, ETTh, and Weather.
Meng: If we're talking about running this at scale, how does this training setup affect computational cost?
Lu: They show that it's parameter efficient; the zero point six five million small variant is smaller than the eight eight point two four million large SAITS variant <ref:2608.19119#pg2>.
Jane: That’s a significant reduction in size, which is good for deployment, especially when we compare it to some of the larger continuous diffusion baselines.
Title and authors: Tom: And on inference speed, they show that the Small MDTIM variant completes inference in about four point zero five seconds, which is much faster than the continuous diffusion baseline which takes up to eight hundred ninety four seconds at large scale.
Jane: That difference in time is huge for real-time applications, so that's a big win for practical use of this model.
Lu: They also show that the performance advantage of MDTIM gets bigger as the sequence length increases, which suggests it manages to capture global context without losing information through attention dilution.
Tom: That’s what we want when dealing with long-term dependencies in time series.
Jane: The paper shows that they used Expectation-based Unmasking to recover the continuous values by computing the probability weighted expectation of the predicted token distribution across multiple independent noise samples.
Tom: So, how does that help with uncertainty?
Lu: They use a Continuous Ranked Probability Score or CRPS for probabilistic settings to assess both calibration and sharpness, which rewards both aspects.
Meng: I need to know about the robustness here. The results show it performs well under geometric masking where most other baselines suffer significant degradation.
Jane: That's important because real-world data often has those kinds of missing patterns, and this model seems pretty resilient to them compared to other methods.
Lu: And they have an ablation study showing that Expectation-based Unmasking consistently reduces error across all datasets, which helps mitigate the quantization error that comes from using the Argmax selection method.
Jane: But what about the Ordinal-Aware Soft Labeling part? Does it always help?
Meng: The contribution of Soft Labeling scales with how regular the underlying data is; it's most effective when there's clear ordinal or periodic structure in the signal.
Lu: They found that on the Sine dataset, where there is known periodicity, Soft Labeling yields substantial additional gain.
Jane: But on datasets like Energy, it didn't offer any further improvement beyond just using Expectation-based Unmasking.
Tom: And they found an optimal vocabulary size of four zero for the best balance between how detailed the discretization is and how complex the classification task becomes.
Jane: So, to summarize what we have on this paper, it successfully adapts masked diffusion training to continuous time series using stochastic discretization and ordinal-aware soft labeling.
Lu: It consistently outperforms both deterministic and generative baselines in reconstruction accuracy across different missing scenarios.
Tom: And when you look at downstream tasks, like forecasting on the Energy dataset, MDTIM reduces the mean absolute error by nineteen point zero percent to twenty six point zero percent compared to SAITS, and by sixteen point zero percent to nineteen point zero percent compared to CSDI.
Title and authors: Jane: That propagation of accuracy is telling; it means that how well this model reconstructs the data directly translates into how much better you can forecast things down the line on things like energy usage.
Meng: I'm still thinking about that PhysioNet two thousand twelve dataset result, where MDTIM achieved the lowest mean absolute error across all settings, outperforming SAITS by a clear margin.
Tom: That’s a strong claim because it shows this approach generalizes well to real-world sparse and irregular observations when you compare it to traditional frequency domain priors which they say become unreliable for those signals.
Lu: But the paper does have some limitations, and one is that the spectral consistency regularization assumes the presence of stable frequency components, so they disabled that part on benchmarks dominated by irregular sampling like PhysioNet A data.
Jane: That's a fair limitation to note; it means the method relies on certain signal characteristics to work best.
Tom: And they point out another future direction: designing a data adaptive discretization scheme that automatically selects the resolution of the vocabulary size K based on the range and granularity of the underlying signal, since they found K=forty was optimal for their tests <ref:2608.19119#pg2>.
Jane: And extending this to extremely long sequences or non-stationary domains like financial tick data is definitely where future work needs to go.
Lu: To wrap up, this paper, "Discretizing Continuous Time Series for Imputation with Masked Diffusion Training," successfully adapts the masked diffusion training paradigm to continuous time series imputation by introducing stochastic discretization and ordinal-aware soft labeling.
Tom: It consistently outperforms both deterministic and generative baselines in reconstruction accuracy and robustness across diverse missing scenarios.
Jane: It’s a solid piece of work because it shows that the way we structure the model, by making the structural separation between valid observations and missing placeholders explicit, provides a real benefit beyond just using masked diffusion training.
Meng: And when you look at inference time, achieving four point zero five seconds on small models is much faster than some continuous diffusion baselines which take up to eight hundred ninety four seconds.
Tom: And the results show that this imputation accuracy translates directly into downstream forecasting performance, reducing the mean absolute error by nineteen point zero percent to twenty six point zero percent on energy forecasts compared to SAITS and CSDI.
Jane: So, we're looking at a model that handles continuous ordinal data in a way that is both computationally feasible and actually improves performance on real-world time series.
The paper's summary: Tom: Essentially, they introduced this Masked Diffusion Time-series Imputation Model or MDTIM. Jane It’s an attempt to bridge the gap between discrete diffusion and continuous time series modeling by using the masked diffusion training paradigm for imputation tasks.
Lu: The core idea is fixing two big problems in existing methods—first, missing and observed values are mixed up in the same representation space without any clear separation. Tom And second, continuous diffusion models usually try to predict some added noise instead of figuring out what the original signal actually was.
Jane: So they solve that by adding a special mask token that is structurally different from any valid observation, which lets the model directly predict the original values instead of just predicting noise. Lu That’s a nice structural trick for cleaning up the representation.
Meng: But how do you handle continuous time series data? They have to deal with them being continuous and ordinal, while diffusion models usually work with discrete tokens. Tom That’s where they introduce Stochastic Discretization to map those continuous values into tokens that respect the ordinal relationships, while still keeping the underlying dynamics smooth.
Jane: And they also layer on this Ordinal-Aware Soft Labeling to make sure those ordinal relationships between the tokens themselves are captured correctly during training. Lu They say this setup leads to a dual-domain objective that combines an Ordinal-Aware Soft Cross Entropy with a Spectral Consistency Loss.
Tom: That Spectral Consistency Loss is important because it forces the model to maintain global temporal coherence by minimizing the distance between the Fourier representations of the reconstructed signal and the true signal. Jane It’s like making sure that when you reconstruct a whole sequence, not only does it look right locally, but it also keeps a consistent pattern across its entire length in time.
Lu: Exactly. The training objective is set up so that if you use those epsilon terms in the loss decomposition, any probability mass the model puts outside the correct ordinal neighborhood of the true value gets penalized. Jane It’s a concrete way to enforce that structural separation we talked about earlier.
Meng: So, what does this mean for actually deploying it? Does this complexity translate into a manageable training process? Tom They show that it’s parameter efficient; the smaller model variant is only point six five million parameters compared to the larger SAITS variant. Jane That size reduction is good for production because it means less memory and faster training times.
Lu: And for inference speed, they show that the small MDTIM variant can complete a run in about four point zero five seconds, which beats some continuous diffusion baselines that need up to eight hundred ninety four seconds at large scale. Tom That difference in time is huge for real-time applications.
Jane: The paper also shows that this approach performs better even when the sequence length gets longer, meaning it captures global context without losing information through attention dilution as the sequence grows. Lu And they use a Continuous Ranked Probability Score or CRPS to measure uncertainty, which gives a better picture of both how well the model predicts and how sure it is about its answer.
Tom: So, in short, they’ve successfully adapted masked diffusion training to continuous time series imputation by using stochastic discretization and ordinal-aware soft labeling. Jane This method consistently beats deterministic and generative baselines in accuracy across different missing scenarios.
Lu: And when you look at downstream tasks, like forecasting on the energy dataset, MDTIM reduces the mean absolute error by nineteen point zero percent to twenty six point zero percent compared to SAITS and by sixteen point zero percent to nineteen point zero percent compared to CSDI. Jane That propagation of accuracy means that how well this model reconstructs data directly translates into how much better you can forecast things down the line.
Meng: I’m still thinking about the PhysioNet two thousand twelve dataset result where MDTIM achieved the lowest mean absolute error across all settings, outperforming SAITS by a clear margin. Tom That’s a strong claim because it shows this approach generalizes well to real-world sparse and irregular observations when you compare it to traditional frequency domain priors which they say become unreliable for those signals.
Jane: But there is one limitation they flag: the spectral consistency regularization assumes the presence of stable frequency components, so they disabled that part on datasets dominated by irregular sampling like PhysioNet A data. Lu That’s a fair caveat; it means the method relies on certain signal characteristics to work best.
Tom: And looking ahead, they suggest a future direction: designing a data adaptive discretization scheme that automatically picks the resolution of the vocabulary size K based on how granular the underlying signal is. Jane That would make it more flexible for different types of data.
Lu: And they also point out that extending MDTIM to extremely long sequences or non-stationary domains like financial tick data is definitely where future work needs to go.
The paper's improvements: Tom: So, we've seen how they adapted masked diffusion for continuous time series imputation using those discretization methods we talked about earlier. Jane Now let's talk about what the authors actually suggest to do next with this work and what that means for future research in this area.
Lu: They point out a few key things they think are important improvements. First, they say we need to design a data adaptive discretization scheme that automatically picks the resolution of the vocabulary size K based on how granular the underlying signal is. Jane That would be smart because K being fixed at forty might not work for every single time series dataset.
Jane: Yeah, if you have a very smooth signal and another one with lots of sharp spikes, having a fixed vocabulary size doesn't make sense for both cases. Tom So they want the model to figure out the best token resolution automatically instead of us guessing it ahead of time.
Tom: Exactly. And beyond just choosing K, Lu says they’re looking at developing an adaptive frequency objective to modulate that spectral consistency loss based on how regular or irregular the local signal is. Jane That means you tune the global coherence part of the model to match the specific local patterns in your data.
Lu: It’s about making it more flexible. They also mention extending this model to handle extremely long sequences and non-stationary domains, like financial tick data, because their current advantage grows with sequence length, which is good but not infinite. Tom That means we need to see if this structure holds up when the time series is really long or constantly changing.
Jane: So they’re looking at making it more robust to those extreme conditions. And they are also evaluating performance on those real-world data sets, like PhysioNet two thousand twelve to see how well it generalizes compared to traditional frequency domain priors for irregular signals. Lu That's a crucial test because those priors often break down when the signal isn't smooth.
Tom: The implication is that if we can automate the discretization choice and the frequency objective, we might create an imputation method that is much more robust across a wider range of real-world applications, not just controlled benchmarks. Jane That moves it from a specific technique to a more universal framework.
Conclusion: Tom: So we've covered how they took masked diffusion and made it work for continuous time series imputation by introducing stochastic discretization and ordinal labeling in this paper called "Discretizing Continuous Time Series for Imputation with Masked Diffusion Training." Jane It really bridges two worlds that usually don't talk to each other.
Jane: Exactly. The main implication is that we can use the powerful training methods from diffusion models to actually reconstruct continuous data, not just discrete tokens. Lu That means we’re getting better at handling those ordinal relationships in time series data.
Lu: I think the big change here is showing that this structural separation between missing and observed states provides a real benefit over standard masked diffusion training alone. Tom It confirms that the way we structure the model makes a difference in how it learns to impute.
Meng: From an engineering standpoint, the inference speed is what really caught my attention because they showed that the small variant runs in about four point zero five seconds, which is much faster than some continuous diffusion baselines that take up to eight hundred ninety four seconds. Jane That’s a massive win for real-time deployment if we can get it stable.
Jane: It really shows the practical benefit of this structural change because speed matters when you have high-frequency data streams, and it does translate into better forecasting accuracy down the line on things like energy usage. Tom Yes, that propagation of accuracy is what makes this work for many real-world systems.
Lu: And I'm confident in the evidence because they showed it generalizes effectively to real-world sparse and irregular observations when tested on PhysioNet two thousand twelve data, which is a pretty tough test for any imputation model. Jane It shows the ordinal-aware discrete formulation holds up even when the signal isn't perfectly smooth.
Tom: And I just want to mention that they have some limitations, like they had to turn off the spectral consistency regularization on some irregular datasets because it assumes stable frequency components. Meng That’s a necessary caveat; we need to know where this method stops working effectively.
Meng: And the future work Lu mentioned is key, so designing a data adaptive discretization scheme that automatically picks the right resolution for vocabulary size K based on signal granularity seems like the most important thing to watch for. Jane It sounds like they are moving toward a system that can adapt to whatever kind of time series you feed it.
Lu: Right, and extending this to extremely long sequences or non-stationary domains, like financial tick data, is where we need to look next because that's the next frontier for these types of models. Tom So, "Discretizing Continuous Time Series for Imputation with Masked Diffusion Training" gives us a solid foundation here. Jane It’s a lot to digest, but it’s definitely pushing the boundaries of how we use diffusion models for continuous data.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation