Evaluating Generative Time-Series Models on Data with Point Masses

Many of the time series that generative time-series models are benchmarked on place a large probability mass on a single value — it does not rain, no ride is requested, no part is ordered. The data law is a mixture: an atom at zero, and a continuous distribution over the positive part. The models are not built that way. Flow matching with data-dependent priors, residual-expert flows, stochastic flow matching, VQ-latent transport and their diffusion predecessors all transport an absolutely continuous source distribution with a learned map. Under the regularity conditions ODE-based flow models are usually analysed with — a velocity field Lipschitz in the state and integrable in time, so that the time-1 flow map is a diffeomorphism — the pushforward of an absolutely continuous law is again absolutely continuous, and P(model)(Y t = c) = 0 for every c. A model is therefore being asked to reproduce, on more than half of the observations in some of these datasets, a feature it cannot represent. The scores used to judge it are marginal — CRPS per timestep, or a distributional distance computed on pooled windows — and are not obviously able to detect the difference between a genuine atom and a thin spike of continuous mass near zero.

This paper began as a test of the hypothesis those two facts suggest: that continuous-state models are systematically handicapped on data with a point mass, and that an explicit occurrence process would fix it. Neither part survived contact with the experiments. Across six datasets the flow is ahead of every explicit occurrence model on one dataset only, and an autoregressive occurrence hurdle — a logistic classifier rolled out one step at a time — beats it on the other five, on rideshare by a factor of 153. The empirical gap is real; what is not supported is that a continuous state is its cause, or that closing it requires a new generative model.

What the experiments did produce is a set of problems with the measurement, and they are the contribution of this paper. We distinguish two kinds of statement throughout. The evaluation-level findings — what a protocol measures, what a marginal score can support — follow from the definitions and apply to any generative model scored this way. The model-level findings are empirical claims about one family of conditional flows on six datasets; we do not read them as properties of continuous-state generative modelling in general.

The evaluation window need not resemble the dataset. The standard rolling-origin protocol scores a model on the final H steps of each series. Where the atom process is non-stationary, those steps can be unrepresentative by a wide margin: covid deaths is 42% zeros as a dataset and 13% in its evaluation windows, rideshare 47% against 5%. This inverted one of our own findings, turning the strongest occurrence model in our study into what looked like a cautionary tale. A series-wise split — hold out whole series, draw windows across their full time axis — removes the problem where the atom process is stationary in time: on rideshare the evaluation-window zero rate moves from 5.3% to 59.1%. It does not rescue covid deaths (14.1%), whose zeros are an artefact of the pre-outbreak period rather than a recurring process; we exclude it below and state the reason rather than reporting a number that cannot mean what it appears to.

A control under which CRPS cannot move. Permuting the sample index independently at each horizon step leaves every per-step predictive marginal — and therefore CRPS — unchanged by construction, while destroying the coupling that carries the occurrence statistics. It measures exactly how much the model’s trajectory coupling contributes to a chosen statistic. We compute the empirical CRPS exactly, evaluating EX − X' with the closed form for the mean absolute pairwise difference of a sorted sample rather than from a random pairing; the measured difference under the permutation is then 0 to floating-point precision on every dataset. (A pairing-based estimator of the second term is not invariant even though CRPS is, and reports a spurious difference of order 10−2.) What the control licenses is narrower than “the part of the temporal structure CRPS cannot see”: it is an exact within-model measurement of the contribution that the model’s trajectory coupling makes to a chosen statistic, with the predictive marginals held fixed. It does not enumerate everything a marginal score misses, and a model whose zero-run error is unchanged by decorrelation may still have learned autocorrelation, seasonality, or magnitude–occurrence coupling that this statistic does not register. Within that scope it gives a reference point a model must beat: if decorrelating its own samples does not worsen a statistic, the coupling it learned contributes nothing to that statistic. On uber and car parts the decorrelated flow is better than the flow, and on rideshare the two are within 12%; the learned coupling pays for itself only on rain, weather and solar.

A benchmark over five seeds, and the model that wins it. We compare, on identical windows and with an identical threshold protocol (threshold selected on a validation half, test-optimal threshold reported as a diagnostic upper bound): a conditional flow; global and per-series Markov occurrence chains; a negative-binomial semi-Markov occurrence process; autoregressive and non-autoregressive hurdle models with a learned occurrence classifier; and a hybrid that keeps the flow’s own positive magnitudes and replaces only its zero pattern with the semi-Markov process.

Three observations follow. An autoregressive hurdle dominates. A learned occurrence classifier rolled out one step at a time is the best model on five of the six datasets, and not narrowly: on rideshare it reaches 0.13 against the flow’s 19.8, a factor of 153. It is also the best or second-best model under the spectral, spell-quantile and survival statistics. The lower panel of Table II shows why this is not simply a better model: on rain, solar and uber the flow has the best CRPS of the seven while being beaten or matched on the occurrence statistics, and on solar it is ahead on both. The two families of number do not order the models the same way, which is the situation a practitioner is actually in. We report this arm prominently because we initially omitted it: under the rolling-origin protocol its rollout drifts and it looks like a cautionary tale about exposure bias, and we had written it up as one. Under the series-wise protocol — where the evaluation windows actually contain the atom — it is the strongest occurrence model we tested. The earlier reading was an artefact of the evaluation windows of Sec. IV-A, which is a second, uncomfortable instance of this paper’s own first finding.

The flow’s atom statistics are seed-unstable; the baselines’ are not. Over five training seeds evaluated on identical held-out series and windows, the flow’s run-length error has a relative spread of 7% to 62%; on solar the seeds range over a factor of four. Every occurrence baseline here is deterministic: the transition probabilities are closed-form and the one learned component has a fixed seed. A single-run number for a flow on these statistics is not reportable at the margins these comparisons are decided by, and the papers we drew the model from report one.

The obvious fix is not one, but the obvious fix was the wrong one. The hybrid keeps the flow’s magnitudes and substitutes an explicit occurrence process, isolating that component. With a semi-Markov occurrence process it improves on the flow on rideshare alone and is worse on the other five. That is the result we would have reported had we stopped there. Table II shows why it is the wrong conclusion to draw: the occurrence process that works is not the semi-Markov one but the autoregressive classifier, and a hybrid built on that component is the natural next experiment rather than a closed question.

Comparing each flow against its own decorrelated version isolates the value of the coupling it learned, with the marginals held fixed. Averaged over seeds, the coupling pays for itself on rain (3.4×), weather (2.6×), uber (1.5×) and solar (1.3×), is neutral on rideshare (1.06×), and is actively harmful on car parts (0.82×); the occurrence ACF gives the same ordering. An earlier three-seed version of this table put uber and car parts both on the harmful side — with five seeds only car parts remains there, which is itself an illustration of the previous paragraph.

Does the conclusion survive a change of statistic? Everything above is measured with the Wasserstein distance between zero-run length laws. That is one hand-built statistic, and a reader is entitled to ask whether the picture is an artefact of it. We therefore repeat the comparison under four more, in two groups. Spell quantiles (mean relative error at the 50th, 90th and 99th percentile) and the sup-norm between run-length survival functions are further summaries of the same object, so they corroborate the W1 rather than test it independently: the W1 weights how far misplaced mass moved, the sup-norm weights how much mass is misplaced, and a model can do well on one and badly on the other. The occurrence autocorrelation and the normalised occurrence periodogram are independent of the run-length construction — they read the same binary process through a second-order and a spectral lens, and are defined for windows containing no completed spell. We also report the two transition rates of the occurrence chain. The model ordering is not the same under five different occurrence statistics, and the two that do not share a construction (ACF, spectrum) agree with each other least: rank correlation between orderings, averaged over datasets, is W1–survS +0.92, W1–spellQ +0.84, W1–spec +0.79, W1–ACF +0.55, ACF–spec +0.45.

In conclusion, a conditional flow is beaten on five of six atom-bearing datasets — on rideshare by a factor of 153 — by an autoregressive occurrence hurdle with no generative machinery to speak of, yet it has the best CRPS of the seven models on three of those same datasets, so which model is better depends on which family of number one reads and on which occurrence statistic within the second family. Three things follow that do not depend on the hypothesis we started from or on the model family we tested: report the atom rate of the evaluation windows, since the standard protocol can differ from the dataset by forty points and inverted the ranking of the best baseline in this study before we caught it; compare a model against its own decorrelated samples before crediting it with temporal structure, which costs one permutation, holds CRPS fixed by construction, and reveals cases where the learned coupling contributes little or negatively; and report a spread, since this flow’s occurrence statistics move by up to 62% across training seeds while every occurrence baseline here is deterministic.

Improvements to AI systems:

1. Add an explicit occurrence-atom component to continuous-state generative models. Instead of forcing a purely continuous flow to approximate point masses (e.g., zeros), augment the model with a discrete occurrence head (e.g., a logistic classifier rolled out autoregressively) that decides whether the next value is exactly zero or drawn from the continuous positive distribution. This directly addresses the structural mismatch where P(model)(Y t = c) = 0 for any constant c.

2. Implement evaluation-window-aware training and reporting. Modify the training or validation protocol to match the atom rate of the actual evaluation windows, not the full dataset. Specifically, if the test protocol uses rolling-origin windows, the model should be trained or fine-tuned on windows with similar zero proportions. Also, automatically report the atom rate of the evaluation windows alongside any benchmark score, so users can detect when the test distribution diverges from the training distribution.

3. Add a permutation-based coupling diagnostic as a standard model check. For any time-series generative model, compute the chosen statistic (e.g., CRPS, zero-run length) on the model’s own samples and on a version where the sample index is permuted independently at each horizon step. If the decorrelated version performs equally or better, flag that the learned temporal coupling contributes nothing or is harmful. This costs one permutation and provides a model-specific baseline for what the coupling should beat.

4. Replace single-seed evaluation with multi-seed spread reporting for stochastic models. For any generative model with random initialization (e.g., flow matching), train over at least five seeds and report the relative spread of key statistics (e.g., run-length error, occurrence ACF). If the spread exceeds a threshold (e.g., 20%), do not report a single-run number as definitive; instead, present the range and note that the model’s occurrence statistics are seed-unstable.

5. Integrate a hybrid model that separates occurrence and magnitude generation. Build a system where a continuous flow generates positive magnitudes, but a separate autoregressive occurrence classifier (not a semi-Markov chain) determines the zero/non-zero pattern. This hybrid should be the default for atom-bearing data, as it outperforms pure flows by up to 153× on occurrence statistics while retaining competitive CRPS.

6. Add a multi-statistic evaluation suite for occurrence processes. Instead of relying on a single hand-built statistic (e.g., Wasserstein distance on zero-run lengths), evaluate under at least five complementary statistics: W1 on run lengths, sup-norm of survival functions, spell quantiles, occurrence autocorrelation, and normalized periodogram. Report rank correlations between these orderings so users can see where conclusions are robust and where they are statistic-dependent.

What the improved AI system can do:

• Generate time series with genuine point masses (e.g., exact zeros for no-rain, no-ride, no-order) without producing thin continuous spikes near zero, because it has an explicit discrete occurrence mechanism.

• Avoid misleading benchmark results by automatically detecting when evaluation windows have a different atom rate than the training data (e.g., 5% vs 47% zeros) and either correcting the protocol or flagging the discrepancy.

• Quantify the value of its own temporal structure by comparing its performance against its own decorrelated samples, allowing users to know whether the model’s learned coupling is actually beneficial for a given statistic.

• Provide reliable, reproducible results by reporting multi-seed spreads, so users are not misled by a single lucky or unlucky run (e.g., a flow whose occurrence statistics vary by 62% across seeds).

• Achieve state-of-the-art performance on atom-heavy datasets by using the autoregressive hurdle hybrid, which beats pure flows by a factor of 153 on rideshare data while maintaining competitive marginal scores.

• Give a complete, multi-faceted evaluation of occurrence behavior, so users can see if a model’s advantage is consistent across run-length, spectral, and autocorrelation views, or if it is an artifact of one statistic.

Many of the series that generative time-series models are benchmarked on place a large probability mass on a single value --- it does not rain, no ride is requested, no part is ordered. We report what happens when such data is evaluated carefully. First, the standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset: on one benchmark the dataset is 42% zeros and the evaluation windows are 13%, on another 47% against 5%. This is not a cosmetic problem --- it reversed one of our own conclusions, turning the strongest occurrence model in our study into what looked like a cautionary tale. Second, we give a control in which CRPS is invariant by construction while the temporal coupling is destroyed, which measures exactly how much that coupling contributes to a chosen statistic. Third, benchmarking seven models on a matched protocol over five seeds, an autoregressive hurdle beats a conditional flow on five of six datasets, by up to a factor of 153, while the flow's own occurrence statistics vary by up to 62% across training seeds and every baseline is deterministic. Finally, the model ordering is not the same under five different occurrence statistics, and the two that do not share a construction agree with each other least.