Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective

summary

The gist

The gist Event sequence forecasting can be framed as a conditional generative modeling problem where no method excels at both accuracy and realism, highlighting a fundamental trade-off between

In short

Event sequence forecasting is framed as a conditional generative modeling problem where no single method achieves both high prediction accuracy and distributional realism simultaneously, highlighting a fundamental trade-off. The work introduces Wasserstein BaryBooster, an algorithm that combines samples from a diffusion model to produce the best possible predictor, proving that distributional realism can be successfully converted into accurate predictions.

Key concepts

Event Sequence Forecasting: This is the task of predicting a future chain of events (like transactions or medical readings) given a history. Each event in the sequence has different types of data, including numbers and categories. The goal is to forecast many future steps ahead based on past observations.
Distributional Realism: This measures how realistic the predicted sequence looks, focusing on the underlying probability distribution of possible events rather than just a single best guess. High realism means the model captures the true variety and structure of real-world event patterns.
Prediction Accuracy (TMS): This metric evaluates how close a forecast is to the actual observed sequence by comparing them. It uses normalization and permutation techniques to measure matching across features and time steps, aiming for high precision in predicting the exact next event.
Wasserstein BaryBooster: This is a novel algorithm that takes multiple random samples from a diffusion model (a type of generative model) and aggregates them into one optimized forecast. It uses a discrete version of the Wasserstein barycenter to find the best prediction that balances high accuracy with maintaining some level of distributional realism.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Flow Matching — explained in 16 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes

This episode discusses

The paper

Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective · Read on arXiv

Dmitry Osin, Egor Surkov, Petr Mokrov, Igor Udovichenko, Ivan Karpukhin, Evgeny Burnaev, Egor Shvetsov, Alexander Korotin

Applied AI Institute 2 Applied AI Institute 3 Vega Institute Foundation 4 SB AI Lab 5 AXXX

Event sequences are chains of random events, each described by heterogeneous (numerical and categorical) features. The problem of event sequence forecasting occurs in finance, healthcare, business process intelligence, and other fields. In this work, we study multi-horizon forecasting of event sequences as a conditional generative modeling problem. We evaluate forecasts not only for prediction accuracy, but also for distributional realism. To this end, we present a benchmark comprising metrics for both quality aspects, strong statistical baselines, recent models adapted from adjacent fields, and curated real-world datasets. Careful evaluation reveals that no method excels at both accuracy and realism, highlighting a fundamental trade-off between the two. Finally, we introduce Wasserstein BaryBooster, an algorithm that aggregates samples from a realism-strong diffusion model into a state-of-the-art predictor, demonstrating that distributional realism can be converted into prediction accuracy. Code is available at https://github.com/xaosina/event-sequence-forecasting.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Realism VS Accuracy".

Jane: The gist Event sequence forecasting can be framed as a conditional generative modeling problem where no method excels at both accuracy and realism,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, to recap, this paper sets up a challenge by comparing generative models for event sequences against two key ideas: prediction accuracy and distributional realism. They're testing if you can get a model that is both good at predicting the next thing and good at generating things that actually look like they could happen.

Jane: Exactly. They’re using datasets to show that you can’t have one method that excels at both metrics simultaneously, which points to a fundamental trade-off in how we approach these forecasts.

Lu: The paper is introducing a specific setup for this comparison, defining the problem as forecasting S H+one:H+L given history S one:H, where each event is defined by three things: time, an event type, and some other feature <ref:2610.12586#pg1>. This formalizes what they are trying to model.

Meng: So it’s not just a simple next-step prediction; it's a longer sequence forecast, L steps ahead, which adds a layer of complexity for the models they are testing.

Lalam: They also introduce two specific metrics to measure these things: Temporal Matching Score, or TMS, which measures prediction accuracy based on aligning predicted and true events using per-feature normalization and permutation of horizon positions.

Tom: And then there's the Conditional Detection Fooling Score, or cDFS, which is their measure for distributional realism—basically how hard it is to fool a classifier into thinking a forecast is real.

The paper's summary: Jane: When we look at the actual summary of "Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective," they show that they've adapted representatives from three different classes of models to fit this event sequence setting.

Lu: They take a data-space diffusion model, which is CDiff, and a latent diffusion model, which adapts TabSyn, and even a non-diffusion method called DEF. This gives them a broad view of what’s out there in the literature for event sequences.

Meng: It seems like they are trying to cover the spectrum from pure data diffusion to these more complex latent space methods and even standard non-diffusion approaches.

Tom: What's interesting is that they use these different model types on the same problem, which helps isolate whether the trade-off we see is due to the model architecture itself or something else.

Jane: The main finding they highlight is that their evaluation shows no method excels at both TMS and cDFS, clearly demonstrating that there's a fundamental conflict between achieving high prediction accuracy and achieving high distributional realism.

Lalam: It’s like if you try to make the forecast perfectly match the most likely path, you end up missing out on generating sequences that are actually plausible under the underlying data distribution.

The paper's improvements: Tom: The authors introduce Wasserstein BaryBooster, which they call a post-hoc procedure for aggregation. It takes N stochastic samples from any base generator and turns them into a single prediction that is optimal according to TMS.

Jane: So this is the mechanism they propose to bridge that gap; it essentially aggregates the potential of multiple samples from one model into one single forecast that maximizes accuracy under TMS, while still keeping some measure of realism.

Lu: The procedure involves finding an arg min where they use a discrete version of the Wasserstein barycenter as a way to find def that minimizes TMS over all those samples, which is pretty heavy computation at inference time.

Meng: That sounds computationally expensive for production if you actually have to run it every time you need a forecast. How does the complexity scale when we try to deploy this?

Tom: The paper shows that Wasserstein BaryBooster produces the highest TMS across all datasets while still retaining a non-trivial cDFS, which means it’s doing better than just picking one of the base models.

Lalam: They also introduce two new metrics: TMS for accuracy and cDFS for realism, which they argue are necessary because previous work focused almost entirely on prediction accuracy and ignored distributional realism.

Conclusion: Jane: So to wrap up, the main implication of this paper is that we need to stop treating prediction accuracy and distributional realism as things you can just optimize independently.

Tom: The Wasserstein BaryBooster procedure gives us a concrete way to harvest the potential of multiple stochastic samples from a realist model, turning it into a single predictor optimal under TMS.

Lu: It shows that even with different architectures like the LDM or DDM, there’s this clear trade-off plot where high prediction accuracy usually means low distributional realism and vice versa.

Meng: From an engineering standpoint, the main caveat is that implementing Wasserstein BaryBooster introduces substantial overhead because it requires generating multiple candidate trajectories and solving that iterative aggregation procedure at inference time.

Lalam: And the paper’s limitation is that they only studied card transactions on a relatively small number of datasets, so we can't say this works perfectly across every domain yet.

More episodes

← Home