Latent-driven Imputation and Forecasting for partially observed Time Series (LIFTS)
summary
The gist
Partially observed multivariate time series pose two coupled challenges: recovering missing measurements and forecasting future trajectories when observation patterns are informative.
In short
LIFTS is a probabilistic framework that jointly handles missing data imputation and future forecasting in multivariate time series when some measurements are missing but the pattern of missingness is informative. It introduces a latent process that governs both the complete measurements and the observation mask, allowing patterns to inform better state estimation for both tasks. The method achieves accurate predictions, improved uncertainty quantification, and faster inference than diffusion models.
Key concepts
- Latent-driven Missingness (LDM)
- This concept assumes that missing data is not random but is instead driven by an underlying hidden state of the system, rather than being purely random noise. The framework models this by assuming the missingness pattern ($O_t$) depends on the latent state ($Z_t$), meaning knowing how a variable is likely to be missing helps infer what its true value might be.
- Forward-Backward Architecture
- This structure organizes the problem into two main parts: a forward pass for filtering and forecasting, and a backward pass for smoothing and imputation. The forward pass uses history to estimate the current state, while the backward pass incorporates future information to refine estimates of past states, enabling both prediction and filling in missing values.
- Joint Training Objective
- The model's generative (forecasting/imputation) and inference components are trained simultaneously using a combined loss function. This loss balances three goals: accurate forecasting, correct imputation of missing data, and ensuring the model's predictions are well-calibrated. This joint training allows the system to learn a more robust representation of the underlying time series.
- Energy-Score Objectives
- These are sample-based objectives used during training to guide the neural networks. They measure how well a proposed model matches real data by quantifying an 'energy score' related to observation laws. Minimizing this score ensures that the learned parameters produce predictions consistent with the observed patterns in the partially observed data.
Terminology used across episodes
This episode discusses
- Latent-driven Imputation and Forecasting for partially observed Time Series (LIFTS) · Paper Radio
- TSI-Bench: Benchmarking Time Series Imputation
- not-MIWAE: Deep Generative Modelling with Missing not at Random Data
- Time Series Generative Learning with Application to Brain Imaging Analysis
- Missingness-aware Data Imputation via AI-powered Bayesian Generative Modeling
- Distributional Principal Autoencoders
- Identifiable Deep Latent Variable Models for MNAR Data
- Emputation: Identification-Guided Neural Imputation Framework · Paper Radio
- Are Transformers universal approximators of sequence-to-sequence functions?
The paper
Latent-driven Imputation and Forecasting for partially observed Time Series (LIFTS) · Read on arXiv
Sanyou Wu, Ying Wei, Kaizheng Wang
Department of Biostatistics, Mailman School of Public Health, Columbia University · Department of IEOR and Data Science Institute, Columbia University
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Latent-driven Imputation and Forecasting for partially observed Time Series (LIFTS)".
Jane: Here's the gist of the paper. Partially observed multivariate time series pose two coupled challenges: recovering missing measurements and forecasting future trajectories when observation patterns are informative.
Tom: LIFTS (Latent-driven Imputation and Forecasting for partly observed Time Series) is a probabilistic framework in which a latent process jointly governs the complete measurement trajectory and observation mask, allowing observation patterns to inform latent-state inference for both forecasting and imputation.
Jane: Problem Formulation The paper introduces the problem setup by representing the complete longitudinal trajectory as a matrix X1:T, where Xt is the full feature vector at time t.
Tom: To model this, a binary mask Ot is introduced to indicate whether the j-th component of Xt is observed or missing.
Paper summary: Tom: So we're looking at this paper called Latent-driven Imputation and Forecasting for partially observed Time Series. The main idea here is tackling the two problems at once: figuring out what's missing from a time series and then predicting what happens next when we only see some of the data.
Jane: It seems like they are aiming to solve something that existing methods often treat as separate issues, assuming the missing parts happen randomly or just using a mask as extra input.
Lu: Yeah, that's exactly right. The core thesis is proposing a probabilistic framework where a latent process controls both the full measurements and the observation mask simultaneously.
Meng: So, instead of just guessing what's missing, they want this hidden state to actually inform how we do both imputation and forecasting.
Tom: Right, so it’s about using that hidden state to guide the inference for both tasks. The structure they build uses a forward-backward architecture, with the forward pass handling filtering and autoregressive forecasting, and the backward pass doing smoothing and conditional multiple imputation.
Jane: That sounds like a nice way to tie retrospective imputation with future prediction together under one umbrella.
Lalam: I see how that unified probabilistic treatment addresses how observation patterns can become informative because they depend on that underlying state rather than just being random.
Tom: Exactly, and the training setup is interesting too. They use sample-based energy-score objectives to train the model, which includes a forecasting loss, an imputation loss, and a synthetic calibration loss for tuning.
Meng: From my side at scale, I’m interested in how they handle that calibration part. How does it translate to actual deployment when we're running models on live traffic?
Jane: The paper claims that this joint training objective allows the model to learn how the underlying process and the observation pattern are linked through that latent state. It sets up a way for observation patterns to actually inform the inference about those hidden states.
Lu: What’s really compelling for me is how they handle identifiability. They establish full-law identification and consistency for a specific class of models under mild regularity conditions.
Tom: Consistency sounds like it’s proving that what the model learns isn't just some lucky fit on the training data, but actually matches the true underlying statistical distribution of the process itself.
Jane: So they show that even when we only have partial data, if we train this way, we can recover the true full law of the time series.
Lu: That’s what Theorem five shows; they prove observation-law consistency and finite horizon full-law identifiability for that specific structural subclass under mild regularity conditions <ref:2610.03916#pg3>. This is a strong statement about the model's ability to learn correctly even when things are partially observed.
Meng: But what does this mean for production? If we’re deploying this, how much effort does it take to train these neural networks compared to, say, a standard diffusion-based baseline?
Tom: That’s where I want the numbers. The summary mentions that LIFTS has substantially faster inference than the diffusion-based baseline.
Jane: It also claims accuracy improvements—not just point predictions—in terms of distributional accuracy and well-calibrated predictive intervals. They even show empirical coverage that is close to nominal levels while keeping prediction intervals relatively narrow.
Lalam: So, when you’re looking at real data, this method seems to give you more reliable bounds on what's coming next than other methods we’ve seen.
Tom: It outperforms competing methods especially when the training data available are limited, and it has substantially faster inference than the diffusion-based baseline. That speed is a big deal for deployment latency.
Lu: The distributional gains are particularly pronounced when training data is limited, which makes sense because the latent state helps regularize things when you don't have a ton of complete examples to learn from.
Jane: So, to sum up this paper on Latent-driven Imputation and Forecasting for partially observed Time Series, it introduces a probabilistic framework that uses a latent process to jointly govern measurements and observation masks. The authors show that this structure allows observation patterns to inform the latent state inference for both imputation and forecasting tasks.
Tom: It’s essentially combining filtering with smoothing in one unified architecture.
Lu: And they prove that under certain conditions, the learned model consistently recovers the true full law of the time series distribution, even when only partially observed.
Jane: The implication is that we can handle complex dynamics where the timing and availability of measurements depend on things like external constraints or operational decisions without needing to specify a separate mechanism for every single missing data pattern.
Meng: It sounds like a solid way to build something that's operationally feasible, provided the training setup doesn't become too heavy at scale.
Tom: The paper also notes that while this framework is powerful for structured classes of informative observation mechanisms, they don't prove consistency for all possible missing-not-at-random mechanisms. That’s a fair limitation to keep in mind when trying to apply it broadly.
Jane: So, the next step seems to be seeing how robust this works across different types of time series dynamics that fall into those structured classes.
Conclusion: Tom: So we’ve been digging into LIFTS, Latent-driven Imputation and Forecasting for partially observed Time Series, and now we’re wrapping up with what this actually means for us.
Jane: Basically, the authors propose this probabilistic framework where a hidden state drives both how they fill in the missing data and how they predict what happens next.
Tom: That hidden state is key because it lets the observation pattern itself inform the model, instead of just treating those missing parts as random noise or something separate.
Lu: They tackle two coupled challenges at once: recovering what's gone and looking ahead based on what we see now. The methodology uses a forward pass for filtering and a backward pass for smoothing and imputation.
Meng: And they train all these neural networks together using sample-based energy-score objectives, which is how they calibrate the model to get those right predictions.
Lalam: The paper establishes full law identification and consistency for a structural subclass of models, meaning if the assumptions hold, the AI learns the true underlying statistical distribution of your time series even with missing pieces.
Jane: That’s a big deal because it means we can trust the imputation and forecasting more when we only have partial data, especially when our training examples are limited.
Tom: It also claims it’s substantially faster at inference than diffusion-based baselines, which is huge for deployment speed. The computational cost seems manageable compared to the gains in accuracy.
Lu: And they show this works well on real-world data from PhysioNet two thousand nineteen giving them good predictive intervals that are well-calibrated.
Jane: So, to put it simply, LIFTS gives us a way to handle complex time series where we have missing information by using a smart hidden state that guides both the filling in and the future predictions.
Tom: It’s moving away from just treating missing data as an extra input and making the observation pattern part of the core inference.
Jane: That opens up new ways to build systems that are more robust when they aren't seeing perfect, complete data streams.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates