The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial

arXiv:2602.06603 · cs.LG · Submitted 2026-02-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "The impact of temporal resampling on the safety of clinical reinforcement learning".

Jane: The gist The hidden risks of temporal resampling in clinical reinforcement learning demonstrate that temporal resampling can reduce model performance by up to 60% relative to unprocessed data and that…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at "The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial," we see that binning data to make it uniform time intervals introduces real risks because it creates flawed training data and leads to performance drops of up to sixty percent in the simulation.

Jane: And what I find particularly concerning is how standard retrospective evaluation can actively hide this problem by predicting returns that are significantly better than actual practice, sometimes by a factor of one point five to three. This means we have a dangerous way of assessing safety if we rely on these resampled datasets alone.

Lu: The paper highlights the persistent gap between research and clinical deployment, and they show that forcing natural variations into a discretized schedule risks training models that only work for those specific schedules and adapt poorly when things become unpredictable in reality.

Meng: From a production standpoint, this suggests we need to be extremely careful about how we preprocess EHR data before feeding it into an RL model, because the discretization step could be silently degrading the model's ability to handle real-world variability.

Lalam: The authors recommend that future research should prioritize datasets that naturally preserve the intervals between decisions so these models can be safely deployed into patient care.

Tom: So, the main implication for us is clear: we need to stop treating resampled data as if it perfectly represents clinical reality, especially when testing model performance. This paper shows us that temporal resampling isn't just a data cleaning step; it's a modeling risk if we don't account for the true timing structure.

Jane: It really puts the onus on us to demand datasets that reflect the natural timing of patient care before we start trusting these AI models with real patient outcomes.

Conclusion: Tom: So, we just finished looking at how binning data in clinical reinforcement learning actually hurts model performance in simulation. Now we're talking about the actual conclusion of this paper, "The impact of temporal resampling on the safety of clinical reinforcement learning."

Jane: The title itself is pretty straightforward, but it points to something really important about making these models safe for real patients. It's not just a technical tweak; it’s a safety issue.

Lu: Yeah, what the authors are saying boils down to this: temporal resampling can fundamentally misrepresent the actual timing of clinical events in the training data.

Meng: So when they look at the results, it seems like if you resample your data—say into ten-minute intervals—your model performance drops by up to sixty percent compared to using the raw, natural timing data.

Tom: That sixty percent number is what really caught my attention. It’s not just a small dip; it’s a significant drop in how good the AI actually gets at making choices.

Jane: And then they show that standard evaluation methods can completely hide this problem because they predict the models will do way better than they actually do in practice, sometimes by a factor of one point five to three times better returns.

Lu: The authors are really highlighting that this overoptimistic evaluation is dangerous. It means you might think an AI is safe when it’s actually fragile and can’t handle real-world unpredictability.

Meng: From a deployment standpoint, that’s worrying because if we rely on those retrospective evaluations to sign off on a model, we're missing the fact that it's already underperforming in the actual environment.

Jane: So what does this mean for someone just listening? It means before you trust these AI models with patient care decisions, you have to seriously question whether they were trained on data that respects the natural flow of time between clinical actions.

Tom: Exactly. We need to shift our focus from just getting a high score in a test set to making sure the model understands the real-world timing structure before we let it interact with patients.

Lu: The paper suggests prioritizing datasets where the decision labels naturally have their own intervals, which is what they call retaining natural intervals between decision labels.

Jane: It sounds like the authors are telling us that for these reinforcement learning models to be viable in healthcare, we need to stop forcing everything into neat time bins and respect how things actually happen in a clinical setting.

Tom: So the next big question is figuring out how we can build AI frameworks that handle those irregular timings properly instead of just throwing data into a uniform schedule.

University College London · University College London Hospitals NHS Foundation Trust

cs.LG

Submitted: 2026-02-06

Updated: 2026-10-15

Comments: 14 pages, 4 figures, 1 table, 6 suppl figures, 2 suppl tables. Manuscript in press, accepted by PLOS Digital Health

Code: https://github.com/tdgfrost/temporal-resampling-experiments

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The hidden risks of temporal resampling in clinical reinforcement learning demonstrate that temporal resampling can reduce model performance by up to 60% relative to unprocessed data and

Key concepts

Temporal Resampling (Binning): This is the process of aggregating or interpolating irregular clinical data into uniform time intervals (like 10-minute or 4-hour blocks) before training an AI model. The paper shows this hides natural, unpredictable decision timings crucial for real patient care.
Offline Reinforcement Learning (ORL): This is a type of AI where models learn optimal actions solely from existing, historical clinical datasets without interacting with the environment during training. It is used in healthcare because direct interaction with patients is unsafe.
Fitted Q-Evaluation (FQE): This technique is used to retrospectively estimate how well a trained model will perform on new data. The paper found that FQE severely overestimates the performance of models trained on resampled data, creating a major risk for clinical deployment.

Terminology

Reinforcement Learning — 2× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes

Summary

The gist The hidden risks of temporal resampling in clinical reinforcement learning demonstrate that temporal resampling can reduce model performance by up to 60% relative to unprocessed data and that standard retrospective evaluation can actively obscure this negative effect with overoptimistic estimates.

Problem Context

Reinforcement learning (RL) is often used in healthcare through offline RL (ORL), which relies on retrospective observational data because online interaction is impractical due to patient safety risks <ref:2602.066031, "Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data."

A prevalent feature in healthcare ORL pipelines is temporal resampling, which involves the aggregation or interpolation of electronic health record (EHR) data into standardized time intervals prior to model training, often known as “binning” <ref:2602.066032, "A particularly prevalent feature of healthcare ORL pipelines is temporal resampling: the aggregation or interpolation of electronic health record (EHR) data into standardised time intervals prior to model training, often known as “binning”." This simplification occurs despite widely available architectures for processing irregularly sampled data directly <ref:2602.066033, Of the 50 studies reviewed, 43 used datasets that had been temporally resampled. The popularity of discretising data is likely due to the convenience offered by regular decision labels, which simplifies the Markov decision process (MDP) framework underpinning most ORL algorithms <ref:2602.066034, "The popularity of discretising data likely relates to the convenience offered by regular decision labels, which greatly simplifies the implementation of the Markov decision process (MDP) framework underpinning most ORL algorithms."

Impact on Model Performance

The study conducted an in silico clinical trial using 30 virtual patients from the FDA-approved UVA/Padova simulator to explore these effects <ref:2602.066039, We chose to explore the effects of this further by conducting an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. When trained on resampled data, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data <ref:2602.066035, When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data. Furthermore, the study observed that 4-hour binning caused all agents to perform worse than the dataset’s baseline <ref:2602.066035, with 4-hour binning causing all agents to perform worse than the dataset’s baseline.

Counterfactual Trajectories and Data Artifacts

Temporal binning can lead to counterfactual training data by reversing the causal ordering of events through data aggregation <ref:2602.066031, temporal binning can reverse the causal ordering of events, leading to inherently flawed training data. This artifact arises because observations are averaged over the window immediately preceding the action <ref:2602.066031, This artefact arises because observations are averaged over the window immediately preceding the action (see Methods). While broad trends for blood glucose may be preserved, insulin actions deviate significantly after binning, as brief spikes are dampened and replaced by gradual changes <ref:2602.066031, However, insulin actions are found to deviate significantly after binning: brief, high-frequency spikes are dampened and replaced by gradual changes.

Evaluation Bias

Retrospective evaluation on resampled data actively obscured the performance degradation of the models by predicting 1.5–3x better returns than agents achieved in practice <ref:2602.066031, Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5–3x better returns than agents achieved in practice. This systematic overestimation likely reflects that models trained on binned datasets solve for actions that appear highly favorable within the binned datasets, even if those actions are suboptimal in reality <ref:2602.066031, This systematic overestimation likely reflects the fact that IQL and CQL are solving for actions that appear highly favourable within the binned datasets, even if such actions are suboptimal in reality.

Proposed Solution

As an alternative to temporal resampling, the authors propose retaining natural intervals between decision labels and utilizing reinforcement learning frameworks that explicitly account for them, such as the semi-MDP (SMDP) framework employed in this work <ref:2602.066031, we propose retaining the natural intervals between decision labels and utilising reinforcement learning frameworks that explicitly account for them (such as the SMDP framework employed in this work). This approach provides the model with the most accurate representation of temporal dynamics, resulting in the greatest capacity to improve upon the dataset’s baseline behaviour while critically retaining accuracy of retrospective evaluation <ref:2602.066031, "As demonstrated by our results, this approach provides the model with the most accurate representation of the environment’s temporal dynamics, resulting in the greatest capacity to improve upon the dataset’s baseline behaviour, whilst critically retaining accuracy of retrospective evaluation."

Conclusion

The findings confirm that temporal resampling can lead to a fundamental misalignment between training data and deployment environments <ref:2602.066031, Our findings confirm that temporal resampling can lead to a fundamental misalignment between the training data and the intended deployment environment. By forcing natural variations into a discretized schedule, researchers risk training models that are only optimal for those schedules—and adapt poorly to any unpredictability <ref:2602.066031, By forcing natural variations in decision intervals into a discretised schedule, researchers risk training models that are only optimal for those schedules—and adapt poorly to any unpredictability. Critically, the fragility of these models is undetectable by standard retrospective evaluation methods when performed on similarly resampled data <ref:2602.066031, Critically, the fragility of these models is undetectable by standard retrospective evaluation methods when performed on similarly resampled data. Any falsely optimistic evaluations could result in the premature justification of a clinical trial for these models <ref:2602.066031, Any falsely optimistic evaluations could result in the premature justification of a clinical trial for these models. Therefore, future research should prioritize methods that preserve natural decision intervals to improve model safety and viability before deployment into patient care <ref:2602.066031, We therefore recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.

How it works

The researchers conducted an in silico clinical trial using 30 virtual patients from the UVA/Padova simulator to test their hypothesis <ref:2602.066039, We chose to explore the effects of this further by conducting an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. They trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals <ref:2602.066035, We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. The evaluation compared prospective performance in the simulator against those predicted by standard retrospective evaluation <ref:2602.066031, We then evaluated each model’s prospective performance in the simulator, and compared these results against those predicted by standard retrospective evaluation.

Dataset Generation

The training dataset was generated from a ground-truth reference dataset created by interacting repeatedly with the environment's irregular version of the simulator <ref:2602.066031, We selected an intermediate checkpoint achieving approximately 70–80% of the agent’s best overall performance to serve as the ‘expert policy’. This process resulted in three dataset versions: (1) Unprocessed dataset, which records all state transitions and a mask identifying independent agent decisions <ref:2602.066031, Unprocessed dataset: Records all state transitions and a mask identifying independent agent decisions, preserving the semi-MDP structure. (2) Interpolated dataset, which treats all transitions as independent decisions by upsampling the decision frequency <ref:2602.066031, Interpolated dataset: Identical to the unprocessed dataset, but treats all transitions as independent decisions. (3) Binned dataset, produced via temporal binning where observations and rewards are aggregated over preceding windows <ref:2602.066031, Binned dataset: Produced via temporal binning. In LavaGap, every second time-step was subsampled.

Offline Training Algorithms

The study evaluated three offline RL algorithms on these datasets: behavioural cloning (BC), implicit Q-learning (IQL), and conservative Q-learning (CQL) <ref:2602.066031, "We evaluated the effect of each dataset using three algorithms: behavioural cloning (BC), implicit Q-learning (IQL), and conservative Q-learning (CQL).

Improvements for AI systems

We should prioritize datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care, to prevent temporal resampling from causing performance degradation of up to 60% in practice when deployed back into simulated environments.
Implement reinforcement learning frameworks that explicitly account for variable decision intervals, such as the semi-MDP (SMDP) framework employed in this work, which provides the model with the most accurate representation of the environment’s temporal dynamics.
Develop safety protocols to detect and mitigate counterfactual training data generated by binning, as this process can lead to inherently flawed training data where observations are averaged over windows that may reverse the real sequence of events.
Utilize fitted Q-evaluation (FQE) with caution; while it provides a measure of performance, its predictions are prone to being hallucinations rather than accurate estimates when applied to binned data, potentially leading to an overestimation of model performance by 1.5–3x.
Design evaluation protocols that compare prospective performance against standard retrospective evaluation more rigorously, ensuring that models are not justified based on evaluations that actively obscured this effect and could be used to justify clinical trials for suboptimal models.

Abstract

Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data. To accommodate naturally irregular clinical records, researchers often resample the data into uniform time intervals before training (known as binning). However, binning may create counterfactual trajectories that do not represent clinical practice. We first quantified these distortions in patients receiving continuous insulin infusions in the MIMIC-IV intensive care database. Binning reversed the apparent causal order for over 70% of paired glucose measurements and insulin adjustments within the same interval, and frequently masked significant dysglycaemic events and merged distinct treatment decisions together. We then conducted an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. The simulator was modified to include stochastic intervals between decisions and used to generate a training dataset for offline RL. We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data, with 4-hour binning causing all agents to perform worse than the dataset's baseline. Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5-3x better returns than agents achieved in practice. We recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.