Predicting Delayed Train Trajectories on the Dutch Railway Network: Explainable AI Evaluation of Topological, Operational and Weather Features with Tree Based Ensemble Methods

arXiv:2609.34692 · cs.LG · Submitted 2026-09-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Predicting Delayed Train Trajectories on the Dutch Railway Network".

Jane: The gist: while feature-rich tree-based models improve withinmonth prediction, their performance experiences a statistically significant reduction when evaluated on future months.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’ve talked about what this paper is trying to achieve and the initial findings about performance dropping when moving from within-month to future predictions, focusing specifically on that Dutch railway network study. We’re now going to look closer at the actual title and who wrote it.

Jane: Yes, we'll start by reading the full title of this work: "Predicting Delayed Train Trajectories on the Dutch Railway Network: Explainable AI Evaluation of Topological, Operational and Weather Features with Tree Based Ensemble Methods." It tells you right away what’s in scope.

Lu: The authors are Jia Long Bao, Ali Mohammed Mansoor Alsahag, and Seyed Sahand Mohammadi Ziabari. They are bringing together topological data—the network structure—operational data about the trains themselves, and environmental features like the weather.

Meng: It’s a pretty comprehensive set of inputs for a single prediction task. It shows they aren't just looking at one thing; they are trying to capture how all these different layers interact to cause delays.

Tom: That’s right, and the title really highlights that they are using explainable AI—specifically tree-based ensembles with SHAP values—to evaluate this network-wide approach. It’s not just a black box prediction; it's about understanding the mechanics behind the numbers.

Jane: Because as we mentioned, those transparency requirements for high-risk infrastructure under rules like the EU AI Act are getting stricter, being able to explain *why* a system predicts a delay is becoming essential.

Lu: The authors chose tree-based ensembles because they bypass some of the strict statistical assumptions that other deep learning methods make, and they generate explicit feature importance rankings for us to look at.

Meng: That’s smart engineering; it gives us something tangible to inspect instead of just looking at millions of weights in a neural network. It makes the model transparent enough for auditors, which is key for critical systems.

Tom: And that leads into the next part—the actual summary of what they did and what they found in this paper about the methodology.

Jane: Right, we need to unpack the summary now to see how they set up their experiment and what specific questions drove their research on temporal robustness.

The paper's summary: Tom: Moving into the summary of "Predicting Delayed Train Trajectories on the Dutch Railway Network: Explainable AI Evaluation of Topological, Operational and Weather Features with Tree Based Ensemble Methods," we see how they structured their research around isolating macroscopic network states from real-time noise.

Jane: They partitioned their operational data into discrete monthly snapshots to try and isolate these larger states, which is a common way to look at long-term trends in time series data. But they admit that this method introduces a methodological trade-off.

Lu: The authors acknowledge that defining a binary target variable across these aggregated monthly periods introduces vulnerability to temporal label shift because the frequency of operational events can change depending on the season.

Meng: That vulnerability is key, right? It means their evaluation isn't perfectly clean, and they have to build in extra steps to handle that shift when testing on future months.

Tom: Exactly. They address this limitation by prioritizing temporal robustness by incorporating a sensitivity analysis to validate the target definition against alternative operational thresholds.

Jane: So they aren't just happy with one way of defining the problem; they are actively trying to test if changing that definition—say, from a general measure to a stricter one, like an absolute operational threshold—changes the results.

Lu: They also look at how their features work together by building feature sets independently before integrating them incrementally, which helps them assess those complex interactions upfront.

Tom: And they are using SHAP values strictly as a correlational diagnostic tool to quantify predictive reliance, which is important because it lets them see where the model is leaning, rather than trying to claim physical causality.

Jane: So their summary sets up a rigorous framework where they test feature robustness over time by comparing simultaneous testing against non-simultaneous testing on unseen future months.

The paper's improvements: Tom: Now we get into the actual improvements the authors suggest based on what they discovered in this work, which is where the practical recommendations come in for building better systems.

Jane: They are pushing toward a new approach to achieve better temporal generalization by moving beyond just accumulating tabular variables. They argue for dynamic, season-aware modeling architectures that are anchored by absolute operational boundaries.

Lu: That sounds like a design philosophy change—instead of just adding more features, you structure the model around known physical constraints of the system.

Meng: That aligns with what we see in other work where you try to constrain the learning process based on known physics or rules, which is much safer for real-time applications.

Tom: They also suggest a target definition sensitivity analysis, testing against an Absolute Operational Threshold—say, a fixed eighty percent punctuality, which is a zero point two zero threshold—and comparing that to a Month Relative Historical Seasonal Threshold.

Jane: So they’re asking: is the performance degradation they see driven by the global median target formulation itself, or is it really caused by external factors like environmental volatility? That’s a crucial diagnostic step.

Lu: They also propose integrating continuous regression targets, things like Schedule Adherence—that’s zero minute delay—and Regulatory Punctuality—which is three minute delay—to check for continuous predictive capacity within the same month.

Meng: Checking those continuous targets gives us a more nuanced look than just using a binary classification of "delayed" or "not delayed." It verifies that the feature sets have predictive power even when looking at fine-grained performance metrics.

Tom: And finally, they advocate for a post-hoc SHAP analysis pipeline specifically designed to track feature stability across temporal contexts, explicitly linking any degradation directly back to environmental feature volatility and instability in the statistical target definition.

Conclusion: Jane: So we’ve covered a lot today. To wrap up on this study of "Predicting Delayed Train Trajectories on the Dutch Railway Network: Explainable AI Evaluation of Topological, Operational and Weather Features with Tree Based Ensemble Methods," the overarching finding is that feature-rich tree-based models are better at predicting within the same month, but they suffer a statistically significant performance reduction when tested on future months.

Tom: That degradation is directly linked to the volatility in environmental features and instability in how we define our statistical targets over time. It’s a clear signal about what to watch out for when building these kinds of models for critical infrastructure.

Lu: The implication is that achieving better long-term generalization isn't just about throwing more variables at the problem; it requires moving toward those dynamic, season-aware architectures anchored by absolute operational boundaries.

Meng: For us in engineering, this means we need to focus on how we structure the inputs to ensure they are stable across different operational contexts before we deploy them for long-term forecasting.

Lalam: I see the vision in this work as pushing us toward models that are inherently more resilient because they prioritize the operational core and control for external noise instead of just chasing higher accuracy numbers.

Tom: That’s it for today on this paper. We’ll be looking at other interesting papers next time, but this one really hammers home the need for stability in temporal modeling.

cs.LG

Submitted: 2026-09-28

Updated: 2026-10-12

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The gist: while feature-rich tree-based models improve withinmonth prediction, their performance experiences a statistically significant reduction when evaluated on future months.

Key concepts

Simultaneous Testing: This testing method involves training and evaluating the model on data from the exact same month. It shows how well a model performs at predicting delays for events happening in the immediate future, capturing short-term patterns.
Non-Simultaneous Testing: This involves training a model on historical monthly data and then testing its performance on entirely new, unseen months in the future. This tests the model's ability to generalize its learned patterns to different time periods.
SHAP Values: SHAP (SHapley Additive exPlanations) is used here as a tool to understand which input features most influenced a prediction. It helps diagnose why performance changes over time by showing how much each feature contributes to the model's output.
Feature Volatility: This refers to how much the statistical distribution of a feature changes from one time period to the next. The study found that weather features were highly volatile, meaning their patterns are unstable and change frequently, which hurts long-term prediction.

Summary

The gist: while feature-rich tree-based models improve withinmonth prediction, their performance experiences a statistically significant reduction when evaluated on future months.

Economic and Operational Context

Railway delays have significant economic consequences, with the Dutch national railway operator distributing €9.7 million in annual compensation and refunds in 2025. Beyond direct fiscal burdens, schedule deviations result in cumulative time losses costing the Dutch economy over €400 million annually. The high-density rail network amplifies sensitivities to operational failures and environmental factors like winter weather. Furthermore, the European Union Artificial Intelligence Act mandates transparency and human oversight for automated decision-making systems in high-risk infrastructure domains such as the Dutch national railway. This research utilizes SHapley Additive exPlanations (SHAP) to align network-wide delay modeling with these emerging regulatory requirements.

Methodological Approach and Research Questions

The study employs interpretable treebased ensembles to integrate granular topological, environmental, and operational features for network-wide analysis. The temporal evaluation framework partitions operational data into discrete monthly snapshots to isolate macroscopic network states from real-time noise. The research questions guide the investigation, specifically asking how predictive performance varies between simultaneous (withinmonth) and non-simultaneous (future unseen months) testing phases.

Feature Engineering and Model Configuration

Predictive features were engineered at the aggregated monthly trajectory level to align with long-term predictive horizons. This involved engineering operational features like a decayed operational edge weight (Wdecayed) to approximate sequential pattern recognition without inducing target leakage. Spatial and topological features were constructed using dynamic, directed network graphs Gt = (Vt, Et) for each month. Environmental weather features were aggregated into monthly station-level features to scale daily predictive validity into macroscopic seasonal network vulnerabilities. The feature set construction involved evaluating feature sets independently before incremental integration to assess complex interactions, with the deliberate choice of omitting explicit feature selection prior to training.

Evaluation Framework and Results

The evaluation framework utilized two primary data splitting schemas: Simultaneous Testing (trained and tested within the same monthly snapshot) and Non-Simultaneous Testing (trained on historical monthly snapshot, tested on unseen future months). The primary metric used was Balanced Accuracy, which is calculated as 1/(2(TP + FN + TN)). The main finding regarding temporal generalization is that while simultaneous testing shows discriminative capacity, non-simultaneous testing exhibits a structural degradation pattern across most feature configurations. Specifically, the fully integrated architecture (All Features) experienced a significant median accuracy loss of 0.069 when shifted from simultaneous to non-simultaneous evaluation.

Model Behavior and Feature Stability Analysis

SHAP values are utilized strictly as a correlational diagnostic tool to quantify temporal shifts in predictive reliance, rather than to infer physical network causality. Analysis of feature importance rankings across multiple temporal horizons reveals substantial instability, with environmental features showing the highest volatility and operational features displaying comparatively greater consistency. The Kendall rank correlation coefficient (τ) was employed to quantify the ordinal consistency of SHAP feature importance rankings across sequential temporal transitions, revealing that the Operational (OPS) configuration demonstrated the highest temporal stability with a mean τ = 0.831. Furthermore, distributional tests showed that Weather features (WTH) exhibited extreme volatility, with statistically significant distributional drift occurring in 84.81% of all sequential temporal transitions.

Conclusion

The overarching finding establishes that feature-rich tree-based models improve withinmonth prediction, but their performance experiences a statistically significant reduction when evaluated on future months. This degradation is explicitly linked to environmental feature volatility and instability in the statistical target definition. Achieving robust temporal generalization necessitates transitioning beyond the mere accumulation of tabular variables toward dynamic, season-aware modeling architectures anchored by absolute, operationally grounded target designs.

Improvements for AI systems

The improved system will employ a dynamic, season-aware architecture anchored by absolute operational boundaries to prevent predictive performance degradation when evaluated across non-simultaneous months, as suggested by the necessity to transition toward dynamic, season-aware architectures anchored by absolute operational boundaries in the abstract. This allows the model to maintain robust generalization beyond immediate monthly snapshots.
Implement a target definition sensitivity analysis, testing against an Absolute Operational Threshold (fixed 80% punctuality, in other words a 0.20 threshold) and a Month Relative Historical Seasonal Threshold, to diagnose whether performance degradation is driven by the global median target formulation itself or external factors like environmental volatility.
Integrate continuous regression targets, such as Schedule Adherence (0-minute delay) and Regulatory Punctuality (3-minute delay), into the model evaluation framework to verify that the feature sets possess continuous predictive capacity within-month, providing a more nuanced assessment than binary classification alone.
Develop a post-hoc SHAP analysis pipeline specifically designed to track feature stability across temporal contexts using Kendall rank correlation coefficients, explicitly linking temporal degradation to environmental feature volatility and instability in the statistical target definition.
The system will utilize conditional feature integration logic where the addition of volatile variables is systematically assessed: the inclusion of environmental variables (weather features) actively degraded predictive capacity; the enhancement hypothesis was completely rejected (p = 1.000) as the median accuracy decreased. This prevents the automatic inclusion of features that cause performance collapse in non-simultaneous testing.