Deep Reinforcement Learning for Dynamic Origin-Destination Matrix Estimation in Microscopic Traffic Simulations Considering Credit Assignment

arXiv:2511.06229 · cs.LG · Submitted 2025-11-09 · Read on arXiv

cs.LG

Submitted: 2025-11-09

Updated: 2026-08-30

Comments: Accepted for publication in Transportation Research Part C: Emerging Technologies

Journal ref: Transportation Research Part C: Emerging Technologies, Volume 194, Part A, 2027, 105991

DOI: 10.1016/j.trc.2026.105991

Code: https://github.com/dgmin-kr/dode-rl-sumo

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

The gist: This paper focuses on dynamic origin-destination matrix estimation (DODE), a crucial calibration process necessary for the effective application of microscopic traffic simulations.

Terminology

Calibration — 33× in this paper · explained in 7 episodes
Credit assignment — 21× in this paper · explained in 3 episodes
Reinforcement Learning — 15× in this paper · explained in 4 episodes
Deep Reinforcement Learning — 10× in this paper · explained in 2 episodes
Actor — 8× in this paper · explained in 3 episodes
Reward Function — 7× in this paper · explained in 2 episodes
Action Space — 6× in this paper · explained in 1 episode
Accuracy — 5× in this paper · explained in 1 episode
Bayesian Optimization — 5× in this paper · explained in 3 episodes
Cosine Similarity — 5× in this paper · explained in 3 episodes
Benchmark — 4× in this paper · explained in 2 episodes
Regularization — 4× in this paper · explained in 3 episodes
Clustering — 3× in this paper · explained in 2 episodes
Core — 3× in this paper · explained in 2 episodes
Deep Reinforcement Learning (DRL) — 3× in this paper · explained in 2 episodes
Markov Decision Process — 3× in this paper
Markov Decision Process (MDP) — 3× in this paper · explained in 11 episodes
Objective Function — 3× in this paper · explained in 2 episodes
Actor-Critic Architecture — 2× in this paper · explained in 1 episode
Adaptation — 2× in this paper · explained in 1 episode
Bayesian Optimization (BO) — 2× in this paper · explained in 3 episodes
Fidelity — 2× in this paper · explained in 2 episodes
Sample Efficiency — 2× in this paper · explained in 4 episodes
alpha — 2× in this paper · explained in 1 episode
Acquisition Function — 1× in this paper · explained in 1 episode
Actor-Critic Framework — 1× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Aggregation — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 3 episodes
Computational Tractability — 1× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Curse of Dimensionality — 1× in this paper · explained in 6 episodes
Digital Twin — 1× in this paper · explained in 4 episodes
Inverse Problem — 1× in this paper · explained in 2 episodes
Loss Function — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes
Spectral clustering — 1× in this paper · explained in 2 episodes
Trace — 1× in this paper · explained in 2 episodes
Transferability — 1× in this paper · explained in 4 episodes
Unified Framework — 1× in this paper · explained in 5 episodes

Abstract

This paper focuses on dynamic origin-destination matrix estimation (DODE), a crucial calibration process necessary for the effective application of microscopic traffic simulations. The fundamental challenge of the DODE problem in microscopic simulations stems from the complex temporal dynamics and inherent uncertainty of individual vehicle dynamics. This makes it highly challenging to precisely determine which vehicle traverses which link at any given moment, resulting in intricate and often ambiguous relationships between origin-destination (OD) matrices and their contributions to resultant link flows. This phenomenon constitutes the credit assignment problem, a central challenge addressed in this study. We formulate the DODE problem as a Markov Decision Process (MDP) and propose a novel framework that applies model-free deep reinforcement learning (DRL). Within our proposed framework, the agent learns an optimal policy to sequentially generate OD matrices, refining its strategy through direct interaction with the simulation environment. This approach was evaluated through a toy experiment on the Nguyen-Dupuis network and a case study utilizing an actual highway subnetwork spanning Santa Clara and San Jose. Experimental results show that the proposed method consistently improves calibration performance relative to the strongest conventional baseline, reducing link-flow MSE by 23.7% in the toy experiment and by 59.2-88.3% in the real-world case study. By reframing DODE as a sequential decision-making problem, our approach addresses the credit assignment challenge through a learned policy and provides a novel framework for calibration of microscopic traffic simulations.

Terminology

Abstract

Related papers