Bridging Impulse Control of Piecewise Deterministic Markov Processes and Markov Decision Processes: Frameworks, Extensions, and Open Challenges
stat.ME, cs.SY, eess.SY
Submitted: 2025-01-07
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Control theory plays a pivotal role in understanding and optimizing the behavior of complex dynamical systems across various scientific and engineering disciplines.
Terminology
Abstract
Control theory plays a pivotal role in understanding and optimizing the behavior of complex dynamical systems across various scientific and engineering disciplines. Two key frameworks that have emerged for modeling and solving control problems in stochastic systems are piecewise deterministic Markov processes (PDMPs) and Markov decision processes (MDPs). Each framework has its unique strengths, and their intersection offers promising opportunities for tackling a broad class of problems, particularly in the context of impulse controls and decision-making in complex systems. The relationship between PDMPs and MDPs is a natural subject of exploration, as embedding impulse control problems for PDMPs into the MDP framework could open new avenues for their analysis and resolution. Specifically, this integration would allow leveraging the computational and theoretical tools developed for MDPs to address the challenges inherent in PDMPs. On the other hand, PDMPs can offer a versatile and simple paradigm to model continuous time problems that are often described as discrete-time MDPs parametrized by complex transition kernels. This transformation has the potential to bridge the gap between the two frameworks, enabling solutions to previously intractable problems and expanding the scope of both fields. This paper presents a comprehensive review of two research domains, illustrated through a recurring medical example. The example is revisited and progressively formalized within the framework of thevarious concepts and objects introduced
Sources
- OpenAI Gym
- Mastering Diverse Domains through World Models
- Playing Atari with Deep Reinforcement Learning
- Proximal Policy Optimization Algorithms
- A new twist for the simulation of hybrid systems using the true jump method
- Dueling Network Architectures for Deep Reinforcement Learning
Related papers
- Doubly robust inference via calibration
- Bayesian Empirical Bayes: Simultaneous Inference from Probabilistic Symmetries
- Optimal Watermark Localization in Mixed-Source Large Language Model Texts
- Flexible Nonparametric Inference for Causal Effects under the Front-Door Model
- A Greedy PDE Router for Blending Neural Operators and Classical Methods
- Policy Learning with Weak Signals