Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Mathematics. The day in mathematics: new theorems, new proofs, and what they open.
Ada: I'm Ada.
Emmy: And I'm Emmy, here with Kolya.
Ada: We're looking at the paper: "Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control".
Emmy: Here's the gist of the paper. Finite-horizon optimal control becomes difficult when a dynamical model can be simulated reliably but its derivatives are unavailable, discontinuous, or unreliable The method is validated on nonsmooth problems with exact solutions, an analytical and learned ReLU biomarker cascade, an amyloid Neural ODE trained from longitudinal ADNI data, and three multidimensional-control problems with up to five simultaneous controls Across these examples, HPDPO closely reproduces exact or reference objective values, trajectories, and control structures The results show that forward simulation and Hamiltonian dual structure can be combined to solve deterministic finite-horizon control problems involving nonsmooth, learned, or black-box dynamics.
Ada: Hamiltonian Primal–Dual Policy Optimization (HPDPO) is introduced as a derivative-free, model-based reinforcement learning framework for this setting The control is represented as a time-indexed finite-horizon policy Candidate actions are evaluated by deterministic suffix rollouts of the supplied transition model, while a regularized Hamiltonian dual formulation supplies structured future-cost information through block-coordinate multiplier updates The main contributions are as follows: 1.
Emmy: We formulate the time-discretized finite-horizon control problem as a deterministic model-based policy–optimization problem while separating the role of discretization from the policy–update mechanism.
Paper summary: Ada: So we've covered what HPDPO is and where they tested it, but let's get a bit more into why this matters for the broader field of control. The paper "Hamiltonian Primal–Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control" is focused on making optimal control practical when the system dynamics are messy.
Emmy: They address the situation where you have a model that you can simulate reliably, but its derivatives are unavailable, discontinuous, or just unreliable for optimization purposes <ref:2610.18524#pg2>. This happens with models like ReLU networks which are continuous but not classically differentiable at their activation boundaries five <ref:2610.18524#pg2,are continuous but not classically differentiable at>.
Kolya: So the paper introduces HPDPO as a derivative-free framework for this setting, treating the control as a time-indexed finite-horizon policy <ref:2610.18524#pg3>. They evaluate candidate actions by running deterministic suffix rollouts of that supplied transition model to get the downstream cost.
Ada: And crucially, they pair that with a regularized Hamiltonian dual formulation to supply structured future-cost information through block-coordinate multiplier updates <ref:2610.18524#pg3>. This avoids the need for explicit differentiation of the state dynamics or any learned critic at all.
Emmy: The main contributions are formulated as three points: first, treating the control sequence as a primal variable and the multiplier sequence as a dual variable <ref:2610.18524#pg3>; second, deriving this regularized Hamiltonian primal-dual objective <ref:2610.18524#pg3>; and third, replacing derivative-based costate integration with bounded block-coordinate optimization of the multiplier trajectory <ref:2610.18524#pg3>.
Kolya: What they mean by that last point is they are using this block-coordinate optimization to replace the costate equation update <ref:2610.18524#pg3>. This is a practical way to handle those missing derivatives in complex systems.
Ada: The validation section shows it performs well across several benchmarks: nonsmooth problems with exact solutions, analytical and learned ReLU cascade models, an amyloid neural ODE from ADNI data four, and three multidimensional control problems with up to five simultaneous controls <ref:2610.18524#pg1>.
Emmy: The results confirm that HPDPO closely reproduces the known optimal control and corresponding state trajectory with substantially smaller numerical errors on this deterministic fixed-initial-state benchmark, like in Example three where they get very small control, state, and objective errors <ref:2610.18524#pg1>.
Kolya: For me, the interesting part is that the computation time for HPDPO is often smaller than what you'd expect from testing PPO or SAC implementations on these same problems <ref:2610.18524#pg1>. So it seems efficient in practice.
Conclusion: Ada: So wrapping up this discussion on the paper "Hamiltonian Primal–Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control," what's the big picture takeaway? It’s about making optimal control methods viable when you can't rely on smooth derivatives of your system model.
Emmy: The authors introduced HPDPO, which combines deterministic model rollout with a regularized Hamiltonian primal–dual objective <ref:2610.18524#pg3>. This framework allows for solving finite-horizon control problems where the transition model is reliable for simulation but unsuitable for derivative-based optimization.
Kolya: What this means in practice is that you can apply this to systems where the dynamics are learned, like those from longitudinal observations, or systems with nonsmooth components <ref:2610.18524#pg2>. It shows a path forward for applying RL and optimal control in biomedical applications, given the challenges of modeling biological processes.
Ada: The paper's validation on exact solutions and learned models suggests that this method is a practical approach for these types of deterministic finite-horizon problems <ref:2610.18524#pg1>. It gives researchers a tool to get control and state trajectories that closely agree with exact solutions or derivative-based references without needing those tricky derivatives.
Emmy: It supports HPDPO as a practical tool for deterministic problems where you have a reliable simulator but need an optimization method that doesn't demand smooth Jacobians <ref:2610.18524#pg3>. It moves the focus to structured, block-coordinate updates rather than relying on traditional derivative propagation.
Kolya: One thing they flag in their work is that the state trajectory is produced only by the supplied transition model <ref:2610.18524#pg3>. So the accuracy of your final control and state sequence is fundamentally tied to how good that simulation model actually is, which makes sense for a black-box setting.
Ada: That's a fair point. The method doesn't magically fix a bad model; it just gives you a better way to optimize *within* the constraints of what your simulator can provide <ref:2610.18524#pg3>. So, this paper provides a solid foundation for deterministic problems with reliable simulation but difficult optimization targets.
Department of Mathematics, The Pennsylvania State University · Department of Computer Science and Engineering, The Pennsylvania State University
math.OC
Submitted: 2026-10-13
Updated: 2026-11-09
Code: https://github.com/LRabiei/HPDPO-derivative-free-control
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: Finite-horizon optimal control becomes difficult when a dynamical model can be simulated reliably but its derivatives are unavailable, discontinuous…
Key concepts
- Hamiltonian Primal–Dual Policy Optimization (HPDPO)
- A derivative-free framework for model-based reinforcement learning. It formulates the control problem using a Hamiltonian structure that separates the policy updates from the system dynamics simulation, allowing it to handle problems with unreliable or nonsmooth models.
- Model-Based Policy–Optimization
- The approach treats the control problem as a deterministic optimization task guided by a provided transition model. Instead of relying on gradients derived from the model, it uses rollouts of this model to evaluate candidate actions, making it suitable when only simulation is reliable.
- Hamiltonian Formulation (LIBP)
- This continuous mathematical representation describes the primal-dual basis of HPDPO. It links the state trajectory, control inputs, and dual variables ($\lambda$) through a specific differential equation structure. The key innovation is shifting derivatives to the multiplier trajectory for block-coordinate updates.
- Block-Coordinate Optimization
- Since exact derivative integration is unavailable, HPDPO replaces it with bounded block-coordinate optimization on the multiplier trajectory. This allows the algorithm to iteratively improve the dual variables by solving simpler subproblems, effectively approximating complex continuous optimizations.
Terminology
Summary
Finite-horizon optimal control becomes difficult when a dynamical model can be simulated reliably but its derivatives are unavailable, discontinuous, or unreliable The method is validated on nonsmooth problems with exact solutions, an analytical and learned ReLU biomarker cascade, an amyloid Neural ODE trained from longitudinal ADNI data, and three multidimensional-control problems with up to five simultaneous controls Across these examples, HPDPO closely reproduces exact or reference objective values, trajectories, and control structures The results show that forward simulation and Hamiltonian dual structure can be combined to solve deterministic finite-horizon control problems involving nonsmooth, learned, or black-box dynamics. p. 0
Methodology Overview
Hamiltonian Primal–Dual Policy Optimization (HPDPO) is introduced as a derivative-free, model-based reinforcement learning framework for this setting The control is represented as a time-indexed finite-horizon policy Candidate actions are evaluated by deterministic suffix rollouts of the supplied transition model, while a regularized Hamiltonian dual formulation supplies structured future-cost information through block-coordinate multiplier updates The main contributions are as follows:
-
We formulate the time-discretized finite-horizon control problem as a deterministic model-based policy–optimization problem while separating the role of discretization from the policy–update mechanism. p. 2
-
We derive a regularized Hamiltonian primal–dual objective in which the control sequence is the primal variable, the multiplier sequence is the dual variable, and the state trajectory is produced only by the supplied transition model. p. 2
-
We replace derivative-based costate integration with bounded block-coordinate optimization of the multiplier trajectory. p. 2
-
We evaluate HPDPO on nonsmooth problems with exact solutions, analytical and learned ReLU cascade models, an amyloid Neural ODE trained from longitudinal ADNI data, and three additional multidimensional–control benchmarks with simultaneous control components. p. 2
Hamiltonian Formulation
The continuous primal–dual basis of HPDPO is represented by LIBP(x, u, λ) = Φ(x(T)) + Z T0 hl(x, u, t) + λ(t)Tf(x, u, t) − x˙(t)i dt This representation is the continuous primal–dual basis of HPDPO The derivative is shifted to the multiplier trajectory, which can be discretized and updated directly through block-coordinate maximization.
Policy Improvement Mechanism
The central block–coordinate updates are u+k ∈ arg min v∈V Q HP k,h(v U,Λ) The candidate-dependent part can be written as Ck + Q roll k(x k, v;U) + X M j=k λT j bk(U)(v) + βu M X−2 j=0 u(j+1)(v) − u(j)(v) The multiplier regularization is constant during the policy–coordinate update and is absorbed into Ck
Dual Update Mechanism
The multiplier sequence is updated by its exact constrained block maximizer λ+k = Π[−λmax,λmax]n bλk This representation makes the curvature of the dual subproblem explicit The multiplier sequence converges to the unique solution Λ⋆(U) of the fixed–U dual maximization problem.
Numerical Validation
HPDPO accurately recovers the known optimal control and corresponding state trajectory with substantially smaller numerical errors on this deterministic fixed–initial-state benchmark For Example 3, HPDPO produces very small control, state, and objective errors In Multidimensional Example 3, HPDPO reproduces the exact switching structure of the two-state compartmental problem The computation time for HPDPO is often smaller than that of tested PPO and SAC implementations.
Conclusion
This work introduced HPDPO, a finite-horizon policy–optimization method that combines deterministic model rollout with a regularized Hamiltonian primal–dual objective HPDPO produces control and state trajectories that closely agreed with exact solutions or derivative-based references This supports HPDPO as a practical approach for deterministic finite-horizon problems in which the transition model is reliable for simulation but unsuitable for derivative-based optimization.
Author Contributions
K. Rabiei developed the methodology, implemented the algorithms, conducted the numerical experiments, analyzed the results, and drafted the manuscript Z. p. 1 Rui contributed to methodological development, scientific discussions, and manuscript review X. p. 0 Lu contributed to reviewing, editing, and checking the manuscript W. p. 0 Hao contributed to the conceptualization of the study, supervised the research, provided methodological guidance, and reviewed and edited the manuscript. p. 0
Acknowledgments
K. R. and W. H. were supported by the National Institute of General Medical Sciences (NIGMS) through Grant No. p. 3 R35GM146894 W. H. was also supported by the National Science Foundation (NSF) through Grant No. p. 0 DMS-2533995 and by the Huck Chair in AI Mathematical Modeling from The Pennsylvania State University’s Huck Institutes of the Life Sciences. p. 0
Conflicts of Interest
The authors declare that they have no conflicts of interest
Use of Artificial Intelligence Tools
ChatGPT (OpenAI) was used for language editing, improving clarity, and organizing portions of the manuscript All scientific content, analyses, interpretations, and final decisions were reviewed and approved by the authors.
Data Availability Statement
The source code supporting the findings of this study is available at the following GitHub repository https://github.com/LRabiei/HPDPO-derivative-free-control. p. 0
References
[1] Lev S. Pontryagin, Vladimir G. p. 2 Boltyanskii, Revaz V. Gamkrelidze, and Evgenii F. Mishchenko The Mathematical Theory of Optimal Processes Interscience Publishers New York 1962
[2] Richard S. Sutton and Andrew G Barto Reinforcement Learning An Introduction The MIT Press Cambridge MA 2 edition 2018
[3] Sheng Zhang and Guang Lin Robust data-driven discovery of governing physical laws with error bars Proceedings of the Royal Society A Mathematical, Physical and Engineering Sciences 474 (2217):20180305, 2018 doi: 10.1098/rspa.2018.0305
[4] Yingying Zhu, Minjeong Kim, Xiaofeng Zhu, Daniel Kaufer, Guorong Wu, and Alzheimer’s Disease Neuroimaging Initiative Long range early diagnosis of alzheimer’s disease using longitudinal mr imaging data Medical Image Analysis 67:101825 2021 doi: 10.1016/j.media.2020.101825
[5] Xavier Glorot, Antoine Bordes, and Yoshua Bengio Deep sparse rectifier neural networks In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics volume 15 of Proceedings of Machine Learning Research pages 315–323 PMLR, 2011
[6] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud Neural ordinary differential equations In Advances in Neural Information Processing Systems volume 31 pages 6571–6583 2018
[7] Andrew R. Conn, Katya Scheinberg, and Luis N. Vicente Introduction to Derivative-Free Optimization Society for Industrial and Applied Mathematics Philadelphia 2009 doi: 10. p. 01137/1.9780898718768
[8] Jeffrey Larson, Matt Menickelly, and Stefan M. Wild Derivative-free optimization methods Acta Numerica 28:287–404 2019 doi: 10. p. 11017/S0962492919000060
[9] Suzanne Lenhart and John T. Workman Optimal Control Applied to Biological Models Chapman and Hall/CRC Boca Raton 2007
[10] Michael McAsey, Libin Mou, and Weimin Han Convergence of the forward–backward sweep method in optimal control Computational Optimization and Applications 53(1):207–226 2012 doi: 10.1007/s10589-011-9454-7
[13] Richard S. Sutton Dyna an integrated architecture for learning ACM SIGART Bulletin 2(4):160–163 1991 doi: 10. p. 11145/122344.122377
[15] Vijay R. Konda and John N Tsitsiklis Actor–critic algorithms In Advances in Neural Information Processing Systems 12 pages 1008–1014 MIT Press 2000
[16] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller Deterministic policy gradient algorithms In Proceedings of the 31st International Conference on Machine Learning volume 32 of Proceedings of Machine Learning Research pages 387–395 PMLR 2014
[17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov Proximal policy optimization algorithms arXiv preprint arXiv:1707.06347 2017
[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor In Proceedings of the 35th International Conference on Machine Learning volume 80 of Proceedings of Machine Learning Research pages 1861–1870 PMLR, 2018
[19] Sanket Kamthe and Marc Deisenroth Data-efficient reinforcement learning with probabilistic model predictive control In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics volume 84 of Proceedings of Machine Learning Research pages 1701–1710 PMLR, 2018
[20] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine Deep reinforcement learning in a handful of trials using probabilistic dynamics models In Advances in Neural Information Processing Systems 31 2018
[21] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine When to trust your model Modelbased policy optimization In Advances in Neural Information Processing Systems 32 pages 12498–12509 2019
[23] Marc Peter Deisenroth and Carl Edward Rasmussen PILCO A model-based and data-efficient approach to policy search In Proceedings of the 28th International Conference on Machine Learning pages 465–472 Omnipress 2011
[24] Dimitri P. Bertsekas Dynamic Programming and Optimal Control volume 1 Athena Scientific Belmont MA 4 edition 2017 doi: 10. p. 11002/9780470316887
[25] Frank L. Lewis and Draguna Vrabie Reinforcement learning and adaptive dynamic programming for feedback control IEEE Circuits and Systems Magazine 9(3):32–50 2009 doi: 10. p. 11109/MCAS.2009.933854
[26] Jeongho Kim and Insoon Yang Hamilton–jacobi–bellman equations for q-learning in continuous time In Proceedings of the 2nd Conference on Learning for Dynamics and Control volume 120 of Proceedings of Machine Learning Research pages 739–748 PMLR 2020
[30] Paul Tseng Convergence of a block coordinate descent method for nondifferentiable minimization Journal of Optimization Theory and Applications 109(3):475–494 2001 doi: 10.1023/A:1017501703105
[32] Luigi Grippo and Marco Sciandrone On the convergence of the block nonlinear gauss–seidel method under convex constraints Operations Research Letters 26(3):127–136 2000 doi: 10.1016/S0167-6377(99)00074-7
[33] M. H. Noori Skandari, A. V. Kamyad, and S. Effati Smoothing approach for a class of nonsmooth optimal control problems Applied Mathematical Modelling 40(2):886–903 2016 doi: 10. p. 01016/j.apm.2015.05.014
[34] Jeffrey R. Petrella, Wenrui Hao, Adithi Rao, and P Murali Doraiswamy Computational causal modeling of the dynamic biomarker cascade in alzheimer’s disease Computational and Mathematical Methods in Medicine 2019:6216530 2019 doi: 10.1155/2019/6216530
[35] J. R. Petrella, J Jiang, K Sreeram, S Dalziel P Murali Doraiswamy W Hao and Alzheimer’s Disease Neuroimaging Initiative Personalized computational causal modeling of the alzheimer’s disease biomarker cascade The Journal of Prevention of Alzheimer’s Disease 11(2):435–444 2024 doi: 10.14283/jpad.2023.134
[36] Michael W. Weiner, Paul S Aisen Jr Jack Clifford R William J Jagust John Q Trojanowski Leslie M Shaw Andrew J Saykin John C Morris Nigel Cairns Laurel A Beckett Arthur W Toga Robert Green Scott Walter Holly Soares Peter Snyder Eric Siemers William Potter Paul E Cole Michael Schmidt and Alzheimer’s Disease Neuroimaging Initiative The alzheimer’s disease neuroimaging initiative Progress report and future plans Alzheimer’s & Dementia 6(3):202–211.e7, 2010 doi: 10.1016/j.jalz.2010.03.007
[37] Bruno M Jedynak, Andrew Lang Bo Liu Elyse Katz Yanwei Zhang Bradley T Wyman David Raunig C Pierre Jedynak Brian Caffo Jerry L Prince and Alzheimer’s Disease Neuroimaging Initiative A computational neurodegenerative disease progression score Method and results with the alzheimer’s disease neuroimaging initiative cohort NeuroImage 63(3):1478–1486 2012 doi: 10.1016/j.neuroimage.2012.07.059
[38] Mostafa Mehdipour Ghazi, Mads Nielsen, Akshay Pai, Marc Modat, M Jorge Cardoso Sébastien Ourselin and Lauge Sørensen Robust parametric modeling of alzheimer’s disease progression NeuroImage 225:117460 2021 doi: 10.1016/j.neuroimage.2020.117460
[39] Franco Blanchini, Paolo Bolzern, Patrizio Colaneri, Giuseppe De Nicolao, and Giulia Giordano Optimal control of compartmental models The exact solution Automatica 147:110680 2023 doi: 10.1016/j.automatica.2022.110680
[40] Iulia Martina Bulai, Francesco Ferraresso, and Francesca Gladiali Optimal control of monomers and oligomers degradation in an Alzheimer’s disease model Journal of Mathematical Biology 91(3):27 2025 doi: 10.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Deterministic model-based policy optimization allows AI systems to optimize finite-horizon control problems directly through forward simulation while retaining a Hamiltonian primal dual structure, avoiding differentiation of state dynamics or learning a global critic.
-
The method can solve optimal control problems involving nonsmooth analytical models, learned ReLU cascades, and amyloid Neural ODEs by evaluating candidate actions via deterministic suffix rollouts and incorporating structured future-cost information through block-coordinate multiplier updates.
-
AI systems can recover exact or reference objective values and trajectories for nonsmooth problems by combining forward simulation with the Hamiltonian dual structure to solve deterministic finite-horizon control problems involving black-box dynamics.
-
The framework enables the simultaneous optimization of multiple control components, as demonstrated in multidimensional examples, allowing AI to find coupled optimal policies for complex systems like flood management or multi-state biological models.
-
HPDPO can be applied directly to learned nonsmooth dynamical models, such as a ReLU Neural ODE trained on longitudinal data, and produce trajectories that closely agree with derivative-based references constructed from known analytical systems.
Sources
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise