Deep Reinforcement Learning with Smith-Predictor Delay Compensation
eess.SY, cs.SY
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: This is an extended version of the paper published in the proceedings of IEEE American Control Conference (ACC)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Time delays can degrade feedback performance and destabilize a closed loop that is stable without delay.
Terminology
Abstract
Time delays can degrade feedback performance and destabilize a closed loop that is stable without delay. This paper studies Smith-predictor-assisted reinforcement learning for systems with constant input delays. A deep deterministic policy gradient (DDPG) actor is trained on a nominal delay-free environment and deployed with a Smith predictor using a fixed nominal delay. For linear controllers, we derive exact closed-loop maps under plant and delay mismatch and provide small-gain and ν-gap robustness conditions. A separate delay-independent Razumikhin condition gives a sufficient stability test for a fixed Lipschitz state-feedback policy. We also study output-error delay identification from a fixed input--output record, deriving the loss gradient and establishing projected-flow descent and local exponential convergence under exact model matching and an excitation condition. These identification results do not imply stability of an online adaptive predictor. Numerical experiments retained from the conference study compare PID, LQR, and DDPG controllers on stable and open-loop unstable first-order plants with valve stiction. The results illustrate the benefit of predictor compensation in the tested configurations and the sensitivity of standalone DDPG to delay and plant changes.
Sources
- Continuous control with deep reinforcement learning
- On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty
- System Identification on the Families of Auto-Regressive with Least-Square-Batch Algorithm
- Neural-Enhanced Micro-Kalman Filtering for Satellite Tracking: A Comparative Study
Related papers
- Industrial Load Modeling and Optimization for Market-Based Interaction with Power Systems
- Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
- Certified Set Convergence for Piecewise Affine Systems
- Symbolic Control for Stochastic Systems via Finite Parity Games
- Multiobjective optimization-based design and dispatch of islanded, hybrid microgrids for remote, off-grid communities in sub-Saharan Africa
- Physiology-informed layered sensing for intelligent human-exoskeleton interaction