On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty

arXiv:2606.32003 · eess.SY, cs.SY · Submitted 2026-06-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty".

Dev: Here's the gist of the paper. This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss.

Rosa: The DDPG controller achieves faster transient responses and improved damping within its training environment, but the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees <ref:2606.32003#pg12> Comparison of Control Methods The study compares Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss <ref:2606.32003#pg13>.

Dev: The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages <ref:2606.32003#pg12>.

Rosa: Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment <ref:2606.32003#pg12>.

Paper summary: Rosa: So, we're looking at this paper "On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty". Basically, they are comparing two major approaches: Adaptive Quantized Control, or AQC, against Deep Deterministic Policy Gradient, or DDPG.

Dev: The core idea here is testing which method handles uncertain linear systems with input quantization over communication channels that have packet loss better. Specifically, the paper claims that while DDPG does things quickly in its training setup, AQC shows much better reliability when the system dynamics change and when packets get lost or switch states.

Taro: So, what's the main tension they're exploring? It seems like AQC is designed to deal with unknown system dynamics using acknowledgment messages to fix those packet losses, but DDPG is trained only on the nominal model without that feedback loop.

Rosa: Right. The paper highlights that the DDPG controller manages to get faster transient responses and better damping when it’s just in its training environment, but when things get messy—when there’s model uncertainty or packet loss happening—the AQC keeps stabilizing the system because of its rigorous Lyapunov stability guarantees.

Dev: That's a big difference. The paper points out that the DDPG controller has to generalize beyond what it was trained on when you introduce those real-world issues, whereas AQC has these explicit mathematical guarantees for stability even under those conditions, which is what makes it more robust.

Taro: It makes sense if we think about how a system actually fails in the real world. The numerical results they show are pretty telling: AQC successfully stabilizes the system across four different scenarios involving dynamic switching from a nominal system to a more unstable one during operation.

Rosa: And when they introduce packet loss into those simulations, it’s where the gap really opens up between the two methods, with AQC clearly winning on robustness compared to DDPG in that specific condition.

Dev: So what does this mean for someone building a system that has to operate over a network? It suggests that just because a learning method like DDPG can get good performance in simulation doesn't mean it will survive when the actual communication channel starts dropping data or the system dynamics drift unexpectedly.

Taro: And that’s where AQC comes in, using those acknowledgment messages to compensate for the packet loss, which gives it that stability under uncertainty. The paper also details how AQC works by decomposing the quantizer into linear and nonlinear parts to show its mathematical structure.

Rosa: I want to talk about the specific mechanism they use for AQC. They describe how they generate the quantized control input from a nominal control law, v equals q of time and input, where that quantizer acts componentwise according to the logarithmic quantization rule.

Dev: That logarithmic quantizer is described mathematically as a time-varying sector-bounded memoryless nonlinearity, which is key because it allows them to represent the quantization in a structured way for stability analysis. They even break down this quantizer into linear and nonlinear components, q equals half of M1 plus half of M2, plus some nonlinear term.

Taro: So when you look at the math, they show that this structure leads to sector bounds where the bound rho i is equal to M1i times M2i for each component, and that relates directly to how the quantizer scales with time and input.

Rosa: The paper then shows how this scaling factor delta, which is related to the difference between M2 and M1, determines the sector bounds. They state that rho i equals one minus delta times delta i plus delta i, where delta is defined using matrix inverses of M2 plus M1 and the difference between them.

Dev: That equation connects the structure of that diagonal matrix M2 minus M1, which must be positive definite, to the actual sector bounds they need for stability proofs. It shows how the relationship between those matrices defines whether a specific system input will fall within a certain range of quantization levels.

Taro: So if you look at the paper, it’s not just about showing that AQC works; it’s showing *why* it works by breaking down the nonlinear quantizer into these structured linear and nonlinear pieces that satisfy those sector bounds.

Rosa: And this is what matters for the practical implications: when we move from training in a perfect world to operating over a lossy channel, AQC has an inherent mechanism—the acknowledgment messages—that helps it adapt and maintain stability where DDPG just gets stuck because it never saw that packet loss during its initial training.

Dev: Exactly. The paper’s conclusion stresses that the DDPG controller can achieve better transient performance within its training environment, but AQC is the one you rely on when you introduce model uncertainty or packet loss because of those explicit Lyapunov stability guarantees.

Taro: For someone who only listens to the show, this means that if your robot or system has to operate in a noisy communication environment, choosing a method with proven stability bounds over one that relies purely on learning from nominal data is crucial for safety.

Rosa: It really puts the strength of adaptive control and reinforcement learning side-by-side: DDPG for speed in controlled settings, but AQC for guaranteed performance when the world misbehaves. That's what this paper is about, "On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty".

Conclusion: Rosa: So we're looking at the study comparing Adaptive Quantized Control and Deep Deterministic Policy Gradient for linear systems dealing with packet loss and uncertainty over communication channels.

Dev: Right, the authors are comparing how these two methods handle those real-world glitches in control loops, specifically focusing on how they react when the system model is uncertain or when data packets get lost.

Rosa: Basically, it shows that Deep Deterministic Policy Gradient does things fast during training but Adaptive Quantized Control has a much better chance of staying stable when the world gets messy.

Dev: That's the core finding, and what this means for us in real-time systems is that learning methods need to be really careful about where they operate outside of their training data.

Rosa: The paper highlights that AQC has these explicit mathematical stability proofs, which is a big deal when you have to guarantee safety in an actual deployment.

Dev: Because DDPG relies on generalizing beyond the nominal model, it might not perform as well when the uncertainty gets worse than what it saw during training.

Rosa: So we're seeing a trade-off between speed in perfect conditions and guaranteed reliability in tough conditions with packet loss.

Dev: It suggests that for critical applications, having those formal stability guarantees from adaptive control might be more important than just chasing the fastest response time learned by reinforcement learning.

Rosa: It really shows how the two approaches have different strengths, even though both are using advanced techniques to handle complex systems.

Dev: And this comparison points toward a future where we might see hybrid systems, maybe combining the speed of learning with the safety net of adaptive control for these kinds of networked environments.

Moh. Kamalul Wafi

Northeastern University

eess.SY, cs.SY

Submitted: 2026-06-30

Updated: 2026-11-02

Comments: Extended and revised version of the paper published in the Proceedings of the SIAM Conference on Control and Its Applications (CT23). doi.org/10.1137/1.9781611977745

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 72/100

The gist: This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input…

Key concepts

Adaptive Quantized Control (AQC)
AQC is a control method designed for systems with unknown dynamics. It uses acknowledgment messages to compensate for packet losses, allowing the controller to adapt. It ensures system stability through rigorous Lyapunov stability guarantees, making it very reliable in uncertain environments.
Deep Deterministic Policy Gradient (DDPG)
DDPG is a reinforcement learning approach where an actor learns the optimal control policy by maximizing expected rewards. It is trained using a nominal system model and does not explicitly account for packet loss or model uncertainty during training, leading to faster transient responses in ideal settings.
Input Quantization
This involves reducing the precision of the control signal before it is sent over communication channels. The paper uses a logarithmic quantization rule, which is mathematically represented as a time-varying sector-bounded nonlinearity. This quantization introduces uncertainty into the system dynamics.
Lyapunov Stability Guarantees
These are mathematical proofs that guarantee a system will remain stable (i.e., not diverge) under certain conditions. AQC relies on these guarantees to prove its robustness against packet loss and model uncertainty, unlike DDPG which must generalize from its training experience.

Terminology

Summary

This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss. p. 0

Where the paper says it

m}@northeastern. edu ABSTRACT This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss.

The DDPG controller achieves faster transient responses and improved damping within its training environment, but the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees <ref:2606. p. 0
Where the paper says it

Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees.

p. 0
Where the paper says it

Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees.

32003#pg12>

Comparison of Control Methods

The study compares Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss <ref:2606.32003#pg13>. The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages <ref:2606. p. 0

Where the paper says it

The considered setting also includes dynamic switching from a nominal unstable system to a more unstable one during operation. The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages.

32003#pg12>. Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment <ref:2606. p. 0
Where the paper says it

The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages. Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment.

32003#pg12>. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees <ref:2606. p. 0
Where the paper says it

Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees.

32003#pg12>.

Adaptive Quantized Control Framework

The Adaptive Quantized Control (AQC) framework considers a discrete-time uncertain linear system where the plant is described by x(k + 1) = Ax(k) + η(k)Bv(k), with A being unknown <ref:2606.32003#pg13>. The quantized control input is generated from the nominal control law v(k) = q(k, u(k)), where u(k) = H(k)x(k), and the quantizer q· acts componentwise according to the logarithmic quantization rule <ref:2606. p. 1

Where the paper says it

The quantized control input is generated from the nominal control law v(k) = q(k, u(k)), (2) where u(k) = H(k)x(k), and H(k) ∈ R m×n denotes the time-varying feedback gain. The quantizer q(·, ·) acts componentwise according to the logarithmic quantization rule qi ,    ϕi(k, j), if ui ∈ (ϕi(k, j + 1), ϕi(k, j)], −ϕi(k, j), if ui ∈ [−ϕi(k, j), −ϕi(k, j + 1)), 0, if ui = 0, j ∈ I, i = 1, .

32003#pg13>. This logarithmic quantizer admits an equivalent representation as a time-varying sector-bounded memoryless nonlinearity <ref:2606. p. 2
Where the paper says it

For negative indices, the quantization levels are defined by ϕi(k, −j) , ai(k) ρ j i (k) , i = 1, . The logarithmic quantizer in (3) admits an equivalent representation as a time-varying sector-bounded memoryless nonlinearity [10], Q , n q : N0 × R m → R m : q(·, 0) = 0, [q(k, u) − M1(k)u] ⊤ [q(k, u) − M2(k)u] ≤ 0, u ∈ R m, k ∈ N0 o .

32003#pg13>. The quantizer is decomposed into linear and nonlinear components as q(k, u) = 1/2 [M1(k) + M2(k)] u + qs(k, u), where qs: N0 × R m → R m denotes the nonlinear component <ref:2606. p. 2
Where the paper says it

For completeness, we briefly summarize the adaptive quantized control framework and refer the reader to [11, 12] for the complete controller design and stability analysis. Following [11], the quantizer is decomposed into linear and nonlinear components as q(k, u) = 1 2 [M1(k) + M2(k)] u + qs(k, u), (8) where qs : N0 × R m → R m denotes the nonlinear component satisfying Qs , n qs : N0 × R m → R m : qs(·, 0) = 0, q ⊤ s (k, u)qs(k, u) − 1 4 u ⊤ [M2(k) − M1(k)]2 u ≤ 0, u ∈ R m, k ∈ N0 o .

32003#pg13>.

Reinforcement Learning Controller Design

The Deep Deterministic Policy Gradient (DDPG) controller is an off-policy actor–critic algorithm in which the critic approximates the action-value function, while the actor learns the policy by maximizing the expected return <ref:2606.32003#pg12>. The objective is to maximize J(µθ) = Z S ρ µ(x) Z A µθ(x, u)r(x, u) du dx <ref:2606. p. 5

Where the paper says it

DDPG is an off-policy actor–critic algorithm in which the critic approximates the action-value function, while the actor learns the policy by maximizing the expected return. The objective is to maximize J(µθ) = Z S ρ µ (x) Z A µθ(x, u)r(x, u) du dx, (16) where ρ µ(x) denotes the discounted state distribution induced by the policy µθ.

32003#pg12>. The state and action spaces are denoted by x ∈ S and u ∈ A, respectively <ref:2606. p. 5
Where the paper says it

Unlike stochastic policy methods, DDPG employs a deterministic policy u = µθ(x), which maps the state directly to the control action, instead of the stochastic policy πθ(u|x) = P[u|x; θ]. The state and action spaces are denoted by x ∈ S and u ∈ A, respectively.

32003#pg12>. The actor network is described by the compact input–output representation u(k) vq(k) = N " x(k) φq(k) 1/N, φq(k) = ∆ (vq (k)) <ref:2606.32003#pg12>.

Numerical Results and Findings

The numerical results compare the performance of AQC and DDPG controllers under four simulation scenarios involving dynamic switching from the nominal system A to the perturbed dynamics Aw at td ∈ 10, 15, 20, 30 s <ref:2606.32003#pg12>. Figure 5 shows that the AQC successfully stabilizes the system under all switching scenarios <ref:2606. p. 9

Where the paper says it

Each simulation considers a dynamic switch from the nominal system A to the perturbed system Aw at the prescribed switching time td for three different initial conditions. Figure 5 shows that the AQC successfully stabilizes the system under all switching scenarios.

32003#pg12>. When packet loss is introduced, however, the AQC exhibits superior robustness compared to the DDPG controller <ref:2606. p. 0
Where the paper says it

Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees.

32003#pg12>. This difference is expected since the AQC is equipped with explicit stability guarantees for uncertain systems with packet loss, whereas the DDPG controller is trained only within the nominal environment and must generalize beyond its training distribution <ref:2606. p. 0
Where the paper says it

The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages. Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment.

32003#pg12>.

Conclusion

The paper concludes that while DDPG achieves improved transient performance within the training environment, AQC exhibits superior robustness in the presence of model uncertainty and packet loss due to its explicit Lyapunov stability guarantees <ref:2606.32003#pg12>. The results highlight the complementary strengths of adaptive control and reinforcement learning, while illustrating the limitations of learning-based controllers when operating beyond the training environment <ref:2606. p. 0

Where the paper says it

However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees. These results highlight the trade-off between data-driven performance and model-based robustness, and provide insight into the applicability of reinforcement learning and adaptive control for networked uncertain systems.

32003#pg12>. Future research will focus on extending the proposed framework to networked multi-agent systems <ref:2606. p. 0
Where the paper says it

This means there exists a probability p¯ of control signal not being sent to the plant, with some variations of the non-linear uncertain system [13]. Similar communication and uncertainty issues also arise in networked estimation and process control applications, including hydraulic multi-tank systems, where reliable state arXiv:2606.

32003#pg12>.

Pruning redundant reasoning tokens

The paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss <ref:2606.32003#pg12>. The DDPG controller achieves faster transient responses and improved damping within its training environment, but the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees <ref:2606.32003#pg12>.

How the training signal is built

The DDPG controller is trained using the nominal system matrix A together with the same quantizer shown in Fig. 2 <ref:2606. p. 0

Where the paper says it

This means there exists a probability p¯ of control signal not being sent to the plant, with some variations of the non-linear uncertain system [13]. Similar communication and uncertainty issues also arise in networked estimation and process control applications, including hydraulic multi-tank systems, where reliable state arXiv:2606.

32003#pg12>. The learned policy is subsequently evaluated under plant variations by replacing A with Aw and introducing packet loss <ref:2606. p. 8
Where the paper says it

During training, the controller is learned using the nominal system matrix A. The learned policy is subsequently evaluated under plant variations by replacing A with Aw and introducing packet loss.

32003#pg12>. This setting provides a direct comparison with the adaptive quantized controller, which is designed for uncertain systems through the stabilizing gain Kg satisfying As = A + BKg <ref:2606. p. 8
Where the paper says it

The learned policy is subsequently evaluated under plant variations by replacing A with Aw and introducing packet loss. This setting provides a direct comparison with the adaptive quantized controller, which is designed for uncertain systems through the stabilizing gain Kg satisfying As = A + BKg.

32003#pg12>.

Where the gains come from

The adaptive control law relies on acknowledgment messages that inform the controller whether the transmitted control packet is successfully received, i.e., the realization of η(k) <ref:2606. p. 0

Where the paper says it

This means there exists a probability p¯ of control signal not being sent to the plant, with some variations of the non-linear uncertain system [13]. Similar communication and uncertainty issues also arise in networked estimation and process control applications, including hydraulic multi-tank systems, where reliable state arXiv:2606.

32003#pg13>. The adaptive quantized control law u(k) = 2 [M1(k) + M2(k)]−1 K(k)x(k), where K is the gain update law. p. 4
Where the paper says it

Assume further that acknowledgment messages are available, so that the controller knows whether packet loss has occurred. If the packet-loss bound p¯ satisfies p¯ 1 − p¯ λmax(P)¯σ 2 AIn − P < γR, (12) then, following the controller synthesis procedure above, the adaptive control law u(k) = 2 [M1(k) + M2(k)]−1 K(k)x(k), (13) where K(k) ∈ R m×n and the quantizer satisfies R − 2K⊤(k)∆(k)B ⊤P B∆(k)K(k) ≥ γR > 0, (14) for every k ∈ N0, together with the quantizer (2) and the gain update law K(k + 1) = K(k) − η(k) 1 + x⊤(k)P x(k) QB† [x(k + 1) −Asx(k) − Bqs(k, u(k))] x ⊤(k), (15) guarantees Lyapunov stability.

The stability guarantee is derived from Theorem 1, which states that if the packet-loss bound p¯ satisfies p¯ 1 − p¯ λmax(P)¯σ 2 AIn − P < γR, then Lyapunov stability is guaranteed. p. 4
Where the paper says it

Assume further that acknowledgment messages are available, so that the controller knows whether packet loss has occurred. If the packet-loss bound p¯ satisfies p¯ 1 − p¯ λmax(P)¯σ 2 AIn − P < γR, (12) then, following the controller synthesis procedure above, the adaptive control law u(k) = 2 [M1(k) + M2(k)]−1 K(k)x(k), (13) where K(k) ∈ R m×n and the quantizer satisfies R − 2K⊤(k)∆(k)B ⊤P B∆(k)K(k) ≥ γR > 0, (14) for every k ∈ N0, together with the quantizer (2) and the gain update law K(k + 1) = K(k) − η(k) 1 + x⊤(k)P x(k) QB† [x(k + 1) −Asx(k) − Bqs(k, u(k))] x ⊤(k), (15) guarantees Lyapunov stability.

The DDPG controller's reward function is defined as r(k) = -100, x > Th, 1/e + ψ, otherwise <ref:2606.32003#pg12>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Robust control for networked systems under packet loss: The improved AI system can maintain stability while transmitting control signals over communication channels subject to packet loss, as demonstrated by Adaptive Quantized Control (AQC) which consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees.

  2. Handling model uncertainty and dynamic switching: The system can operate effectively when the underlying dynamics change unexpectedly, as the AQC is designed for unknown system dynamics and handles a dynamic switching from a nominal unstable system to a more unstable one during operation.

  3. Enhanced transient response in training environments: For tasks where the agent is trained in a known environment, this AI system can achieve faster performance because the DDPG controller achieves faster transient responses and improved damping within its training environment.

  4. Continuous and infinite control actions: The deep reinforcement learning component allows for complex control strategies by enabling continuous and infinite control actions, which is a key advantage over discrete methods like DQN.

  5. Policy generalization beyond training data: While DDPG performs well in the nominal setting, the improved system's reliance on explicit stability guarantees suggests it can be better at generalizing to new environments where the AQC remains stable over a significantly wider range of parameter variations.

Abstract

This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss. The considered setting also includes dynamic switching from a nominal unstable system to a more unstable one during operation. The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages. Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees. These results highlight the trade-off between data-driven performance and model-based robustness, and provide insight into the applicability of reinforcement learning and adaptive control for networked uncertain systems.

Sources

Related papers