Deep Reinforcement Learning-Aided Frequency Control of LCC-S Resonant Converters for Wireless Power Transfer Systems

arXiv:2505.01850 · eess.SY, cs.SY · Submitted 2025-05-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Deep Reinforcement Learning-Aided Frequency Control of LCC-S Resonant Converters for Wireless Power Transfer Systems".

Dev: The gist The proposed DRL-based control strategy for LCC-S resonant converters achieves precise and robust output voltage regulation for wireless power transfer applications by leveraging Twin Delayed Deep Deterministic Policy…

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: We just covered how they use TD3 to tune the PI gains for LCC-S resonant converters, focusing on how this DRL approach handles the complex dynamics using DPWA modeling. Now, let's get into a bit more detail about what the paper actually claims in "Deep Reinforcement Learning-Aided Frequency Control of LCC-S Resonant Converters for Wireless Power Transfer Systems".

Dev: The core thesis is that they are proposing a novel DRL-based control strategy to achieve precise and robust output voltage regulation for wireless power transfer applications, moving away from the old methods.

Rosa: They claim that by leveraging the Twin Delayed Deep Deterministic Policy Gradient algorithm, they can systematically optimize the PI controller parameters directly rather than relying on manual tuning or those older heuristic approaches.

Dev: And to do that optimization, they are using a specific modeling technique called Direct Piecewise Affine modeling to capture the nonlinear dynamics of these converters in a structured way.

Rosa: They argue this combination eliminates the need for manual tuning while making the control much better at handling changing operating conditions.

Taro: So if I'm thinking about what this means for autonomous systems, it suggests that instead of having fixed controller settings, we could have an AI that learns to adjust those settings based on the system's experience <ref:2505.01850#pg2>.

Dev: Exactly. And the paper points out that because DRL learns through accumulating experience, it doesn't need you to provide a complete mathematical model of the environment upfront or all its environmental knowledge twenty-one <ref:2505.01850#pg2>.

Rosa: That’s a big deal because building those detailed models for power electronics is often incredibly difficult, and this method bypasses that barrier by letting the AI figure it out through interaction.

Taro: But what about the actual numbers? The abstract doesn't give us specific performance metrics yet; we need to know how much better it actually is compared to the traditional methods.

Dev: Well, they do provide some results, saying that their proposed DRL-based tuning approach significantly outperforms traditional methods in terms of stability, robustness and response time <ref:2505.01850#pg1>.

Rosa: So they’re not just making vague claims; they are backing up the idea with comparative data showing superior performance in those key areas.

Taro: That’s the naive question a listener might ask: does this mean we can take this control strategy and drop it into any kind of power system right away?

Dev: The answer is that while the framework itself—the DRL tuning coupled with DPWA modeling—is general, they are applying it specifically to LCC-S resonant converters for wireless power transfer applications.

Rosa: So for now, we see it as a highly effective solution tailored to this specific converter type, showing what’s possible when you combine these advanced control techniques in practice.

Taro: It shows the potential for using DRL not just as an experiment, but as a scalable way to design controllers for complex power electronics that might need to be adaptive behavior.

Conclusion: Dev: So we've gone through the paper "Deep Reinforcement Learning-Aided Frequency Control of LCC-S Resonant Converters for Wireless Power Transfer Systems" and seen how they use TD3 to tune PI gains combined with DPWA modeling for these converters.

Rosa: The authors are essentially showing that this DRL method replaces the need for manual tuning with an AI that learns the optimal parameters.

Dev: They found it works well in simulation and experiments, demonstrating better stability, robustness, response time compared to conventional PI controllers.

Rosa: It really boils down to this: DRL can be used to iteratively learn those optimal control settings through a loop of experience.

Taro: For the implications for the wider world, it means we have a way to design controllers that can be more adaptive than what we currently design by letting the system learn its own best behavior over time <ref:2505.01850#pg2>.

Dev: It moves us toward systems where control is not just fixed but something that evolves based on operating conditions.

Rosa: So this paper confirms the potential for DRL to be a powerful tool in power electronic control, offering a data-driven alternative to conventional controller design methods.

Taro: It suggests that future work should look at applying this idea more broadly across different types of converters if we want to see how far it can go.

Dev: Right, so the paper "Deep Reinforcement Learning-Aided Frequency Control of LCC-S Resonant Converters for Wireless Power Transfer Systems" provides a practical example of using DRL to tackle nonlinear control challenges.

University of Tehran · Shahid Rajaee Teacher Training University

eess.SY, cs.SY

Submitted: 2025-05-03

Updated: 2025-05-03

Journal ref: IEEE Journal of Emerging and Selected Topics in Power Electronics, vol. 14, no. 2, pp. 2723-2733, 2026

DOI: 10.1109/JESTPE.2025.3638683

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 72/100

The gist: The gist The proposed DRL-based control strategy for LCC-S resonant converters achieves precise and robust output voltage regulation for wireless power transfer applications by leveraging Twin

Key concepts

LCC-S Resonant Converter: This is a type of power converter used in wireless power transfer systems. It uses an LCC tank (inductor and capacitors) on the primary side to efficiently transfer energy wirelessly. The goal is to regulate the output voltage precisely for applications like charging devices.
Direct Piecewise Affine (DPWA) Modeling: This is a mathematical technique used to simplify complex, nonlinear converter dynamics into a structured model. It divides the system's operating space into different 'cells' where the behavior is linear. This allows engineers to analyze and control the system more effectively by handling its inherent nonlinearities systematically.
Twin Delayed Deep Deterministic Policy Gradient (TD3): TD3 is an advanced Deep Reinforcement Learning algorithm used here to find the best PI controller settings automatically. It uses two 'critic' networks to prevent overestimating rewards and delayed policy updates for stability, ensuring the learned control strategy is robust and reliable.

Terminology

Deep Reinforcement Learning — 2× in this paper · explained in 3 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Adaptability — 1× in this paper · explained in 1 episode
Deep Reinforcement Learning (DRL) — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes

Summary

The gist The proposed DRL-based control strategy for LCC-S resonant converters achieves precise and robust output voltage regulation for wireless power transfer applications by leveraging Twin Delayed Deep Deterministic Policy Gradient (TD3) to optimize PI controller parameters, which significantly outperforms traditional tuning methods in terms of stability, robustness, and response time

Modeling the System Dynamics

The complex converter dynamics are captured using the Direct Piecewise Affine (DPWA) modeling technique, providing a structured approach to handling its nonlinearities. The DPWA model is formulated as x˙ = Aix + bi and y = Cix + di, where i denotes the cell in which the state trajectory resides. This model introduces tunable cell boundaries, making it ideal for applications requiring adaptive switching, such as in PECs like resonant converters. The DPWA representation of the LCC-S model is expressed as x˙(t) = Aix(t) + bi and y(t) = Cx(t), where i = φ(s, x(t)).

Control Strategy Implementation

A PI controller is designed and fine-tuned using the TD3 algorithm, enabling dynamic optimization of system performance and ensuring robust control under varying operating conditions. The deep deterministic policy gradient (DDPG) technique is one of the most popular DRL algorithms; nevertheless, it is sensitive to hyperparameters and frequently overestimates Q values obtained from mimicked Q functions, which may render the learned policy erroneous. The TD3 technique was developed with three significant enhancements to address these problems: 1. Reduced overestimation bias by clipped double Q-learning; 2. Improved stability through delayed policy updates; and 3. Reduced detrimental policy variations through regularization of the smoothing of the target policy.

DRL Training for Controller Tuning

To demonstrate the effectiveness of the TD3 algorithm in tuning the PI controller of an LCC-S converter, it was implemented in MATLAB/SIMULINK. The state vector for the agent was defined as x = Z e(t).dt e(t) where e(t) = Vref (t)−Vout(t) is the voltage error. The reward function was designed to motivate the agent to minimize the voltage error: R = −(Vref (t) − Vout(t))2. The initial values for the PI gains, Kp and Ki, were set to 5.28 and 0.05, respectively. After 500 episodes, the learning process converged to a maximum reward, resulting in optimized PI controller parameters: Kp = 0.0553 and Ki = 12.9637.

Performance Comparison

The simulation results highlight the superior performance of the TD3 tuned PI controller in regulating the output voltage of the LCC-S resonant converter compared to the DDPG tuned controller. The TD3-tuned controller demonstrates reduced overshoot and better stability, ensuring a smoother response at t = 0 s when the step input is triggered to regulate the output voltage to 200 volts. Throughout the simulation, under the influence of random noise at 85 kHz (matching the switching frequency), the TD3-tuned controller exhibits enhanced robustness by reducing voltage ripple, making it more reliable for real-world applications despite not being faster in response.

Experimental Validation

The experimental setup was conducted in the Photovoltaic Lab at the University of Tehran and includes the WPT system featuring an LCC-S resonant converter. Under closed-loop control, the microcontroller firmware adjusts the output voltage at a pulse frequency of one hertz, maintaining the target of 28 V with minimal overshoot (under 2%), a 5 ms rise time, and an average output voltage of 27.49 V, indicating a 2% error. The coil voltages reveal a drop due to air gaps and environmental noise, which leads to waveform distortion.

Conclusion

By leveraging DRL, the approach allowed the PI controller to iteratively learn optimal parameters, significantly improving robustness and control performance. Both simulation and experimental results demonstrated the effectiveness of DRL-based tuning, confirming its potential to enhance robust control in complex power electronic systems.

How it works

The paper introduces a novel deep reinforcement learning (DRL)-based control strategy for achieving precise and robust output voltage regulation in LCC-S resonant converters, specifically designed for wireless power transfer applications. Unlike conventional methods that rely on manually tuned PI controllers or heuristic tuning approaches, the method leverages the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to systematically optimize PI controller parameters. The complex converter dynamics are captured using the Direct Piecewise Affine (DPWA) modeling technique, providing a structured approach to handling its nonlinearities. This integration not only eliminates the need for manual tuning, but also enhances control adaptability under varying operating conditions.

The DPWA model is employed to analyze the internal dynamics of converters, as inherently non-linear systems [7], is crucial regardless of their architecture and any accompanying benefits or drawbacks. The DPWA model introduces tunable cell boundaries, making it ideal for applications requiring adaptive switching, such as in PECs like resonant converters. The logic input s is defined as s = (+1 Kx + m ≥ 0-1 Kx + m < 0) where K = [k1, k2, k3, k4, k5, k6, k7] and m represent the parameters of the switching surfaces that determine the variable s.

The control objective in the DRL setup is to minimize the voltage error by defining a reward function R = −(Vref (t) − Vout(t))2. The agent learns through an action-state-agent-environment loop, where the agent's action is the PI controller parameters K, and the environment is the LCC-S converter dynamics. The TD3 algorithm minimizes a loss function Lk = 1/2M Σ i=1 (yi − Qk (Si, Ai; ϕk)) squared for critic k. The actor network is updated to maximize the expected return using the gradient ∇θJ ≈ 1/M Σ i=1 GaiGπi(8).

Key Components and Parameters

The LCC-S resonant converter utilizes a resonant tank on the primary side, consisting of an inductor (L1) and two capacitors (C1 and Cs1). The magnetic coupling between the primary and secondary sides is represented by equivalent inductors (Lp, Ls, and M). Key components listed in Table II include L1 Ser. Pri. Ind. (µH) 73.4, C1 Par. Pri. Cap. (nF) 46.7, Cs1 Ser. Pri. Cap. (nF) 15.5, and Vdc Input Volt of 200 V. The main resonant frequency was set to 85 kHz, which falls within the standard operating frequency range of 79 to 90 kHz for WPT systems.

Advantages of TD3

TD3 employs two critic networks to independently evaluate the same action, which reduces the risk of overestimating the potential rewards of an action. Furthermore, TD3 introduces delayed policy updates, meaning that the actor network (which controls the actions) is updated less frequently than the critic networks. Finally, TD3 incorporates target noise, a technique that adds random noise to target actions used to update the critic networks.

Improvements for AI systems

Bold Header: Automated PI Controller Tuning via TD3

The improved AI system can autonomously tune PI controller parameters by leveraging the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to systematically optimize PI controller parameters, thereby eliminating the need for manually tuned controllers and ensuring robust control under varying conditions.

Bold Header: Enhanced Dynamic Modeling with DPWA

The system gains superior accuracy in capturing complex dynamics by employing the Direct Piecewise Affine (DPWA) modeling technique, which allows the converter to handle nonlinearities through tunable cell boundaries.

Bold Header: Robust Frequency Control Strategy

The AI can implement a novel control strategy for frequency regulation, as the research demonstrates that By considering the frequency control strategy shown in Fig. 4, the LCC-S resonant converter inherently operates as a switched system with affine subsystems.

Bold Header: Superior Control Performance over DDPG

The TD3-tuned controller exhibits reduced overshoot and better stability compared to DDPG-tuned controllers, indicating that the AI system provides more stable and resilient control performance under dynamic and noisy conditions.