Robust and Adaptive Deep Reinforcement Learning for Enhancing Flow Control around a Square Cylinder with Varying Reynolds Numbers
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.
Kai: I'm Kai.
Mira: And I'm Mira, here with Lev.
Kai: We're looking at the paper: "Robust and Adaptive Deep Reinforcement Learning for Enhancing Flow Control around a Square Cylinder with Varying Reynolds Numbers".
Mira: Here's the gist of the paper. Robust DRL for enhancing flow control around a square cylinder with varying Reynolds numbers applies a Deep Reinforcement Learning algorithm to modulate the flow of synthetic jets on a square cylinder across four different Reynolds numbers to effectively suppress vortex shedding while significantly diminishing drag and lift fluctuations.
Kai: Flow Characteristics Across Reynolds Numbers The investigation begins by analyzing the baseline flow in the absence of active control, observing that at Re = 100 and Re = 200, the vortex shedding exhibits mono-frequency characteristics.
Mira: Conversely, at Re = 300 and Re = 400, the vortex shedding is dominated by multiple frequencies, which is indicative of more complex flow features.
Kai: As the Reynolds number increases to 300 and 400, both the amplitude of CD (Fig.5 (j)) and CL (Fig.5 (k)) surpass those observed at other Re <ref:2404.1212359, When the Re reaches 400, both the amplitude of CD (Fig.5 (j)) and CL (Fig.5 (k)) surpass those observed at other Re accompanied by a more complex fluctuation frequency spectrum <ref:2404.12123#pg7>.
Paper summary: Kai: To build on what we just said, this paper introduces a Deep Reinforcement Learning approach specifically for Active Flow Control, which is using synthetic jets to manipulate the flow around a square cylinder. The core idea is that you use an algorithm called Soft Actor-Critic, which blends Deep Learning with maximum entropy theory to help the system explore its options and find robust strategies.
Lev: When we look at the math behind this, it’s framed as a Markov Decision Process, which is defined by a tuple of states, actions, rewards, transitions and an initial distribution rho zero. The optimization goal for the Soft Actor-Critic algorithm is to find a policy pi star that maximizes the expected sum of rewards plus alpha times the entropy of the policy given the state.
Mira: That mathematical setup tells us they are focusing on exploration alongside performance; they aren't just looking for one perfect solution, but a strategy that can handle different flow regimes reliably. They are defining this as finding pi star equals argmax pi sum t E(st,at)∼rho pi
r(st,at) + alpha H (pi(·st)): .
Kai: They test this across four distinct Reynolds numbers: one hundred, two hundred, three hundred, and four hundred. The initial observation is that at re equals one hundred and two hundred the vortex shedding shows mono-frequency characteristics.
Lev: That’s the baseline physical behavior they start with; it’s simpler to model because there’s only one dominant frequency in the wake flow when Re is low. But then, as we increase Re to three hundred and four hundred, that shedding becomes dominated by multiple frequencies, which points to more complex physical features.
Mira: And they note that this complexity also leads to larger amplitudes for both the drag coefficient and the lift coefficient at those higher Reynolds numbers compared to what’s seen in lower re cases. This is why they need a robust method, not just a one-off solution for each Re number.
Kai: The paper essentially claims that this DRL framework can effectively suppress that vortex shedding while simultaneously reducing drag and lift fluctuations significantly across all those Reynolds numbers. They are showing the control strategy works dynamically for each condition.
Lev: When we look at what the agents actually do, they successfully learn control strategies tailored to these different dynamic conditions, demonstrating how adaptable DRL can be in designing solutions for complex fluid dynamics problems like this one.
Conclusion: Kai: So to wrap up this paper on Robust and Adaptive Deep Reinforcement Learning for Enhancing Flow Control around a Square Cylinder with Varying Reynolds Numbers, the main idea is that using the Soft Actor-Critic algorithm allows the system to dynamically adjust synthetic jet flow rates based on the specific Reynolds number of the flow.
Mira: It shows that DRL is capable of controlling intricate, multi-frequency dominated vortex shedding phenomena by reacting to different scales of frequency in the fluid dynamics. That adaptability across those four different Re values is what makes this work interesting.
Lev: The implication for real hardware is that while these agents are smart, we still need a detailed investigation into the underlying physical mechanisms that allow them to react effectively to those multi-scale frequencies in chaotic and turbulent active flow control scenarios.
Kai: The authors acknowledge that while their SAC strategy meets their expectations regarding the ability to handle those different conditions, it’s still important to understand exactly how the DRL agent is making these decisions in such a complex environment.
Mira: It suggests that the future work here should focus on delving into those physical mechanisms behind how this DRL-based AFC technology manages such intricate flow patterns across different Re.
Wang Jia, Hang Xu
State Key Lab of Ocean Engineering, School of Naval Architecture, Ocean and Civil Engineering, Shanghai Jiao Tong University
physics.flu-dyn
Submitted: 2024-04-18
Updated: 2024-09-26
Journal ref: Physics of Fluids 36, 054103 (2024)
DOI: 10.1063/5.0207879
Code: https://github.com/jerabaul29/Cylinder2DFlowControlDRLParallel65
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: Robust DRL for enhancing flow control around a square cylinder with varying Reynolds numbers applies a Deep Reinforcement Learning algorithm to modulate the flow of synthetic jets on a square…
Key concepts
- Deep Reinforcement Learning (DRL)
- A type of machine learning where an agent learns to make decisions by interacting with an environment. It uses neural networks to learn complex control policies, allowing the system to adapt its actions in real-time based on the flow conditions.
- Soft Actor-Critic (SAC)
- A specific DRL algorithm that combines reinforcement learning with maximum entropy theory. This approach encourages the agent to explore widely and find robust, high-performing control strategies while maintaining a balance between exploration and exploitation.
- Reynolds Number (Re)
- A dimensionless number representing the ratio of inertial forces to viscous forces in fluid flow. The study investigates how flow characteristics, like vortex shedding complexity, change as this number varies across four distinct values (100, 200, 300, and 400).
- Vortex Shedding
- The process where alternating vortices are shed from the surface of a cylinder in fluid flow. The DRL system was trained to actively modulate synthetic jet flow to suppress this shedding, which is crucial for reducing drag and lift instability.
Terminology
Summary
Robust DRL for enhancing flow control around a square cylinder with varying Reynolds numbers applies a Deep Reinforcement Learning algorithm to modulate the flow of synthetic jets on a square cylinder across four different Reynolds numbers to effectively suppress vortex shedding while significantly diminishing drag and lift fluctuations. p. 0
Flow Characteristics Across Reynolds Numbers
The investigation begins by analyzing the baseline flow in the absence of active control, observing that at Re = 100 and Re = 200, the vortex shedding exhibits mono-frequency characteristics. Conversely, at Re = 300 and Re = 400, the vortex shedding is dominated by multiple frequencies, which is indicative of more complex flow features. p. 0 As the Reynolds number increases to 300 and 400, both the amplitude of CD (Fig. p. 85 (j)) and CL (Fig. p. 2 p. 25 (k)) surpass those observed at other Re <ref:2404. p. 81212359, When the Re reaches 400, both the amplitude of CD (Fig. p. 85 (j)) and CL (Fig.5 (k)) surpass those observed at other Re accompanied by a more complex fluctuation frequency spectrum <ref:2404. p. 812123#pg7>.
How the training signal is built
The study employs the Soft Actor-Critic (SAC) algorithm, which merges the principles of DRL with maximum entropy theory to promote exploration and robust strategies. The mathematical foundation of RL is encapsulated by the Markov Decision Process (MDP), which is characterized by a tuple (S,A,R,P,ρ0) where S represents the set of all possible st within the environment. p. 5 The core optimization goal of the SAC algorithm is defined as π∗ = argmax π ∑t E(st,at)∼ρπ [r(st,at) +αH (π(·st))]. p. 5
Where the gains come from
The quantitative results demonstrate that the data-driven active control strategy results in a drag reduction of approximately 14.4%, 26.4%, 38.9%, and 47.0% for Re = 100, 200, 300, and 400, respectively. p. 0 The SAC-based flow control results show that under four different Re flow configurations, the CL quickly drops to a level close to zero after the control is initiated and stabilizes at a level close to zero. The near-zero magnitude of the jet velocity also implies that minimal energy is required to sustain the stabilized flow for long-duration drag reduction. p. 2
Control Mechanism and Flow Field Comparison
The DRL agent successfully identifies control strategies tailored to dynamic conditions at varying Re, showcasing the adaptability of DRL in devising distinct control strategies. The stream line comparison shows that under the influence of the synthetic jets, the recirculation region downstream of the square cylinder is significantly elongated and expanded. p. 2
Conclusion on Robustness
The deployment of intelligent agents trained via the SAC algorithm has facilitated the learning of a control strategy capable of dynamically adjusting the mass flow rate of synthetic jets. These research outcomes underscore the adeptness of DRL in controlling intricate, multi-frequency dominated vortex shedding phenomena <ref:2404. p. 012123#pg7>. The capability to react to multi-scale frequencies provides promising aspects of the adaptability of DRL in chaotic and turbulent AFC scenarios. p. 0 The strategy provided by SAC has met our expectations in this regard <ref:2404. p. 012123#pg7>.
Acknowledgments
The authors thank Dr. Jean Rabault (University of Oslo, Oslo, Norway) and Mr. p. 0 Qiulei Wang (The University of Hong Kong, Hong Kong SAR, China) for making their open-source codes for deep reinforcement learning and numerical simulation available online at https://github. p. 0com/jerabaul29/Cylinder2DFlowControlDRLParallel and https://github.com/venturi123/DRLinFluids <ref:2404. p. 012123#pg7>. The authors report no conflict of interest. p. 1
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
1
Improvement in DRL algorithm selection: The paper utilizes Soft Actor-Critic (SAC) because it is an advanced framework that merges the principles of DRL with maximum entropy theory,
which leverages deep neural networks to serve as approximators for value and policy functions, aiming to maximize the entropy of at.
This choice ensures a balance between exploiting known rewards and exploring novel at,
making it highly versatile for continuous action spaces.
2
Improvement in state representation: The state st is constituted by instantaneous flow field or pressure data collected from specific locations within the computational domain,
with 201 probes strategically placed to capture transient states. This detailed input allows the agent to observe the pertinent information describing the current circumstances of both the environment and the agent.
3
Improvement in action space constraint: The action at of the agent are defined as the velocity of the synthetic jets, with the stipulation that the magnitude of these action cannot exceed 2% of the inlet velocity.
This constraint is explicitly designed with the intention of achieving drag reduction using
the smallest possible jet velocity, thus ensuring the energy efficiency.
4
Improvement in reward function design: The reward function rTi incorporates both CD and CL, taking the form rTi = CD,0 −(CD)Ti − ω (CL)Ti,
where CD,0 represents a baseline drag coefficient. This structure drives the overall reward to positive values to help with the convergence during training.
5
Improvement in parallelization strategy: The DRL training and CFD simulations are performed on a high-performance computing system utilizing 64 cores distributed across two sockets. This allocation and utilization of CPU cores during the coupled training of DRL algorithms and CFD simulations, showcasing the fundamental parallelization strategy.
Sources
- Emergence of Locomotion Behaviours in Rich Environments
- Optimal Parallelization Strategies for Active Flow Control in Deep Reinforcement Learning-Based Computational Fluid Dynamics
- Soft Actor-Critic Algorithms and Applications
Related papers
- Data-driven impeller model for efficient large eddy simulations of metastable von K'arm'an flows
- Numerical study of nitrogen oxides chemistry during plasma assisted combustion in a sequential combustor
- Emergent Transfer of a Physics Foundation Model from Simulation to Laboratory Turbulence
- Early onset of secondary shear instability in Kelvin-Helmholtz braids at high Reynolds number
- Rotational covariance, continuous spectrum and non-modal growth of three-dimensional planar weak viscous shock layers
- Deep Learning-based Algebraic Reynolds Stress Closures for RANS Simulations of Turbulent Flows