4.1. Mesh convergence study

As shown in figure 3, the computational domain is defined by three parts for the ease of mesh generation. Here, $x_1$ and $x_2$ are the lengths of the two rectangle subdomains in the streamwise direction. The two rectangle subdomains are discretised using $n_1 \times n_3$ and $n_2 \times n_3$ element nodes, respectively. The middle square domain with a side length of 2.0 is discretised using an ‘O’-type mesh with $4 \times n_3 \times n_4$ element nodes (4 here denotes the four compartments delimited by the blue lines in the figure; $n_3=8$ and $n_4=6$). To obtain a reasonable computational mesh, we study the influences of both the computational domain size and the mesh resolution on the numerical results. As shown in table 1, five computational sizes are investigated for $Re=200, \beta =0.5$ ($Re=200$ is in the upper limit of the Reynolds numbers we investigate in this work) and $E$ is the total number of elements. For each computational size, we compare numerical results by using meshes of two resolution levels (L1 and L2). The L1 mesh is finer, which is generated by doubling the element numbers used in the L2 mesh. The time-averaged $C_d$ evaluated by different meshes are shown in table 1 and the reference value evaluated by Sahin & Owens (Reference Sahin and Owens2004) is 2.4245. The choice of five computational domains does not significantly influence the numerical results and all are close to the reference. This means that using the smallest computational domain (D1) is sufficient. Compared with the finer L1 mesh, the L2 mesh merely introduces an error of less than $0.003\,\%$. In the following, we will use D5-L1 as the configuration for all the stability analyses below and in the case of RL control, to reduce the computational burden, we use D1-L2.

Figure 3.

The computational mesh is composed of two rectangle domains and a square domain.

Table 1.

Resolution parameters of L2 meshes in five computational domains and the time-averaged $C_d$ on the confined cylinder with $\beta =0.5$ and $Re=200$ evaluated by different meshes. Here, $E$ is the total number of elements in the spectral element method.

4.2. Critical Reynolds numbers

The dynamics of the confined cylinder wake flow is governed by the Reynolds number and the blockage ratio. The vortex shedding in the wake flow occurs through a symmetry-breaking Hopf bifurcation beyond the critical Reynolds number ($Re_c$). In the confined flow past a cylinder, $Re_c$ varies with the blockage ratio. To determine the vortex shedding region in the $Re - \beta$ plane, we solve the eigenvalue problem in the linear stability analysis based on the SFD base flow.

As shown in figure 4, two values of $\beta$ are studied. For the confined flow with $\beta =0.5$, the growth rate of the leading eigenmode increases monotonically with a rise of $Re$ from 100 to 150. When the Reynolds number is greater than $Re_c = 123.6$, the base flow becomes unstable and then the wake starts meandering. However, for the confined flow with the blockage ratio $\beta =0.75$, two critical $Re_c$ values, $Re_{c1} = 108.9$ and $Re_{c2} = 169.6$, are identified. With the increase of $Re$ from 100, the flow stability is lost via the first Hopf bifurcation point at $Re_{c1}$ and the wake vortex starts shedding subsequently. If the $Re$ further increases, passing the other bifurcation point at $Re_{c2}$, the flow becomes stabilised. As shown in figure 5, the confined flows with $\beta =0.25$ and $\beta =0.5$ merely have one recirculation zone; whereas for the larger blockage ratio $\beta =0.75$, in addition to the recirculation bubble just downstream of the cylinder, two additional recirculation bubbles develop close to the walls further downstream when $Re$ is large enough. Similar to observations by Sahin & Owens (Reference Sahin and Owens2004), the recirculation bubbles on the confinement walls become larger with the increase of $Re$ and lead to the other Hopf bifurcation point at $Re_{c2}$.

Figure 4.

Eigenvalues of the leading eigenmode at different Reynolds numbers for two blockage ratios $\beta =0.5,0.75$. The curves are interpolated using data of the scatter points. The grey shade indicates flow instability. Note that that the data points are our computational results and the lines are fitted to guide the eyes.

Figure 5.

Vorticity of SFD base flows of different confined cylinder wake flows. The purple dashed lines and black dash–dotted lines are the recirculation zones boundaries of the SFD base flows and time-mean flows, respectively: (a) $\beta =0.25$; (b) $\beta =0.5$ and (c) $\beta =0.75$.

By examining the critical Reynolds numbers with different blockage ratios, the vortex shedding region is shown in figure 6 as a grey background in the $Re - \beta$ plane. The region is determined by two branches of critical points, and we have used 13 and 8 data points in the lower branch and upper branch, respectively. The favourable agreement with the results by Chen et al. (Reference Chen, Pritchard and Tavener1995) and Sahin & Owens (Reference Sahin and Owens2004) demonstrates the good accuracy of the numerical methods and meshes used in this work. The confined flow becomes more stable as $\beta$ increases up to 0.5. Then, the flow becomes destabilised as the block ratio increases. When $\beta$ is larger than ${\sim}0.85$, the flow is again stabilised. For a relatively large $\beta$ ($\lessapprox 0.85$), another stability region can be identified when $Re$ is sufficiently large (when $Re\le 200$). It is noted that the flow phenomena are richer and more complex in the upper-right region of the $Re - \beta$ plane (see figure 5 of Sahin & Owens Reference Sahin and Owens2004) and they are not studied here. We will focus on the grey region where vortex shedding occurs and will use RL control to abate it.

Figure 6.

Vortex shedding region (grey) delimited by the neutral stability curve. Flows in the white region are stable. A comparison is made to two previous publications, as shown in the legend.

4.3. Vortex shedding phenomenon

In general, the oscillating Kármán vortex street downstream the cylinder wake occurs owing to the loss of instantaneous reflection symmetry through a Hopf bifurcation. For the unconfined flow past a cylinder, it is suggested by Maurel, Pagneux & Wesfreid (Reference Maurel, Pagneux and Wesfreid1995) and Noack et al. (Reference Noack, Afanasiev, Morzyński, Tadmor and Thiele2003) that the amplitude of the oscillating wake saturates when the time-averaged flow (mean flow) is marginally stable and the mechanism for nonlinear saturation of the oscillating wake flow is the mean flow correction/modification through the formation of Reynolds stresses. The mean flow provides a good profile to predict the shedding frequency of the unconfined cylinder wake (Yang & Zebib Reference Yang and Zebib1989; Pier Reference Pier2002; Barkley Reference Barkley2006). To understand in detail the stability property of the confined cylinder wake, hereafter, we perform the global linear stability analysis of its mean flow.

Figure 7 shows the leading eigenvalues of the confined cylinder wake with $\beta =0.25$, $\beta =0.5$ and $\beta =0.75$. Compared with the stability analysis based on the SFD base flow (black solid lines), the frequencies solved by the linear analysis based on the mean flow (blue dashed lines) agree better with the results of nonlinear DNS (red filled triangles), and the relative discrepancies are almost within 1 %. The growth rates of the mean flows are close to zero, which implies that the confined mean flows are marginally stable. The conclusions hold for both the weak and strong confinement cases. This implies that the confined cylinder wake flows approximately have the real-zero imaginary-frequency (RZIF) property (Turton, Tuckerman & Barkley Reference Turton, Tuckerman and Barkley2015), which is similar to the unconfined cylinder wake. The RZIF property implies that the eigenfrequency of a nonlinearly saturated oscillating flow can be well approximated by a linear analysis based on the time-mean flow (Pier Reference Pier2002; Barkley Reference Barkley2006; Sipp & Lebedev Reference Sipp and Lebedev2007). Furthermore, the linear stability analysis using the SFD base flow (black lines) generates apparently different results than those using the mean flow and of DNS (except the frequency in the case of $\beta =0.75$).

Figure 7.

Frequencies and growth rates as a function of $Re$ in the global linear stability analysis of confined cylinder flows for three values of confinement ratio $\beta$.

To reveal the relationship between the SFD base flow and the mean flow, we perturb and evolve the SFD base flow at $\beta =0.25$ at $Re=150$ to see how it develops to the saturated state (with vortex shedding). This analysis follows that for the unconfined wake flows in Barkley (Reference Barkley2006). As shown in figure 8, because the SFD base flow is unstable, the amplitude of the $C_l$ oscillation increases and eventually saturates at a periodic vortex shedding state. The cone-like shape of drag evolution has been explained by Loiseau, Noack & Brunton (Reference Loiseau, Noack and Brunton2018) based on the sparse identification of nonlinear dynamics, SINDy (Brunton, Proctor & Kutz Reference Brunton, Proctor and Kutz2016). Similarly to the unconfined flow past a cylinder, the evolution of the unstable SFD base flow in the confined wake is a nonlinear saturation of oscillations (as shown in panels a and b), and the selection of vortex shedding amplitude and frequency is based on the marginal stability of the mean flow. In figure 8(c), we show the growth rates and frequencies of the SFD base flow and the saturated mean flow.

Figure 8.

Relationship between the SFD base flow and the mean flow at $Re=150$ with $\beta = 0.25$: (a) $C_l$ as function of $t$ from the SFD base flow to the saturated flow with vortex shedding; (b) phase diagram of $C_l$ and $C_d$ from $t=0$ to $t=150$; (c) growth rates and frequencies of the SFD base flow and the mean flow solved by the linear stability analysis.

4.4. Structural sensitivity

The above section details the frequency and growth rate of the dominant linear mode in the confined cylinder wake flow. Regarding the flow control and manipulation, much more useful information can be obtained by conducting a sensitivity analysis. Structural sensitivity reveals the most sensitive spatial part of the flow to perturbation; this region is traditionally dubbed as a wavemaker region. We investigate the influence of blockage ratios and Reynolds numbers on the wavemaker region in this section. Three Reynolds numbers, $Re=115, 150, 185$ are selected, covering subcritical and supercritical cases. The growth rates and frequencies of the leading eigenvalues obtained in the global linear stability analysis are shown in figure 9. When the blockage ratio $\beta$ is smaller than 0.5, increasing $\beta$ stabilises the confined cylinder wake, and the leading eigenfrequency decreases simultaneously. For the confined wake flow at $Re=115$, the vortex shedding is even fully suppressed (i.e. the flow becomes stable) when $0.4<\beta <0.6$. Then, further increasing $\beta$ destabilises the wake, which becomes unstable again when $0.6<\beta < 0.8$. These results on the growth rate are consistent with those in figure 6. In confined cylinder wakes at $Re=150$, a similar stabilising–destabilising trend is observed when $0.4 < \beta < 0.6$ but the flow is always unstable with a positive growth rate. A stabilising effect is observed when $\beta > 0.7$ and the flow becomes stable when $\beta = 0.8$. This is because the second critical Reynolds number ($Re_{c2}$) exists in the confined cylinder wakes with $\beta > 0.7$ and it decreases significantly with the increase of $\beta$, see figure 6. For wake flows at $Re=185$, the stabilising effect for $\beta > 0.7$ is more significant and leads to a more stable flow when $\beta = 0.8$. Despite the stability, the wake flow is in an asymmetric status (see figure 10(c) with $\beta =0.8$ below), and this phenomenon has been reported by Sahin & Owens (Reference Sahin and Owens2004). However, the effects of $Re$ on the leading eigenfrequency are not significant when $0.4<\beta < 0.7$, as shown in figure 9(b). When $\beta > 0.7$, the leading eigenfrequency of $Re=185$ decreases more significantly.

Figure 9.

Growth rates and frequencies of SFD base flows of confinement cylinder wakes with different blockage ratios and Reynolds numbers.

Figure 10.

Structural sensitivity analysis of confined cylinder wakes with different blockage ratios and Reynolds numbers. The adjoint mode, direct mode and wavemaker are shown by red, blue and grey colour maps. The recirculation zone is illustrated by the purple dashed line. Note that only part of the domain is shown: (a) $Re=115$; (b) $Re=150$ and (c) $Re=185$.

Next, we show the direct, adjoint modes and wavemaker region in the confined cylinder wake flow with different blockage ratios and Reynolds numbers in figure 10. The amplitudes of adjoint and direct modes for velocity are represented by the red and blue colour maps, respectively. The wavemaker region is shown by nine contour lines using the grey colour map, which illustrates the sensitive domain with $\eta > 5\,\% \times \eta ^{max}$, where $\eta$ is the overlap function in structural sensitive analysis evaluated by (3.6) and $\eta ^{max}$ is the largest $\eta$ in the computational domain. Similar to unconfined cylinder wakes (Giannetti & Luchini Reference Giannetti and Luchini2007; Marquet et al. Reference Marquet, Sipp and Jacquin2008), the wavemaker region of the confined wake flow is located downstream of the cylinder, and thus to some extent, the upstream domain is less important in terms of flow sensitivity and needs not to be monitored in the suppression of wake vortex shedding. Furthermore, when $\beta <0.7$, the cylinder wake flow with a larger blockage ratio (i.e. a stronger confinement effect) possesses a longer and wider recirculation zone, and correspondingly the wavemaker region also expands and moves downstream. Regarding the length of the recirculation region, Chen et al. (Reference Chen, Pritchard and Tavener1995) also reported longer recirculation zones as a function of $Re$, whereas the result of the elongated wavemaker region with $\beta$ seems not to have been reported in the literature for the confined cylinder wake flow. However, increasing the Reynolds number also leads to longer recirculation zones, and the wavemaker region is pushed downstream (at least for the three $Re$ values investigated here). Regarding the flow control of the confined cylinder wake flows, the significance of these results is that to more efficiently suppress the vortex shedding, one should monitor the perturbations further downstream from the cylinder when $\beta$ or $Re$ increases because the most sensitive region (wavemaker) is located further downstream. Finally, for some wake flows with larger $\beta$, i.e. $Re=150, \beta =0.8$ or $Re=185, \beta =0.7, 0.8$, recirculation bubbles are developed on the confinement walls downstream of the cylinder wake (similar to the results in figure 5), which stabilises the cylinder wake and leads to a second critical $Re$ as analysed in the previous section.

5.1. Vortex shedding suppression via reinforcement learning

After characterising the linear instability and flow sensitivity of the confined cylinder wake flows, we adopt the deep RL to control the vortex shedding in this flow using the two synthetic jets on the cylinder. In the literature, Rabault et al. (Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019) has studied an RL-based control method to reduce the drag on the cylinder.

To begin, we would like to first obtain some preliminary results by defining the reward as

(5.1)\begin{equation} r_0={-}\sum_i^{n_{node}} {((u^i)^2 + (v^i)^2)}, \end{equation}

to have a peek of what the controlled wake may become. Here, $r_0$ is the negative value of a plain sum of the kinetic energy in the computational domain and $n_{node}$ is the number of grid nodes in DNS. Two points that deserve to be mentioned are that the kinetic energy will not decrease to zero as long as we feed the flow domain with an inflow and that the kinetic energy will decrease with the suppression of the vortex shedding. Thus, the RL algorithm will control the confined wake flow towards the lowest kinetic energy where the vortex shedding is weakest. All the control investigations from §§ 5.1–5.4 are based on 86 velocity probes covering the wavemaker region (see probe distributionin figure 19b). The reason for this selection will be explained in § 5.6.

Following Rabault et al. (Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019), we use a random reset in the training process so that new episodes have a 20 % possibility to start from the given uncontrolled initial condition and otherwise they start from the last state of the previous episode. As shown in figure 11(a), the averaged energy decreases with the training of the RL agent, especially after 100 episodes. The energy spikes of the learning curve result from the starting points of new episodes from the given initial condition. After $400 \sim 500$ episodes, the RL training approximately converges. As shown in figure 11(b), we test the RL-based control from $t=16$ to $t=200$ (the flow starts from the initial condition at $t=0$). Although the testing control time is shorter than that in the fluidic pinball stabilisation (Maceda et al. Reference Maceda, Li, Lusseyran, Morzyński and Noack2021), the controlled flow shows a convergence to a periodic state in this case. The obtained control policy reduces the kinetic energy and the wake vortex shedding is suppressed (figure 12). At the beginning of the control (briefly after $t=16$), the energy first increases, and a slightly larger lift is observed consequently (see the inset in figure 11b). Then, the energy decreases exponentially and the oscillations of the lift coefficient are also reduced before reaching a steady oscillation. This result seems to be a common characteristics of the RL control applied to the wake flow in our case and will be seen later in our results. A larger actuation has also been observed in the RL control of Tang et al. (Reference Tang, Rabault, Kuhnle, Wang and Wang2020) whose aim is to reduce the drag on the cylinder. Eventually, the value of the lift coefficient converges to an oscillation with a small amplitude. Figure 12 shows the vorticity of the baseline and controlled flows. RL control increases the recirculation zone when it suppresses the oscillation. Nevertheless, slight vortex shedding downstream from the recirculation zone can still be observed, though the extent and amplitude are largely decreased.

Figure 11.

RL-based vortex shedding suppression control using the plain sum of the kinetic energy ($\beta =0.25$ and $Re=150$). In panel (b), the grey shade means that no control is applied until $t=16$. The reward function is (5.1). (a) Training process and (b) RL control performance.

Figure 12.

Vorticity of the baseline confined cylinder flow with $\beta =0.25$ at $Re=150$ and the controlled flow with the RL agent trained by the total energy. RL control using the kinetic energy damps the oscillation by increasing the recirculation zone. Nevertheless, the vortex shedding is not fully suppressed. The reward function is (5.1).

5.2. SFD base flow

From figure 12, we see that when the confined wake flow is controlled, one salient feature is that the recirculation zone elongates. This is in fact a strong clue indicating that the controlled wake flow may converge to a flow that is close to the SFD base flow, because the recirculation zone in the latter case is relatively long (see figure 5).

To confirm this hypothesis, we use the fluctuation of the kinetic energy to monitor the strength of the vortex shedding, which is computed by

(5.2)\begin{equation} s_e = \sum_i^{n_{node}} { ( (u^i - u^i_0)^2 + (v^i - v^i_0)^2 )}. \end{equation}

The shedding wake flow ($\boldsymbol {u}$) can be decomposed as a sum of a reference part $\boldsymbol {u}_0=(u_0,v_0)$ and a shedding part $\boldsymbol {u}_s$, as in $\boldsymbol {u} = \boldsymbol {u}_s + \boldsymbol {u}_0$. To suppress the vortex shedding, we can use $r = -s_e$ as the reward function in RL, which will minimise the fluctuation shedding energy. The idea is then to use the time-mean flow or the SFD base flow to evaluate the shedding energy in training RL control agents; that is, the mean flow $\boldsymbol {u}_b$ (obtained by time-averaging the periodic vortex shedding) and the SFD base flow $\bar {\boldsymbol {u}}$ (obtained by the SFD method) are used in (5.2) for $\boldsymbol {u}_0$. If the RL control with the SFD base flow being used in the reward can yield $r\rightarrow 0$, then we can understand that the controlled wake flow indeed converges to the SFD base flow. The time-mean flow is also tested for a comparison.

Figure 13 shows the RL control performances of the corresponding agents. Using the mean flow in the shedding energy evaluation cannot effectively suppress the vortex shedding. The result indicates that the time-averaged mean flow is modified to another status where the vortex shedding is slightly suppressed. Thus, the mean flow is not the flow to which the RL algorithm will lead the controlled wake flow. However, using the SFD base flow in the shedding energy evaluation (5.2) leads to a complete suppression. The shedding energy is reduced to approximately zero, which means that the controlled wake is almost the same as the SFD base flow. The fluctuations of the lift coefficient are also almost fully suppressed and no vortex shedding is observed in the vorticity contours. It can be concluded that vortex shedding suppression (by RL) in the confined cylinder wake flow is realised by modifying the wake flow to be a status which is very close to the SFD base flow. Interestingly, Flinois & Colonius (Reference Flinois and Colonius2015) found similar results in the adjoint-based optimal control of an unconfined cylinder wake that the controlled flow is the same as the SFD base flow. Because of its good performance, we will use the SFD base flow to evaluate the shedding energy for RL control in the following.

Figure 13.

RL-based vortex shedding suppression of the confined cylinder wake ($\beta =0.25$ and $Re=150$) with RL agents trained using different reference flows (mean flow and SFD base flow). It can be seen that using the SFD base flow as the reference in the RL reward evaluation leads to an effective control with the vortex shedding fully suppressed. The reward function is (5.2).

5.3. Control starting time

The above RL control starts at $t=16$, which is arbitrarily chosen. We next test the robustness of the RL control policy in terms of its application at other phases in the vortex-shedding period. Figure 14 shows the control processes of the confined cylinder wake with the RL-based control starting from three different time stamps. The interval between adjacent time points is 0.8 unit time ($t_2 - t_1 = t_3 - t_2 = 0.8$), which is approximately $1/3$ period of the vortex shedding. Among the three cases explored here, a general trend is that the trained RL always uses large positive jet flow rates ahead of the next minimum point of the lift coefficient, when the upper vortex rolls up. Afterwards, relatively small jet rates are needed, and the shedding energy starts to decay. Figure 14 also shows the vorticity fields at the moments with largest jet flow rates, which are exactly the moment of the upper vortex rolling-up. The large blowing of the upper jet produces positive vorticity to offset the rolling-up vorticity, which can be considered to be destructive to the vortex development. This suggests that using large flow rates when the vortex rolls up is an important step in suppressing the cylinder wake fluctuations. The trained RL agent is adaptive to the shedding phase and captures the ‘right’ timing. Owing to the symmetry of the problem at hand, it can be understood that there is a similar RL control policy that uses significant blowing of the lower jet to suppress the rolling-up of the lower vortex.

Figure 14.

(a) Confined cylinder wakes ($\beta =0.25$ at $Re=150$) with the RL control starting at different time points ($t_1$, $t_2$ and $t_3$). (b) Wake vorticities at $t=7.5$. It can be seen that no matter when the control starts, RL always chooses to use large flow rates to offset the vorticity at the moment of the upper vortex rolling-up.

5.4. Necessity of a persistent oscillating control

According to the linear stability analysis, the SFD base flow is unstable; the RL agent maintains this unstable status by actively modifying the jet flow rate in real time. To probe how the stability property of the flow changes with time, we will use dynamic mode decomposition (DMD; $N_{snapshot}=20$ and ${\rm \Delta} t = 0.2$) to monitor the status of the flow stability/instability in the control process. In additional tests (not shown), we used more DMD snapshots and found that the results agreed well with those to be presented in the following. In figure 15, the control starts at $t=16$, and in the beginning, again, relatively large jet flow rates are imposed by the RL agent. Then, as the lift coefficient is reduced, the flow rate of the synthetic jets decreases to a small value with a very small oscillating amplitude (see the inset in panel a). Even though the amplitude of the fluctuation is insignificant, we find that this fluctuation in the flow rate is vital to the successful continuous suppression of vortex shedding (we note in passing that the jet flow rate does not oscillate exactly around 0 because the overall effect of blowing/suction will introduce a small degree of asymmetry with respect to the channel centreline in the flow). We perform a numerical experiment in figure 15 to impose a sudden modification to the jet flow rate after $t=80$. If nothing is changed in the RL control, when $t>80$, the controlled wake flow is stable with the slightly fluctuated flow rates (the black lines). However, using a similar but constant jet flow rate ($u=0.00352$, see the red dashed line in panel $a$) at $t=80$ soon triggers flow instability (panel $c$) and gradually leads to vortex shedding in the cylinder wake (panel $b$). Figure 16 shows the leading DMD modes of the two flows from $t=85$ to $t=100$. With the active jet flow rate fluctuation, significant vorticity is observed around the jets in the leading mode of the RL controlled flow. However, the leading DMD mode remains almost unchanged in the (successful) controlled process from $t=85$ to $t=100$. Yet if a constant jet flow rate is forced, the vorticity around the jets of the leading mode vanishes. At the same time, the leading mode becomes unstable and gradually evolves to a state with much stronger vorticities downstream the cylinder. Finally, the flow develops to a saturated periodic vortex shedding state, which has a similar $C_l$ magnitude to the uncontrolled flow (see figure 15b). To sum up, in the vortex shedding suppression of cylinder wake, the RL-based policy tends to spend large energy at the start of the control in modifying the mean flow structure. Then, much less effort is required to maintain the stabilised flow. A similar tendency is found in the RL-based drag reduction control of a confined cylinder (Rabault et al. Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019). Nevertheless, a persistent active control using the RL agent, even though its oscillating amplitude is small, is required to suppress the vortex shedding in the cylinder wake. This is because the controlled flow is almost identical to the SFD base flow, which is unstable. Thus, when the persistent control is suppressed, the stabilised flow will become unstable and the vortex starts shedding again, as demonstrated in our numerical experiment above.

Figure 15.

DMD analysis of the RL-based control process. The grey shade is the confined cylinder wake without control and the RL-based control starts at $t=16$. We test a constant jet flow rate in the controlled cylinder wake from $t=80$, which is shown as the red shade. The black solid lines show results of RL-based control and the red lines are results with the control switched to the constant flow rate.

Figure 16.

Vorticities of the leading DMD modes in the controlled flows ($\beta =0.25$ at $Re=150$). Only part of the computational domain is shown for clarity. After the vortex shedding is damped, using a similar but constant jet flow rate soon triggers flow instability and gradually leads to vortex shedding in the cylinder wake.

In real-world applications, the control system always faces uncertain noise. Tang et al. (Reference Tang, Rabault, Kuhnle, Wang and Wang2020) and Paris et al. (Reference Paris, Beneddine and Dandois2021) have shown that the RL-based control can be robust to the Reynolds number variation and systematic noise. There is another uncertainty which is the incoming disturbances penetrating into the inlet domain. To study this influence, we place an external forcing term $\boldsymbol {F}(x,y) \gamma (t)$ following Hervé et al. (Reference Hervé, Sipp, Schmid and Samuelides2012). Here, $\gamma (t)$ is a random scaling factor of standard deviation $\sigma _{\gamma } = 0.1$ and $\boldsymbol {F}(x,y)$ is the spatial structure with a Gaussian shape

(5.3)\begin{equation} \boldsymbol{F}(x,y) = A \exp{\left( \frac{-(x-x_0)^2}{2 \sigma_{F}^2} \right)} \exp{\left( \frac{-(x-x_0)^2}{2 \sigma_{F}^2} \right)} (1,1)^T, \end{equation}

where $\sigma _{F} = 0.1$ and the spatial centre is $x_0=-1.5$ and $y_0=0$. The magnitude of the external force can be modified by changing $A$, and two values ($A=2.0$ and $A=10.0$) are studied. Figure 17 shows the RL-based control of the cylinder wake flow with noisy forces imposed at $t \in [60,80]$. To stabilise the flow, jet flow rates controlled by the RL agent are increased slightly, and the increase magnitude is positively correlated with $A$. Although the noisy spatial forces trigger instabilities in the controlled wake flow, the wake does not develop significant vortex shedding. After the noisy force is removed ($t > 80$), the wake gets fully stabilised again after 20 unit time. This numerical experiment shows that RL-based control is robust to spatial noise from the inflow and it is vital to sustain a persistent RL-based control.

Figure 17.

DMD analysis of the RL-based control process with noisy forces in the inflow domain, which is imposed at $t \in [60,80]$. We test two force magnitudes ($A=2.0$ and $A=10.0$), which are shown as the red and blue lines, respectively. The controlled wake flows are not developed to significant vortex shedding and become stabilised again after the noisy force is removed.

5.5. Stability-enhanced reward

Next, we discuss how the reward in RL can be changed to incorporate some information on flow stability/instability to improve the control performance. Increasing the Reynolds number destabilises the confined wake flows, especially for cylinders with $\beta \le 0.5$. In the RL-based drag reduction control of a confined cylinder wake with $\beta \sim 0.25$, Tang et al. (Reference Tang, Rabault, Kuhnle, Wang and Wang2020) found that small drag oscillations still existed in the controlled flow at $Re=200$ although four synthetic jets were used; and with a further increase of $Re$, although a significant reduction of the averaged drag was achieved, the RL agent could not find a fully stabilised control strategy to completely suppress oscillations in the drag coefficient. This means that the strong instability with the increase of $Re$ in confined cylinder wakes brings significant challenges to RL-based control, and thus full suppression of the vortex shedding is difficult. We increase the Reynolds number to $Re=200$ (note the difference in the definitions of $Re$ in our work ($Re= U_{{max}}D/ \nu$) and their works ($Re= \bar {U} D/ \nu$); $Re=200$ in our work corresponds to $Re=400/3$ in their work) and investigate its influence on vortex shedding suppression with different $\beta$. As shown in figure 18, the increase of $Re$ further pushes the wavemaker domains downstream of the cylinder. We place the probes for RL control to cover the wavemaker region (to be discussed in § 5.6) and train RL agents for vortex suppression at $Re=200$ using the original reward in (5.2). It can be seen that not all the controlled wake flows are fully stabilised owing to the increase of $Re$.

Figure 18.

RL control performance of cylinder wakes with different $\beta$ at $Re=200$. The black region indicates the wavemaker region and the probes are placed to cover it (see the discussion in § 5.6). The stability-enhanced reward reduces unstable instants in the controlled wake flows. From $t=128$, the RL-based control is shut down (shown as the red shade), and the controlled wake flows re-develop to significant vortex shedding, which shows the necessity of a persistent control.

We used DMD ($N_{snapshot}=20$ and ${\rm \Delta} t = 0.2$) to analyse the controlled flows and found that the growth rate may flip to be positive even after the shedding energy is significantly reduced, which leads to flow instabilities and a subsequent increase of shedding energies. This unstable control is not preferable in practice. To obtain an effective control policy, we use a stability-enhanced reward function for RL, which embeds the information of flow stability/instability in the original reward:

(5.4)\begin{equation} \text{Reward} ={-}s_e {e^g}, \end{equation}

where $s_e$ is the shedding energy defined in (5.2) and $g$ is the largest growth rate of the flow, which is evaluated by DMD. For a stable flow, the growth rate is smaller than zero and $e^g < 1.0$, which means a larger reward is given to motivate such control policies. For a neutral flow, the growth rate is zero, and (5.4) is equivalent to using the original reward defined by the shedding energy. For unstable flows, the growth rate is larger than zero. In this circumstance, the stability-enhanced term $e^g > 1.0$, and the reward decreases correspondingly. With such a reward function, the trained RL agent will come up with a control policy that compensates for the decrease of the reward arising from flow instability. This mitigates the adverse effect of flow instability in the control. We train RL agents using the stability-enhanced reward for vortex suppression with different $\beta$ at $Re=200$. As shown in figure 18, using the stability-enhanced reward, the instants when the growth rate is larger than zero become fewer in the controlled wake flows. Thus, instabilities in the controlled flows are significantly damped. Such an instability-abated effect is of great value in practical applications such as prevention of aerodynamic buffeting (Gao et al. Reference Gao, Zhang, Kou, Liu and Ye2017) and aeroelastic flutter (Jonsson et al. Reference Jonsson, Riso, Lupp, Cesnik, Martins and Epureanu2019). In most cases (figure 18), the shedding energy with the stability-enhanced reward is reduced to a lower value that can be maintained for a long run, and, in these circumstances, the amplitude of the control actions is decreased as well. This implies that it is favourable to maintain the controlled cylinder wake flow close to the base flow because it costs lower control energy, which is a beneficial side effect of the proposed stability-enhanced reward. Nevertheless, there is still no guarantee to get a fully stabilised wake flow by using the proposed stability-enhanced reward (see $t \in [100, 128]$ in the case of $\beta =0.6$). We also investigate the destabilising process of the controlled flow with the shut-down of control actions (shown as the red shades in figure 18). The results show that a continuous control is necessary; otherwise, the flow will become unstable.

We provide some discussions on these results in terms of flow physics. In the literature of 2-D cylindrical wake flow, the shift mode has been found to be indispensable in a successful reduced-order model (ROM) of the transient dynamics in this flow (Noack et al. Reference Noack, Afanasiev, Morzyński, Tadmor and Thiele2003). Inclusion of this mode in a ROM realises a mean field correction, pointing from the unstable steady flow to the time-averaged mean flow. Our results on the RL of the confined wake flow can be understood in a similar manner. The controlled flow is the SFD base flow, which is unstable and tends to evolve to the periodic flow once the control is turned off (see figure 15). Thus, when we improve the RL agent by telling it how to abate the instability, the control can be more effective and efficient, as shown in our tests (see figure 18). Admittedly, our stability-enhanced reward is still crude as we simply incorporate the information of the first DMD mode in the new reward. In answering a comment by one of the reviewers, we realise that the analysis and RL control can be conducted more delicately by abating the flow instability in the direction of the unstable base flow drifting to the time-mean flow (currently we do not know how much the abatement effect of using the first DMD mode can contribute in this direction). This may lead to a more efficient RL control. More research is required to explore the new design of reward function in RL in the future.

5.6. A heuristic strategy of probe placement

The reinforcement learning agent observes environment changes via the probes. In preparing this work, we found that Paris et al. (Reference Paris, Beneddine and Dandois2021) studied the probe placement by modifying the architecture of RL-based control by adding a stochastic gated input layer between the states and agent. The modified RL can select an optimal subset from the initial probe sensors. However, a good control performance in this case may be largely dependent on the initial placement of the probes. It may be the case that the initial placement of the probes is not able to encompass the global optimal solution of the control. To achieve an effective RL-based control, probes should be placed in domains that can convey all the essential changes in the flow. In the following, we take a heuristic approach to evaluate the placement strategy of the probes by harnessing the results of the structure sensitivity.

As the most sensitive region to flow perturbations, the wavemaker region may be the ideal monitoring position for the probe placement. Strykowski & Sreenivasan (Reference Strykowski and Sreenivasan1990) showed that the wavemaker is similar to the region where one can place a small control cylinder to suppress the vortex shedding. We hypothesise that the most effective probes should be placed to cover (at least partially) the wavemaker domain in RL-based control of vortex suppression in confined cylinder wakes. We place the probes according to the wavemaker region calculated using the SFD base flow. In the following, three sets of tests have been conducted as shown in figures 19–21, which are explained below.

Figure 19.

Probes covering the wavemaker zone lead to the best control of the confined cylinder wake ($\beta =0.25$ at $Re=150$).

First, in figure 19, we would like to demonstrate that the probes should be placed to cover the wavemaker region. To verify this, we consider three kinds of probes: (1) probes upstream of the wavemaker; (2) probes inside the wavemaker; (3) probes downstream of the wavemaker. As shown in figure 19, placing the probes upstream of the wavemaker significantly reduces control effectiveness (even though they are very close to the cylinder); while placing the probes downstream of the wavemaker cannot train a stable control policy either, even though the RL performance is better than the previous case. Probes placed inside the wavemaker have an effective and stable RL control. Because the wavemaker is obtained by overlapping the direct mode and adjoint mode, we further investigate the control effectiveness with probes being placed to cover the direct mode and adjoint mode, respectively. As shown in figure 19, the RL-based control with probes covering the direct mode is ineffective, while the control with probes covering the adjoint mode is effective and the performance is almost close to those covering the wavemaker. This may arise from the fact that the adjoint mode covers most of the wavemaker domain while the direct mode is located further downstream. Although the direct mode describes the spatial structure of vortex shedding and most residual non-stationarities in the controlled wake flows are located in the region of the direct mode, it is more effective for RL-based control to monitor flow changes in the core domain of instability.

Next, in figure 20, we would like to demonstrate that a small difference in the probe distribution will not affect the general good function of RL as long as the main part of the wavemaker is covered, which will confirm the robustness of its performance. All the control policies also display the similar features found in the last section; that is, large flow rates of the synthetic jets are used in the beginning to modify the mean flow structure and much smaller time-varying jet flow rates are required to maintain the status. In the same figure, we have also added a result of a gradient-based optimisation method in reducing the shedding energy. This consideration is to understand how the RL-based policy compares with other control methods. In the gradient-based optimisation method, the flow rates at different time steps are discretised as independent design variables with a control horizon of 32 time units. The objective function of the control design optimisation is the total shedding energy in the control time-horizon. The gradient of the objective function with respect to the flow rates is solved by using the finite difference method. More details are provided in Appendix C. It can be seen in figure 20 that placing the probes in the wavemaker region achieves a control performance that is close to that solved by the gradient-based optimisation. The control policy found by gradient-based optimisation leads to more reductions of the shedding energy in the short-term ($t\in [5,15]$), because this strategy can achieve the smallest total shedding energy in the chosen control horizon (meaning that the controlled solution based on the gradient-based optimisation method is optimal in the sense that the total energy with $t\in [0,32]$ is minimum). Nevertheless, it cannot be guaranteed that other solutions may exist that have locally smaller shedding energies than the policy found by the gradient-based optimisation. For example, our RL-based controls have slightly better long-term performances (for instance, look at the results at $t=30$ in the figure), which can be more preferable in practice.

Figure 20.

Even with slight differences in the placement, probes covering the wavemaker zone can always lead to effective and stable RL control of the confined cylinder wake ($\beta =0.25$ at $Re=150$). The optimal control policy solved by the gradient-based optimisation achieves better short-term performance and RL agents lead to a better performance in the long term.

The wavemaker domain of the confined cylinder wake changes with the blockage ratio or the Reynolds number. We investigate the influence of different blockage ratios on RL-based control performance at $Re = 150$. As shown in figure 22 of Appendix D, increasing the blockage ratio pushes the wavemaker to the downstream region, as we have discussed above. We place probes for RL control of confined wakes with different blockage ratios at the corresponding wavemaker regions. With the RL-based control, the shedding energies are all reduced to a small value. This means that regardless of the blockage ratio, as long as the probes are placed to cover the wavemaker region, the RL-control policy is able to effectively control the confined flow to damp the vortex shedding (at least for the Reynolds number investigated, $Re = 150$). In practice, if the probes are fixed in a series of control tests with different parameters (such as $\beta$ or $Re$), the optimal placement of the probes should be determined to be suitable for all the parameters (the trend of how the flow properties change with these flow parameters has been analysed in the previous sections).

Finally, we consider the cases where the number of probes is insufficient. We choose to train the RL agent using ten probes. The probe distributions and control performances are shown in figure 21. Distributions $1\sim 3$ are evenly placed in the wavemaker region, and Distributions $4\sim 5$ are placed in the most sensitive part (black). We can see that even though all the five placements successfully reduce the shedding energy, difference exists in their performance. The best performance belongs to Distribution 1, where the shedding energy has been abated to almost zero when $t>20$. This result indicates that Distribution 1 should be close to the global optimal solution in the RL (in which the shedding energy is zero). However, Distributions 2,3, even though they are also trying to cover the wavemaker region as fully as possible, somehow perform poorly owing to the insufficient number of probes. Note that, compared with figure 20, the results of Distributions 2,3 should be considered to be worse. Finally, when clustering all the available probes in the most sensitive region (black colour) as in Distributions 4,5, the RL control does not necessarily yield good performance. All these results indicate that when insufficient probes are used, the performance of the RL control is scattered depending on the placement strategy of the probes. Our heuristic approach indicate that Distribution 1 is the best choice; however, this fortuitous result may not be carried over to other situations. Based on this result, future works can consider coupling the proposed criterion of probe placement (to cover the wavemaker region) with an optimisation method with the former providing a general good choice of the initial placement of the probes and the latter fine-tuning the selection of the probes by the optimisation of the probes, exemplifying the combination of the a priori knowledge on flow physics and the power of RL algorithm. For example, the modified RL developed by Paris et al. (Reference Paris, Beneddine and Dandois2021) can be used to select the optimal subset from the probes which are initially covering or not covering the wavemaker region (and we hypothesise that the probes covering the wavemaker region will be selected when vortex shedding is to be abated).

Figure 21.

RL control of the confined cylinder wake ($\beta =0.25$ at $Re=150$) using ten probes. Different distributions of probes lead to a significant divergence in the control performance.