4.1. Dynamics of the 1-D linearised KS equation

As introduced in § 2, the 1-D linearised KS equation is an idealised equation for modelling the perturbation evolving in a Blasius boundary layer flow, with features such as non-normality, convective instability and a large time delay. Without control, the spatiotemporal dynamics of the KS equation subjected to upstream Gaussian white noise is presented in figure 6(a), where the perturbation is amplified significantly while travelling downstream, and the pattern of parallel slashes demonstrates the presence of a time delay. Figure 6(b) presents the temporal signal of noise with unit variance, and figure 6(c) displays the output signal at $x=700$.

Figure 6.

Dynamics of the 1-D linearised KS equation when subject to Gaussian white noise with unit variance: (a) spatiotemporal response of the system; (b) temporal signal of noise input $d(t)$ at $x = 35$; (c) output signal $z(t)$ measured by sensor at $x = 700$. The blue arrow represents input, and the red arrow represents output.

Due to the existence of linear instability in the flow, the amplitude of perturbation grows exponentially along the $x$-direction, which is verified by the black curve in figure 7(b), where the r.m.s. value of perturbation calculated by

(4.1)\begin{equation} v^{\prime}(x)|_{rms}=\sqrt{\left(\overline{v^{\prime}(x)^{2}}-\overline{v^{\prime}(x)}^{2}\right)} \end{equation}

is plotted along the 1-D domain, and here $\overline {v^{\prime }(x)}$ represents the time average of the perturbation velocity at a particular position $x$. In addition, by comparing figures 6(b) and 6(c), we find that the perturbation amplitude increases but some frequencies are filtered by the system. This is because when a white noise is introduced, a broadband of frequencies are excited. However, only frequencies corresponding to a certain range of wavelengths are unstable and amplified in the flow. This is the main trait of a convectively unstable flow, plus the nonlinearity of the flow system, rendering it difficult for flow control. Such a flow is usually coined as a noise amplifier in the hydrodynamic stability context (Huerre & Monkewitz Reference Huerre and Monkewitz1990).

Figure 7.

DRL-based control for reducing the downstream perturbation evolving in the 1-D linearised KS equation. (a) Average perturbation amplitude at point $z$ recorded during the training process, where the red dashed line denotes the time when the experience memory is full. (b) The r.m.s. value of perturbation along the 1-D domain plotted for both cases, with and without control.

4.2. DRL-based control of the KS system

In this subsection, we present the performance of DRL-based control on reducing the downstream perturbation evolving in the KS system. The training process is shown in figure 7(a), where the average perturbation amplitude at point $z$ ($x = 700$; see figure 2) is plotted for all the training episodes. In the initial stage of training, the perturbation amplitude is large and shows no sign of decreasing, which corresponds to the filling process of experience memory in the DDPG algorithm. Once the memory is full (denoted by the red dashed line in figure 7a), the learning process begins and the amplitude decreases significantly to a value near zero, and remains almost unchanged after 350 episodes, indicating that the policy network has converged and the optimal control policy has been learnt. Note that this explains the periodical training of 350 episodes when we implement the PSO algorithm, as mentioned in § 3.4. Then we test the learnt policy by applying it to the KS environment for 10 000 time steps, with both the external disturbance input $d(t)$ and control input $u(t)$ turned on.

The spatiotemporal response of the controlled KS system is shown in figure 8(a), subjected to the noise input as shown in figure 8(b). Compared with that in figure 6(a), the downstream perturbation is significantly suppressed after the implementation of control action at $x = 400$ starting from $t = 1450$ (denoted by the blue dashed line). By observing figures 8(c) and 8(d), we find that there is a short time delay between the control starting time and the time when the downstream perturbation $z(t)$ starts to decline. This is due to the convective nature that the action needs some time to make an impact downstream. In addition, we also plot the r.m.s. value of perturbation along the 1-D domain, as shown by the blue curve in figure 7(b). In contrast to the originally exponential growth, with the DRL-based control, the perturbation amplitude is largely reduced downstream of the control position ($x = 400$). For instance, the amplitude at $x = 700$ is reduced from about 20 to 0.03, which demonstrates the effectiveness of the DRL-based method in controlling the perturbation evolving in the 1-D KS system.

Figure 8.

Dynamics of the 1-D linearised KS equation with both the external disturbance and control inputs: (a) spatiotemporal response of the system; (b) temporal signal of noise input $d(t)$ at $x = 35$; (c) temporal signal of control input $u(t)$ at $x = 400$ provided by the DRL agent; (d) output signal $z(t)$ measured by sensor at $x = 700$. The blue arrows represent inputs, and the red arrow represents output; the blue dashed line denotes the control starting time.

4.3. Comparison between DRL and model-based LQR

In order to better evaluate the performance of DRL applied to the 1-D KS equation, we compare the DRL-controlled results with those of the classical linear quadratic regulator (LQR) method. The LQR method for controlling the KS equation has been explained in detail in Fabbiane et al. (Reference Fabbiane, Semeraro, Bagheri and Henningson2014). For the basics of LQR, the reader may consult Appendix D.

Ideally, if there are no action bounds imposed on LQR, then it will generate the optimal control performance as shown in the red curves in figure 9(a), and we call it the ideal LQR control. It is shown that the perturbation amplitude downstream of the actuator position is reduced dramatically, and the corresponding action happens to be in the range $[-5, 5]$. Thus for a consistent comparison of LQR and DRL, we apply the same action bound $[-5, 5]$ to DRL-based control, and present the resulting control performance as blue curves. It is found that the DRL-based control is slightly better than the LQR control in reducing the downstream perturbation, as shown in the right-hand plot of figure 9(a). Despite the fact that in the near downstream region from $x = 400$ to $x = 470$, the amplitude of DRL-based control is higher than that of LQR control, the reduction of perturbation is more significant in the DRL control in a further downstream region.

Figure 9.

Comparison between DRL-based control and LQR control when applied to the 1-D linearised KS system with different action bounds. In each panel, the top left plot presents the time-variation curve of the control action $u(t)$, and the bottom left plot presents the corresponding sensor output $z(t)$; the right-hand plot shows the r.m.s. value of perturbation velocity along the 1-D domain. (a) Control performance of ideal LQR and of DRL with action bound ${[-5,5]}$. (b) Control performance of LQR and DRL both with action bound ${[-3,3]}$. (c) Control performance of LQR and DRL both with action bound ${[-2,2]}$. (d) Control performance of LQR and DRL both with action bound ${[-1,1]}$.

In real control applications, one needs to consider the saturation effect of the actuators, and this is realised by imposing the action bound (Grundmann & Tropea Reference Grundmann and Tropea2008; Corke, Enloe & Wilkinson Reference Corke, Enloe and Wilkinson2010). Here, we adopt the method used by Fabbiane et al. (Reference Fabbiane, Semeraro, Bagheri and Henningson2014) to bound the LQR control action as

(4.2)\begin{equation} u_{L Q R}= \begin{cases}u_{L Q R} & \text{if } \bar{u}_{min }< u_{L Q R}<\bar{u}_{max }, \\ \bar{u}_{min } & \text{if } \bar{u}_{min } \geqslant u_{L Q R}, \\ \bar{u}_{max } & \text{if } \bar{u}_{max } \leqslant u_{L Q R}. \end{cases} \end{equation}

We first impose the same action bound $[-3, 3]$ on both LQR and DRL-based control, and compare their control performance in figure 9(b). When the saturation function is applied to the LQR control signal, the controller becomes sub-optimal, and its performance to reduce the perturbation becomes worse. It is observed that at about $t = 17\,000$, the action is saturated, thus there appears a small spike later in the output signal $z(t)$, as shown in the bottom left plot of figure 9(b). On the other hand, the DRL-based control also performs slightly worse with the smaller action range, but it still outperforms LQR control with the same action bound. Next, we further narrow down the action range to $[-2, 2]$, and this time both LQR and DRL-based control deteriorate due to the saturation of action, and their control performances are at a similar level, as shown in figure 9(c). Finally, we further limit the action range to $[-1, 1]$ and compare the corresponding control performances of the two methods in figure 9(d). In this condition, the LQR controller deteriorates severely, and the reduction of downstream perturbation is rather limited. In contrast, DRL-based control performs better, with a relatively clear trend of reduction of perturbation downstream of the actuator position, although the extent of reduction is not significant compared to the previous cases with larger action bounds, as shown in figures 9(a–c). We would like to emphasise that compared to the model-based LQR method, the DRL-based method is totally model-free, which means that the control law is learnt with no awareness of the assumed models of the plant, as mentioned in § 3.1. In addition, the control action is determined based solely on states collected by some sensors from the flow field, instead of the full knowledge of the plant as required by LQR control.

The current LQR controller is based on a specific selection of weights, i.e. $w_z = w_u = 1$ as given in (D2). This selection was adopted by Fabbiane et al. (Reference Fabbiane, Semeraro, Bagheri and Henningson2014), indicating that equal weight has been assigned to the two parts, i.e. sensor output $z(t)$ and control input $u(t)$. These parameters influence the control performance of LQR. For example, the LQR performance can be improved by increasing the weight $w_z$ from 1 to 10 and 100 with a fixed $w_u$ at 1 (which means that we focus more on the output part). The results are shown in figure 10(a). However, the ensuing problem is that with the increase of $w_z$, the magnitude of control action will experience a rapid increase at the very initial stage of the control process, as shown in figure 10(b), which is unavoidable because larger weight is considered to force the output to the target value as soon as possible, and this is realised by a larger action in the beginning. In realistic applications, the choice of weights is a trade-off between the control performance and the available range of control action.

Figure 10.

Comparison of LQR control performance with different values of weight $w_z$. (a) The r.m.s. value of perturbation velocity along the 1-D domain. (b) The time-variation curves of control action $u(t)$, with the control starting at $t = 1450$.

In addition to the control performance, the cost of energy effort is another important factor in active flow control. The cost of energy effort in our context can be quantified by the average magnitude of action during the control process. We compare this aspect for the two methods in table 1, which shows that the average magnitudes of action for the two methods are in general comparable to each other, except that in the case of the smallest action bound $[-1,1]$, the averaged action of DRL is about 20 % less than that of LQR.

Table 1.

Average magnitude of action for different methods with different action bounds.

4.4. Robustness of the DRL policy in controlling convectively unstable flows

In this section, we test the robustness of the learnt control policy to two types of noise, i.e. measurement noise and external noise. In realistic conditions, the states collected by sensors are always subjected to noise, and we call it measurement noise. We consider this effect by adding Gaussian noise with standard deviations $\sigma _N=0.01$, 0.05 and 0.1, respectively, to the original states, and test the robustness of the learnt policy, which was trained with $\sigma _N=0$ to the measurement noise of different levels. As shown in figures 11(a–c), with the increase of noise level, the overall result is still satisfactory except that the DRL-based control performance deteriorates in the sense that there are some small oscillations being monitored in the sensor output $z(t)$. We plot the corresponding r.m.s. value of perturbation along the 1-D domain in figure 11(d), where the control performance with uncontaminated states, i.e. $\sigma _N = 0$, is also shown by the black curve, for comparison. With the noise level increasing to 0.1 (accounting for about 10 % of the originally normalised state signals), the controlled amplitude of $z(t)$ at $x = 700$ increases from 0.03 to 1.0. Despite the slight deterioration of the control performance with the increased noise level, the current policy is still effective in reducing the downstream perturbation (recall that the uncontrolled amplitude at $x=700$ is 20). We have also attempted to train a DRL agent with $\sigma _N = 0.1$ and test it in the same condition. The resulting control performance is depicted by the orange dashed curve in figure 11(d), and it can be seen that the result is close to the orange solid curve, which was obtained from the agent trained with $\sigma _N = 0$ but tested with $\sigma _N = 0.1$. Therefore, the DRL agent is robust in the sense that once trained, it can effectively control a new flow situation with some additional noise. This robustness property benefits from the closed-loop nature of the control system and the decorrelation of noise among state observations, as explained by Paris et al. (Reference Paris, Beneddine and Dandois2021). More specifically, the action error caused by measurement noise does not accumulate over time since the feedback state is able to rectify the previous erroneous action and prevent it from deviating too far.

Figure 11.

DRL-based control performance with states contaminated by Gaussian noise of three different standard deviations $\sigma _N$. (a–c) The sensor output $z(t)$ plotted for the three cases. (d) The r.m.s. value of perturbation along the 1-D domain plotted for the three cases. Also, the black curve for control with uncontaminated states, and the orange dashed curve for control with the agent trained with $\sigma _N = 0.1$, are shown for comparison.

The second type of noise is the external noise. The current policy was trained under a fixed external noise condition, i.e. white noise input at $x_d = 35$ with a unit standard deviation $\sigma _{d(t)} = 1.0$. We want to test whether the current policy is still valid in new external noise situations. For instance, we can move the noise position from $x_d = 35$ to $x_d = 75$ with the noise level unchanged, or increase the noise level from $\sigma _{d(t)} = 1.0$ to $\sigma _{d(t)} = 1.5$ with the noise position unchanged, or change both of them. We test the control policy in such new environments, and plot the obtained control performance in figure 12. Due to the increase of noise level and/or the shift of noise position, the perturbation fields upstream of the control position are only trivially different from each other, as shown in figure 12(d). Once the DRL-based control is applied, the downstream perturbation is reduced significantly, as demonstrated in figure 12. This robustness property benefits from the state normalisation process as mentioned in § 3.3, which has also been demonstrated in Paris et al. (Reference Paris, Beneddine and Dandois2021). Among the three testing cases, the second one ($x_d = 35$ and $\sigma _{d(t)} = 1.5$), depicted by the green curve in figure 12, leads to a relatively large downstream perturbation. This is due to the fact that the uncontrolled perturbation level increases (depicted by the green dashed curve in figure 12d) and sometimes is out of the current action range $[-5, 5]$, which will lead to the small spikes as shown in figure 12(b).

Figure 12.

DRL-based control performance under three different upstream noise conditions: Gaussian noise input at $x_d$ with standard deviation $\sigma _{d(t)}$. (a–c) The sensor output $z(t)$ plotted for the three cases. (d) The r.m.s. value of perturbation along the 1-D domain plotted for the three cases, and that for control testing with the same environment as training, i.e. $x_d = 35$ and $\sigma _{d(t)} = 1.0$, is also given by the black curve, for comparison.

Finally, we would like to mention that in the confined cylinder wake flow as studied by Rabault et al. (Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019), Paris et al. (Reference Paris, Beneddine and Dandois2021) and Li & Zhang (Reference Li and Zhang2022), the nature of flow instability is locally absolute; cf. Monkewitz & Nguyen (Reference Monkewitz and Nguyen1987) and Giannetti & Luchini (Reference Giannetti and Luchini2007) for the general theory of flow instability in unconfined wake flows, or see the flow stability and sensitivity analyses in Li & Zhang (Reference Li and Zhang2022) for the confined wake flow. In such a flow, the noise level is only secondary compared to the primary absolute instability mechanism accounting for the perturbation amplification; cf. Huerre & Monkewitz (Reference Huerre and Monkewitz1990) for a classical review of the absolute and convective instabilities. On the other hand, in the case of the KS equation or boundary layer flows, the nature of instability is convective, for which the perturbation amplification depends more critically on the upstream noise. Our numerical demonstration of the DRL control in the convectively unstable flow is a stricter test of its robustness and proves its applicability in such flows.

4.5. Optimisation of the sensor placement

All the above results are obtained based on the optimal sensor placement. In this subsection, we explain how the optimal sensor placement has been identified. As mentioned in § 3.4, we adopt the PSO method to find the optimal sensor placement. Before searching for the optimal sensor positions, we need to specify lower and upper bounds for the searching space. In the current work, we investigate different cases with 1, 2, 4, 6, 8 and 10 sensors located in $x\in (370,430)$. This range is determined through our preliminary tests. We have attempted to add more sensors outside the current range, and found that they had unimportant effects on the final control performance.

The optimisation processes for all the considered cases are presented in figure 13(a), where the trends of the curves are similar. Here, we take the case with two sensors as an example to illustrate this process. The initial placement of the sensors is random. After the first iteration, the algorithm returns an initial sensor placement (denoted by the green point) that all the particles in the swarm have ever found, which corresponds to the lowest objective function $r_b$ so far, as defined in § 3.4. Then, based on the initial placement, the particles attempt to find a new placement, which leads to a lower $r_b$ in the next iteration. Finally, after a number of iterations, the searching process converges, and the optimal placement is identified, denoted by the blue point. To demonstrate the effectiveness of the optimisation, we compare the control performance given by the initial sensor placement and the optimised one. As shown in figure 13(b), the optimised placement indeed results in a better control performance, i.e. a lower perturbation amplitude at the downstream position $x_z = 700$ than that of the unoptimised one.

Figure 13.

Optimal sensor placement in DRL-based flow control. (a) The optimisation process using PSO presented for cases with different numbers of sensors; the optimal placement is found, which leads to the lowest perturbation amplitude $r_b$. (b) Control performance comparison between the initial placement and the optimised one (taking 2 sensors as an example).

The optimised sensor layouts with different numbers of sensors are presented in figure 14(a), where dots represent the optimised sensor placements (OSPs), named as OSP plus the number of sensors, and the actuator position is also given as a reference. As suggested by one of the reviewers, we will test whether using sensors only upstream of the actuator is sufficient for control. To do so, we consider three newly added sensor placements, as shown by crosses in figure 14(a), among which: SP1 is a four-sensor layout with the downstream sensors removed from OSP8; SP2 is a five-sensor layout with the downstream sensors removed from OSP10; and SP3 is the uniformly spaced four-sensor layout upstream of the actuator that is scaled by the parallel slash pattern as shown in figure 6(a).

Figure 14.

Optimal sensor placement and the effect of the number of sensors. (a) The optimal placement is presented for cases with different numbers of sensors. In addition, three specific layouts with all sensors placed upstream are also considered. (b) Comparison of the corresponding control performance with different sensor placements.

The corresponding control performance is presented in figure 14(b), where the r.m.s. value of perturbation along the 1-D domain is plotted. It is shown that if only one sensor is used, it is better to place it upstream of the actuator rather than downstream, to help the agent detect the upcoming disturbance. Nevertheless, due to the implementation of sticky actions where the control action keeps constant every 30 time steps, a single upstream sensor can inform the agent of only a single upcoming parallel slash in a packet of slashes that crosses the actuator over the duration that the action is ‘stuck’. As the slashes are generated by random noise, it is not possible for the agent to choose an action that accounts for the packet of slashes based on a single measurement. With more sensors placed upstream, it is more likely that the agent will be fully informed to select the best sticky action to suppress the packet. This hypothesis is verified by the fact that the performance of the three layouts SP1, SP2 and SP3 (crosses in figure 14a) is better than that of OSP1 and OSP2 (one upstream and one downstream). However, the four-sensor layout with all sensors placed upstream is inferior to OSP4 (two upstream and two downstream), which indicates that the downstream sensors are also necessary for a better control performance. This is determined by the inherent feature of the DRL framework, where the agent interacts with the environment in a closed loop. After the agent imposes an action on the environment, the environment needs to return states as feedback that determines the next action. Overall, with the number of sensors increasing from 1 to 8, the resulting control performance is improved continuously, as expected since more sensors will provide a more complete measurement of the flow environment. However, in the current case, as the number of sensors is increased further, from 8 to 10, no further improvement is observed, which indicates that the information given by the two additional sensors is redundant. That is the reason why we adopt eight sensors in § 3.3, and the placement of them is depicted by the purple points in figure 14(a).

Some discussions on the role of sensors in DRL control are in order. In the model-free DRL framework, sensors are used to collect states that not only provide a measurement of the flow environment but also reflect the effect of the action after it is taken. In this context, a proper number of sensors should be placed both upstream and downstream of the actuator. In contrast, in traditional model-based control methods such as the linear quadratic Gaussian regulator, the sensor is used to estimate the full states of the plant via the Kalman filter, and then the control action is calculated based on the estimated states. In this situation, one sensor can usually work well since this demonstration does not implement sticky actions and is able to observe each slash individually that passes through, and actuate accordingly. For instance, Belson et al. (Reference Belson, Semeraro, Rowley and Henningson2013) investigated the effects of different types and positions of sensor and actuator on the model-based control of boundary layer flows, and found a specific sensor–actuator pair with good control performance and robustness.

4.6. DRL-based control performance in 2-D boundary layer flows

We have demonstrated that the DRL-based method is effective in controlling the convectively unstable perturbation evolving in the 1-D KS system. Since the KS equation is a reduced model for the dynamics of the streamwise perturbation velocity evolving in the boundary layer flow at the wall-normal position $y=1$, it is reasonable to expect that the optimised sensor placement that we have derived in the KS system can be translated to the control of 2-D Blasius boundary layer flows. We note that the nonlinear NS equations are the governing equations in this case.

Here, we adopt the same sensor placements with different numbers of sensors as shown in figure 14(a) in the DRL training for controlling boundary layer flows. A typical training process is shown in figure 15(a), where the red dashed line again denotes the time when the experience memory is full, and then the absolute value of the average reward per episode, i.e. the local perturbation energy around point $z$ as defined in (2.12), generally decreases until convergence after about 50 episodes. Note that the reward is defined as the negative local perturbation energy; nevertheless, the global perturbation level in the whole flow field decreases. As shown in figure 15(b), we test the learnt control policy by applying it to the boundary layer flow for 6000 action steps that are 10 times longer than the training episode, with both the disturbance input $d(t)$ and control input $u(t)$ activated. It is found that the global perturbation energy downstream of the actuator is significantly reduced and maintained at a level close to zero throughout the test process. Here, the global downstream perturbation energy $e(t)$ is calculated as

(4.3)\begin{equation} e(t)=\sum_{i = 1}^{M}\left[\left(u^{i}(t)-U^{i}\right)^{2}+\left(v^{i}(t)-V^{i}\right)^{2}\right], \end{equation}

where $M$ is the number of uniformly distributed, customised grid points in the downstream domain $(410, 800) \times (0, 30)$, and here $M=2052$; $u^i(t)$ and $v^i(t)$ are the instantaneous $x$- and $y$-direction velocity components at point $i$, and $U^i$ and $V^i$ are the corresponding base flow velocities.

Figure 15.

DRL-based control for the 2-D Blasius boundary layer flow, subjected to an upstream disturbance input $d(t)$. (a) Average local perturbation energy around point $z$ recorded during the training process. (b) Global perturbation energy downstream of the actuator recorded during the test process. (c) The r.m.s. value of streamwise velocity along $y=1$ plotted for DRL-based control with different numbers of sensors. (d) The r.m.s. value of streamwise velocity along $y=1$ plotted for DRL-based control with disturbance input of different amplitudes $\sigma _{d(t)}$. Curves for cases without control are also shown for reference.

We then extract the r.m.s. value of the streamwise velocity along $y=1$ and plot it in figure 15(c), where the black curve is for the case without control, and other coloured curves represent DRL-based control with different numbers of sensors. Similar to that in the KS system (cf. figure 14b), when no control is implemented, the random noise introduced upstream exhibits exponential growth while convecting downstream. Such behaviour of linear instability is observed when the external disturbance $d(t)$ is small enough, with standard deviation $2 \times 10^{-4}$ used here. With the control activated, the perturbation amplitude downstream of the actuator is reduced significantly. Also, the effect of the number of sensors on the control performance is similar to that in the KS system. With the number of sensors increasing from 1 to 8, the resulting control performance is continuously improved, while with a further increase to 10, the control performance slightly degrades, similar to the situation for the KS equation.

Next, the amplitude of the input disturbance $d(t)$ is increased to investigate the effect of nonlinearity on the DRL-based control performance. Each disturbance level represents an agent trained in that specific system, which means that the training conditions are the same as the test conditions. With the standard deviation $\sigma _{d(t)}$ increasing from $2 \times 10^{-4}$ to $5 \times 10^{-4}$ and $1 \times 10^{-3}$, the nonlinear behaviour gradually starts to play a role in the dynamics, as shown by the solid curves in figure 15(d), where the linear exponential growth is lost in the downstream region. The nonlinearity in turn slightly degrades the DRL-based control performance, as shown by the dashed curves in figure 15(d). The comparison of flow contours of streamwise perturbation velocity at $y = 1$ with and without control for the three input disturbance levels is shown in figures 16–18, respectively. Figures 16(a), 17(a) and 18(a) represent the flow field triggered solely by an upstream disturbance input $d(t)$, and figures 16(b), 17(b) and 18(b) display that with both $d(t)$ and DRL-based control $u(t)$ being activated. It is shown that in all three cases, the downstream perturbation is suppressed to some extent with DRL control, but the degree of effectiveness differs. With the increase of disturbance amplitude, the required magnitude of control action $u(t)$ increases (still within the predefined range $[-0.01, 0.01]$) but the resultant control performance deteriorates; compare contours in figures 16(b), 17(b) and 18(b), and also compare the recorded output signal $z(t)$, i.e. the localised perturbation energy around $(550,1)$, in figures 16(c), 17(c) and 18(d).

Figure 16.

(a) Contour of streamwise perturbation velocity at $y=1$ induced by an upstream disturbance $d(t)$ with standard deviation $\delta _{d(t)} = 0.0002$. (b) Contour of streamwise perturbation velocity with both disturbance input $d(t)$ and control input $u(t)$. (c) Output signal $z(t)$ comparison between the uncontrolled and controlled cases, i.e. the localised perturbation energy monitored around (550,1). Blue arrows represent inputs, and the red arrows represent outputs.

Figure 17.

(a) Contour of streamwise perturbation velocity at $y=1$ induced by an upstream disturbance $d(t)$ with standard deviation $\delta _{d(t)} = 0.0005$. (b) Contour of streamwise perturbation velocity with both disturbance input $d(t)$ and control input $u(t)$. (c) Output signal $z(t)$ comparison between the uncontrolled and controlled cases, i.e. the localised perturbation energy monitored around $(550,1)$. Blue arrows represent inputs, and the red arrows represent outputs.

Figure 18.

(a) Contour of streamwise perturbation velocity at $y=1$ induced by an upstream disturbance $d(t)$ with standard deviation $\delta _{d(t)} = 0.001$. (b) Contour of streamwise perturbation velocity controlled by $u(t)$ with the original forcing spatial distribution. (c) Contour of streamwise perturbation velocity controlled by $u(t)$ with the new forcing spatial distribution. (d) Output signal $z(t)$ comparison among the uncontrolled and controlled cases, i.e. the localised perturbation energy monitored around $(550,1)$. (e) Control performance comparison in terms of the r.m.s. value of streamwise velocity at $y=1$. The blue arrows represent inputs, and the red arrows represent outputs.

The performance degradation can be understood as follows. First, the optimised sensor placement applied here is obtained from the linearised KS system, which may be sub-optimal in the nonlinear conditions. Second, with the increase of disturbance level, such a control method via a localised volumetric forcing may not be effective due to its confined region of influence compared to the extended region influenced by the increasing disturbance input. This argument is verified by performing an additional DRL training for a spatially enlarged localised forcing and then comparing its control performance with the original one. The original control forcing is localised around point $(400,1)$ with 2-D Gaussian supports with $\delta _x = 4$ and $\delta _y = 1/4$, and the new forcing is also localised around $(400,1)$ but with a slightly enlarged spatial region of influence with $\delta _x = 5$ and $\delta _y = 1/3$. All the other training parameters are the same as for the original one, and the resultant control performance is shown in figure 18(c) and the red dotted curves in figure 18(d,e). It is demonstrated that the control performance can be improved by slightly enlarging the spatial region of the localised forcing, which can be seen by comparing the downstream contour in figures 18(b,c), where the perturbation is better suppressed with the new forcing distribution, and from the recorded output signal $z(t)$ in figure 18(d), and also from the r.m.s. value of streamwise velocity along $y=1$ plotted in figure 18(e).

All the results shown in figures 15–18 are related to the perturbation reduction along a single straight line. Next, we plot the time-averaged perturbation energy field downstream of the actuator with different amplitudes of the disturbance input in figures 19(a–c). For clarity of comparison, all the raw data of the perturbation energy field are post-processed by first normalisation using its maximum value and then taking the logarithm. Thus the right bound of the colour code shown in figure 19 is zero, which corresponds to the region having the maximum perturbation energy among the considered cases. As shown in figure 19(a), when no control is implemented, the perturbation velocity keeps increasing as convected downstream, and develops from the boundary layer region into the outer region. In contrast, when DRL-based control is activated, the downstream perturbation is suppressed significantly. The effect of the number of sensors is also illustrated here. For DRL control with only one sensor, some small perturbations can still be observed along $y=1$, while for DRL control with two sensors, some small perturbations appear at the outlet. For DRL control with 8 sensors, the downstream perturbations are almost completely suppressed.

Figure 19.

Contours of the time-averaged perturbation energy field downstream of the actuator with different amplitudes $\sigma _{d(t)}$ of disturbance input. In each panel, the comparison of perturbation energy field with and without control is presented. The colour bar displays the logarithm of the normalised perturbation energy. (a) Disturbance level $\sigma _{d(t)} = 0.0002$. (b) Disturbance level $\sigma _{d(t)} = 0.0005$. (c) Disturbance level $\sigma _{d(t)} = 0.001$.

Moreover, the effect of the amplitude of disturbance input is illustrated by the comparison among the first rows of figures 19(a–c), using the same eight-sensor placement for consistency. With the increase of disturbance level, the region with high perturbations is obviously enlarged, thus the DRL-based control performance deteriorates as explained before. A quantitative comparison is summarised in table 2, where the time-averaged global perturbation energy $E$, (i.e. $E = \overline {e(t)}$, where $e(t)$ is defined in (4.3)) downstream of the actuator is reported and compared. It is found that when the disturbance level is low, DRL-based control is remarkably efficient, with an over 96 % reduction of the downstream perturbation energy. With the disturbance level increasing to 0.001, DRL-based control is not that efficient but still effective, with an 89.68 % reduction of the downstream perturbation energy.

Table 2.

DRL-based control performance quantified by the reduction of perturbation energy.

4.7. Interpretation of the learnt control policy

In this subsection, we make an attempt to understand the DRL-based flow control from a physical point of view. We stress that the convectively unstable flow system under control is a selective frequency amplifier, subject to upstream random noise. To make this point clear, we perform the stability analysis of the 2-D Blasius flow and discuss its relevance to the control.

We solve numerically the Orr–Sommerfeld equation with the solution form $\psi (x, y, t)=\varphi (y) \exp ({\mathrm {i}(\alpha x-\omega t)})$, where $\varphi (y)$ is the complex amplitude, $\alpha$ is the wavenumber, $\omega = \omega _r + \mathrm {i}\omega _i$, with $\omega _r$ being the angular frequency and $\omega _i$ the exponential growth rate, and $c = \omega /\alpha = c_r + \mathrm {i}c_i$, where $c_r$ is the phase velocity of the wave, and the sign of $c_i$ determines whether the wave is stable or unstable. The limiting case $c_i = 0$ yields the neutral stability curves as shown in figure 20(a) for $Re\unicode{x2013} \omega _r$, non-dimensionalised by the free-stream velocity $U_\infty$ and the local displacement thickness $\delta ^*$ of the boundary layer. It is shown that the critical $Re$ is 519.4, and for $Re > 519.4$, a narrow range of frequencies is amplified. For our control case whose inlet has $Re = 1000$, the local displacement thickness of the boundary layer at the location of control $x = 400$ is about $\delta ^* = 1.474$, thus the local value is $Re_c = 1474$. Moreover, sensors used to collect state are also positioned around this location. Thus we can extract the corresponding stability curve at this section, denoted by the black dashed line in figure 20(a), and plot it in an $\omega _r\unicode{x2013} c_i$ plane in figure 20(b). It can be seen from figure 20(b) that the unstable frequency range at this section is about $(0.027, 0.079)$, denoted by two blue points. To make the control effective, this frequency range should be well resolved by state observations in DRL. In the current case, we adopt a time step of 0.05 unit times for temporal integration in 2-D simulations. The action is constant for 30 time steps, and there is only one observation collected by eight sensors at the end of each of the 30 time steps that is returned as state. Thus the control frequency is the same as the state sampling frequency, namely, $2{\rm \pi} /(30\times 0.05) = 4.189$, which is much larger than the aforementioned amplified frequency, thus satisfying Nyquist criteria and avoiding the aliasing effect.

Figure 20.

Stability curves for Blasius flow by solving the Orr–Sommerfeld equation. (a) Neutral curve: $Re$ versus $\omega _r$. (b) Curve for $Re_c = 1474$: $\omega _r$ versus $c_i$.

Moreover, we make an attempt to understand the learnt control policy through an analysis of the macroscopic flow features induced by the control action, since it is difficult to interpret the policy from the weights of neural networks due to the black box feature of ML algorithms (Schmidhuber Reference Schmidhuber2015; Rabault et al. Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019). We plot the instantaneous snapshots of streamwise perturbation velocity at three different time instants $t_1$, $t_2$ and $t_3$ in figures 21(a–c), respectively, corresponding to the three spikes labelled in figure 15(b). In each panel, the top plot corresponds to the black curve in figure 15(b), which represents the situation with only the upstream noise input at $x_d = 35$, while the bottom plot corresponds to the blue curve in figure 15(b), which represents the situation with both upstream noise input at $x_d = 35$ and DRL-based control implemented at $x_u = 400$. During the control process, we record the action sequence produced by the agent and then perform a new simulation with only the recorded control actions input, and obtain the resultant field as shown by the middle plot. It is observed in all the three panels that the specific wave induced by the control action (middle plot) is of almost the same magnitude but with phase opposite to that generated by the disturbance input (top plot), which imposes a destructive inference to the incoming TS wave, and thus the downstream perturbations are well suppressed, as shown in the bottom plot of each panel. This is exactly the opposition control mechanism as has been discussed in e.g. Choi, Moin & Kim (Reference Choi, Moin and Kim1994), Grundmann & Tropea (Reference Grundmann and Tropea2008), Brennan, Gajjar & Hewitt (Reference Brennan, Gajjar and Hewitt2021), Sonoda et al. (Reference Sonoda, Liu, Itoh and Hasegawa2022) and Güemes et al. (Reference Güemes, Martín, Castellanos, Oscar and Discetti2022). To summarise, DRL control is effective in controlling the convective instability evolving in boundary layer flows, using the optimised sensor placement obtained from the KS system. The learnt control policy based on the ‘sticky action choice’ acts as opposition control to cancel the incoming perturbation TS wave.

Figure 21.

Instantaneous snapshots of streamwise perturbation velocity at three different time instants $t_1$, $t_2$ and $t_3$. In each panel: the top plot represents the velocity field with only the upstream disturbance input; the middle plot represents the velocity field with only the corresponding control input; the bottom plot represents the velocity field with both the disturbance input and control input turning on. The position of disturbance input and control input is denoted by the blue triangle. Plots are for (a) $t_1 = 1000$, (b) $t_2 = 3100$, (c) $t_3 = 5600$.