2. Interaction dynamics in a viscous environment

Dynamics of the flow induced by a micro-swimmer (say, for example, a swimming micro-robot or a motile micro-organism) can be described by the Stokes equations

(2.1)\begin{equation} \boldsymbol{\nabla} P = \mu \nabla^{2} \boldsymbol{u} + \boldsymbol{F}, \quad \boldsymbol{\nabla} \boldsymbol{\cdot} \boldsymbol{u} = 0, \end{equation}

where $\mu$ is dynamic viscosity of the surrounding fluid, $P$ represents the pressure field, $\boldsymbol {u}$ is the velocity field and $\boldsymbol {F}$ is the notion of body forces per unit volume. In the absence of external (magnetic or electric) fields a self-propelled buoyant micro-swimmer exerts no net force/torque to the surrounding fluid. Therefore, in the most general form, far field of the disturbing flow induced by a micro-swimmer is well described by the flow of a force dipole ($\pm f_s \boldsymbol {e}$) composed of the thrust force (generated by the swimmer's propulsion mechanism) and the viscous drag acting on its body (figure 1a). To be more precise, for a micro-swimmer swimming with speed $v_s$ toward direction $\boldsymbol {e}$ through an unbounded fluid domain, the induced flow field can be formulated (see, e.g. Elgeti, Winkler & Gompper Reference Elgeti, Winkler and Gompper2015) as $\boldsymbol {u}_{\text {SD}} = \mathscr {D}/(8{\rm \pi} \mu \ r^{3}) [ -1+ 3 ( \boldsymbol {r} \boldsymbol {\cdot } \boldsymbol {e}/r )^{2} ] \boldsymbol {r}$, where $\boldsymbol {r} = \boldsymbol {x}- \boldsymbol {x}_0$ for any generic point $\boldsymbol {x}$ in space and $\boldsymbol {x}_0$ represents the instantaneous position of the swimmer. The dipole strength, $|\mathscr {D}| \sim v_s L_s^{2} \sim f_s L_s$, has a positive (negative) sign for pusher (puller) swimmers, and its value can be inferred from experimental measurements (see, e.g. Drescher et al. Reference Drescher, Dunkel, Cisneros, Ganguly and Goldstein2011). Here the characteristic length $L_s$ is of the order of swimmer dimensions, and we use $\{ U_s = v_s, L_s, T_s = L_s/v_s \}$ to make the quantities dimensionless throughout this work.

Figure 1.

(a) Schematic representation of the archetypal puller and pusher swimmers. The force dipole ($\pm f_0 \boldsymbol {e}$) exerted by each swimmer to the surrounding fluid is shown by red arrows. Vector $\boldsymbol {e}$ denotes the swimming direction, and curly blue arrows demonstrate the direction of the induced disturbing flows in each case. (b) Schematic representation for a system of interacting micro-swimmers. The position vector of swimmer $i \in \{ 1 \ldots N \}$, swimming with speed $V_s$ in direction $\boldsymbol {e}^{i}$, is denoted by $\boldsymbol {r}^{i}$ with respect to the fixed frame of reference. Here $N=6$, and the system includes both pusher and puller swimmers.

Swimming in the Stokes regime is significantly affected by the presence of long-range hydrodynamic interactions. In fact, disturbing flows induced by the motion of each micro-swimmer highly affects dynamics of other nearby swimming objects. Here we take into account both hydrodynamic and steric interactions between the swimmers. To this end, let us consider a generic flock of ‘$N$’ interacting micro-swimmers as presented schematically in figure 1(b). For any swimmer $i \in \{ 1, \ldots ,N \}$ in the flock, dynamics of the position vector, $\boldsymbol {r}^{i}$, and the swimming direction, $\boldsymbol {e}^{i}$, are governed by

(2.2a)\begin{gather} \dot{\boldsymbol{r}}^{i} = v_s \boldsymbol{e}^{i} + \sum_{j\neq i} \left[ \boldsymbol{u}^{j} ( \boldsymbol{r}^{i} ) + \frac{1}{6{\rm \pi} \mu L_s} \boldsymbol{\mathcal{F}}^{j} ( \boldsymbol{r}^{i} ) \right] + \boldsymbol{u}_{f} ( \boldsymbol{r}^{i} ) , \end{gather}

(2.2b)\begin{gather}\dot{\boldsymbol{e}}^{i} = \left\{ \frac{1}{2} \boldsymbol{\omega}_f( \boldsymbol{r}^{i} ) + \frac{1}{2} \sum_{j\neq i} \boldsymbol{\omega}^{j}( \boldsymbol{r}^{i} ) + \mathcal{B} \boldsymbol{e}^{i} \times \left[\boldsymbol{\mathsf{E}}_f( \boldsymbol{r}^{i} ) + \sum_{j\neq i} \boldsymbol{\mathsf{E}}^{j}( \boldsymbol{r}^{i} ) \right] \boldsymbol{\cdot} \boldsymbol{e}^{i} \right\} \times \boldsymbol{e}^{i}, \end{gather}

where $\boldsymbol {u}^{j} ( \boldsymbol {r}^{i} ) = \boldsymbol {u}_{\text {SD}} ( \boldsymbol {r}^{i} -\boldsymbol {r}^{j} )$ is the disturbing flow induced by swimmer ‘$j$’ at the position of swimmer ‘$i$’ and $\boldsymbol {\omega }^{j} = \boldsymbol {\nabla } \times \boldsymbol {u}^{j}$ is the corresponding vorticity field. Similarly, $\boldsymbol {u}_{f} ( \boldsymbol {r}^{i} )$ denotes the background (external) flow at the position of swimmer ‘$i$’ and $\boldsymbol {\omega }_f = \boldsymbol {\nabla } \times \boldsymbol {u}_f$ is the corresponding vorticity field. Here each swimmer is modelled as an infinitesimal spheroid of aspect ratio $\lambda$, for which the Bretherton constant (see, e.g. Kim & Karrila Reference Kim and Karrila2013) is defined as $\mathcal {B} = ( \lambda ^{2}-1 ) / ( \lambda ^{2}+1)$. The rate of strain tensor, defined as $\boldsymbol{\mathsf{E}} ( \boldsymbol {r}^{i} ) = (1/2)[ \boldsymbol {\nabla } \boldsymbol {u} (\boldsymbol {r}^{i}) + \boldsymbol {\nabla } \boldsymbol {u}^{\top } (\boldsymbol {r}^{i}) ]$, is also denoted by $\boldsymbol{\mathsf{E}}^{j}$ and $\boldsymbol{\mathsf{E}}_f$, for $\boldsymbol {u}^{j}$ and $\boldsymbol {u}_{f}$, respectively. To regularize our dipole models (diverging as $1/r^{2}$), we also define a purely repulsive force (c.f. Ryan et al. Reference Ryan, Haines, Berlyand, Ziebert and Aranson2011), based on the Lennard–Jones-type potential,

(2.3)\begin{equation} \boldsymbol{\mathcal{F}}^{j} ( \boldsymbol{r}^{i} ) = -\partial \mathscr{L}^{ji} /\partial \boldsymbol{r}^{ji}, \quad \mathscr{L}^{ji} = 4 \epsilon_{L} \left[ \left( \frac{\sigma}{r^{ji}} \right)^{12} - \left( \frac{\sigma}{r^{ji}} \right)^{6} \right] + \epsilon_{L}, \end{equation}

where $r^{ji} = |\boldsymbol {r}^{i}-\boldsymbol {r}^{j}|$, the constant $\sigma$ indirectly specifies the equilibrium distance (i.e. $2^{1/6} \sigma \sim 2L_s$) and $\epsilon _{L}$ tunes the strength of the steric interactions. This ensures the excluded volume constraints, and, thus, pushes the swimmers away from each other when approaching closer than a certain distance (determined by $2^{1/6} \sigma$).

It is worth noting that the presented framework (2.2) can be readily extended by incorporating a more detailed description of $\boldsymbol {u}^{j} ( \boldsymbol {r}^{i} )$. In particular, near field of the flow induced by micro-swimmers can be described more accurately through including an appropriately chosen combination of higher-order terms from the multipole expansion (see, e.g. Spagnolie & Lauga Reference Spagnolie and Lauga2012; Ghose & Adhikari Reference Ghose and Adhikari2014). However, in this work we are interested in the span of swimmers’ induced disturbances and their consequent detection region, for which far field of the flow is of primary interest. Therefore, as discussed earlier, the disturbing flow induced by each micro-swimmer is formulated here, in the most general form, as the flow of a force dipole. This simple model has been validated and widely used in the literature (see Lauga & Powers Reference Lauga and Powers2009; Elgeti et al. Reference Elgeti, Winkler and Gompper2015). In particular, for archetypal pusher and puller swimmers (i.e. Escherichia coli bacteria and Chlamydomonas reinhardtii alga) used as benchmarks throughout this work, the validity of this model is further confirmed via direct comparison to the disturbing flows (experimentally) measured around individual swimming cells (Drescher et al. Reference Drescher, Goldstein, Michel, Polin and Tuval2010, Reference Drescher, Dunkel, Cisneros, Ganguly and Goldstein2011). In the case of E. coli bacteria, for instance, values of $f_s=0.42$ pN and $L_s=1.9\ \mathrm {\mu }$m have been suggested (Drescher et al. Reference Drescher, Dunkel, Cisneros, Ganguly and Goldstein2011) in agreement with resistive force theory (Darnton et al. Reference Darnton, Turner, Rojevsky and Berg2007) and optical trap measurements (Chattopadhyay et al. Reference Chattopadhyay, Moldovan, Yeung and Wu2006). The corresponding dipole model (with $\mathscr {D}\sim f_s L_s$) is proven to be highly accurate in predicting the magnitude of the induced disturbing flows, even at distances comparable to the characteristic length of the swimmer (Drescher et al. Reference Drescher, Dunkel, Cisneros, Ganguly and Goldstein2011). To be more specific, the predicted values are in agreement with those experimentally measured around an individual swimming cell at ranges $r\gtrsim 3L_s$. At shorter distances, the induced disturbing flows are shown to be less significant than those estimated by the dipole model (Drescher et al. Reference Drescher, Dunkel, Cisneros, Ganguly and Goldstein2011), and, thus, steric interactions (‘soft’ collisions) are known to dominate the hydrodynamic interaction. Therefore, it is expected that the dipolar hydrodynamics, when supplemented by Lennard–Jones-type excluded volume interactions (2.3), well-describe interaction dynamics in a semi-dilute suspension (see, e.g. Ryan et al. Reference Ryan, Haines, Berlyand, Ziebert and Aranson2011, Reference Ryan, Berlyand, Haines and Karpeev2013). This agent-based model has also been reported (Ariel et al. Reference Ariel, Sidortsov, Ryan, Heidenreich, Bär and Be'er2018) successful in predicting collective dynamics of two-dimensional bacterial swarms for swimmer concentrations (in terms of surface fractions) of up to 0.6.

To keep the model simple and tractable in this work, we specifically consider swimmers with spherical body shapes (with $\mathcal {B}=0$) for our swimmers. Nevertheless, the presented framework (2.2) explicitly incorporates the effects of swimmers’ geometry. Thus, it can be used to describe the interaction dynamics of micro-swimmers with different shapes. In the case of ellipsoidal body shapes with eccentricity $\lambda$, for instance, the Bretherton constant can be evaluated as $\mathcal {B} = ( \lambda ^{2}-1 ) / ( \lambda ^{2}+1)$. This stands for the values of $\mathcal {B} = 0/1$ for spheres/needles, and takes positive (negative) values for prolate (oblate) objects.

3. Cloaking a micro-swimmer in Stokes flows

A cloak is commonly referred to as a patch enclosing an object to make it invisible. In an aquatic environment, however, swimming organisms nearly always use fluid disturbances (caused by motions in the surrounding fluid) as hydrodynamic signals to detect the nearby objects. Therefore, to be invisible in such environments, one needs to generate no disturbances to the ambient fluid. Accordingly, here we define the cloak as a patch virtually covering/enclosing a swimmer to cancel out its induced disturbances. This will then keep the enclosed swimmer ‘invisible’ to others in the medium – by stifling its flow signature. Similar to other forms of cloaking, a suitable cloak is also required to be omni-directional, that is, to keep itself and the enclosed swimmer ‘invisible’ from any direction. It is worth highlighting that the induced disturbing flows have a long-range nature at the Stokes regime pertinent to the motility of micro-swimmers. This means that despite high-Reynolds-number motions in which the hydrodynamic signature is relatively confined to the immediate neighbourhood, the motion of an object at a low-Reynolds-number regime can be felt tens of body-lengths away. As a result, movements of micro-swimmers significantly alter their surrounding environment. Therefore, specific to biological applications, the cloaking may be more important in protecting sensitive regions (e.g. organs) against disturbing flows induced by intruders – such as passing-by swimming organisms or deployed biomedical micro-robots. This, indeed, is equivalent to making such intruders hydrodynamically invisible (i.e. cloaked) within the surrounding environment.

We assess the effectiveness of an invisibility cloak in the Stokes regime by evaluating its efficiency in stifling the swimmer's induced disturbing flows. A measure of distortion caused by swimming objects to the ambient fluid can be obtained through directly computing the mean disturbing flow magnitude ($\mathscr {U}$) over a surrounding ring of radius $R$ (c.f. figure 2a) – here denoted by $\mathcal {C}(R)$. The cloaking efficiency ($\eta$) can then be calculated as

(3.1)\begin{equation} \eta = 1- \frac{\mathscr{U}_c}{\mathscr{U}_i}, \quad \mathscr{U} = \frac{1}{2{\rm \pi} R} \oint_{\mathcal{C}(R)} |\bar{\boldsymbol{u}}| \, \text{d} s, \end{equation}

where $\mathscr {U}_c$ measures the induced disturbances once the cloak is implemented (i.e. corresponding to the net flows generated by the swimmer and its implemented cloak), and is directly compared to disturbances induced by the isolated swimmer ($\mathscr {U}_i$) – when no cloaking is in place. Here, ‘$\text {d} s$’ is the differential length along the surrounding ring and $\bar {\boldsymbol {u}}=\boldsymbol {u}/v_s$ represents the net dimensionless flow field induced by the swimmer/system.

Figure 2.

(a) Schematic representation of the presented method for cloaking an intruder (here a puller swimmer is shown in green) using properly positioned single ($\psi =0$) or double ($\psi \neq 0$) cloaking agents (here a pusher swimmer is shown in blue). To compute the cloaking efficiency of different arrangements, the induced fluid disturbances are measured over a surrounding (dashed) ring of radius $R$. The dipole strength of each smart agent is half of the intruder's (i.e. $\mathscr {D}/2$). The agents are positioned on a (highlighted dashed) ring of radius $r_0$ around the intruder – i.e. their separation distance from the intruder is set to $r_0$, and their relative position vectors are denoted by $\boldsymbol {r}_1$ and $\boldsymbol {r}_2$, respectively. The angle between these two vectors is denoted by $\psi$ and represents a measure of symmetry. Also, $\beta$ is defined as the angle between the intruder's swimming direction (green arrow) and the position vector $\boldsymbol {r}_1$. (b) Magnitude of the net fluid disturbances ($\mathscr {U}_c$) induced by the cloaked intruder and its cloaking agents, normalized by that of an isolated intruder ($\mathscr {U}_i$), is plotted for various cloak arrangements as a function of distance from the swimmer (R). Specifically, $\mathscr {U}_c/\mathscr {U}_i$ is plotted as a function of $R/r_0$ for $\psi = 0$, ${\rm \pi} /4, {\rm \pi}/2, 3{\rm \pi} /4$ and ${\rm \pi}$, where $\beta$ is set to ${\rm \pi} /2$. (c) The cloaking efficiency of the arrangements associated with each of the $\psi$-angles presented in panel (b) is plotted as a function of $R/r_0$. Insets also demonstrate how the cloaking performance (in terms of $\eta \%$) varies with $\psi$ (I) and $\beta$ (II), respectively. For all cases presented in (I), $\beta$ is fixed to ${\rm \pi} /2$, and for those presented in (II), $\psi$ is fixed to ${\rm \pi}$. The presented values are measured at $R/r_0 = 5$, 10 and 20.

The simplest strategy in cloaking a single swimmer (an intruder) at any given instance of time is to properly position, next to it, another swimmer of the same dipolar strength, but from the opposite type. This basically replaces the dipolar (vanishing as $1/r^{2}$) leading order of the overall disturbing flows (in the far field) with a quadrupole (rapidly decaying as $1/r^{3}$), and, thus, can effectively cloak the intruder by drastically stifling its induced disturbances in the ambient fluid. To give an example, a single cloaking agent positioned next to the intruder with a separation distance of $r_0$ dramatically reduces the magnitude of the overall detectable disturbing flows (measured by $\mathscr {U}$) to 11.6 % and 4.4 % of its original value, once measured at distances $R/r_0= 20$ and 50 from the intruder, respectively (figure 2b for $\psi =0$). This is equivalent to more than 88.4 % and 95.6 % in cloaking efficiencies at the ranges of $R/r_0 \geq 20$ and $\geq 50$, respectively (figure 2 for $\psi =0$). For this simple approach, however, the underlying hydrodynamic interaction between the intruder and its cloaking agent will inevitably disturb the intruder from its original path. Therefore, using a single cloaking agent, although effective, is not the smartest approach.

Here, we are particularly interested in implementing non-invasive active cloaking, where the cloaked swimmer (i.e. the intruder) is not deviated from its original path by the cloak implementation. This is fundamentally impossible using only a single cloaking agent. Alternatively, a proper positioning of two half-sized cloaking agents around an intruder (e.g. see figure 2a for $\psi ={\rm \pi}$ and $\beta ={\rm \pi} /2$) can potentially prevent its deviation from the original path. This also significantly improves the cloaking efficiency (by switching the leading order to an octupole rapidly vanishing as $1/r^{4}$) compared to the single-agent method (with quadrupolar far field decaying as $1/r^{3}$).

The schematic representations of cloaking an intruder using single ($\psi =0$) and double ($\psi \neq 0$) cloaking agents are presented in figure 2(a). We also present cloaking performances corresponding to various arrangements of the agents around an intruder in figure 2 – both in terms of the relative magnitude of induced disturbances ($\mathscr {U}_c/\mathscr {U}_i$) and the cloaking efficiency ($\eta$). Note that the presented efficiencies correspond to a set of optimally designed cloaking agents which can provide an ideally cancelling set of dipole strengths for the specific intruder of interest. Our results show that the effect of angular positioning (measured by $\beta$) on cloaking efficiency is negligible (figure 2cII), whereas $\psi$ (as a measure of symmetry) primarily controls the efficiency (figure 2cI). Variation of $\psi$ in the range of $[ 0,{\rm \pi} ]$ basically covers the whole spectrum from single-agent ($\psi = 0$) to symmetric double-agent ($\psi = {\rm \pi}$) cloaking. It is worth highlighting that the symmetric arrangement of cloaking agents on the sides of an intruder (i.e. the arrangement with $\psi ={\rm \pi}$ and $\beta ={\rm \pi} /2$ in figure 2a) represents not only a non-invasive but also the most efficient double-agent cloaking strategy. The latter, in particular, is not immediately clear a priori, and has been revealed through the presented results (figure 2). In practice, such an optimal arrangement can potentially be used to realize a virtually perfect non-invasive cloak, the efficiency of which reaches beyond 99 % and 99.9 % at distances $R/r_0 \geq 20$ and $50$, respectively (figure 2).

Active cloaking of an arbitrarily moving intruder in the Stokes regime, however, still remains a challenge. For a naive (i.e. non-smart) pair of cloaking agents, even if they are initially placed accurately in their predefined positions, the arrangement will soon be distorted due to the presence of long-range hydrodynamic interactions – causing the swimming agents to either diverge from one another or collide. In fact, disturbing flows induced by the intruder (although cancelled out by its cloak in the far field) are internally disruptive for the cloak itself – i.e. disturbs the agents’ arrangement and, thus, breaks the cloak. Therefore, to realize active cloaking in the presence of such complex flow-mediated interactions, key questions still remain to be addressed in the following section. First, what is the optimal sequence of actions (corresponding to the shortest non-invasive path) for each cloaking agent to position itself in a desired arrangement around an arbitrary intruder? Second, what is the optimal behaviour policy for the cloaking agents to dynamically adjust their swimming actions in response to nonlinearly varying hydrodynamic loads, so as to robustly keep their concealing arrangements around an intruder? Lastly, to what extent is the presented approach generalizable? That is, can we use it to actively cloak multiple arbitrary intruders within crowded suspensions, where each swimmer moves toward an arbitrary direction and experiences frequent close encounters?

4. Learning to cloak random intruders via reinforcement learning

Here we present a systematic methodology to equip micro-swimmers with adaptive decision-making intelligence, in response to flow-mediated interactions. We then use these smart agents to elucidate active cloaking of arbitrary intruders within a crowded environment. Note that once an agent is equipped with such an artificial intelligence, it will consider potential consequences of its actions when making any decision – hence, is called a planning agent. One may alternatively suggest the implementation of an external active control to lead each of the employed swimming micro-robots (i.e. cloaking agents) toward the desired arrangements. However, the efficacy of such approaches suffer from nonlinearly varying hydrodynamic loads due to the presence of flow-mediated interactions. In fact, the dynamical system representing a group of interacting micro-swimmers is a complicated four-way-coupled system, where any tiny perturbation/deviation in prescribed motions of the swimmers can induce unpredicted complex choreographies (see, e.g. Mirzakhanloo, Jalali & Alam Reference Mirzakhanloo, Jalali and Alam2018). It is also worth noting that the reflex agents (by definition) are incapable of identifying optimal pathways to form non-invasive cloaks, especially in crowded suspensions or in the presence of obstacles. Therefore, here our approach is to let the cloaking agents learn optimal action policies by their own experience through a reinforcement learning algorithm (Sutton & Barto Reference Sutton and Barto2018). By accumulating experience, our cloaking agents learn how to optimally collaborate, and dynamically adjust their swimming actions, to form a robust non-invasive cloak for any randomly moving intruder.

The implemented reinforcement learning algorithm was initially inspired by the concept of animal learning (Niv, Daw & Dayan Reference Niv, Daw and Dayan2006), and has been shown effective in learning previously unknown strategies, solely based upon the received feedback on performance (Tesauro Reference Tesauro1995). The great potential of this approach has also been recently demonstrated in fish schooling (Gazzola et al. Reference Gazzola, Tchieu, Alexeev, de Brauer and Koumoutsakos2016), soaring of birds through turbulent environments (Reddy et al. Reference Reddy, Celani, Sejnowski and Vergassola2016), flow navigation of gravitactic particles (Colabrese et al. Reference Colabrese, Gustavsson, Celani and Biferale2017) and many other applications in the realm of active matters (Cichos et al. Reference Cichos, Gustavsson, Mehlig and Volpe2020; Tsang et al. Reference Tsang, Demir, Ding and Pak2020a,Reference Tsang, Tong, Nallan and Pakb).

Here, we formulate the reinforcement learning algorithm as a Markov decision process. This allows us to use the $Q$-learning framework, which not only benefits from algorithmic simplicity, but also is proven (Watkins & Dayan Reference Watkins and Dayan1992) to converge to an optimal behaviour policy. In this framework, the agent (here a micro-swimmer capable of decision making) gradually learns the optimal behaviour policy through exploring the environment (see figure 3a). At any given instance of time (the $n$th learning step), the agent is able to sense some information about the environment (state, $s_n$), depending on which, it will choose an action ($a_n$) according to its current policy (${\rm \pi} _n$). Taking this action will then transit the agent to a new state ($s_{n+1}$), and it will be given a reward ($r_{n+1}$) quantifying its immediate success (figure 3a). The experience acquired by the agent (after going through each learning step) is stored as an action-value function $Q(s,a)$ in the quality matrix ‘$Q$’ – hence, the name of the algorithm. To be more precise, this $Q$-matrix encodes an action-value function under the current policy, which represents the expected sum of discounted future rewards when taking the action $a_n$ at the current state ($s_n$) and following the current policy (${\rm \pi} _n$) thereafter; i.e.

(4.1)\begin{equation} Q ( s_n,a_n ) = r_{n+1} + \gamma r_{n+2} + \gamma^{2} r_{n+3} + \cdots . \end{equation}

Figure 3.

(a) Schematic representation of the reinforcement learning algorithm coupled with the presented flow-mediated interaction dynamics in viscous environments. (b) Schematic illustration of the state space $(\xi ,\theta )$, when two of our smart micro-swimmers (agents A and B shown in blue) attempt to form a desired active cloak around a randomly moving intruder (green).

This $Q$-matrix is updated throughout the learning process, and is proven (Watkins & Dayan Reference Watkins and Dayan1992) to encode optimal action-value function ($Q^{*}$) at convergence. The optimal behaviour policy (${\rm \pi} ^{*}: s_n \rightarrow a_n$) is then readily available. To be more precise, in any given state ($s_n$) the optimal action ($a_n^{*}$) is the one that maximizes the expected sum of future rewards (which is encoded in $Q^{*}$). This policy (${\rm \pi} ^{*}$), in fact, serves as the agent's adaptive decision-making intelligence.

Here we implement an off-policy control using the described $Q$-learning scheme. During the learning phase, an $\epsilon$-greedy behaviour policy has been employed to ensure exploration of new solutions while appropriately exploiting the gained knowledge. Specifically, at the $n$th learning step, the policy (${\rm \pi} _n: s_n \rightarrow a_n$) is to choose the action ($a_n$) that maximizes current evaluation of the action-value function ($Q$). Except for a small probability ($\epsilon$), in which case, a random action will be chosen (independent of $Q$) from the set of possible actions ($\mathcal {A}$) to further explore the state space; i.e.

(4.2)\begin{align} a_n =\begin{cases} \text{arg max}_{a} Q(s_n,a) & \text{probability of } 1-\epsilon,\\ \text{random action } a \in \mathcal{A} & \text{probability of } \epsilon . \end{cases} \end{align}

After going through each learning step, the current estimation of the action-value function ($Q$) is then updated according to

(4.3)\begin{equation} Q ( s_n,a_n ) \leftarrow Q ( s_n,a_n ) + \alpha [ r_{n+1} + \gamma max_{a} Q ( s_{n+1},a ) - Q ( s_n,a_n ) ] , \end{equation}

where the learning rate, $0 \leq \alpha \leq 1$, specifies the rate at which previously learned experience is overwritten by the newly gained information. The discount factor, $0 \leq \gamma <1$, determines to what extent the value of expected future rewards is incorporated into the agent's decision making: for $\gamma \rightarrow 0 (\rightarrow 1)$, its behaviour will be myopic (farsighted) tending to maximize immediate (future) rewards. Here, the state-transition ($s_n \rightarrow s_{n+1}$), i.e. evolution of the system dynamics, is directly simulated by taking into account the four-way coupled nature of the system through flow-mediated interactions (see § 2 for details). Once in the new state ($s_{n+1}$), the agent will then take another action, based on the updated policy, and the process will be repeated (figure 3a). Note that the updated action policy (${\rm \pi} _{n+1}$) in the new state will still be the same $\epsilon$-greedy policy, yet based on the updated estimation of $Q$. Therefore, the behaviour policy is continuously improved through the learning phase, and eventually converges to the optimal behaviour policy ($n \rightarrow \infty : {\rm \pi}_n \rightarrow {\rm \pi}^{*}$). We then employ a purely deterministic (greedy) behaviour policy (i.e. $\epsilon =0$) for our cloaking agents, according to the obtained optimal action-value function, $Q^{*}$, to further assure the success of any assigned task.

Let us now consider the problem of cloaking arbitrarily moving intruders using pairs of swimming agents. We first outline a rigorous approach to equip our cloaking agents with adaptive decision-making intelligence – i.e. realizing the concept of smart micro-swimmers. As a benchmark, we consider puller intruders (say, for example, motile C. reinhardtii cells) swimming in an infinite two-dimensional fluid domain. Nevertheless, the results will be similar for pushers, and our study can be inherently generalized to a three-dimensional domain. Each of the cloaking agents is modelled as a self-propelled micro-swimmer of the opposite type (here pushers) with a half-sized dipole strength ($\mathscr {D}/2$). We are particularly interested in realizing the active (and robust) form of the most effective cloak presented in § 3 (figure 2) – for which, $\eta >99\,\%$ at $R/r_0 \geq 20$. Therefore, the approach is to position two cloaking agents symmetrically arranged (with a predefined separation distance $r_0$) on the sides of each intruder throughout its motion. This non-invasive cloak will effectively keep the enclosed swimmer hydrodynamically ‘invisible’ within the surrounding fluid.

In order to form this active cloak, our cloaking agents must learn optimal behaviour policy in the presence of long-range hydrodynamic interactions. A pair of these smart micro-swimmers can then be assigned to cloak each of the randomly moving swimmers within any arbitrarily crowded suspension. The learning objective for each of these two cloaking agents can be translated into two specific tasks: (i) learning how to optimally catch the assigned (randomly moving) intruder by forming a desired cloaking arrangement around it; and once the target positions (e.g. points A and B in figure 3b) are reached then (ii) learning how to keep on following the assigned intruder while robustly retaining the desired arrangement. Hereafter, we refer to these two consecutive learning tasks as to ‘catch’ and ‘follow’ a random intruder – both required in realizing an active cloak. Note that optimal execution of the former requires the cloaking agents to identify the shortest non-invasive paths toward positioning themselves (from anywhere in the state space) into predefined desired arrangements around any moving subject.

To accomplish these tasks, each swimming agent can sense (or is provided with) an estimation of its distance and relative orientation with respect to the intruder. In practice, a range of imaging techniques (see Pané et al. Reference Pané, Puigmartí-Luis, Bergeles, Chen, Pellicer, Sort, Počepcová, Ferreira and Nelson2019) can be used to feed such information into the agent's decision-making unit. For instance, magnetic resonance imaging has successfully been integrated into robotic platforms to provide the information required for real-time navigation of untethered micro-robots (or bacteria) swimming through human micro-vasculature (see, e.g. Martel et al. Reference Martel, Felfoul, Mathieu, Chanu, Tamaz, Mohammadi, Mankiewicz and Tabatabaei2009a,Reference Martel, Mohammadi, Felfoul, Lu and Pouponneaub).

The acquired information by each agent mathematically translates to the normalized distance $\xi = |\boldsymbol {x}^{t}-\boldsymbol {x}|/L_s$ and relative orientation $\theta$ with respect to the assigned target point, as depicted in figure 3(b) for the catching phase. This representation of the state space is also consistent with previous studies (e.g. Gazzola et al. Reference Gazzola, Tchieu, Alexeev, de Brauer and Koumoutsakos2016) on swarm control of swimming agents. Note that once the agents successfully catch the intruder (by forming the desired cloaking arrangement around it) they still will need to follow the intruder, so as to actively keep the desired arrangement throughout its motion. Therefore, once in the cloaking positions, $\theta$ is defined as the deviation angle that each agent senses for its own swimming direction relative to the assigned intruder's.

Although the micro-swimmers are free to move in a continuous two-dimensional space, here we map the state space, $s = (\xi , \theta )$, on a finite-size two-dimensional pseudo-grid. This way of representing the state space dramatically mitigates the curse of dimensionality, and makes the learning process computationally feasible. Also, from the practical point of view, the agents will then only need an estimation of their distance and deviation from their assigned target points. The required accuracy of such estimation is specified by the grid size. Therefore, let us consider the discretized representation of our state space ($\mathcal {S}$), which is the product of two finite-size subsets, i.e. $\mathcal {S} = \mathcal {S}_{\xi } \times \mathcal {S}_{\theta }$. Specifically, $\mathcal {S}_{\xi }$ represents the distance $\xi$ as $n_{\xi }$ discrete states within the range of ${\rm \Delta} \xi$, and $\mathcal {S}_{\theta }$ indexes the angle $\theta$ as $n_{\theta }$ discrete states within the range of ${\rm \Delta} \theta$. Hence, at every instance of time ($t_n$), the state of an agent is represented by a tuple $s_n = (\xi _n, \theta _n)$, where

(4.4a)\begin{gather} \xi_n = min ( {\rm \Delta} \xi, \lfloor{ \xi n_{\xi}/{\rm \Delta} \xi }\rfloor \cdot {\rm \Delta} \xi/n_{\xi}), \end{gather}

(4.4b)\begin{gather}\theta_n = min ( {\rm \Delta} \theta, \ \lfloor{ \theta n_{\theta}/{\rm \Delta} \theta }\rfloor \cdot {\rm \Delta} \theta/n_{\theta} ). \end{gather}

For instance, here we set $n_{\xi }=100, {\rm \Delta} \xi = 50, n_{\theta }=36$ and ${\rm \Delta} \theta = 2{\rm \pi}$. This means a total of 3600 discrete possible states for each agent.

The deployed cloaking agents are able to swim in three distinct speeds: $v^{0}$ (nominal), $v^{+} = v^{0} + \delta v$ (fast) and $v^{-} = v^{0} - \delta v$ (slow). They can also instantly turn to the right ($\theta \leftarrow \theta + \delta \theta$) or left ($\theta \leftarrow \theta - \delta \theta$) with different choices of angle $\delta \theta$. Therefore, the motion of each swimming agent during the learning process can be described as a repeated occurrence of the following two-step process: (i) the agent updates its swimming velocity (through either a change in orientation by $\delta \theta$, or a modification in speed by $\delta v$) according to the current policy; and then (ii) it moves with the new velocity for a fixed time $\tau _r$ ($=1$ in non-dimensional time units scaled by $T_s = L_s/v^{0}$). Note that each agent takes an exclusive swimming action (e.g. turning to the right/left with an angle $\delta \theta$, or speeding up/down with $\delta v$) based on the current policy (${\rm \pi} _n$) at intervals of $\tau _r$ during the learning process. Once an action is taken, the agent swims steadily forward (i.e. ‘runs’) until a new action (potentially a change in orientation, i.e. ‘tumbling’) is taken after $\tau _r$. This is inspired by the observed behaviour of natural swimming micro-organisms, and is reminiscent of the so-called run-and-tumble locomotion. In practice, realization of the smart version of this locomotion dynamics seems also feasible in the context of artificial and model micro-swimmers. The recently proposed Quadroar swimmer (Jalali, Alam & Mousavi Reference Jalali, Alam and Mousavi2014; Mirzakhanloo & Alam Reference Mirzakhanloo and Alam2018), for instance, propels on straight lines (runs), and can perform full three-dimensional reorientation (tumbling) maneuvers (Saadat et al. Reference Saadat, Mirzakhanloo, Shen, Tomizuka and Alam2019).

Here, the nominal swimming speed for the agents is set to that of the intruders (i.e. $v^{0} = v_s$), and we consider $\delta v = 0.05 \ v^{0} (0.5 v^{0})$ while following (catching) an intruder. Nevertheless, our numerical experiments reveal that a learned policy (under this specification) can be readily adopted by agents with different speed characteristics ($v^{0}, \delta v$), and so in cloaking intruders with various swimming speeds (see appendix A.1 for details). Furthermore, the space of possible values for $\delta \theta$ inherently covers the entire range of $[ 0, {\rm \pi}]$. However, selecting only a finite-size set of options for $\delta \theta$ substantially reduces the computational costs associated with the recurrent identification of optimal actions during the reinforcement learning process. Therefore, here we assume the agents are only able to turn right/left with $\delta \theta \in \{ {\rm \pi}/18,{\rm \pi} /4,{\rm \pi} /2 \}$. This finite-size simplified set is particularly devised to provide each agent with the ability to (i) make major changes in the swimming direction (with $\delta \theta = {\rm \pi}/2$), typically done in the early stages of a catching process; (ii) perform moderate reorientations (with $\delta \theta = {\rm \pi}/4$) commonly used during a catching process; and (iii) fine tune slight deviations from a desired orientation (with $\delta \theta = {\rm \pi}/18$), which is particularly required at final stages of a catching process or while following an intruder. Therefore, our set of possible actions ($\mathcal {A}$) consists of nine specified swimming actions. This, along with the described discretized state space, result in 32 400 entries for the space of state-action pairs to be used in evaluating the action-value function $Q(s_n,a_n)$. We note that the frequency of taking an action by each agent (i.e. $\tau _r^{-1}$) can also be tuned according to the system of interest.

For the swimming agents, realization of an active cloak around an arbitrary intruder is translated into taking an optimal sequence of actions to actively catch and follow the assigned target points in a predefined arrangement. The immediate success of each agent in satisfying this assigned task is mathematically interpreted as a numerical reward signal

(4.5)\begin{equation} r_n = [1/(\xi_n+\delta \xi) - \xi_n] + [\delta(v_n-v^{0})-1] + \mathscr{C}_n + \mathscr{P}_n, \end{equation}

where $\delta \xi$ is the system precision (i.e. the grid size) in measuring the distance. The $\delta (x)$ is defined such that $\{ \delta (x) =0 \ | \ \forall x \neq 0 \}$ and $\delta (0) =1$. In our reward signal definition, the first term reflects how well the agent is following (or catching) an assigned target point, while the second term penalizes any unnecessary speed-ups or -downs. This penalty is assigned because the dipole strength of the swimmers is directly proportional to their swimming speed ($\mathscr {D} \sim v_sl_s^{2}$), and, thus, any change in the value of swimming speeds can disturb the balance of dipole strengths. However, the relative value of these speed modifications is kept negligibly small (here, for example, $\delta v/v^{0} = 0.05$) while the agents are actively cloaking an intruder (i.e. robustly following it in a predefined arrangement). Contribution of the third term in the reward signal ($\mathscr {C}_n$) is two-fold: (i) it encourages the agent to accurately get into the assigned target point once fairly close (here, for example, $\mathscr {C}_n = 100$ when $|\boldsymbol {x}^{t}-\boldsymbol {x}| \leq \delta \xi$); and (ii) it strictly penalizes wandering of the agent far off the target (here, for example, $\mathscr {C}_n = -100$ when $|\boldsymbol {x}^{t}-\boldsymbol {x}| \geq {\rm \Delta} \xi$). Finally, the last term, $\mathscr {P}_n = - \cos ^{-1} (\boldsymbol {e} \boldsymbol {\cdot } \boldsymbol {e}^{i})$, ensures that the agents learn how to smartly collaborate in a way that their induced disturbing flows do not disturb the cloaked object (i.e. the intruder) from its original path ($\boldsymbol {e}^{i}$) – hence, realizing non-invasive active cloaking.

It is to be noted that realizing an active cloak around the assigned intruder, as the ultimate goal of this learning algorithm for each agent, is mathematically encoded as achieving maximal long-term accumulated rewards. Therefore, a negative nature of the reward signal $r_n$ (4.5) further ensures that our cloaking agents learn the shortest paths toward positioning themselves in the desired cloaking arrangement – as any unnecessary action will result in an extra accumulation of negative rewards.

Here, the cloaking agents are trained in pairs, and we employ a shared policy approach among them to accelerate the learning process. Training is conducted through consecutive learning episodes, denoted by $E = 1,2, \ldots , n_E$, where $n_E$ is the total number of episodes in the learning phase. Each episode starts with an intruder randomly entering a guarded (sensitive) region – i.e. from a random position and toward a random direction. The cloaking agents are also initially located on either side of the entrance (see figure 4). The first episode ($E=1$) is initialized by zero entries for all elements of the $Q$-matrix (i.e. an optimistic initialization) to further encourage exploration of new solutions and avoid trapping in local optima – recall the negative nature of the reward signal. For the subsequent episodes ($E = i+1, i\geq 1$), the action-value function, $Q$, will be initialized by that obtained at the end of the previous episode (i.e. $E=i$). The end of each training episode is declared when either (i) the agents have formed a desired cloak around the intruder – i.e. when they both have reached the assigned target points in a predefined arrangement (see, e.g. figure 4d); or (ii) a certain number of state changes have passed without any success (e.g. $N_{s, max}=250$ in figure 4a). The process is in turn repeated until the action-value function coverages to an optimal value (i.e. $Q_n \rightarrow Q^{*}$ as $n_E \rightarrow \infty$) – or, practically, when the policy (${\rm \pi} _n$) converges to the optimal behaviour policy (${\rm \pi} _n \rightarrow {\rm \pi}^{*}$).

Figure 4.

(b,c,d,e,f,g). (a) The learning process is assessed in terms of the agent's success rate and number of the required swimming actions. After every $1000$ training episodes, a set of $100$ random active cloaking tests have been performed using a purely deterministic policy based on the most updated $Q$-matrix at the moment. Each of the testing episodes starts with an intruder randomly entering a guarded region (shaded in grey; see panels b–g) from a random position and swimming toward a random direction, while the cloaking agents are initially positioned on the sides of the entrance (see, e.g. panels b–g). The test results are presented throughout the learning phase in terms of (i) the success rate (shown in green) of the agents in catching the intruders and forming the desired active cloak; as well as (ii) the number of total state changes (average $\pm$ standard error) in each set of testing episode (shown in red). (b–d) Benchmarks of different learning stages. The panels represent a sample test on realizing an active cloak around a random intruder, performed in three different learning stages, i.e. non-adaptive (b), intermediate-adaptive (c) and well-adaptive (d) stages. The episode number ($E$) corresponding to each of the panels (b–d) is marked in panel (a), and agents use the most updated behaviour policy at each stage. The intruder (here a puller shown in green) and the cloaking agents (pushers shown in blue) are also represented both at the initial and final positions with stripe and solid schematics, respectively. Trajectories of the swimmers are shown in each panel by dashed lines, and colour coded based on the swimming speed at the moment – see the legend in panel (c). The assigned target points (to realize the desired symmetrical active cloak) for each of the agents is also marked in each panel (pink markers). (e–g) Sample tests performed using well-trained cloaking agents, equipped with well-adaptive behaviour policy, which is obtained after convergence of the success rate to $100\,\%$ (see panel a). The number of required swimming actions (or, equivalently, the number of state changes, $N_s$) is denoted on each panel. At this stage, the agents are capable of identifying the shortest non-invasive paths in forming desired active cloaks around any randomly moving intruder. The target arrangement here is to symmetrically position on the sides of the intruder, with a separation distance $r_0/L_s =5$ (see, e.g. panel e). The learning hyper-parameters are set to $\alpha =0.3, \gamma =0.95$ and $\epsilon =0.01$.

To assess the learning process at different stages, we run a set of $100$ random active cloaking tests throughout the training phase – here, specifically after every $1000$ learning episodes. In testing the agents’ intelligence, we deploy a purely deterministic (greedy) policy (i.e. $\epsilon =0$) based on the most updated acquired $Q$-matrix. During each of these testing episodes, the agents are subject to cloak a random intruder using their so-far gained experience (encoded in the given $Q$-matrix). For each set of the ($100$) performed tests, we then compute the average number of total state changes ($N_s$), along with the success rate of the agents in catching the intruders and forming desired active cloaks. Both measures are monitored throughout the learning phase as demonstrated in figure 4(a). The learning is evident in the gradual increase (decrease) of the agents’ success rate (required number of swimming actions) in collaborative formation of the desired active cloak around random intruders (figure 4). As a benchmark, we also demonstrate, in panels (b–d) of figure 4, a specific sample test performed in three different learning stages – corresponding episode numbers ($E$) are marked in panel (a). While the agents are too naive at the early stages (e.g. figure 4b) and fail to catch and cloak the subject, they eventually learn how to actively cloak the moving intruder through an optimal set of non-invasive actions (e.g. figure 4d). Note that both of the monitored measures (i.e. the success rate and the number of swimming actions required to form a desired cloak) eventually converge to their optimal values as the number of learning episodes increases sufficiently (here, after $E \sim 25\ 000$). The converged value of $100\,\%$ success rate for the cloaking agents clearly shows their ability to smartly form a non-invasive active cloak around any randomly moving intruder (see, e.g. the samples presented in figure 4e–g). In doing so, they find the shortest non-invasive paths using only their own adaptive decision-making intelligence – hence, the name smart micro-swimmers.

Convergence of the cloaking agents’ adaptive decision-making intelligence (encoded in the $Q$-matrix) is better demonstrated in figure 5, where we show evolution of the element-wise sum of the $Q$-matrix ($\varSigma$) normalized by its converged value. Additionally, the experientially learned behaviour policy is visualized by colour coding over a sector of the state space in four different learning stages (marked in figure 5 by I-IV). Here, convergence of the action-value function (and, thus, the behaviour policy) is achieved when the number of successive training episodes ($E$) exceeds ${\sim }25\ 000$ (figures 4 and 5).

Figure 5.

Evolution of the normalized element-wise sum of the $Q$-matrix ($\bar {\varSigma }$) through successive training episodes ($E$) during the learning phase. The agents’ behaviour policy is also visualized (see insets I–IV) by colour coding over a selected sector of the state space. Its evolution is monitored at four different learning stages (marked over the curve by I-IV), and eventually converges to the optimal behaviour policy. Here, each swimming action is presented by a colour code (see the colour-bar legend), and every point in the state space is shaded by a colour representing the agents’ current understanding of the best action at that specific state. The training episodes are the same as those presented in figure 4 for which the learning hyper-parameters are set to $\alpha =0.3, \gamma =0.95$ and $\epsilon =0.01$.

Once the learning process is converged, the cloaking agents are readily equipped with an optimal behaviour policy. To be more precise, the optimal behaviour policy refers to the policy (${\rm \pi} ^{*}: s \rightarrow a$) that in any given state ($s$) determines the action ($a$) which maximizes the expected sum of future rewards (encoded in $Q^{*}$). Here, the reward signal ($r_{n+1}$) received (during the learning process) by the agents after taking an action ($a_n$) has two salient features – c.f. (4.5): (i) it reflects a penalty for disturbing the intruder from its original path; and (ii) it is designed to always have a negative value, except at the target state. Therefore, maximizing the discounted sum of future rewards will ensure not only to successfully realize the desired non-invasive cloak, but also to do so with a minimum number of actions (figure 4a). Note that spending more time (by taking excessive actions) will simply result in accumulating more negative rewards, which is discouraged by the optimal policy. This enables the agents to find the shortest non-invasive paths toward positioning themselves in the desired cloaking arrangements around arbitrarily moving intruders (see, e.g. the samples presented in figure 4e–g).

However, there is still a caveat here: the presence of long-range flow-mediated interactions, if not responded adaptively by taking proper swimming actions, will soon distort the arrangement and reveal the cloaked object. Our swimming agents, however, have been trained not only to ‘catch’ and form desired cloaks around random intruders, but also to ‘follow’ them robustly, while maintaining the implemented cloak throughout their motion. As a benchmark, let us monitor the time evolution of a sample crowded suspension of intruders (figure 6), where the deployed pairs of our smart micro-swimmers have already formed the desired cloak around each of the subjects – see the examples presented in figure 4. Periodic boundary conditions are imposed on the presented panels, as if they represent just a window of an infinite domain of suspension. Such a complex random system is mainly characterized by each of the swimmers moving toward a random direction in space (figure 6a), with frequent close encounters happening between them (e.g. figure 6b,c). It is worth noting that even an isolated intruder swimming (alone) on a straight line in an unbounded domain can distort the arrangement of an implemented cloak, and cause naive cloaking agents to diverge or collapse. The situation here is further exacerbated by the presence of other nearby swimmers, given their disturbing flows and long-range flow-mediated interactions – let alone close encounters happening in such a crowded system, and the consequent disruptions in cloak arrangements. However, as it is evident throughout the system evolution (see also supplementary movie S1), the proposed smart active cloaking strategy is robust. In particular, by taking optimal adaptive swimming actions, the cloaking agents not only successfully retain their arrangement in the presence of hydrodynamic interactions, but also immediately reform the desired cloaking arrangements after any sudden disruption (figure 6, supplementary movie S1).

Figure 6.

Time evolution of a crowded suspension of intruders (shown in green), each of which are actively cloaked by a pair of smart micro-swimmers (shown in blue). Here the intruders are freely moving toward random directions in space – specified by green arrows in (a). Periodic boundary conditions are imposed on the presented panels – i.e. each panel represents just a window of an infinite domain at that specific moment. As demonstrated through snapshots (a–d) of the system's time evolution (see also supplementary movie S1 available at https://doi.org/10.1017/jfm.2020.665), using their adaptive decision-making intelligence, our agents are able to robustly maintain the cloak formation in the presence of complex hydrodynamic interactions. They are also able to immediately restore the desired formation after any severe close encounters (b,c) which cause major disruptions to the cloak. As a benchmark, the evolution dynamics of four sample groups (G.1–4) are tracked in the panels. Specifically, the close encounter between G.1 & G.2 as well as G.3 & G.4 are marked by red dashed circles in panels (b) and (c), respectively. Also, trajectories of the swimmers in G.1 are shown by solid lines throughout the time, while other groups’ trajectories are only demonstrated (by dashed lines) within the last $25T_s$ of their motion. The time corresponding to each snapshot is noted in each panel, and swimmer schematics are added to panels (a) and (d) for readability. Here, the goal was to realize an active version of the symmetric cloak presented in figure 2. Thus, for reference, rings of radii $r/L_s=5$ (that is equal to the predefined desired separation distances) are depicted around intruders in the final snapshot (d).

In practice, it is often of interest to conceal a single micro-swimmer (e.g. a biological micro-robot) within a crowded suspension (e.g. inside the human body) using only a pair of cloaking agents. However, for the sake of generality, here we have used a flock of cloaking agents to cloak all the (randomly moving) swimmers in the presented crowded suspension (see figure 6). To assess performance of the deployed flock in concealing the assigned swimming objects, we then monitor the overall cloaking efficiency ($\eta \%$) throughout the system's evolution (figure 7a). The magnitude of disturbing flows (i.e. flow signatures) induced by the system is also monitored throughout the represented time evolution (supplementary movie S1), and is compared to the same system with no cloaking agents (figure 7b–e).

Figure 7.

(a) Time evolution of the overall cloaking performance ($\eta \%$) corresponding to the flock of smart micro-swimmers deployed within the crowded suspension presented in figure 6 to cloak random intruders. Here $\eta$ ($\%$) is monitored over the entire time evolution at ranges $R/L= 1$ (red solid line), $R/L= 10$ (black dashed line) and $R/L= 100$ (black dash–dotted line), where $L$ is the length of each presented window in figure 6. (b–e) Snapshots of the induced fluid disturbances (see also supplementary movie S1) are visualized (by colour shading) over the entire system (cloak: on), and also compared to those induced when no cloaking agents are deployed (cloak: off). The moment corresponding to each of the snapshots in (b–e) is marked on the curve in (a). Specifically, we demonstrate the snapshots at (i) an early stage, where the desired cloaks ($\eta \approx 98\,\%$) have been just formed (figures 6a and 7b); (ii) the moments of close encounters happening between G.1 & G.2 (first disruption, see figures 6b and 7c) as well as G.3 & G.4 (second disruption, see figures 6c and 7d), respectively, where the implemented cloaks undergo severe disruptions; and (iii) a final recovered stage (figures 6d and 7e) where the agents are once again perfectly in the desired cloaking arrangements ($\eta \approx 93\,\%$).

The implemented cloaks are dynamically adjustable, as the cloaking agents behave adaptively in response to fluid-mediated interactions. This enables them to robustly keep their specified arrangement, and actively conceal any moving subject throughout its motion (see, e.g. figure 6). Not to mention that the agents first need to smartly collaborate in pairs and find optimal paths to separately catch each of the assigned subjects (see the samples in figure 4) forming an active set of perfect cloaks (with $\eta > 98\,\%$) around these randomly moving intruders (figures 6a and 7b). Once realized, throughout the system evolution, the cloaking efficiency remains well above 75 % even in the presence of nonlinearly varying complex flow-mediated interactions (figure 7a). The two relatively notable sudden drops in the cloaking performance (i.e. to $\eta \sim 70\,\%$) occur right after the reported close encounters, which cause major disruptions to the cloaks’ arrangements (figures 6b,c and 7c,d). However, the implemented active cloaks are robust, thus are immediately reformed and the cloaking efficiency is quickly recovered. For instance, note the final stage (figures 6d and 7e) with a remarkable $93\,\%$ cloaking efficiency that is recovered after going through many interactions and encounters.

It is noteworthy that the deployed flock of cloaking agents have never been trained for, neither exposed to, this (or any other) specific scenario with multiple intruders. This means that once the agents undergo the described catch and follow training processes (see figure 4), they can be readily used in pairs to actively cloak any number of arbitrary intruders within unexposed crowded suspensions.

5. Concluding remarks

In this article we presented a rigorous approach to actively cloak swimming objects in the Stokes regime using micro-swimmers equipped with adaptive decision-making intelligence. Through a reinforcement learning algorithm, our cloaking agents experientially learn an optimal adaptive behaviour policy in the presence of nonlinear flow-mediated interactions. This artificial intelligence enables them to dynamically adjust their swimming actions so as to optimally form, and robustly retain, desired cloaking arrangements around any arbitrarily moving object. In doing so, our smart agents optimally cooperate, such that their overall generated flows not only cancel out the cloaked object's induced fluid disturbances, but also do not disturb it from its original path. Therefore, the presented active cloaking approach is also non-invasive to the subject's motion. We then generalized our methodology, and demonstrated that a cooperative flock of well-adapted cloaking agents can be readily used in a crowded environment to actively cloak any number of arbitrary intruders.

Our study provides a clear road map toward realizing hydrodynamic invisibility cloaks for externally or internally controlled artificial swimming micro-robots (e.g. Li et al. Reference Li, de Avila, Gao, Zhang and Wang2017). This will be significant in the non-invasive intrusion of swimming micro-robots with a broad range of biomedical applications (see, e.g. Nelson, Kaliakatsos & Abbott Reference Nelson, Kaliakatsos and Abbott2010). Moreover, our findings demonstrate the great potential of reinforcement learning in paving the path toward engineering of smart micro-swimmers capable of accomplishing a new class of group objectives. We, therefore, hope that this article will spur further research on this field at the intersection of fluid mechanics and artificial intelligence.