1. Introduction
Machine learning for fluid mechanics has garnered considerable attention in recent years (Brunton, Noack & Koumoutsakos Reference Brunton, Noack and Koumoutsakos2020; Karniadakis et al. Reference Karniadakis, Kevrekidis, Lu, Perdikaris, Wang and Yang2021). Reinforcement learning, one of the machine learning methods, employs an agent that learns to interact with its environment by taking actions and learning from the rewards that it receives, with the aim of maximising the cumulative reward (Sutton & Barto Reference Sutton and Barto2018). It has been successfully applied in fluid mechanics, such as swimming strategy (Qiu et al. Reference Qiu, Huang, Xu and Zhao2020), tracking (Mirzakhanloo, Esmaeilzadeh & Alam Reference Mirzakhanloo, Esmaeilzadeh and Alam2020), schooling strategy (Gazzola et al. Reference Gazzola, Tchieu, Alexeev, de Brauer and Koumoutsakos2016) and turbulence modelling (Bae & Koumoutsakos Reference Bae and Koumoutsakos2022).
Classic reinforcement learning uses iteration methods, dynamic programming, Monte Carlo methods and tabular methods, whereas deep reinforcement learning (DRL) uses a deep neural network (Garnier et al. Reference Garnier, Viquerat, Rabault, Larcher, Kuhnle and Hachem2021). The DRL is suitable for high-dimensional or nonlinear tasks with a very large number of states, as it avoids modelling complex systems and exploits the feature extraction capabilities of deep neural networks. It has been successfully applied to various control tasks, such as walking (Haarnoja et al. Reference Haarnoja, Ha, Zhou, Tan, Tucker and Levine2019a ; Su & Gutierrez-Farewik Reference Su and Gutierrez-Farewik2023; Itahashi et al. Reference Itahashi, Itoh, Fukumoto and Wakuya2024), flying (Reddy et al. Reference Reddy, Celani, Sejnowski and Vergassola2016; Becker-Ehmck et al. Reference Becker-Ehmck, Karl, Peters and van der Smagt2020; d’Apolito & Sulzbachner Reference d’Apolito and Sulzbachner2021) and swimming (Verma, Novati & Koumoutsakos Reference Verma, Novati and Koumoutsakos2018; Chen et al. Reference Chen, Lu, Yang and Hu2022; Rodwell & Tallapragada Reference Rodwell and Tallapragada2023), to enhance propulsion efficiency or reduce drag, and active flow control for flows past cylinders (Fan et al. Reference Fan, Yang, Wang, Triantafyllou and Karniadakis2020; Han, Huang & Xu Reference Han, Huang and Xu2022; Chen et al. Reference Chen, Zhang, Liu and Ding2024).
Specifically, for a walking task, DRL receives joint angles and sensor readings of a minitature robot and gives proper motor positions to make it walk efficiently (Haarnoja et al. Reference Haarnoja, Ha, Zhou, Tan, Tucker and Levine2019a ). For flight control, DRL controls a drone to a marked position based on sensor data (Becker-Ehmck et al. Reference Becker-Ehmck, Karl, Peters and van der Smagt2020). For a swimming task, DRL ascertains the torque acting on the mass centre of a Joukowski aerofoil, enabling it to follow a predefined path (Rodwell & Tallapragada Reference Rodwell and Tallapragada2023). For drag reduction, DRL takes the surrounding flow velocity of a two-dimensional (2-D) cylinder as input and determines the optimal angular velocity that minimises drag torque on the surface (Han et al. Reference Han, Huang and Xu2022).
The DRL applications are based on various reinforcement learning methods, e.g. the actor–critic algorithm (Konda & Tsitsiklis Reference Konda and Tsitsiklis1999) and deep Q-learning (Mnih et al. Reference Mnih2015), and with various neural network structures, e.g. the multilayer perceptron (Rosenblatt Reference Rosenblatt1962; Hornik, Stinchcombe & White Reference Hornik, Stinchcombe and White1989), recurrent neural network (Rumelhart, Hinton & Williams Reference Rumelhart, Hinton and Williams1986), long short-term memory (Hochreiter & Schmidhuber Reference Hochreiter and Schmidhuber1997) and transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017).
These studies have highlighted the potential of machine learning techniques, particularly DRL, in enhancing fluid dynamic performance through active flow control (Fan et al. Reference Fan, Yang, Wang, Triantafyllou and Karniadakis2020; Liu et al. Reference Liu, An, Nohmi, Obuchi and Taira2020; Han et al. Reference Han, Huang and Xu2022; Kubo & Shimizu Reference Kubo and Shimizu2022; Li & Zhang Reference Li and Zhang2022; Rodwell & Tallapragada Reference Rodwell and Tallapragada2023; Vignon, Rabault & Vinuesa Reference Vignon, Rabault and Vinuesa2023; Xie et al. Reference Xie, Zheng, Ji, Zhang, Bi, Zhou and Zheng2023; Xu & Zhang Reference Xu and Zhang2023; Chatzimanolakis, Weber & Koumoutsakos Reference Chatzimanolakis, Weber and Koumoutsakos2024). Most of these applications primarily focused on relatively straightforward tasks, where the actions can swiftly exert their effects on the controlled object, with minor influences from fluid–structure interaction (FSI). Several works have investigated the flow control tasks with delay between action and effect by including history information (Verma et al. Reference Verma, Novati and Koumoutsakos2018; Renn & Gharib Reference Renn and Gharib2022; Singh et al. Reference Singh, van Breugel, Rao and Brunton2023) or using dynamical models (Gazzola, Van Rees & Koumoutsakos Reference Gazzola, Van Rees and Koumoutsakos2012; Brunton, Rowley & Williams Reference Brunton, Rowley and Williams2013). Additionally, there are more open challenges in DRL for fluid mechanics, such as partial observability and robustness to noises (Viquerat et al. Reference Viquerat, Meliga, Larcher and Hachem2022).
Despite the challenges, DRL can be extended to complex flow control problems with significant FSI effects. Jellyfish are believed to have one of the highest locomotion efficiencies among natural creatures (Gemmell et al. Reference Gemmell, Costello, Colin, Stewart, Dabiri, Tafti and Priya2013; Costello et al. Reference Costello, Colin, Dabiri, Gemmell, Lucas and Sutherland2021), which involves body deformation and strong interaction with surrounding flow. Besides, a jellyfish-like robot has potential in ocean exploration (Reaka-Kudla Reference Reaka-Kudla2001; Wölfl et al. Reference Wölfl2019) to understand the ecosystems and discover natural resources in the ocean (DeVries et al. Reference DeVries2019). Previous experiments and simulations found that the jellyfish adopts an optimal locomotion of swimming forwards and consuming less energy (Joshi Reference Joshi2012; Frame et al. Reference Frame, Lopez, Curet and Engeberg2018; Xu & Dabiri Reference Xu and Dabiri2020; Matharu et al. Reference Matharu, Gong, Guntaka, Almubarak, Jin and Tadesse2023). Its swimming pattern and morphology are relatively simple, allowing for good manoeuvrability through controlling the swimming direction (Hoover & Miller Reference Hoover and Miller2015).
Extensive research has been conducted on the forward propulsion mechanisms of jellyfish (Dular, Bajcar & Širok Reference Dular, Bajcar and Širok2009; Hoover, Griffith & Miller Reference Hoover, Griffith and Miller2017; Miles & Battista Reference Miles and Battista2019a , Reference Miles and Battistab ), whereas investigations into their turning capabilities remain relatively scarce (Pallasdies et al. Reference Pallasdies, Goedeke, Braun and Memmesheimer2019). The mechanism of forward propulsion is the formation and shedding of symmetric vortex rings with symmetric muscle contraction (Hoover et al. Reference Hoover, Griffith and Miller2017), which is also similar to the flapping flight mechanism observed in natural birds (Wang & Wu Reference Wang and Wu2010, Reference Wang and Wu2012). The mechanism of turning involves two sets of nerve nets that control different parts of the muscle, enabling the asynchronous contraction of muscles (Pallasdies et al. Reference Pallasdies, Goedeke, Braun and Memmesheimer2019; Hoover et al. Reference Hoover, Xu, Gemmell, Colin, Costello, Dabiri and Miller2021). For a model jellyfish swimming in a 2-D flow, the mechanism of forward propulsion has also been well studied, which involves vortex shedding with acting force on certain parts of the jellyfish muscle with parameter studies on the jellyfish’s diameter, contraction frequency and tentacles (Hoover & Miller Reference Hoover and Miller2015; Miles & Battista Reference Miles and Battista2019a , Reference Miles and Battistab ).
In the present paper, we studied the control and navigation of a flexible jellyfish-like swimmer capable of freely turning in 2-D flows. Once both forward propulsion and turning are accomplished, it is possible for the jellyfish to navigate. In particular, we proposed a challenging DRL task wherein we controlled the motion of the flexible swimmer that can both swim forwards and turn to track specific targets. The locomotion of jellyfish typically involves strong FSI and deformation of objects. Therefore the control of its motion can be challenging due to the intricate influence of wake vortices generated by its previous action. Instead of using history information or a dynamical model to account for such difficulties, we used only instantaneous information for tracking tasks, assuming that jellyfish in a natural environment may not have memory of the past but still have the ability to navigate. Moreover, we improved previous jellyfish muscle models to enable navigation of the flexible swimmer, and introduced action regulation to account for the FSI effect and reduce the difficulty of training and the complexity of the DRL model.
The outline of the paper is as follows. In § 2, we introduce methods of computational fluid dynamics (CFD) and reinforcement learning, and describe set-ups of simulation cases and tracking tasks. In §§ 3 and 4, we demonstrate the capability of DRL for tracking fixed and moving target points in jellyfish-like swimming, respectively. Some discussions are presented in § 5.
2. The DRL
The overall workflow is sketched in figure 1. We first run multiple simulations to collect data of jellyfish-like swimming (figure 1 a). The dataset is then used to train the neural network offline (figure 1 d). The network is tested in various tracking tasks (figure 1 f). Supplementary movies 1 and 2 are available at https://doi.org/10.1017/jfm.2025.10470 illustrate how the swimmer tracks a moving target. Moreover, figure 1(b,c,e) show the jellyfish’s state, action space and decision-making process. The source code of the simulation and training is available on GitHub (https://github.com/YYgroup/DRLjellyfish).
Diagram for the overall workflow. (a) Data obtained from multiple simulations are used for offline training. (b) Geometry and state of the jellyfish-like swimmer. The red parts indicate where the forces are applied. (c) Action space with four actions
$A_i$
,
$i=0,1,2,3$
, representing typical jellyfish actions (from left to right): symmetric forces on the two sides, larger force on the right-hand side, larger force on the left-hand side, and no force. (d) Multiple simulations with random actions (left-hand dashed box) and collection of the experience tuple
$(\boldsymbol{s}_{t},a_{t},r_{t},\boldsymbol{s}_{t+\Delta t},D)$
(right-hand dashed box). (e) The DQN module receives the state vector and outputs a Q-value for each action. The action with the highest Q-value is finally chosen. ( f) The trained agent is tested in various tracking tasks. Supplementary movies 1 and 2 illustrate how the swimmer tracks a moving target.
2.1. Data preparation
The 2-D flow data for the DRL of jellyfish-like swimming are obtained from numerical simulations. The immersed boundary method (Peskin Reference Peskin2002; Tong, Yang & Wang Reference Tong, Yang and Wang2021) is used to treat the fluid–solid coupling at the moving boundary. A unit density, incompressible flow is governed by the Navier–Stokes equations
\begin{align} \frac { \partial \boldsymbol{ u } } { \partial t } + \boldsymbol{ u } \boldsymbol{\cdot }\boldsymbol{\nabla }\boldsymbol{ u } &= - \boldsymbol{\nabla }p + \nu\, {\nabla} ^ { 2 } \boldsymbol{ u } + \boldsymbol{ f }, \end{align}
\begin{align} \boldsymbol{\nabla }\boldsymbol{\cdot }\boldsymbol{ u }&=0, \end{align}
where
$\boldsymbol{ u }$
,
$p$
,
$\nu$
and
$\boldsymbol{f}$
denote the velocity, pressure, kinematic viscosity and body force exerted by a jellyfish-like swimmer.
The immersed boundary is represented by Lagrangian markers. A regularised delta function
$\delta _h$
(Peskin Reference Peskin2002) is employed to interpolate and spread
$\boldsymbol{f}$
between Eulerian and Lagrangian points, whose coordinates are denoted by
$\boldsymbol{x}$
and
$\boldsymbol{X}$
, respectively. The Eulerian force in (2.1) is calculated as
\begin{equation} \boldsymbol{ f } ( \boldsymbol{ x }) = \int _{S} \boldsymbol{ F } ( \boldsymbol{ X })\, \delta _h ( \boldsymbol{ x } - \boldsymbol{ X } )\, {\rm d} \boldsymbol{X}, \end{equation}
where
$\boldsymbol{F}$
denotes the Lagrangian force at
$\boldsymbol{X}$
, and
$S$
is the domain of the immersed boundary. The no-slip condition is satisfied by exerting
$\boldsymbol{F}$
on the immersed boundary. The velocity on the immersed boundary satisfies
\begin{equation} \int _ { \mathcal{D} } \boldsymbol{ u } ( \boldsymbol{ x } )\, \delta _h (\boldsymbol{ x } - \boldsymbol{ X })\, {\rm d} \boldsymbol{ x } = \boldsymbol{ U } _ { b } ( \boldsymbol{ X }), \end{equation}
where
$\mathcal{D}$
denotes the entire fluid domain, and
$\boldsymbol{ U } _ { b }$
is the velocity at Lagrangian points.
The simulations were conducted using the code IBAMR (Griffith et al. Reference Griffith, Hornung, McQueen and Peskin2007), which is a distributed-memory parallel implementation of the immersed boundary method with adaptive mesh refinement for the Cartesian grid. The total number of grid points is of the order of
$10^5$
. Additionally, we conducted convergence tests using double the grid resolution to ensure that the target tracking trajectory converges.
2.2. Deep Q-learning
A reinforcement learning system consists of two parts – the agent and the environment. The agent observes the status of environment and takes an optimal action based on this observation. The environment then changes, and the agent receives a reward by evaluating the action. These steps loop until the end of a learning process.
The deep Q-learning (Mnih et al. Reference Mnih2015), a reinforcement learning method, uses the deep Q-network (DQN) to generate the optimal action. This neural network receives state
$s_{t}$
at time
$t$
as input, and outputs a value
$Q(s_{t},A_i)$
for each action
$A_i$
, representing the highest total reward for the agent after taking action
$A_i$
. This Q-value satisfies the Bellman optimality equation (Sutton & Barto Reference Sutton and Barto2018)
\begin{equation} Q(s_{t},A_{i})=r_{t}+ \max_{A_{\!j}} Q(s_{t+\Delta t},A_{\!j}). \end{equation}
Namely, given state
$s_{t}$
and action
$A_{i}$
, the maximum total reward at
$t$
consists of two parts: one is the reward
$r_{t}$
after taking this action, and the other is the maximum total reward at the next state
$s_{t+\Delta t}$
among all possible actions
$A_{\!j}$
, where
$\Delta t$
denotes the time interval. For clarity, we define
$t^*= t/\Delta t$
, along with current state
$s_{t^*}$
and next state
$s_{t^*+1}$
. Note that the number of available actions is finite in the DQN, and the DQN has been successfully used for navigating tasks (Loisy & Eloy Reference Loisy and Eloy2022). There are several difficulties in a reinforcement learning task – proper forms of state vector and reward function, overall convergence for a Bellman optimality equation, and gap between the convergence and actual performance of the agent. Therefore, multiple training and performance tests are needed for our task.
2.3. Case set-up and action regulation
In our 2-D flow simulation, we adopt the bell-like geometry shown in figure 1(b) for jellyfish-like swimming. The outflow boundary condition (zero velocity gradient) is applied to all boundaries of the computational domain. The domain size is sufficiently large so that the boundary conditions do not influence the simulation results.
In the jellyfish-like locomotion, a pair of sinusoidal forces is applied to the swimmer’s tips (marked in red), as illustrated in figure 1(b). The force densities (force per unit length)
\begin{equation} \boldsymbol{F}_{i}= F_i\sin(2\unicode{x03C0} ft)\,\boldsymbol{\tau }_{i},\quad i=1,2, \end{equation}
acting on the left- and right-hand halves of the swimmer are denoted as
$\boldsymbol{F}_1$
and
$\boldsymbol{F}_2$
, respectively, with the force magnitude
$F_i$
and the unit tangent vector
$\boldsymbol{\tau }_{i}$
on
$S$
. As sketched in figure 1(c),
$F_1$
and
$F_2$
can be different, while they have the same frequency
$f$
and zero initial phase. Other set-up details are similar to those in Hoover & Miller (Reference Hoover and Miller2015), except that the model for the force density due to the swimmer’s elastic deformation is adapted for jellyfish-like swimming.
The model swimmer consists of 159 Lagrangian points. Each point is connected to its neighbouring ones with linear springs with large stiffness to prevent significant displacement among the Lagrangian points. Every set of three neighbouring points is connected with a beam that generates a restoring force on those three points when bent, in order to maintain the curvature of the structure.
The structure of the swimmer is sketched in figure 2. The springs and beams together constitute the muscle and geometry of our model swimmer. For jellyfish-like swimming, the beam needs to resist deformation such as bending, and does not resist translation and rotation, so that the whole swimmer can turn without influence from potentially non-zero internal torque.
The jellyfish-like model swimmer consists of 159 Lagrangian points. Every pair of adjacent points is connected by a spring, and every three adjacent points are connected by a beam. Black solid lines and dashed lines represent springs and beams, respectively. Different colours correspond to different beams. The discrete fibre beam model of three neighbouring points is shown on the right. The three circles represent three neighbouring points of a beam model (red dashed line),
$\boldsymbol{ X}_{\!L},\boldsymbol{ X}_{\!M},\boldsymbol{ X}_{\!R}$
are the displacement vectors, and
$\boldsymbol{ F}_{\!L},\boldsymbol{ F}_{\!M},\boldsymbol{F}_{\!R}$
are the Lagrangian forces of the beam’s left, middle and right points, respectively.
The elastic force density in Hoover & Miller (Reference Hoover and Miller2015) is chosen as
\begin{equation} \boldsymbol{F}_{\!E}(q,t) = \frac {\partial }{\partial q}\left (k_{s}\left |\frac {\partial }{\partial q}\boldsymbol{ X }(q,t)\right |-1\right )\boldsymbol{\tau }(q,t)- k_{b}\frac {\partial ^{4}}{\partial q^{4}}\left (\boldsymbol{ X }(q,t)-\boldsymbol{ X }_{b}(q)\right )\!, \end{equation}
where
$\boldsymbol{ X }(q,t)$
is the Cartesian coordinate of the Lagrangian material point
$q$
,
$\boldsymbol{\tau }(q,t)=(\partial \boldsymbol{ X }/\partial q)/|\partial \boldsymbol{ X }/\partial q|$
is the unit tangent vector,
$k_{b}\geqslant 0$
and
$k_{s}\geqslant 0$
are respectively the bending and stretching stiffnesses of the muscle, and
$\boldsymbol{X}_{b}$
characterises the shape of the swimmer’s body at rest. On the right-hand side of (2.7), the first term represents a linear spring connecting neighbouring Lagrangian points, and the second term represents a linear beam applied on three neighbouring points.
In the present study, the fibre model for the second term is changed from the linear beam to a torsional spring (Battista, Strickland & Miller Reference Battista, Strickland and Miller2017) to incorporate rotation, so that the muscle is more flexible than that in Hoover & Miller (Reference Hoover and Miller2015). When the entire torsional spring rotates around an arbitrary point, the force acting on it rotates accordingly, while the torque applied to it remains zero. Our CFD simulations showed that this change to the muscle model is pivotal for effectively controlling the turning of the jellyfish-like swimmer.
The forces in the torsional spring model are
\begin{align}& \boldsymbol{ F}_{\!L} = k_{b}\varDelta _\vartheta (\boldsymbol{ X}_{\!M}-\boldsymbol{ X }_{\!R})\times \boldsymbol{ e}_{z}, \end{align}
\begin{align}& \boldsymbol{ F}_{\!M} = k_{b}\varDelta _\vartheta (\boldsymbol{ X}_{\!R}-\boldsymbol{ X }_{\!L})\times \boldsymbol{ e}_{z}, \end{align}
\begin{align}& \boldsymbol{ F}_{\!R} = k_{b}\varDelta _\vartheta (\boldsymbol{ X}_{\!L}-\boldsymbol{ X }_{\!M})\times \boldsymbol{ e}_{z}. \end{align}
Here,
$\boldsymbol{ e}_{z} = \boldsymbol{ e}_{x}\times \boldsymbol{ e}_{y}$
is the unit vector perpendicular to the
$x$
–
$y$
plane;
$\boldsymbol{ X}_{\!L},\boldsymbol{ X}_{\!M},\boldsymbol{ X}_{\!R}$
denote the displacement vectors, and
$\boldsymbol{F}_{\!L},\boldsymbol{ F}_{\!M},\boldsymbol{F}_{\!R}$
denote the Lagrangian forces of the beam’s left, middle and right points, respectively (see figure 2). Also,
$\varDelta _\vartheta =\boldsymbol{e}_z\boldsymbol{\cdot }((\boldsymbol{X}_{\!R} - \boldsymbol{X}_{\!M})\times (\boldsymbol{X}_{\!M} - \boldsymbol{X}_{\!L}) - (\boldsymbol{X}_{R0} - \boldsymbol{X}_{M0})\times (\boldsymbol{X}_{M0} - \boldsymbol{X}_{L0}))$
denotes the change in the angle between
$\boldsymbol{X}_{\!R} - \boldsymbol{X}_{\!M}$
and
$\boldsymbol{X}_{\!M} - \boldsymbol{X}_{\!L}$
, where
$\boldsymbol{ X}_{L0},\boldsymbol{ X}_{M0},\boldsymbol{ X}_{R0}$
denote the corresponding displacement vectors at the initial time when the swimmer is at rest.
The torque referenced to the mass centre
$\boldsymbol{ X}_{C} = (\boldsymbol{ X}_{\!L} + \boldsymbol{ X}_{\!M}+\boldsymbol{ X}_{\!R})/3$
, or any arbitrary point, is
\begin{align} \boldsymbol{T} &= (\boldsymbol{ X}_{\!L}-\boldsymbol{ X}_{C})\times \boldsymbol{ F}_{\!L} + (\boldsymbol{ X}_{\!M}-\boldsymbol{ X}_{C})\times \boldsymbol{ F}_{\!M} + (\boldsymbol{ X}_{\!R}-\boldsymbol{ X}_{C})\times \boldsymbol{ F}_{\!R}\nonumber \\& =\boldsymbol{ X}_{\!L}\times \boldsymbol{ F}_{\!L} + \boldsymbol{ X}_{\!M}\times \boldsymbol{ F}_{\!M} + \boldsymbol{ X}_{\!R}\times \boldsymbol{ F}_{\!R}. \end{align}
Substituting (2.8)–(2.10) into (2.11) yields
$\boldsymbol{T}=0$
.
For comparison, the discrete restoring forces of the beam model used in (2.7) are
\begin{align}& \boldsymbol{ F}_{\!L} = -k_{b}(\boldsymbol{ X}_{\!L}+\boldsymbol{ X }_{\!R}-2\boldsymbol{ X}_{\!M}-\boldsymbol{X}_0), \end{align}
\begin{align}& \boldsymbol{ F}_{\!M} = 2k_{b}(\boldsymbol{ X}_{\!L}+\boldsymbol{ X }_{\!R}-2\boldsymbol{ X}_{\!M}-\boldsymbol{X}_0), \end{align}
\begin{align}& \boldsymbol{ F}_{\!R} = -k_{b}(\boldsymbol{ X}_{\!L}+\boldsymbol{ X }_{\!R}-2\boldsymbol{ X}_{\!M}-\boldsymbol{X}_0), \end{align}
with
$\boldsymbol{X}_0 = \boldsymbol{ X}_{L0}+\boldsymbol{ X }_{R0}-2\boldsymbol{ X}_{M0}$
. Substituting (2.12)–(2.14) into (2.11) yields a non-zero anti-rotational torque
\begin{equation} \boldsymbol{T}=k_b(\boldsymbol{ X}_{\!L}+\boldsymbol{ X }_{\!R}-2\boldsymbol{ X}_{\!M})\times \boldsymbol{X}_0. \end{equation}
If the three points of a beam at its initial rest state are all rotated around an arbitrary point, then (2.15) gives a non-vanishing anti-rotational torque.
Therefore, the torsional spring model in (2.8)–(2.10) refrains from imparting extraneous torques, notably anti-rotational forces, onto the swimmer’s flexible structure. This model makes the swimmer’s structure more flexible for mimicking jellyfish-like swimming.
The simulation parameters are listed in table 1. All parameters are non-dimensionalised by the reference length
$l=1$
m and reference time 1 s. The diameter
$d_0$
of the jellyfish-like swimmer is defined as the distance between the left and right ends of the swimmer at rest. The frequency is selected based on the natural behaviour of jellyfish (e.g. Aurelia aurita). The stiffnesses are referred to those in Hoover & Miller (Reference Hoover and Miller2015). Since we use a muscle model different from that in Hoover & Miller (Reference Hoover and Miller2015), these parameters were tuned so that the swimmer moves relatively fast as a real jellyfish.
Parameters in the simulation for jellyfish-like swimming.
As listed in table 2, we divide a period
$t = 0{-}T$
of the sinusoidal force pair
$(F_1,F_2)$
applied to the swimmer into four quarters, with
$T=1/f$
. The force pair is non-dimensionalised by the reference force density 1
$\mathrm{N}\ \mathrm{m}^{-1}$
. During each quarter, a DQN is employed to process the current state and produce
$(F_1,F_2)$
. The values of
$(F_1,F_2)$
remain the same during each quarter, until new values of
$(F_1,F_2)$
are given in another quarter. In order to reduce the training complexity, we restrict the choices for
$(F_1,F_2)$
to
$(0.003,0.003)$
,
$(0.001,0.003)$
,
$(0.003,0.001)$
and
$(0,0)$
. These choices are labelled as actions
$A_i$
with
$i=0,1,2,3$
, as listed in table 2 and illustrated in figure 1(c).
Action regulation: forces are applied only during the first and third quarters of each period, except for
$A_0$
and
$A_3$
, which can last for the next quarter following specific actions.
We introduce an action regulation (see table 2) in the performance test to mimic the jellyfish locomotion and achieve high propulsion efficiency and manoeuvrability: forces are applied only during the first and third quarters of each period, except for
$A_0$
and
$A_3$
(with symmetric force magnitudes), which can last for the next quarter following specific actions. In other words, the action taken at the second and fourth quarters of a period is decided by the previous action with the regulation rule, instead of given by the trained agent. This action regulation is similar to the burst-and-coast strategy used in optimal intermittent swimming of fish (Li et al. Reference Li, Ashraf, François, Kolomenskiy, Lechenault, Godoy-Diana and Thiria2021) and jellyfish-like swimming (Kang et al. Reference Kang, Gao, Han, Cui and Lu2023).
To evaluate the efficiency of model performance, we use the total time taken for the swimmer to complete a task. The task is considered ended when the swimmer reaches the target or the swimmer gets too close to the boundary of the simulation domain.
2.4. Training
As sketched in figure 1(b), the swimmer’s state is characterised by a 12-dimensional vector
\begin{equation} \boldsymbol{s}=(x_{1},y_{1},x_{2},y_{2},x_{3},y_{3},u_{1},u_{2},d,\theta ,\varOmega ,n), \end{equation}
where
$(x_{i},y_{i}),\ i=1,2,3$
, denote the coordinates of the swimmer’s left, middle and right endpoints referenced to its mass centre,
$\boldsymbol{u}_r = (u_{1},u_{2})$
is the velocity of the mass centre relative to target,
$d$
is the distance between the mass centre and target,
$\theta$
is the angle between the swimmer’s symmetric axis and the line segment connecting the mass centre and target,
$\varOmega$
is the angular velocity of the symmetric axis, and
$n=0,1,2,3$
indicates the quarter of a period. Note that the symmetric axis is defined as the line connecting the midpoint of the swimmer and its mass centre, and it does not necessarily imply that the swimmer geometry is symmetric all the time. The state vector in (2.16) encapsulates the information perceived by a swimming jellyfish when it chooses to navigate to a specific location within its natural environment, and the choice of state variables is important for the further training and tracking performance.
We choose a function
$r(\boldsymbol{s},a) = -\min(\theta ^{2},3)+\mathcal A\,\mathrm{clip}(v,-0.1,0.1)-\mathcal B d$
to characterise the reward received at state
$\boldsymbol{s}$
after taking action
$a$
. Here,
$v$
is the projection of the mass centre’s velocity onto the line segment connecting the mass centre and target point; constants
$\mathcal A$
and
$\mathcal B$
are tuned during the training so that these three terms have the same order of magnitude; and the clip function
$\mathrm{clip}(x,b,c)$
returns
$x$
,
$b$
and
$c$
for
$b\leqslant x\leqslant c$
,
$x\lt b$
and
$x\gt c$
, respectively. Figure 3 shows the contour of
$r(\boldsymbol{s},a)$
with typical parameters
$\mathcal A=20$
,
$\mathcal B=10$
and
$d=0.1$
. This reward function encourages alignment of the swimmer’s symmetric axis with the target, velocity towards the target, and short distance between the swimmer and target.
Contour of the reward function
$r(\boldsymbol{s},a)$
with typical parameters
$\mathcal A=20$
,
$\mathcal B=10$
and
$d=0.1$
.
The DQN agent receives state and estimates
$Q^{*}(\boldsymbol{s},a)$
. The loss function is defined by
\begin{equation} \mathrm{loss} = (Q^{*}(\boldsymbol{s},a)-(r(\boldsymbol{s},a)+(1-D)\max_{A_{\!j}} Q^{*}(\boldsymbol{ s}^{\prime },A_{\!j})))^{2}, \end{equation}
where
$D=1$
if the task is ended, and
$D=0$
otherwise;
$\boldsymbol{s}^\prime$
denotes the next state, and its subscript for time is omitted for clarity. This loss function optimises the DQN parameters to satisfy the Bellman optimality equation in (2.5) with a slight modification. The term
$(1-D)$
means that the Q-value at the penultimate state is equivalent to the reward obtained. More details on reinforcement learning and training are provided in Appendix A, including the optimisation formulation, dataset preparation and DQN algorithm.
Note that the action regulation was imposed only in the performance test, where the actions at the second and fourth quarters of a period were imposed by the regulation role in table 2 instead of given by the trained agent.
3. Tracking a fixed target
3.1. Tracking performance
We first examine the DQN performance in tracking a fixed target. In this case, the swimmer can adjust its movement direction gradually towards the target. As shown in figure 4(a–c), we set three target points. The swimmer trajectories suggest that the DQN has learned how to make turns using the combination of symmetric and asymmetric actions.
(a–c) Swimmer’s trajectories for tracking fixed targets (front left, straight and front right). Red dots represent the start and target points. (d) The process of making a right turn by the swimmer with action regulation, along with the contour of the vorticity magnitude, at
$t/T=0,0.2,0.4,0.6,0.8,1$
. The action sequence is
$A_1$
,
$A_3$
,
$A_0$
and
$A_0$
. (e) Swimmer’s trajectory for tracking the front right target, along with the contour of the vorticity magnitude. ( f) Temporal evolution of
$\theta$
and
$\varOmega$
for tracking the front right target.
Figures 4(d) and 4(e) illustrate how the swimmer moves towards the target point, where the symmetric and asymmetric actions are marked in red and blue, respectively, on the swimmer trajectory. The first action with asymmetric forces causes the swimmer to turn right by slightly rotating its body, forming a pair of asymmetric wake vortices. Under the action regulation (see table 2), the subsequent actions with symmetric forces make the swimmer move straight to the target with a small angular velocity
$\varOmega$
and decreasing
$\theta$
(see figure 4
f), and then make the orientation of its symmetric axis turns slowly to align with its velocity, with the decrease of
$\alpha$
(see figure 1
b). The decreasing of both
$\theta$
and
$\alpha$
is ideal for tracking a fixed target, because the swimmer can adjust to proper orientation and then swim forwards. This adjustment is critical, which will be shown in tracking of a moving target.
3.2. Effect of action regulation
The detailed process of vorticity formation during the right turn of a swimmer is illustrated in figure 4(d), with the sequence of actions
$A_1,A_3,A_0,A_0$
in a period from
$t=0\,$
to
$t=T$
. First, the force exerted on the right-hand side is larger than that on the left in action
$A_1$
, generating a stronger vortex at the right trailing edge. This vortex then induces a leftward rearward velocity on the right-hand half of the swimmer, effectively dragging the entire body towards the right.
The turning process depicted in figure 4(d) emphasises the importance of action regulation. By applying forces only in the first and third quarters of each cycle, the swimmer can execute a swift turn within a single period, and the vortices generated in the two quarters have enough time to shed off from the trailing edges within the second and fourth quarters. Experimental studies showed that the starting vortex at the trailing edge for a impulsively started wing rapidly escapes to the wake within
$t/T=0.167$
(Huang et al. Reference Huang, Wu, Jeng and Chen2001). This motion is similar to the flapping by the muscle of the jellyfish-like swimmer. Therefore, the time
$T/4$
is enough for these vortices to shed off, with minor influence on lateral motion of the swimmer.
By contrast, frequent exertion of forces can make the swimmer difficult to control for the present 2-D model. Without the action regulation, the asymmetric forces can be applied in consecutive quarters as the action sequence
$A_1$
,
$A_1$
,
$A_0$
and
$A_0$
in figure 5. As a result, the vortices generated in the consecutive quarters can interfere with each other. In contrast to figure 4(d), where the swimmer’s geometry becomes nearly symmetric, the vortex (red patch highlighted in the blue circle at
$t/T = 0.4$
) formed by the right-hand side of the swimmer stays at the trailing edge when the next symmetric action is executed. This interaction between the previously formed vortex and the newly generated vortex leads to an asymmetric shape of the swimmer. The neighbouring vortices (red and blue patches highlighted in the blue circle at
$t/T = 0.7$
, 0.8 and 1) with opposite signs cancel out each other, which significantly reduces thrust and makes the motion difficult to control. Besides, the regulation prevents unnatural behaviours of the swimmer, such as forcing for a full period, that pull the bell-shaped body apart and lead to a very large diameter. As the regulation mimics the contracting and expanding motion of a jellyfish, it has time to recover the swimmer’s shape.
The process of making a right turn by the swimmer without action regulation, along with the contour of the vorticity magnitude, at
$t/T=0,0.2,0.4,0.7,0.8,1$
. The action sequence is
$A_1$
,
$A_1$
,
$A_0$
and
$A_0$
, with asymmetric forces in the second quarter. The trailing vortices with opposite signs are generated in different quarters. They cancel out with each other (highlighted in the blue circle), reducing the propulsion efficiency.
4. Tracking a moving target
4.1. Circular target trajectory tracking
We then examine the DQN agent performance in tracking a moving target. Here, the target moves much more slowly than the characteristic speed of the swimmer. Thus tracking the moving target is equivalent to tracking a fixed point at each individual time step, and the DQN agent trained by tacking fixed targets can be applicable.
The target trajectory (green curve) in figure 6(a) is part of a circle centred at
$(0.8, 0.8)$
with radius
$0.4$
. The target point starts from
$(1.2, 0.8)$
and moves anticlockwise with angular velocity
$\unicode{x03C0} /120 \ \mathrm{rad}\ \mathrm{s}^-{^1}$
or
$1.5^\circ\ \mathrm{s}^{-1}$
, which is speed
$0.0105$
in figure 6(a).
(a) Trajectories of the swimmer’s mass centre (red–blue line) and the moving target (green line). The red and blue line segments denote the actions with and without forces, respectively. Six snapshots are presented with the swimmer shape (black curves), vorticity distribution around the swimmer (colour contours), and the target point positions (green dots) at the six time points. (b) Temporal evolution of
$\theta$
and
$\varOmega$
for tracking the moving target. The circular target trajectory tracking is also presented in supplementary movie 1.
The swimmer effectively tracks the moving target, indicating that the trained neural network is capable of adapting to dynamic scenarios. Figure 6(a) and supplementary movie 1 illustrate how the swimmer tracks a moving target in detail, where the red trajectory denotes the action with forces, and the blue one denotes the action without force. Initially, the target is on the front right of the swimmer. In phase 1, the swimmer turns right and drifts for a certain period (marked by the blue trajectory) to align its symmetric axis direction with the velocity direction. It then swims straight towards the target (marked by the red trajectory). In phase 2, as the target moves left, the swimmer chooses to drift again (marked by the blue trajectory), allowing it to slow down and adjust its symmetric axis direction towards the moving target. In phase 3, the swimmer at low speed already turns its symmetric axis towards the target and starts to move forwards, and then makes further adjustments to reduce
$\theta$
.
These observations suggest that the swimmer has acquired the skill of employing drifting (action
$A_3$
) as a strategy to adjust its direction with information on the velocity and angular velocity from its state vector, enabling it to efficiently follow the moving target. Moreover, the large radius of the circular target trajectory provides the swimmer with enough space and time to adjust its swimming direction. Figure 1(e) sketches the decision process at
$t=0$
. The DQN takes the state vector as the input and gives the Q-value for each action, and the action with the largest Q-value is chosen.
Figure 6(b) plots the temporal evolution of
$\varOmega$
(angular velocity) and
$\theta$
(angle). In general,
$\theta$
and
$\varOmega$
exhibit opposite signs for the majority of the time, indicating that the swimmer is actively adjusting its direction to swim towards the target. The delay in the change of
$\varOmega$
relative to
$\theta$
arises from the fact that once an action is performed, it does not instantaneously influence the swimmer’s motion. The swimmer learns to drift, waiting for the previous action to take effect, then adjust its direction. Initially, the swimmer employs asymmetric forces to generate an angular velocity, swiftly adjusting its direction towards the target in a short time. Then the swimmer swims straight forwards with minor changes in
$\theta$
. When
$\theta$
becomes large, the swimmer finds the deviation and responds by producing a counter
$\varOmega$
to reduce
$\theta$
.
4.2. Figure-eight target trajectory tracking
The trajectory of the target in § 4.1 is a circle with a relatively large radius, moving at a low speed. This provides the swimmer with sufficient time and space to adjust its velocity direction to align with its symmetric axis.
Figure 7 and supplementary movie 2 present a more challenging tracking task – the target moves along a figure-eight trajectory, characterised by a smaller radius and higher speed than those in the previous case. Specifically, the radius of each circle is 0.3, and the angular velocity of the target that moves along it is
$\unicode{x03C0} /45 \ \mathrm{rad}\ \mathrm{s}^-{^1}$
or
$4^\circ\ \mathrm{s}^{-1}$
. Consequently, the target’s speed is
$0.021$
, which is twice the speed of the target in figure 6(a).
Trajectories of the swimmer’s mass centre (blue line) and the moving target (green line). Red arrows represent the direction of the target’s movement. Blue and green dots with different numbers represent the positions of the swimmer and target at the same moments, at
$t/T=9,18,\ldots ,90$
. Red dots represent the starting points of the swimmer and target. The figure-eight target trajectory tracking is also presented in supplementary movie 2.
Given the delay between the swimmer’s motion and action, the swimmer’s trajectory does not strictly align with that of the target. Instead, it goes through a process of deviating and adjusting its direction. Figure 8(a) plots the evolution of
$\varOmega$
and
$\theta$
. Similar to figure 6(b),
$\theta$
and
$\varOmega$
exhibit opposite sign for the majority of the time. The event for the jumps of
$\theta$
around
$t/T=45$
is that the swimmer reaches and surpasses the target as shown in figure 8(b). These behaviours indicate that the swimmer’s actions are based solely on its current state, without prior knowledge of the target’s motion. Note that the primary objective is to track the target, and the swimmer accomplishes this task for the first time at station 5 in figure 7. Then the swimmer restarts to track the moving target.
(a) Temporal evolution of
$\theta$
and
$\varOmega$
in the figure-eight target trajectory tracking. Here,
$\theta$
changes significantly around
$t/T=45$
(near station 5 in figure 7) where the swimmer surpasses the target. (b) The swimmer positions relative to the target when it surpasses the target at
$t/T=40,44,45,50$
. Green dots and lines represent the position and trajectory of the target, respectively. The swimmer first moves close to the target, at which point the target changes its trajectory from clockwise to anticlockwise. This S-shaped manoeuvre makes it challenging for the DQN agent, which relies solely on the current state, to keep up, resulting in the swimmer temporarily deviating further from the target.
The trajectory of the swimmer for tracking the first circle (stations 1–5) aligns relatively well with that of the target. After reaching the target at station 5, the two trajectories begin to show notable discrepancy. Following a period of questionable manoeuvres, the swimmer decelerates and manages to reorient itself correctly and catch up at stations 6–8. Difficulties arise at stations 8–9, where the swimmer’s velocity turns excessively, leading to further deviation. The issues in this complex target trajectory tracking are discussed in detail in Appendix B. Nonetheless, the swimmer ultimately adjusts and catches up at stations 9–10. Overall, the swimmer still performs satisfactory tracking by completing an approximate figure-eight trajectory.
Figure 9 plots the input power of the swimmer:
\begin{equation} P(t)=\sum _{\textit{left}}F_{1}\sin(2\unicode{x03C0} ft)\,v_{\textit{left},i}+\sum _{\textit{right}}F_{2}\sin(2\unicode{x03C0} ft)\,v_{\textit{right},i}, \end{equation}
where
$v_{\textit{left},i}$
and
$v_{\textit{right},i}$
represent the velocities projected on the tangent vectors of left and right parts of the swimmer muscle, respectively. The average total energy spent in a period of sinusoidal force application is
$1.84\times 10^{-4}$
J, and the average input power is
$0.92\times 10^{-4}$
W, which can be used as a reference for building a robot jellyfish. To account for the efficiency (10–30 %) of electric machinery (Wang Reference Wang2014) and energy consumption such as in sensor devices (Xu et al. Reference Xu, Li, Fang, Wang, Hong, Yang, Han, Zhao and Wang2024) with
$5\times 10^{-3}\,$
W, the power requirement for a robot jellyfish is estimated to be
$0.01$
W, which is close to that of a jellyfish in natural environment (Xu & Dabiri Reference Xu and Dabiri2020).
Input power of the swimmer during the figure-eight target trajectory tracking.
We then examine the actions output by the agent. Figure 10 shows the actions in terms of
$\theta$
and
$\varOmega$
. They are divided into two groups,
$d\lt d_{0}$
and
$d\geqslant d_{0}$
. As shown in figure 10(a), as the swimmer maintains a substantial distance from the target, the agent exhibits a pronounced preference for certain actions. Action
$A_0$
, with symmetric forcing, predominates for a large portion of the time when
$\theta$
is small. This preference is particularly evident when both
$\theta$
and
$\varOmega$
are small (highlighted by the dashed box in figure 10
a), reflecting that the agent prefers straightforward progression with negligible self-rotation under these conditions.
Scatter plots of
$\theta$
and
$\varOmega$
with different actions output by the DQN agent in the figure-eight target trajectory tracking. The data in the first and third quarters with action regulation are not included. (a) Scatter plot for
$d\geqslant d_0$
. The data points in the dashed box indicate that the agent frequently selects
$A_0$
for small
$\theta$
. (b) Scatter plot for
$d\lt d_0$
.
Action
$A_3$
ranks second in the preference, and spans a broader spectrum of
$\theta$
, corresponding to states characterised by varying velocities. This preference implies that the DQN learns a strategy of drifting that enables the swimmer to naturally decelerate or adjust its trajectory by ceasing to apply forces, especially when moving at high speeds in a misaligned direction. Conversely,
$A_1$
and
$A_2$
are predominantly selected when
$\theta$
has finite positive and negative values, respectively, indicating that the swimmer recalibrates its symmetry axis towards the target.
As shown in figure 10(b), when the swimmer closes in on the target, however, the action selection becomes less predictable. Within the vicinity defined by
$d\lt d_0$
, the swimmer either approaches immediate proximity to or overshoots the target. In this critical range, determining the optimal action becomes increasingly challenging, as any manoeuvre can precipitate substantial alterations in
$\theta$
, compounded by the lack of information regarding the target’s trajectory. The agent’s decision-making process, therefore, becomes more stochastic.
These observations highlight the current challenges of the DQN in handling complex scenarios where precise adjustments and quick responses are required to track a continuously moving target. Further fine tuning of the training process is necessary to overcome these challenges, such as using an extra network to predict the target’s trajectory, and adding extra action regulations.
4.3. Effects of the Reynolds number on tracking performance
We investigate effects of
$Re$
on tracking performance by varying viscosity. Note that the agent was trained at
${Re}=100$
(as listed in table 1) but tested at
${Re}=20$
and 2000. The trajectories of tracking fixed targets in figure 11 show that the agent is still able to track the target at different
$Re$
, despite viscosity having an influence on the tracking performance and optimal move.
Trajectories of tracking a fixed target (green dot) at (a)
${Re}=20$
and (b)
${Re}=2000$
, along with the vorticity contour at the ending time. The red and blue line segments denote the actions with and without forces, respectively.
As shown in figure 11(a), the drag from fluid viscosity is significantly increased at lower
${Re} = 20$
, resulting in lower speed of the jellyfish and longer time to track. The tracking time
$37.25\ T$
is more than three times that at
${Re} = 100$
. The red–blue trajectory line, marked by small zigzag patterns, indicates that the jellyfish exerts more effort to turn right. The surrounding vorticity of the swimmer is weak due to faster vorticity dissipation at low
$Re$
. This vorticity suppression simplifies the control task by reducing vortex interactions. The high viscosity of the fluid restricts the range of the swimmer’s movements. This decreases the propulsion speed while enhancing manoeuvrability by reducing the amplitude of motion.
As shown in figure 11(b), the drag from viscosity is significantly decreased and the swimmer moves faster at higher
${Re} = 2000$
. The tracking time
$8.75\ T$
is 75 % of that at
${Re} = 100$
. The moving swimmer generates many more and stronger shedding vortices. These vortices dissipate more slowly and shed faster and further than those at
${Re} = 100$
. The turbulent wake causes the swimmer to deviate slightly from the optimal tracking trajectory, but the agent is able to quickly adjust to finally catch the target. Thus the swimmer exhibits a larger range of motion at high
$Re$
, leading to faster propulsion but reduced controllability.
5. Conclusions
We develop a deep reinforcement learning method for tracking a moving target in jellyfish-like swimming. This control strategy, with a DQN agent, is based on the instantaneous state of a 2-D jellyfish-like swimmer. The swimmer is a flexible object with a muscle model of torsional spring that has no torque introduced when deformed. We apply a pair of sinusoidal forces to the muscle part of the swimmer, and adjust their magnitudes to control the swimmer’s motion.
The data for the swimmer motion in fluid are obtained from the simulation using the immersed boundary method, and they are utilised to train and validate the DQN agent. To make swimming more natural and reduce the difficulty of training, we introduce an action regulation. This regulation mitigates the cancellation of wake vortices generated by the swimmer’s flapping motion by suspending the application of forces during specific time periods.
Equipped with the DQN agent and action regulation, the swimmer demonstrates capability to track both fixed and moving targets. The action regulation reduces the influence from the flow induced by previous actions, enabling the swimmer to output action to control its movement based only on its current state.
In the basic test for tracking a fixed target, the swimmer with the DQN agent performs much more efficiently than that with the baseline strategy outlined in Appendix A, and its tracking performance is robust at a range of Reynolds numbers. In the challenging test for tracking a moving target, there is an inherent latency between the action for exerting forces and the response for rotating the swimmer’s body, due to the hydrodynamic interactions between the shedding vortices and the swimmer’s own locomotion. The swimmer is still able to dynamically adjust its course based on its instantaneous state. This resilience highlights the DQN agent’s robust decision-making capabilities, enabling it to counteract external perturbations and maintain a focused pursuit trajectory.
When confronted with more intricate target trajectories, such as the figure-eight curve, the agent’s performance may exhibit certain limitations. This observation implies the complex interplay of factors that govern natural navigation strategies, particularly those employed by organisms in their native environments. For example, the jellyfish leverages a multifaceted array of sensory inputs, encompassing both internal state cues (e.g. velocity) and external environmental indicators (e.g. fluid dynamics) to navigate precisely towards designated locations.
The present study advances the capabilities of a 2-D jellyfish-like swimmer beyond forward swimming, explores its control strategy, and broadens the application of reinforcement learning in fluid dynamics. Given that the swimmer’s motion involves significant FSI, the agent may require additional information to take optimal actions. To enhance the tracking performance, the swimmer could leverage more information from its surrounding fluid flow, predict the target’s position, and incorporate data from previous states. Besides, more complex network architectures, such as RNN (Rumelhart et al. Reference Rumelhart, Hinton and Williams1986) and transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), could be employed to capture more features and temporal dependencies from past and current states to achieve a higher level of control strategy.
Supplementary movies
Supplementary movies are available at https://doi.org/10.1017/jfm.2025.10470.
Acknowledgements
Numerical simulations were carried out on the TH-2A supercomputer in Guangzhou, China.
Funding
This work has been supported by the National Natural Science Foundation of China (grant nos 12525201, 12432010 and 12588201), and the Xplore Prize.
Declaration of interests
The authors report no conflict of interest.
Author contributions
Y.Y. and Y.C. designed the research. Y.C. performed the research. All the authors discussed the results and wrote the paper. All the authors have given approval for the paper.
Appendix A. Details on reinforcement learning and training
The tracking task is formulated as a sequential decision problem solved using reinforcement learning. The decision-making is modelled as a Markov decision process where the agent makes a decision based solely on the current state (Sutton & Barto Reference Sutton and Barto2018).
The trajectory of the swimmer can be written as
\begin{equation} \varGamma _{t}=(s_{0},a_{0},r_{0},\ldots ,s_{t^*-1},a_{t^*-1},r_{t^*-1},s_{t^*},a_{t^*},r_{t^*}), \end{equation}
where
$s_{t^*}$
,
$a_{t^*}$
and
$r_{t^*}$
denote the state, the action taken by the agent, and the reward received at a given time
$t=t^*\Delta t$
, respectively. In the present study,
$\Delta t=T/4$
, corresponding to the time interval for force application. The agent takes action
$a_0$
at
$t=0$
, then it receives reward
$r_0$
, and its state changes from
$s_0$
to
$s_1$
. The same process continues until time
$t$
.
The action
$a_{t^*}$
taken by the agent is determined solely by the current state
$s_{t^*}$
and the agent’s policy
$\unicode{x03C0} (\varTheta , {\cdot })$
, which is implemented as a neural network with parameters
$\varTheta$
. The reward function
$r_{t^*}=r_{t^*}(s_{t^*},a_{t^*})$
depends on the current state and action. The goal for our DQN agent is to maximise the expected discounted sum of rewards over time by adjusting
$\varTheta$
(Sutton & Barto Reference Sutton and Barto2018). This can be expressed as an optimisation problem
\begin{equation} \max _{\unicode{x03C0} }\mathrm{E}_{s_{0},a_{0},s_{1},a_{1},\ldots }\left (\sum _{t^*=0}^{\infty } \gamma ^{t^*}\,r(s_{t^*},a_{t^*}) \right )\!, \end{equation}
where
$\mathrm{E}_{s_{0},a_{0},s_{1},a_{1},\ldots }$
denotes the expected value taken over the trajectory of states and actions
$(s_0, a_0, s_1, a_1, \ldots )$
that the agent encounters. The action
$a_{t^*}$
is selected according to policy
$\unicode{x03C0}$
, and
$\gamma \in (0, 1]$
is a discount factor that assigns less importance to the reward received in the future. We chose this model, considering that the jellyfish has a simple neural structure (Katsuki & Greenspan Reference Katsuki and Greenspan2013; Pallasdies et al. Reference Pallasdies, Goedeke, Braun and Memmesheimer2019), and the current-state information can be sufficient for our model swimmer to navigate and also makes the agent training simpler.
The training dataset consists of approximately 40 000 samples called experience tuples. Each sample is described by
\begin{equation} (\boldsymbol{s}_{t^*},a_{t^*},r_{t^*},\boldsymbol{s}_{t^*+1},D), \end{equation}
where
$\boldsymbol{s}_{t^*}$
,
$a_{t^*}$
,
$r_{t^*}$
,
$\boldsymbol{s}_{t^*+1}$
and
$D$
are the current state, the choice of action, the reward received, the next state, and a flag indicating whether it is the end state of a task, respectively. Here,
$D=1$
denotes the end state when the distance between mass centre and target point is smaller than a certain value
$d_{c}$
, and
$D=0$
otherwise.
Our training dataset is partly computed from a raw dataset, which is obtained by running simulations with randomly chosen actions. The state vector
\begin{align} \boldsymbol{s}_{t^*} ={}&\left(x_{1}^{t^*-1},y_{1}^{t^*-1},x_{2}^{t^*-1},y_{2}^{t^*-1},x_{3}^{t^*-1},y_{3}^{t^*-1},x_{sw}^{t^*-1},y_{sw}^{t^*-1},x_{ta}^{t^*-1},y_{ta}^{t^*-1},n^{t^*-1},\right .\nonumber \\ &\left .\ x_{1}^{t^*},y_{1}^{t^*},x_{2}^{t^*},y_{2}^{t^*},x_{3}^{t^*},y_{3}^{t^*},x_{sw}^{t^*},y_{sw}^{t^*},x_{ta}^{t^*},y_{ta}^{t^*},n^{t^*}\right) \end{align}
of samples in the raw dataset is slightly different from that defined in (2.16), where the superscripts
$t^*-1$
and
$t^*$
denote the data from previous and current time steps, respectively. Here,
$(x_{sw},y_{sw})$
represents the coordinates of the swimmer’s mass centre, and
$(x_{ta},y_{ta})$
represents the coordinates of the target; other variables are the same as in (2.16).
This extended state allows the training dataset to be generated from simulations with multiple different target locations
$(x_{ta}, y_{ta})$
, and enables the generation of large and diverse training datasets. The large-scale datasets facilitate the DQN agent learning a robust and adaptable policy for various tracking tasks.
The other variables in (2.16) at time
$t$
are calculated as follows:
\begin{align} \big(u_{1}^{t^*},u_{2}^{t^*}\big)&=\boldsymbol{ \xi }^{t^*} - \boldsymbol{ \xi }^{t^*-1},\nonumber\\ \theta ^{t^*} &= \mathrm{arccos}\left (\frac {\boldsymbol{ \xi }^{t^*}\boldsymbol{\cdot }\boldsymbol{ \eta }^{t^*}}{|\boldsymbol{ \xi }^{t^*}||\boldsymbol{ \eta }^{t^*}|}\right )\times\mathrm{sign}\big(\big(\boldsymbol{ \xi }^{t^*}\times \boldsymbol{ \eta }^{t^*}\big)\boldsymbol{\cdot}\boldsymbol{e}_{z}\big),\nonumber\\ \varOmega ^{t^*} &= \theta ^{t^*} - \theta ^{t^*-1}, \end{align}
with
\begin{align} \boldsymbol{ \xi }^{t^*} &= \big(x_{ta}^{t^*},y_{ta}^{t^*}\big)-\big(x_{sw}^{t^*},y_{sw}^{t^*}\big),\nonumber\\ \boldsymbol{ \eta }^{t^*}&= \big(x_{2}^{t^*},y_{2}^{t^*}\big) - \big(x_{sw}^{t^*},y_{sw}^{t^*}\big). \end{align}
Scatter plot of
$d$
and
$\theta$
. Blue and red dots represent samples from the raw dataset and the synthetic training dataset, respectively.
The training dataset consists of two parts. One part consists of samples directly computed from the raw dataset without changing
$(x_{sw},y_{sw})$
. The other, the synthetic part, is computed with varying
$(x_{sw},y_{sw})$
by shifting the target in the raw dataset, and without CFD simulations. The
$d$
–
$\theta$
distributions of the samples in the raw and synthetic data in the training dataset are shown in figure 12. For reference, the diameter
$d_0$
of the jellyfish-like swimmer (the distance between the left and right ends of the swimmer at rest) is 0.1 (see table 1). In the raw dataset (blue dots), sample points spread over the whole range of
$\theta$
and concentrate in
$\theta \gt 0^\circ$
with mostly
$d/d_0\lt 6$
due to the finite size of the simulation domain. To balance this concentration, we vary
$(x_{sw},y_{sw})$
so that the synthetic part (red dots) has more samples with
$\theta \lt 0^\circ$
, which complements the unbalance of
$\theta$
in the training dataset. This data augmentation helps to improve the lack of exploration of state space, and reduces computational cost for sampling from CFD simulations.
The training process is outlined in figure 1. The data generated via simulations are utilised in offline training. The performance of the trained DQN agent is then evaluated. Since the convergence of reinforcement learning training is generally not guaranteed, multiple training sessions and evaluations are required.
In our implementation, we employed a combination of two variants of the DQN, the duel DQN (Wang et al. Reference Wang, Schaul, Hessel, Hasselt, Lanctot and Freitas2016) and the double DQN (Van Hasselt, Guez & Silver Reference Van Hasselt, Guez and Silver2016), facilitating the fast convergence and training stability. The duel DQN is used as the backbone network structure, where the network learns to estimate both the state value function and the advantage function separately. This allows the agent to make informed decisions by comprehending the inherent value of various states and the comparative advantages of different actions. The double DQN utilises the policy network and the target network during training. The former is responsible for generating Q-values, while the latter is used to compute the target Q-values, helping to mitigate the overestimation of action values and stabilise the training process. The two networks have the same structure as illustrated in figure 13, and they are commonly used in DRL tasks with the deep Q-learning algorithm.
Structure of the duelling network. The number in parentheses denotes the number of neurals for the input and output dimension for a linear layer. Each linear layer follows a ReLU activation layer. The agent’s network consists of feature net, advantage net and value net. The output of feature net is the input of advantage net and value net. The sum of the two nets’ output is the final output of the agent, the Q-value for four actions.
Algorithm 1 outlines the procedure for updating the policy and target network parameters throughout the learning process. Here, the hyper-parameters are the discount rate
$\gamma =0.99$
and learning rates
$\varepsilon =\varepsilon ^{\prime }=10^{-4}$
for deep Q-learning and supervised learning. The relaxation factor is
$\tau =0.9$
. Batch sizes for deep Q-learning and supervised learning are both 64. The loss calculated in line 4 in algorithm 1 implements the loss function defined in (2.17). The cross-entropy function in line 12 is defined as
$H(p,q)=-\sum _{i}p_i\log(q_i)$
, where
$p_i$
and
$q_i$
are target and output distributions, respectively, with
$i=0,1,2,3$
. For the supervised learning task here,
$q_i=\exp(Q_i)/\sum _i\exp(Q_i)$
, where
$Q_i$
is the Q-value output of the DQN, and
$p$
is defined as a one-hot categorical distribution (Bishop Reference Bishop2006) with
$a_t=A_j$
and
$p_i=\delta _{ij}$
, such as
$p=(1,0,0,0)$
for
$a_t = A_0$
. The delta function is defined as
$\delta _{ij}=1$
for
$i=j$
, and
$0$
otherwise.
Training of the network
The choice of the reward function has an impact on the convergence and overall performance of the DQN agent. The reward function
\begin{equation} r(\boldsymbol{s},a)=-\mathrm{min}(\theta ^{2},3)+\mathcal A\,\mathrm{clip}(v,-0.1,0.1)-\mathcal B d \end{equation}
takes into account the swimmer’s various behaviours. We set constants
$\mathcal A = 20$
and
$\mathcal B = 10$
to balance the importance of the three terms on the right-hand side of (A7). This reward function encourages that the agent learns to align the swimmer’s orientation with the target, maintain an appropriate velocity towards the target, and minimise the distance to the target. The first term on the right-hand side of (A7) penalises the squared deviation of
$\theta$
from the optimal alignment with the target, up to a maximum value 3. This is an important component of the reward function because the direction of the swimmer’s body symmetric axis is crucial. When symmetric forces are applied, the swimmer’s velocity quickly adjusts to the direction of the symmetric axis. The second term rewards the swimmer’s mass centre velocity towards the target, with the velocity clipped between
$-0.1$
and 0.1 to avoid excessive speed. This term encourages the swimmer to maintain an appropriate velocity towards the target. The third term minimises the distance between the swimmer and the target.
Comparison of the tracking performance with two DQN agents and the baseline strategy. Trajectories of the swimmer and surrounding vortices are plotted. The red and blue line segments on the swimmer’s trajectory denote the actions with and without forces, respectively. Green dots denote the target point. Plots for (a) agent 1, (b) agent 2, (c) baseline strategy
$a=A_0$
for
$|\theta |\lt \unicode{x03C0} /6$
,
$a=A_1$
for
$\theta \gt \unicode{x03C0} /6$
, and
$a=A_2$
for
$\theta \lt -\unicode{x03C0} /6$
. The total times to complete the tracking task for agents 1 and 2 are
$11.75T$
and
$14.25T$
, respectively.
Compared with the standard DQN algorithm, we add an extra supervised learning step into the training process (see lines 9–15 in algorithm 1). This comes from the observation that during training and testing, there are more than one action sequences that can reach the target point. As shown in figures 14(a,b), agents 1 and 2 have the DQN trained with different steps to output action. Although both agents complete the task, agent 1 has a shorter time.
As shown in figure 14(c), we also introduce a synthetic agent as baseline for comparison. This baseline strategy has
$a=A_0$
for
$|\theta |\lt \unicode{x03C0} /6$
,
$a=A_1$
for
$\theta \gt \unicode{x03C0} /6$
, and
$a=A_2$
for
$\theta \lt -\unicode{x03C0} /6$
, and also has action regulation. It is a very straightforward one – it proceeds straight when the swimmer’s direction falls within the tolerance range, and turns left or right when it deviates from that range. The swimmer with the baseline strategy fails to reach the target, because it turns so frequently that the shedding vorticies mix together. The vortex interaction produces strong velocity perturbations, causing the swimmer to be hard to control.
Furthermore, the action sequence provided by one network can be replaced by another in training. Among these, we retain the one with the simplest form and shortest duration. As depicted in figure 15, the network’s performance may not consistently improve with the number of training steps, so we employ a supervised learning approach to enforce the network to maintain its choice of actions for certain states. This training method constrains the network to satisfy the Bellman optimality equation while maintaining known optimal choices for certain states, thereby enhancing the agent’s performance.
Convergence of the loss value in one training process without supervised learning, and the corresponding tracking performance (insets a–d) with
$(1,1.5,2,2.5,3)\times 10^{6}$
training steps. As training step increases, the performance of the agent is not guaranteed to become better.
Note that our training process was run multiple times, whereas the agent’s performance was tested only on a few of the tracking tasks. To address the issue of reproducibility crisis (Agarwal et al. Reference Agarwal, Schwarzer, Castro, Courville and Bellemare2021), where an agent’s performance cannot be tested for all possible scenarios, we apply an off-policy evaluation (OPE) (Uehara, Shi & Kallus Reference Uehara, Shi and Kallus2022) to evaluate our learned policy from given history trajectories. The OPE score for each trajectory is calculated as
\begin{equation} \sum _{t^*=0}^{N}\frac {p(a_{t^*}|s_{0,k},\ldots ,s_{t^*})}{p_{b}(a_{t^*}|s_{0},\ldots ,s_{t^*})}\gamma ^{t^*}r_{t^*}, \end{equation}
where
$N$
is the total time steps of the trajectory,
$p(a_{t^*}|s_{0,k},\ldots ,s_{t^*})$
and
$p_{b}(a_{t^*}|s_{0},\ldots ,s_{t^*})$
are the probabilities of choosing action
$a_{t^*}$
at time step
$t^*$
given trajectory
$s_0,a_0,\ldots s_{t^*}$
for the evaluated policy and the baseline policy, respectively,
$\gamma$
is a discount factor, and
$r_{t^*}$
is the reward at time step
$t^*$
. The probability terms
\begin{equation}p(a_{t^*}|s_{0,k},\ldots ,s_{t^*})=p(a_{0}|s_{0})\ldots p(a_{t^*}|s_{t^*})\end{equation}
and
\begin{equation}p_{b}(a_{t^*}|s_{0},\ldots ,s_{t^*})=p_b(a_{0}|s_{0})\ldots p_b(a_{t^*}|s_{t^*})\end{equation}
can be calculated directly from given states and actions.
This OPE score evaluates the effectiveness of a policy given a historical trajectory, compared to a baseline. Specifically, if the reward received after taking an action in a certain state is high, and the policy demonstrates a higher likelihood of selecting this action in that state compared to the baseline, then the score will be higher. We used a data augmentation technique similar to that in our training dataset mentioned above so that we have sufficient history trajectories (over 400) with different target points. The histogram of the OPE score in figure 16 shows that the trained agent has high scores for over 70 % of given trajectories. Since the reward is negative in most scenarios in figure 3, a score approximatehly larger than
$-1$
indicates that the agent has good performance on this trajectory. Therefore, our trained agent performs better than the baseline in general.
Histogram of OPE values over 400 trajectories.
Appendix B. Issues in complex target trajectory tracking
In complex target trajectory tracking in § 4.2, we find that the agent’s capability to formulate appropriate actions is compromised when the relative velocity’s magnitude
$|\boldsymbol{u}_r|$
is substantial, and the angle
$\alpha$
between
$\boldsymbol{u}_r$
and symmetric axis is large (see figure 1
b). This deficiency becomes apparent when the target velocity deviates from the swimmer velocity, the swimmer overtakes the target, and the swimmer motion is influenced by the wake flow.
The first scenario emerges when the direction of the target velocity significantly deviates from that of the swimmer. Note that the swimmer’s mass centre velocity
$\boldsymbol{u}_m$
is different from
$\boldsymbol{u}_r$
, and they are the same in tracking a fixed target. Besides, large
$\alpha$
, even for small
$\theta$
, also challenges the control of swimmer. This condition is exemplified in figure 17(a), where the swimmer struggles to adjust its course to maintain alignment with the target’s path. Although the symmetric axis aligns well toward the target, its
$\boldsymbol{u}_r$
deviates, so that the swimmer needs more space and time to reduce the large
$\alpha$
.
(a) The swimmer position relative to the target at
$t/T=72$
(station 8 in figure 7). (b) Evolution of
$\alpha$
, and the actions chosen at a series of times.
Figure 17(b) shows that action
$A_0$
is chosen for the majority of the time when
$\alpha$
is large, indicating that symmetric force application helps the swimmer to adjust
$\boldsymbol{u}_r$
to align with the symmetric axis. Note that a large
$\alpha$
occurs ultimately because the target can move freely in any direction that deviates from the swimmer velocity. When the target velocity deviates from
$\boldsymbol{u}_m$
, the symmetric axis adjusts its orientation relatively quickly, which leads to large
$\alpha$
. These observations imply that optimal tracking performance necessitates not only a minimal
$\theta$
but also a small
$\alpha$
.
The second scenario occurs when the swimmer reaches and surpasses the target, as observed at station 5 in figure 7. This event triggers a considerable alteration in
$\theta$
(see figure 8
b), so that the swimmer is required to execute a sharp U-turn within a confined spatial range to re-establish proximity to the target. However, this manoeuvre is generally unfeasible, because the swimmer is unable to instantly modify its velocity with complex FSI.
Additionally, the flow induced by earlier movements of the swimmer can disrupt its subsequent motion, particularly when navigating towards the left-hand circle. Figure 18 illustrates the vorticity and velocity field surrounding the swimmer as it approaches its starting position (station 5 in figure 7). The flow generated by its previous motion exerts a retarding force, interfering with the swimmer’s directional control.
The vorticity contour and velocity field at
$t/T=45$
(at station 5 in figure 7), along with the mass centre trajectory (grey line). The flow induced by previous actions of the swimmer is not negligible. It drags the swimmer to move right upwards (marked by the blue arrow), deviating from the figure-eight target trajectory.
These issues can hinder the swimmer’s ability to promptly and efficiently pursue the target. They underscore a limitation in the current task, wherein the agent operates without anticipating the target’s future movements. The unpredictable trajectory of the target poses challenges for the swimmer in adjusting its course accordingly with strong FSI.
Appendix C. Discussion on the open challenges of DRL in fluid mechanics
As mentioned in § 1, open challenges of DRL in fluid mechanics include noises, action delay and partial observability. To build a robot jellyfish, one challenge for control is that the sensor may not have high accuracy, equivalent to the scenario where the state input has noises. This issue is addressed using the soft actor–critic (SAC) method (Haarnoja et al. Reference Haarnoja, Ha, Zhou, Tan, Tucker and Levine2019b
). The SAC method contains a Q-network, similar to DQN, while it has an extra policy net for outputting a distribution of actions based on the Q-values accounting for noises in the state vector. Here, we added
$\pm 10\,\%$
noise to the state vector. Results in figure 19 show that the noise does not have significant influence on the tracking performance, except that the trajectory may slightly deviate from the optimal one. The tracking times for completing the tasks are
$17.5\ T$
,
$12\ T$
and
$13\ T$
, compared to
$11.75\ T$
,
$10\ T$
and
$11.75\ T$
in the same tracking tasks without noise, respectively.
Trajectories of the SAC agent with noisy input for tracking fixed targets: (a) front left, (b) straight, and (c) front right.
Action delay and partial observability are typically addressed by incorporating historical information, whereas we utilised only instantaneous state information coupled with action regulation to streamline our model. To assess the impact of historical information, we employed the SAC algorithm with noisy input, and integrated the long short-term memory method (Hochreiter & Schmidhuber Reference Hochreiter and Schmidhuber1997). However, we found that incorporating historical information did not significantly enhance performance in tracking a moving target. This may be due to the state vector lacking information about the surrounding flow of the swimmer. Consequently, unlike study of flapping aerofoils (Theodorsen Reference Theodorsen1935; Sears Reference Sears1938, Reference Sears1941), where the time history of the flapping kinematics fully determines the lift and the effect of shed vortices on the aerofoil, including historical states may not provide explicit information about the flow for the jellyfish-like swimmer with strong FSI. Another potential reason is that even if the swimmer is aware of the surrounding vortices, the highly unpredictable wake vortices make it challenging for the swimmer to make optimal decisions. Nonetheless, it is still possible that more careful training techniques, more varied datasets and more complicated networks are needed to train a better agent, as it is generally more difficult to train such a multi-steps network.
