1. Introduction
The pursuit-evasion problem in multi-robot systems simulates the collective behaviors of biological organisms in nature and originates from the predatory activities observed in the biological world [Reference Mu, Pan, Zhou, Yu and Cao1]. With the rapid development of artificial intelligence and autonomous decision-making technologies, pursuit-evasion games have become an important research direction in the field of multi-agent cooperative control and decision-making [Reference Bellman2]. Particularly in underwater environments, achieving effective collaborative pursuit and evasion strategies among multiple agents has attracted increasing attention in recent years. AUV, characterized by strong maneuverability, wide operational range, and high concealment, have been widely applied in typical underwater Internet of Things (IoUT) tasks, including marine resource exploration [Reference Wang, Zhang, Wang, Jiang, Wei and Ren3], information acquisition [Reference Guan, Wang, Jiang, Duan, Ren and Quek4], and real-time search and rescue operations [Reference Wei, Wang, Du, Fang, Jiang and Ren5]. However, due to the high maneuverability of targets, uncertain evasion strategies, and the limited perception capability of underwater robots, studying the underwater pursui-evasion (UPE) game between multiple AUV and evasive targets has become particularly important [Reference Liu, Zhang, Zhang, Pan and Yu6].
Multi-agent pursuit-evasion games require both pursuers and evaders to continuously adjust their behaviors and strategies, either simultaneously or sequentially, according to the observable environmental information, in order to maximize their respective benefits and ultimately achieve successful capture or escape. In such scenarios, the evading target typically exhibits high maneuverability, uncertain motion states, and adaptive escape strategies. Compared with single-agent pursuit-evasion games, which often possess a Nash equilibrium during the game process [Reference Bhattacharya and Hutchinson7], multi-agent pursuit-evasion games present typical non-zero-sum characteristics due to the cooperative benefits and positive externalities among pursuers. The interaction of individual strategies can improve the overall capture efficiency through spatial coordination and cooperative actions, enabling the total team reward to exceed the traditional zero-sum constraint in which “one party’s gain corresponds to another party’s loss.” Consequently, a cooperative Nash equilibrium oriented toward team-level optimization can emerge. Therefore, effective coordination among pursuers is essential for improving capture efficiency, for example, by employing cooperative strategies to jointly intercept and capture evaders with higher speeds [Reference Gerkey, Thrun and Gordon8]. However, research on multi-agent pursuit-evasion strategies exhibits a noticeable imbalance across different environments. Owing to the limited perception capability and complex operating conditions of underwater environments, studies on Underwater Pursuit-Evasion (UPE) tasks are still in their early stages. Compared with aerial and ground robotic systems, underwater AUV systems face more severe perception constraints during pursuit tasks. Due to the attenuation of electromagnetic signals in water, underwater robots mainly rely on acoustic sensing devices such as sonar to perceive surrounding obstacles and targets. However, acoustic perception usually suffers from limited effective sensing range, measurement noise, and incomplete local observations. These constraints make it difficult for multiple AUVs to obtain accurate environmental and target-state information, especially in obstacle-rich pursuit scenarios. As a result, underwater cooperative pursuit requires robust decision-making under partial observation and noisy sonar measurements, which makes UPE tasks more challenging than their terrestrial and aerial counterparts. Most existing decision-making approaches for UPE scenarios are derived from methods originally developed for aerial and ground robotic systems. Representative traditional pursuit-evasion approaches include model predictive control (MPC), artificial potential field methods, geometric analysis techniques, neural network-based methods, and game-theoretic approaches. Although these methods are supported by solid theoretical foundations, they typically rely on accurate system models and substantial computational resources. As a result, they face significant challenges when applied to complex, dynamic, and highly nonlinear real-world environments. Their limited scalability and adaptability also restrict their effectiveness in highly dynamic multi-agent UPE tasks.
Due to its strong capability for feature representation and its ability to learn through interactive feedback with the environment, deep reinforcement learning (DRL) has emerged as an effective solution for decision-making in multi-agent unmanned systems within pursuit-evasion tasks. In existing studies, Lin et al. [Reference Lin, Gao and Xia9] investigated the pursuit-evasion problem of fixed-wing UAV swarms and introduced a curriculum learning strategy to address complex dynamic environments. Chen et al. [Reference Chen, Yu, Li, Tang, Ji, Yang, Xu, Yang and Wang10] trained quadrotor UAVs in a three-dimensional environment and achieved successful zero-shot transfer from simulation to real-world deployment by integrating an evader prediction network with an adaptive environment generator. However, their training framework suffered from high complexity and failed to adequately address the issue of inaccurate value estimation among multiple agents. To further improve pursuit performance, Chen et al. [Reference Chen, Shi, Dai, Meng and Yu11] proposed the TP-MADDPG algorithm incorporating LSTM-based trajectory prediction, enabling pursuers to anticipate the motion of evaders and perform early interception, thereby significantly improving the encirclement efficiency. Nevertheless, the robustness of this method under complex obstacle environments and long-term partial observability still requires further improvement. In addition, most existing studies primarily focus on aerial or ground unmanned systems, whereas multi-agent pursuit-evasion tasks in underwater environments involve higher interaction costs and more challenging environmental conditions. To address this issue, Xu et al. [Reference Xu, Zhang, Wang, Han and Ren12] introduced offline reinforcement learning combined with a decision transformer into the multi-agent pursuit-evasion problem for AUV and proposed the MMOTS framework, which effectively reduced the high cost of online interactions. However, this approach relies heavily on the availability of high-quality offline datasets and does not sufficiently consider the frequent collisions caused by exploratory behaviors during the early stages of training, which remains a key bottleneck that hinders efficient policy convergence. In subsequent studies, Xu et al. [Reference Liu, Huang and Zhao13] adopted offline reinforcement learning for underactuated AUV obstacle avoidance to avoid online training risks. However, it is only applicable to single-robot scenarios and cannot support multi-AUV cooperative pursuit-evasion decision-making.
In summary, although substantial progress has been achieved in existing studies focusing on specific platforms or individual issues, several critical challenges remain in multi-AUV cooperative pursuit tasks within underwater environments. First, in traditional multi-agent reinforcement learning, algorithms either rely on a single global critic or use fully decentralized critics. Relying solely on a global value network makes it difficult to accurately evaluate individual contributions and leads to serious credit assignment problems and training instability. Meanwhile, completely decentralized critics lack global guidance, resulting in poor coordination performance. Second, during the early stages of training, random exploration strategies tend to induce collisions between AUVs and environmental obstacles, causing frequent episode termination and extremely low training efficiency. Third, most existing studies on underwater pursuit-evasion (UPE) are conducted in traditional two-dimensional or simplified dynamic simulation environments, often implemented in MATLAB, which makes it difficult to transfer research outcomes to realistic marine environments.
To address the aforementioned challenges, this paper proposes a Scheduled Value Reweighting Network-enhanced Multi-Agent Deep Deterministic Policy Gradient (SVRN-MADDPG) algorithm integrated with an Active Sonar Obstacle Avoidance (ASOA) strategy for multi-AUV cooperative pursuit control. Different from existing dual-critic and value decomposition approaches, the proposed SVRN-MADDPG is designed to achieve complementary value evaluation across heterogeneous information levels rather than simply employing multiple critics for overestimation reduction or value factorization. Specifically, MATD3 employs two homogeneous critics to construct conservative value estimates for alleviating overestimation bias, while COMA introduces a centralized critic with a counterfactual baseline to address the credit assignment problem by estimating the marginal contribution of each agent. QMIX and QTRAN follow the value decomposition paradigm, where the joint action-value function is represented or constrained through individual value functions. In contrast, the proposed framework introduces heterogeneous global and local critics with different information scopes and functional roles. The global critic evaluates joint state-action information to provide cooperative value guidance, whereas the local critic evaluates individual observation-action information to provide auxiliary behavioral feedback under partial observability. The scheduled value reweighting mechanism further coordinates the contributions of these two critics during training, enabling global cooperative information to play a dominant role in the final value estimation. The main contributions of this work are summarized as follows:
First, based on the MADDPG framework, a dual-critic architecture is proposed using the Scheduled Value Reweighting Network (SVRN). The framework introduces local critics to enhance the individual responsiveness of agents, while global critics are employed to enable multi-agent cooperative decision-making. In addition, a time-varying scheduling strategy is designed to dynamically reweight the contributions of the two types of critics. This mechanism enables a smooth transition from individual exploration to cooperative tracking, thereby improving both convergence efficiency and task execution performance.
Second, an ASOA strategy is proposed and integrated into the reinforcement learning training framework as a safety-oriented constraint. This strategy establishes a hierarchical control mechanism in which reward-driven learning is prioritized while obstacle avoidance is enforced through hard safety constraints. As a result, the frequency of episode termination caused by collisions with obstacles during the early training stage is significantly reduced, thereby lowering training costs and accelerating policy convergence.
Third, a high-fidelity underwater simulation environment is constructed based on Gazebo and the UUV Simulator. By incorporating realistic dynamic models and physical interaction mechanisms, the developed simulation platform provides a more practical experimental environment for studying multi-AUV cooperative pursuit-evasion tasks.
Finally, a hierarchical reward function is designed to guide efficient policy learning. The reward mechanism consists of three layers – an immediate layer, a tactical layer, and a strategic layer – where the immediate layer encourages fundamental behaviors, the tactical layer facilitates intelligent interception, and the strategic layer promotes long-term cooperative strategies, forming a comprehensive incentive mechanism for efficient multi-AUV learning.
The pursuit-evasion framework adopted in this study is illustrated in Figure 1, which presents a hierarchical joint control architecture for AUV. In this architecture, the SVRN-MADDPG deep reinforcement learning algorithm serves as the high-level decision-making module, generating the desired horizontal velocity and angular velocity commands for the AUV based on observation information obtained from the simulation environment. Meanwhile, a depth PID controller independently performs depth control and outputs the desired vertical velocity. At the control layer, a velocity PID controller and an acceleration PID controller sequentially perform closed-loop tracking of the motion commands. The corresponding control torques are then generated through a propeller allocation module to drive the AUV, thereby enabling motion execution and forming an interactive closed-loop control process within the environment. By integrating the autonomous decision-making capability of reinforcement learning with the robust stability of PID control, the proposed architecture enables effective autonomous obstacle avoidance and precise motion control for AUV operating in complex underwater environments.
Based on the deep reinforcement learning (DRL) framework for pursuit-evasion, the AUVs are capable of completing pursuit tasks within a simulated marine environment.
Figure 1. Long description
A diagram of the control system for an autonomous underwater vehicle (AUV) in a simulated environment. The diagram is divided into three main sections: Environment, Simulation Environment, and Control system. The Environment section shows an underwater scene with obstacles. The Simulation Environment section depicts a top-down view of the underwater area with obstacles. The Control system section outlines the control processes for the AUV, including SVRN-MADDPG, Depth PID, Velocity PID, Acceleration PID, and Thrust Allocation. The AUV interacts with the environment through observation and action, with control commands sent to the AUV based on modeling. The diagram includes labels for various axes and control processes, indicating the flow of information and control commands.
2. Related work
The pursuit-evasion problem, as an important research direction in the field of multi-agent cooperative control and decision-making, has significant application value in areas such as cooperative control of unmanned systems, target interception, and autonomous decision-making. Early studies on pursuit-evasion problems typically modeled them as adversarial decision-making tasks in known environments, where the pursuer aims to minimize the distance to the evader, while the evader attempts to maximize the distance between the two agents. The differential game theory proposed by Isaacs [Reference Isaacs14] provides an important analytical framework for pursuit-evasion problems by establishing a dynamic game model to derive the optimal strategies for both the pursuer and the evader. With the continuous advancement of research, various approaches have been developed to address pursuit-evasion problems from different perspectives, including methods based on geometric analysis, artificial potential fields, MPC, and DRL.
Early studies often addressed pursuit-evasion problems from the perspective of geometric analysis, among which the Apollonian circle is a representative analytical tool. Dorothy et al. [Reference Dorothy, Maity, Shishika and Von Moll15] introduced the Apollonian circle to transform the pursuit-evasion problem into a nonlinear control problem and analyzed the capture condition of the pursuer. Xue et al. [Reference Xue, Wang, Wu, Yuan and Liu16] constructed a multi-agent pursuit-evasion framework based on Apollonius partitions, where the state space was divided into dominance regions and a potential game model was established. Furthermore, Al-Talabi et al. [Reference Al-Talabi17] combined fuzzy reinforcement learning with Apollonian circle techniques and formation control strategies to improve pursuit decision-making. Although these geometric methods provide clear analytical interpretations for pursuit-evasion problems, they usually rely on idealized assumptions and have limited adaptability to obstacle-rich environments, complex dynamic constraints, and multi-agent cooperative interactions. Therefore, their applicability to realistic underwater multi-AUV pursuit scenarios remains limited.
As research has progressed, artificial potential field (APF) methods have been introduced to address pursuit-evasion and obstacle avoidance problems. By constructing attractive potentials toward targets and repulsive potentials around obstacles, APF-based approaches provide intuitive and computationally efficient solutions for local motion guidance and obstacle avoidance. Chen et al. [Reference Chen, Yang, Zhang and Li18] integrated APF with the Apollonian circle method to dynamically redistribute pursuers and improve cooperative pursuit efficiency. Dong et al. [Reference Dong, Zhang and Jia19] combined an improved dynamic APF with differential game theory and regional planning strategies, providing an integrated solution for pursuit-evasion decision-making under complex interactions. Sun et al. [Reference Sun, Sun, Li and Zou20] further incorporated Apollonius circle constraints into an improved APF framework to achieve cooperative pursuit and obstacle avoidance for multiple USVs in marine environments. These studies demonstrate that APF-based methods are effective for local motion guidance and obstacle avoidance. However, when APF is employed as an independent planning or decision-making approach, its performance may be affected by local optima and limited adaptability in complex dynamic environments. This limitation does not indicate that distance-based obstacle avoidance principles are ineffective; instead, it suggests that such mechanisms should be integrated with higher-level decision-making frameworks for complex cooperative tasks. Therefore, in this work, distance-based obstacle information is incorporated into the reinforcement learning framework as a supplementary safety constraint rather than an independent planning strategy.
MPC has also been widely investigated for pursuit-evasion and interception problems due to its ability to predict future system states and optimize control inputs online. Huang et al. [Reference Huang, Zhang, Zhan and Ma21] combined an insensitivity mechanism with nonlinear MPC to achieve effective interception of maneuvering targets. Qiu et al. [Reference Qiu, Zhang and Wang22] proposed the MPC-DPEG framework, which integrates MPC with inverse optimal control to derive Nash strategies for non-zero-sum multi-player pursuit games. Hu et al. [Reference Hu, Zhao and Pan23] further incorporated fuzzy Q-learning into MPC to improve decision-making capability under uncertain environments. These studies demonstrate that MPC-based methods can provide effective control solutions when reliable system models and prediction information are available. However, they generally depend on accurate dynamic models or target motion prediction, and their computational burden increases significantly in multi-agent cooperative pursuit scenarios, which limits their real-time applicability in complex environments.
With the advancement of artificial intelligence technologies, DRL has gradually become an important approach for solving complex pursuit-evasion problems. DRL approximates policy and value functions using deep neural networks, enabling agents to learn optimal decision-making strategies through continuous interaction with the environment. As a result, algorithms such as Deep Deterministic Policy Gradient (DDPG) [Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra24], Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO), and Deep Q-Network (DQN) have been widely applied in decision-making and control tasks for unmanned systems.
Building upon these advances, multi-agent reinforcement learning (MARL) has been further extended to cooperative planning and coordination tasks in unmanned systems, particularly in multi-UAV scenarios. Chen et al. investigated several MARL-based methods for multi-UAV cooperative tasks, including cooperative path planning, behavior control, and target encirclement. For cooperative path planning, they proposed a dual-centralized Q-network-based reinforcement learning method for homogeneous UAVs, where collision avoidance, motion continuity, and energy consumption constraints were considered to generate feasible and collision-free paths [Reference Chen, Ren, Hu, Zhang, Lu, Li, You and Rodrigues25]. For cooperative behavior control, they developed a global-and-local attention-based reinforcement learning framework to enhance coordination among multiple UAVs in dynamic environments [Reference Chen, Li, Zhang, You, Lu, Tiwari and Kumar26]. For cooperative target encirclement, they further introduced an extrinsic-and-intrinsic reward-based MARL method for heterogeneous UAVs to improve exploration ability and encirclement performance under sparse reward conditions [Reference Chen, Wang, Zhang, Lu, Shu and Hu27]. These studies demonstrate the effectiveness of MARL in multi-UAV cooperative planning and control tasks.
Compared with general UAV cooperative planning and coordination tasks, pursuit-evasion problems are more challenging because they involve not only multi-agent cooperation but also adversarial interactions between pursuers and evaders. In pursuit-evasion research, early reinforcement learning approaches mainly focused on single-agent or simple adversarial scenarios. Wang et al. [Reference Wang, Wang, Cui, Guo and Yan28] applied an online reinforcement learning method to the pursuit-evasion problem of unmanned surface vehicles (USVs), modeling the game interaction between the pursuer and the evader using a relative motion model and adopting a “one-Critic-two-Actor” network structure for policy learning. Subsequently, deep reinforcement learning has been gradually extended to multi-agent pursuit-evasion tasks. Li et al. [Reference Li, Chen, Zhou and Sun29] investigated orbital multi-agent pursuit-evasion problems using deep reinforcement learning, optimizing strategies through a discrete Markov decision process framework and an improved distributed deterministic policy gradient algorithm. Under complex environmental conditions, several studies have further enhanced algorithm performance by integrating environmental modeling and training mechanisms. Tan et al. [Reference Tan, Sun, Ding, Zhou and Liu30] proposed a scalable hierarchical pursuit-evasion framework for fixed-wing UAVs. This framework combines dynamic target assignment and hierarchical reinforcement learning and introduces stable auxiliary gradients to ensure smooth flight. It effectively improves scalability and training efficiency in large-scale multi-UAV scenarios. Furthermore, Zhuang et al. [Reference Zhuang, Gao, Wu, Zhang and Jia31] proposed a hybrid reinforcement learning approach that combines an improved deep DDPG algorithm, fuzzy logic, and the BFGS algorithm optimization strategy, significantly improving the cooperative control performance of multi-agent systems in dynamic environments. Huang et al. [Reference Huang, Zeng and Xu32] proposed a D4PG-based autonomous interception decision-making method for close-range pursuit-evasion problems, which enhances the algorithm’s convergence speed and robustness in highly maneuverable adversarial environments through state entropy maximization exploration and minimum variance selection among multi-object value networks. Nevertheless, the aforementioned studies still face several challenges in multi-agent cooperative pursuit-evasion tasks, including inaccurate value estimation, difficulties in credit assignment, and insufficient training stability.
In summary, existing studies have achieved considerable progress in addressing pursuit-evasion game problems. However, several challenges remain, particularly in complex environment modeling, multi-agent collaborative decision-making, and stable and efficient training. In underwater scenarios, these challenges become even more pronounced due to environmental characteristics such as perceptual uncertainty and complex hydrodynamic interactions. As a result, most existing studies remain limited to simplified simulations or idealized environments, while relatively little research has been conducted on multi-AUV cooperative pursuit in realistic physical simulation platforms. To address these issues, this paper investigates the multi-AUV cooperative pursuit task and constructs an underwater pursuit-evasion scenario within the Gazebo marine physical simulation environment. Within a deep reinforcement learning framework, an SVRN-MADDPG algorithm is proposed to improve cooperative decision-making among multiple agents. The proposed approach improves multi-agent value evaluation by introducing a dual-critic architecture and a time-varying reweighting strategy and further enhances training stability and safety by integrating the ASOA mechanism. As a result, the learning efficiency and task success rate of multi-AUV cooperative pursuit in underwater environments are significantly improved.
3. Problem modeling
3.1. Pursuit scenario
The pursuit problem investigated in this study focuses on the control and decision-making task in which multiple AUVs pursue an intelligent target within a constrained environment with limited spatial scale. To increase the difficulty of the task, the intelligent target is designed to have a higher escape speed than that of the pursuing AUV while maintaining the same turning rate.
A rule-based fixed evasion strategy is designed for the intelligent evader in this study. Under the motion parameter settings adopted in this work, the maximum linear velocity of the evader is higher than that of the pursuers, which implies that a single pursuer is unable to successfully capture the evader under ideal conditions. Therefore, this study promotes the formation of cooperative pursuit strategies by increasing the number of pursuers, enabling the evader to be gradually confined within a restricted region and eventually captured. Specifically, the evasion strategy is defined along the extension of the line connecting the nearest pursuer and the evader. Under this strategy, the angular velocity output of the evader is decomposed into two key components: the fundamental evasion angular velocity
$W_{e,\mathrm{base}}$
, which drives the evader away from the nearest pursuer, and the obstacle-avoidance correction angular velocity
$W_{e,\mathrm{obs}}$
, which enables the evader to avoid environmental obstacles.
\begin{align} v_e = \begin{cases} 0.9\ \mathrm{m/s}, & p_e - p_{\rm pur} \gt d_{v_e} \\[3pt] 1.2\ \mathrm{m/s}, & p_e -\rm p_{pur} \leq d_{v_e} \end{cases} \qquad \qquad \qquad \\[-10pt] \nonumber \end{align}
\begin{align} W_{e,{\rm base}} = \mathrm{clip}\left (\Delta \psi \times k_1, -\omega _{\max} \times k_2, \omega _{\max} \times k_2\right )\!,\quad p_e - p_{\rm pur} \leq d_{\rm cap} \\[-9pt] \nonumber \end{align}
\begin{align} W_{e,{\rm obs}} = \mathrm{clip}\left (turn_{\rm dir} \times \omega _{\max} \times k_2, -\omega _{\max}, \omega _{\max}\right )\!,\quad p_e - p_{\rm obs} \leq d_{\rm obs} \\[-10pt] \nonumber \end{align}
\begin{align} \omega _e = \begin{cases} W_{e,{\rm obs}}, & p_e - p_{\rm obs} \leq d_{\rm obs} \\[3pt]W_{e,{\rm base}}, & p_e - p_{\rm obs} \gt d_{\rm obs} \end{cases} \qquad \qquad \qquad \\[10pt] \nonumber \end{align}
Here,
$v_e$
denotes the linear velocity of the evader, and
$\omega _e$
represents its angular velocity.
$p_e$
denotes the position of the evader, while
$p_{\rm pur}$
and
$p_{\rm obs}$
represent the positions of the nearest pursuer and the obstacle, respectively.
$d_{v_e}$
denotes the distance threshold for switching the linear velocity, which is set to 20 m in this study.
$\Delta \psi$
represents the heading angle error between the current heading of the target and the desired evasion direction.
$k_1$
is the proportional gain associated with the heading error and is set to 0.8 in this study.
$\omega _{\mathrm{max}}$
denotes the maximum angular velocity of the evader, which is set to 0.5 rad/s.
$k_2$
represents the angular velocity limiting coefficient and is set to 0.8.
$d_{\rm cap}$
denotes the distance threshold for activating the pursuit-related angular velocity component and is set to 30 m. The parameter
$turn_{\rm dir}$
represents the turning direction coefficient, which is determined based on sonar sensor data to select the turning direction with greater free space. Finally,
$d_{\rm obs}$
denotes the distance threshold for activating the obstacle-avoidance angular velocity component and is set to 1.8 m in this study.
The evasion strategy designed in this study is developed based on principles of game fairness, behavioral realism, and engineering safety, ensuring its overall rationality. Specifically, the linear velocity adopts a distance-adaptive hierarchical control mechanism, allowing the evader to cruise at a lower speed when the pursuer is far away to maintain stable task execution, while switching to a higher escape speed when the pursuer approaches, thereby increasing the challenge of the pursuit scenario. The angular velocity follows a dual-mode hierarchical decision mechanism. In obstacle-free environments, stable steering is achieved based on the heading error, whereas in cluttered environments, sonar information is utilized to perform rapid obstacle avoidance, strictly following the obstacle-avoidance priority principle. This strategy not only conforms to the motion constraints of underwater robots and the escape behaviors observed in biological systems but also provides a reasonable and challenging evaluation environment for multi-agent pursuit-evasion algorithms.
For the pursuers, successful capture is defined when the Euclidean distance between the geometric centers of any pursuer and the evader is smaller than the capture threshold
$d_{\mathrm{trigger}}$
. The capture distance is defined based on the geometric centers of the pursuer and the evader rather than the distance between their outer hulls. Since each AUV is represented as a rigid-body model with finite physical dimensions in the Gazebo simulator, the capture threshold should account for both the vehicle size and the uncertainty caused by sonar-based perception. Therefore, the capture threshold
$d_{\mathrm{trigger}}$
is empirically set to
$4.5\,\mathrm{m}$
, which provides a balance between reliable cooperative capture determination and robustness against perception noise. Meanwhile, the reinforcement learning policy is optimized by maximizing the cumulative reward function, and the corresponding objective function is presented in Eq. (5).
\begin{equation} \arg \max \left \{ J(\pi ) = \mathbb{E}\left [ \sum _{t=0}^{\infty } \gamma ^t r_t \right ] \right \}, \quad \text{s.t.} \quad \min _i \left \| p_i - p_e \right \| \leq d_{\mathrm{trigger}} \end{equation}
Here,
$J(\pi )$
denotes the cumulative reward objective function corresponding to policy
$\pi$
,
$\gamma ^t$
represents the reward discount factor raised to the
$t$
-th power,
$r_t$
denotes the immediate reward at time step
$t$
, and
$p_i$
represents the position of the
$i$
-th pursuer.
Each pursuer departs from a designated initial position and cooperatively tracks the evader, with the objective of completing the capture within a specified time limit. Due to limitations in communication range and sensing capability, each AUV can only obtain environmental information within its local perception range as well as limited state information from its teammates. Since the evader moves faster than the pursuers, it is necessary to design effective strategies that enable the pursuers to make appropriate decisions based on limited information and successfully capture the target within a short period of time while avoiding collisions with environmental obstacles.
3.2. Reinforcement learning framework
In the multi-AUV pursuit-evasion task considered in this study, the decision-making process of an individual pursuer can be approximately modeled as a Markov Decision Process (MDP). However, in a multi-agent cooperative environment, since each agent can only obtain partial observations of the global environmental state, the problem can be further formulated as a Partially Observable Markov Decision Process (POMDP) [Reference Kaelbling, Littman and Cassandra33]. A standard POMDP can be formally defined by the following seven-tuple
$(S, A, T, R, O, Z, \gamma )$
. Among these components,
$S$
and
$A$
denote the state space and action space, respectively, while
$R$
represents the reward function.
$T\;:\; S \times A \times S \rightarrow [0,1]$
denotes the state transition probability function, which satisfies the formulation given in Eq. (6).
\begin{equation} T(s' | s, a) = P(S_{t+1} = s' | S_t = s, A_t = a) \end{equation}
Among them,
$\gamma \in [0,1)$
denotes the discount factor. Compared with the MDP framework, the POMDP introduces two additional core components to characterize partial observability: the observation space
$O$
and the observation probability function
$Z\;:\; S \times A \times O \rightarrow [0,1]$
. The observation probability function represents the conditional probability that the agent obtains observation
$o$
after executing action
$a$
and transitioning to state
$s'$
. Since the true state
$s \in S$
cannot be directly observed by the agent, the decision-making process needs to maintain a belief distribution
$b_t(s)$
over the system state based on the observation history
$\{o_1, o_2, \ldots , o_t\}$
, which can be expressed as follows.
\begin{equation} b_t(s) = P\left (S_t = s \mid o_{1:t}, a_{1:t-1}\right ) \end{equation}
Where
$b_t(s)$
represents the probability that the agent estimates the system to be in state
$s$
at time
$t$
. On this basis, strategy
$\pi \;:\; \mathcal{B} \times A \rightarrow [0,1]$
is optimized by maximizing the expected discounted cumulative reward, that is
\begin{equation} J(\pi ) = \mathbb{E}\left [ \sum _{t=0}^{\infty } \gamma ^t r_t \mid b_0, \pi \right ] \end{equation}
where
$r_t$
denotes the immediate reward at time
$t$
.
3.2.1. Action
To reduce the decision dimensionality of the agent’s action space, the pursuer’s speed is defined as a continuous action
$a_i$
. For the
$i$
-th agent, its action space at time
$t$
can be expressed as :
\begin{equation} a_i = [v_i, \omega _i] \end{equation}
Among them,
$v_i$
represents the forward linear velocity of the AUV, and
$\omega _i$
denotes the yaw angular velocity of the AUV. Both variables are defined as continuous control actions. Considering the dynamic characteristics of underwater AUVs and the mission requirements, the action space is strictly constrained to ensure the physical feasibility of the control outputs. The linear velocity
$v_i$
ranges within
$[0, v_{\max}]$
, where
$v_{\max} = 1$
m/s. The non-negative setting of the linear velocity is adopted because backward motion is not required for the pursuer in the pursuit-evasion task, while the upper bound is determined according to the actual propulsion capability of the pursuer. The angular velocity
$\omega _i$
ranges within
$[-\omega _{\max}, \omega _{\max}]$
, where
$\omega _{\max} = 0.5$
rad/s. Both positive and negative angular velocities are allowed to satisfy the left and right yawing requirements of the pursuer. This range balances the turning flexibility and motion stability of the AUV.
3.2.2. Rewards
The reward function provides effective incentives for training DRL agents and plays a crucial role in learning performance. The objective of the pursuers is to capture the evader while avoiding collisions with environmental obstacles. Therefore, this paper designs a three-layer hierarchical reward function, consisting of the immediate layer, tactical layer, and strategic layer. The reward
$r_i^t$
received by the
$i$
-th pursuer at time
$t$
is defined as shown in Eq. (10).
\begin{equation} r_i^t = r_i^{\mathrm{inst}}(o_i, a_i) + r_i^{\mathrm{tac}}(o_i, a_i) + r_i^{\mathrm{str}}(s, {a}) \end{equation}
Among them,
$r_i^{\mathrm{inst}}$
denotes the immediate-layer reward,
$r_i^{\mathrm{tac}}$
denotes the tactical-layer reward, and
$r_i^{\mathrm{str}}$
denotes the strategic-layer reward.
$o_i$
represents the local observation of agent i, and
$a_i$
represents the action of agent i.
$\mathbf{s} = (o_1, \ldots , o_N)$
denotes the global state, while
$\mathbf{a} = (a_1, a_2, \ldots , a_N)$
represents the corresponding global information used for value evaluation. The proposed three-layer reward structure guides the pursuers to approach the target at the immediate level, facilitates obstacle avoidance and early interception at the tactical level, and promotes multi-agent cooperative capture at the strategic level. Together, these components drive the pursuers to learn an efficient cooperative pursuit policy. The specific design of each reward component will be described in detail in Section 4.1.
3.2.3. Algorithm framework
To achieve collaborative pursuit control of multiple AUV, this paper constructs the SVRN-MADDPG algorithm framework based on the MADDPG algorithm. During the centralized training phase, each agent updates its critic network using global state information while relying solely on local observations for decision-making during the execution phase. The algorithm updates the value function through the TD objective and introduces both global and local critic networks to evaluate information at different scales. This hierarchical evaluation mechanism improves the stability of policy learning and enhances the agents’ collaborative pursuit capability.
Firstly, a TD objective is constructed based on the Bellman equation to guide the parameter update of the critic network, as shown in Eq. (11).
\begin{equation} y_i = r_i + \gamma (1 - d) Q_i'(s', a_1', a_2', \ldots , a_N') \end{equation}
Among them,
$y_i$
represents the TD target value of the i-th agent, and
$\gamma$
denotes the discount factor, which is set to 0.95 in this paper.
$Q'_i$
represents the target critic network,
$s'$
denotes the global state at the next time step, and
$a'_1,a'_2,\ldots ,a'_N$
represents the actions of all agents at the next time step. The variable
$d\in \{0,1\}$
denotes the episode termination flag. When
$d=1$
, the current episode terminates due to predefined conditions, including successful target capture, collision with obstacles, or reaching the maximum episode length, after which the simulation environment is reset to the initial state. When
$d=0$
, the episode continues without termination. In the TD target calculation,
$(1-d)$
prevents the discounted future value from being propagated beyond terminal states, ensuring that value updates are performed only for valid state transitions. Subsequently, the TD objective is used to update the global and local critic networks, with the loss functions defined in Eqs. (12) and (13).
\begin{align} L_i^{\mathrm{g}} &= \mathbb{E}\big[\big(Q_i^{\mathrm{g}}(\mathbf{s}, \mathbf{a}) - y_i\big)^2\big ] \\[-10pt] \nonumber \end{align}
\begin{align} L_i^{\mathrm{l}} &= \mathbb{E}\big [\big (Q_i^{\mathrm{l}}(\mathbf{o}_i, a_i) - y_i\big )^2\big ] \\[-10pt] \nonumber \end{align}
\begin{align} Q_i^{\mathrm{g}} &= Q_i^{\mathrm{g}}(\mathbf{s}, \mathbf{a}) \\[-10pt] \nonumber \end{align}
\begin{align} Q_i^{\mathrm{l}} &= Q_i^{\mathrm{l}}(\mathbf{o}_i, a_i) \\[10pt] \nonumber \end{align}
Among them,
$Q_i^{\mathrm{g}}$
represents the global critic network, which evaluates the value of joint actions of all agents under the global state.
$Q_i^{\mathrm{l}}$
represents the local critic network, whose inputs consist of the local observation
$o_i$
of agent
$i$
and its action
$a_i$
.
During the update process of the Actor network, the policy parameters are optimized by the policy gradient method, and the gradient expression is given by Eq. (16).
\begin{equation} \nabla _{\theta _i} J(\theta _i) = \mathbb{E}[ \nabla _{\theta _i} \mu _i(o_i) \nabla _{a_i} Q_i(s, {a}) ] \end{equation}
S Among them,
$\mu _i(o_i)$
denotes the policy network of the
$i$
-th agent, and
$Q_i(s, a)$
denotes the SVRN-based dual-critic network of the
$i$
-th agent.
The core differences between the two critic networks are shown in Table I.
Comparison of core differences between two critic networks.
Table I. Long description
A table comparing core differences between two critic networks. The table has two columns and four rows. Column headers are Local critic and Global critic. Row labels are Input information, Evaluation criterion, Information source, and Functional role. Row 1: Input information, o_i + a_i, s + a. Row 2: Evaluation criterion, Individual local tasks, Team global tasks. Row 3: Information source, Agent’s own perception, Centralized global information. Row 4: Functional role, Ensure local rationality, Guide global cooperativeness.
4. Methods
The proposed SVRN-MADDPG algorithm introduces an auxiliary local critic alongside the conventional global critic, forming a dual-critic architecture with a scheduled value reweighting mechanism. As illustrated in Figure 2, the overall framework consists of two main stages: policy improvement and model training. In the first stage, the global and local critics are fused through a time-varying reweighting mechanism to guide policy learning. The local critic provides informative feedback for individual pursuers in the early training phase, facilitating fast acquisition of reward signals and encouraging exploration behavior. As training progresses, the influence of the global critic gradually increases, promoting the emergence of cooperative pursuit strategies among multiple agents. In the second stage, each pursuer performs policy learning using the MADDPG algorithm. The agents update their actor-critic networks through interaction with the environment and experience replay, and the trained policy is ultimately deployed to each pursuer to achieve collaborative pursuit control.
4.1. Reward function design
To improve tracking efficiency during the pursuit process, this paper adopts a three-layer reward structure, consisting of the real-time layer, tactical layer, and strategic layer. The real-time layer includes proximity rewards, spin penalties, and low-speed penalties, which provide rapid feedback and guide basic motion behaviors. The tactical layer incorporates prediction rewards and obstacle avoidance rewards, enabling mid-term decision-making and facilitating effective interception. The strategic layer contains team rewards and capture rewards, which encourage long-term objectives and promote cooperative pursuit among agents. Through this hierarchical reward design, pursuers are guided to gradually learn effective strategies, enabling cooperative pursuit of high-speed evaders by low-speed pursuers within bounded environments.
4.1.1. Instant layer rewards
The real-time layer reward
$r_i^{\text{inst}}$
includes proximity reward, turn penalty, and low-speed penalty. Real-time reward output can quickly motivate pursuers to effectively explore approaching targets and quickly suppress ineffective turn behavior. In the first 100 training rounds, the actions output under the influence of exploration noise are relatively random, and the design of the real-time layer can quickly help pursuers find signals from evaders. In order to guide the pursuer to approach the evader as soon as possible, the pursuer receives rewards at each time step.
Approaching Reward
$r_1$
: Encourage the pursuer to approach the target.
\begin{equation} r_1 = k_a (d_{t-1} - d_t) \end{equation}
SVRN-MADDPG framework: The framework consists of three modules: decentralized execution, experience replay buffer, and centralized training. During the decentralized execution phase, each agent interacts with the environment to generate experience tuples, including states, actions, and rewards. These interaction data are stored in the experience replay buffer for unified management and sampling. In the centralized training phase, batches of experience are sampled from the buffer to update the Actor and Critic networks, enabling continuous optimization of the policy and improving the collaborative learning capability of the multi-agent system.
Among them,
$k_a$
denotes the proximity reward coefficient, which is set to 1 in this paper.
$d_t$
represents the distance between the pursuer and the target at the current time step, while
$d_{t-1}$
denotes the distance between the pursuer and the target at the previous time step. This reward design ensures that the pursuer receives a positive reward whenever it approaches the evader, effectively guiding the pursuer to complete the pursuit task efficiently.
Rotation penalty
$r_2$
: used to prevent the pursuer from spinning.
\begin{align} r_2 = \begin{cases} p_{\text{spin}}, & \sum \limits _{t'=t-2}^{t} I(|\omega _{t'}| \gt \omega _{\text{thr}}) \geq T_w \ \text{and} \ d_{\text{sonar}} \gt 1.8 \\[8pt] 0, & \text{otherwise} \end{cases} \end{align}
Among them,
$p_{\text{spin}}$
represents the penalty value for spinning, which is set to 0.5 in this study.
$\omega _t$
denotes the angular velocity of the pursuer at time t, and
$\omega _{\text{thr}}$
represents the angular velocity threshold, which is set to 0.35 rad/s.
$T_w$
denotes the continuous time-step window, which is set to 3. The rotation penalty is triggered only when the angular velocity of the robot exceeds the threshold
$\omega _{\text{thr}}$
for three consecutive time steps.
$d_{\text{sonar}}$
represents the distance to the nearest obstacle detected by the sonar, and
$d_{\text{sonar}} \gt 1.8$
is introduced to eliminate the influence of obstacle-avoidance maneuvers and turning behaviors.
Low-speed penalty
$r_3$
: This term is introduced to prevent the pursuer from stalling and assist in avoiding fixed obstacles.
\begin{align} r_3 = \begin{cases} p_{\text{low}}, & v_i \leq v_{\text{thr}} \\[3pt] 0, & v_i \gt v_{\text{thr}} \end{cases} \end{align}
Among them,
$p_{\text{low}}$
represents the low-speed penalty coefficient, which is set to −0.1 in this study.
$v_{\text{thr}}$
denotes the low-speed threshold, which is set to 0.1 m/s. When the linear velocity falls below this threshold, the pursuer is considered to be in a low-speed or stagnation state.
4.1.2. Tactical-level rewards
The tactical-layer reward
$r_i^{\text{tac}}$
consists of a prediction reward and an obstacle-avoidance reward and is designed to support mid-term decision-making for intelligent interception. Specifically, the obstacle-avoidance component ensures operational safety, while the prediction component improves interception efficiency. Through the coordinated effect of these two components, the tactical layer promotes intelligent, safe, and efficient pursuit behavior.
Obstacle-Avoidance Reward
$r_4$
: This reward term is designed to prevent the pursuer from colliding with environmental obstacles by encouraging the agent to maintain a safe distance during navigation.
\begin{equation} r_4 = \begin{cases} p_{\text{col}}, & d_{\text{sonar}} \lt d_{\text{col}} \\[3pt]k_o (d_{\text{sonar}} - d_{\text{safe}}), & d_{\text{col}} \leq d_{\text{sonar}} \lt d_{\text{safe}} \\[3pt]0, & d_{\text{sonar}} \geq d_{\text{safe}} \end{cases} \end{equation}
Among them,
$p_{\text{col}}$
represents the collision penalty value, which is set to −10 in this study.
$d_{\text{col}}$
denotes the collision distance threshold and is set to 1.1 m.
$k_o$
represents the obstacle-avoidance coefficient, which is used to scale the penalty magnitude and balance its weight relative to other reward components; in this study, it is set to 0.25.
$d_{\text{safe}}$
denotes the safe distance threshold, which is set to 2 m. When the distance falls within this region, the reward value is negative, and the penalty increases as the agent approaches the obstacle. When
$d_{\text{sonar}} \geq d_{\text{safe}}$
lies within the safe zone, no additional reward or penalty is applied.
Prediction reward
$r_5$
: guides pursuers to perform anticipatory intercept by estimating the motion trend of the target.
\begin{align} \dot {P}_e = P_e + \frac {P_{e,t} - P_{e,t-1}}{\Delta t} \cdot \tau _p \\[-10pt] \nonumber \end{align}
\begin{align} \;\dot{\!\!d}_t = \| p_i - \dot {P}_e \|_2 \qquad \\[-10pt] \nonumber \end{align}
\begin{align} r_5 = \left ( \;\;\dot {\!\!d}_{t-1} - \;\;\dot {\!\!d}_t \right ) \cdot k_p \qquad \\[10pt] \nonumber \end{align}
Among them,
$\dot {P}_e$
denotes the predicted future position of the evader, while
$P_{e,t-1}$
represents the previous position of the evader.
$\tau _p$
denotes the prediction horizon, which is set to five time steps in this study.
$\;\;\dot {\!\!d}_t$
represents the current distance between the pursuer and the predicted target position, and
$\;\;\dot {\!\!d}_{t-1}$
denotes the distance between the pursuer and the predicted target position at the previous time step.
$k_p$
is the prediction reward coefficient, which is set to 0.5 in this paper.
4.1.3. Strategic-level rewards
Strategic Reward
$r_i^{\text{st}}$
consists of team rewards and capture rewards, which are designed to support long-term objectives and promote cooperative pursuit behavior. The strategic layer emphasizes global optimality rather than individual optimality, thereby addressing the credit assignment problem and coordination challenges in multi-agent systems.
Team Reward
$r_6$
: This reward is designed to promote cooperative encirclement among multiple pursuers and is implemented using a conditional triggering mechanism.
\begin{equation} r_6 = \begin{cases} k_t (d_{\text{thr}} - \bar {d}), & \text{if } \forall j \in \{1,2,3\}, d_j \lt d_{\text{thr}} \\[3pt]0, & \text{otherwise} \end{cases} \end{equation}
Among them,
$k_t$
represents the team reward coefficient, which is set to 0.5 in this paper.
$d_{\text{thr}}$
denotes the triggering threshold for the team reward. The team reward is activated only when the distance between each pursuer and the target is smaller than this threshold, ensuring that the pursuers form an overall encirclement of the target. In this paper, the threshold is set to 15 m.
Capture reward
$r_7$
: used for differentiated reward allocation upon task completion, thereby addressing the credit assignment problem in multi-agent systems.
\begin{equation} r_7 = \begin{cases} Ra_m, & \text{if } i = \arg \min _j d_j \text{ and } d_i^{\min } \leq d_{\text{trigger}} \\[3pt]Ra_t \left (1 - \frac {d_i}{d_{\text{coop}}}\right )\!, & \text{if } i \neq \arg \min _j d_j \text{ and } d_i \leq d_{\text{coop}} \\[3pt]0, & \text{otherwise} \end{cases} \end{equation}
Among them,
$R$
represents the total capture reward, which is set to 50 in this study.
$a_m$
denotes the main capturer reward coefficient and is set to 0.6.
$\arg \min _j d_j$
represents the pursuer closest to the target, that is, the agent that ultimately completes the capture.
$d_i^{\min }$
denotes the minimum distance between the
$i$
-th agent and the target.
$a_t$
represents the collaborator reward coefficient, which is set to 0.4.
$d_{\text{coop}}$
denotes the collaboration reward threshold. The smaller the distance, the larger the value of this term and the higher the reward, thereby encouraging cooperative encirclement behavior. In this study, it is set to 20 m.
Finally, the total reward
$r_i$
of pursuer
$i$
is defined as the sum of the seven reward components described above.
\begin{equation} r_i = r_1 + r_2 + r_3 + r_4 + r_5 + r_6 + r_7 \end{equation}
Among them,
$r_1$
denotes the proximity reward,
$r_2$
denotes the rotation penalty,
$r_3$
denotes the low-speed penalty,
$r_4$
denotes the obstacle avoidance reward,
$r_5$
denotes the prediction reward,
$r_6$
denotes the team reward, and
$r_7$
denotes the capture reward.
4.2. SVRN-based dual-critic network design
Due to the inherent non-stationarity in multi-agent reinforcement learning, training with a single global critic network often suffers from instability and difficulties in credit assignment. Specifically, the global critic struggles to accurately evaluate the individual contributions of each agent, while relying solely on a local critic lacks the global coordination perspective required for collaborative tasks.
To address the above challenges, this paper proposes a Scheduled Value Reweighting Network. The network introduces a local critic in addition to the conventional global critic to provide auxiliary value estimation from a local perceptual perspective, while dynamically reweighting their contributions through a linear scheduling function. During training, the contributions of the global and local critics are dynamically adjusted through the scheduled value reweighting mechanism, where the global critic maintains the dominant role in final value estimation. This design enables a smooth transition from local auxiliary evaluation to global optimization dominance. Based on the proposed scheduled reweighting mechanism, the fused Q-value is computed as follows:
\begin{equation} Q_i^h(s, a) = \lambda _g(t) Q_i^g(s, a) + \lambda _l(t) Q_i^l(o_i, a_i) \end{equation}
Among them,
$Q_s^h(s,a)$
represents the fused action-value function of agent
$s$
.
$\lambda _g(t)$
and
$\lambda _l(t)$
denote the time-varying weights, which satisfy
$\lambda _g(t) + \lambda _l(t) = 1$
. The global weight gradually increases from the initial value
$\lambda _g^{\text{init}}$
to the final value
$\lambda _g^{\text{final}}$
as the number of training steps increases.
\begin{equation} \lambda _g(t) = \lambda _g^{\text{final}} + \left ( \lambda _g^{\text{init}} - \lambda _g^{\text{final}} \right ) \left ( 1 - \frac {t}{T_{\text{decay}}} \right ) \end{equation}
Among them,
$T_{\text{decay}}$
represents the total number of steps for weight decay, which is set to 40,000 in this study.
The scheduled value reweighting mechanism is inherently coupled with the proposed heterogeneous dual-critic architecture rather than being an independent annealing operation. The necessity of this mechanism originates from the complementary information provided by the global and local critics. The global critic evaluates joint state-action information to capture cooperative decision-making patterns among multiple AUVs, while the local critic estimates individual observation-action values under partial observability and provides additional behavioral guidance. Since these two critics focus on different levels of task information, their contributions should be dynamically coordinated during policy learning. During the early training stage, the local critic provides informative individual-level feedback to facilitate the acquisition of basic pursuit behaviors and improve local decision-making capability. As the learned policies gradually develop cooperative behaviors, the global critic becomes increasingly important for evaluating joint actions from a team-level perspective. Therefore, the contribution of the local critic is gradually reduced while maintaining the dominant role of the global critic throughout the training process. Although linear annealing strategies have been widely adopted in reinforcement learning, the proposed linear scheduling strategy differs from conventional annealing methods that are usually applied to independent optimization parameters, such as exploration rates or learning rates. In this framework, the scheduling function directly regulates the interaction between two heterogeneous critics with different information scopes and functional roles. A linear scheduling strategy is adopted because its constant transition rate enables a smooth and monotonic adjustment of critic contributions without introducing additional shape parameters. Compared with nonlinear scheduling strategies, such as exponential or piecewise decay, linear scheduling reduces parameter sensitivity and avoids abrupt variations in value fusion. This design enables the value estimation process to gradually transition from auxiliary individual-level guidance toward global cooperative evaluation while maintaining stable training dynamics.
This design enables agents to gradually transition from relying on individual strategies to leveraging global guidance as the training process progresses. As illustrated in Figure 3, the SVRN-based dual-critic architecture stabilizes multi-agent training through dual-channel value estimation, making SVRN-MADDPG particularly suitable for scenarios where the target motion pattern is unpredictable.
Neural network architecture.
Figure 3. Long description
Panel A: A sequence of images showing an underwater scene with multiple objects. Panel B: A flowchart depicting the neural network architecture. The architecture includes an Actor network and two Critic networks, labeled as Local Critic network and Global Critic network. The Actor network processes inputs from various states, including the state of the AUV, team state, target state, and sonar state, to generate actions. These actions are then evaluated by the Local and Global Critic networks, which provide feedback in the form of Q-values. The Experience Replay Buffer stores past experiences for training the networks.
4.3. Active sonar obstacle avoidance
In multi-AUV reinforcement learning pursuit tasks, the training process of reinforcement learning (RL) agents is highly susceptible to environmental disturbances. The core challenge lies in the difficulty for agents to rapidly learn safe obstacle avoidance behaviors under sparse reward signals, which often leads to collisions with obstacles, inter-agent collisions, and local deadlocks. These issues may result in training interruptions, slow convergence, and even failure to learn effective cooperative pursuit strategies. To address these problems and ensure stable training and policy convergence of RL agents, this paper designs a reactive obstacle avoidance strategy based on forward-looking sonar. This mechanism is embedded as a low-level safety constraint within the RL training framework, forming a hierarchical control scheme that combines hard obstacle avoidance safety guarantees with high-level RL-based autonomous obstacle avoidance learning.
The forward-looking sonar is configured as a fan-shaped multi-beam ranging sensor in the UUV Simulator and mounted at the front of the AUV. The sonar array consists of 700 beams, where each beam corresponds to a predefined angular direction in the sonar coordinate frame. At each control step, the sonar returns a one-dimensional range measurement vector with an effective sensing range of 1–5 m. Before being fed into ASOA, the raw sonar measurements are preprocessed to extract low-dimensional obstacle-related features. The 700-beam range vector is uniformly divided into 44 consecutive angular sectors according to the beam order. Invalid measurements, including infinite and non-numerical values, are removed, and the minimum valid distance within each sector is retained as the representative sector feature. This sector-based nearest-distance extraction reduces the input dimension while preserving the closest obstacle information required for safety-oriented obstacle avoidance. Based on the obtained 44-dimensional sector representation, the feature vector is divided into two symmetric regions corresponding to the left and right sides of the AUV, respectively. The average valid distance of each region is calculated to estimate the surrounding free space. When the minimum sector distance falls below the ASOA activation threshold, the AUV selects the direction with a larger average distance and generates an avoidance maneuver toward the more open region.
At the control level, ASOA serves as the underlying safety layer and has the highest execution priority. As illustrated in Figure 4, the red dashed line represents the obstacle avoidance trajectory generated by ASOA, while the black and red rays denote the sonar measurements in the left and right regions, respectively, with a sensing range of 1–5 m. The core principle of ASOA is to determine the turning direction based on the average sonar distance in the left and right regions. When the minimum sonar distance falls below the safety threshold, the system immediately triggers the obstacle avoidance behavior, overriding the action commands generated by reinforcement learning. In this case, the linear velocity is forcibly reduced, and the angular velocity is determined according to the sonar-based decision, enabling the AUV to quickly turn toward a more open area. Once the obstacle is cleared, control is automatically returned to the reinforcement learning policy. It is worth noting that ASOA does not completely replace the obstacle avoidance learning capability of RL. Instead, it achieves a balance between safety and learning through a hierarchical design. Specifically, the obstacle avoidance penalty trigger distance in the reward function is set slightly larger than the ASOA activation threshold. When an obstacle lies between these two thresholds, the RL policy remains active but receives a negative reward due to the short distance, encouraging the agent to move away from the obstacle proactively. This mechanism – reward guidance as the primary approach supplemented by hard obstacle avoidance constraints – enables RL agents to gradually acquire obstacle avoidance capabilities in a safe environment while avoiding training interruptions caused by collisions during the early training stages.
Schematic diagram of the ASOA obstacle avoidance principle.
This design follows a risk-aware hierarchical control principle, where the control authority is adjusted according to the environmental risk level. The proposed framework consists of three risk-level regions, including the safe region, warning region, and danger region. In the safe region, the reinforcement learning policy maintains full control authority to optimize long-term task performance. In the warning region, the reinforcement learning policy remains active, while safety-oriented reward shaping is introduced to encourage obstacle-aware behaviors. When the AUV enters the danger region, the ASOA controller temporarily overrides the learned policy to guarantee collision avoidance. This hierarchical design prioritizes operational safety while preserving the autonomous decision-making capability of the reinforcement learning agent.
Active Sonar Obstacle Avoidance (ASOA)
Algorithm 1 Long description
A textbox containing a step-by-step algorithm for Active Sonar Obstacle Avoidance (ASOA). The algorithm starts by computing the minimum sonar distance. It then determines the control mode and reward shaping based on the distance to the obstacle. If the minimum distance is greater than or equal to a penalty distance, the system operates in a safe zone with full reinforcement learning control and no obstacle penalty. If the minimum distance is less than or equal to the penalty distance but greater than a danger distance, the system operates in a warning zone with reinforcement learning control and negative reward shaping. If the minimum distance is less than the danger distance, the system enters a danger zone with a hard takeover by ASOA. The algorithm also includes a collision check and updates the reward accordingly.
The ASOA module is designed as a low-level safety filter embedded within the reinforcement learning framework. Although this mechanism provides safety intervention, the executed action may differ from the action directly generated by the actor network. When an AUV enters the danger zone, the action generated by the actor network is temporarily replaced by a safety-oriented control command. Since part of the sampled transitions are generated under safety-filtered actions rather than directly from the original policy, the critic may estimate state-action values influenced by safety intervention, resulting in potential value estimation bias. However, the influence of this bias is effectively bounded in the proposed framework. ASOA is activated only under safety-critical conditions and does not continuously intervene throughout the entire training process. Meanwhile, the warning-zone reward shaping provides additional obstacle-related feedback and reduces the probability of entering the danger zone. Therefore, as training progresses, ASOA intervention gradually decreases, and the learned reinforcement learning policy increasingly dominates the decision-making process. Although ASOA and APF-based methods both utilize obstacle-distance information for local obstacle avoidance, their functional roles and intervention mechanisms are different. Conventional APF methods generate motion commands through continuously acting attractive and repulsive potential fields, where the repulsive component directly influences trajectory generation. In contrast, ASOA adopts an event-triggered safety mechanism within the reinforcement learning framework and temporarily modifies actions only when collision risks become critical. In the safe and warning zones, the reinforcement learning policy remains the primary decision-making component. Thus, ASOA provides local safety protection within the reinforcement learning framework rather than replacing the learned policy or acting as an independent planning strategy.
The core value of this active obstacle avoidance strategy lies in establishing reliable low-level safety constraints for RL agents. By regularizing the fundamental requirement of obstacle avoidance, the RL agents are no longer required to spend extensive training steps learning basic collision avoidance behaviors from scratch. Instead, they can focus on learning high-level task strategies such as multi-agent cooperation, target encirclement, and dynamic pursuit. This design significantly improves the training convergence speed, training stability, and overall task success rate of the RL algorithm, thereby providing essential support for the effective implementation of multi-AUV reinforcement learning pursuit tasks.
5. Simulation results and discussion
5.1. Experimental platform scene modeling and simulation results
This paper constructs a simulation environment based on real ocean data for the modeling and validation of multi-AUV collaborative pursuit tasks. In this environment, the evader adopts a fixed evasion strategy integrated with game-theoretic principles to achieve autonomous escape, while the pursuers learn cooperative pursuit strategies through multi-agent reinforcement learning methods. The simulation experiments are conducted on a high-performance computing platform equipped with an Intel
$\circledR$
Core
$^{\text{TM}}$
i9-10920X CPU @ 3.50 GHz x 24 and an NVIDIA RTX 4000 GPU, running the Ubuntu 20.04.6 LTS operating system. The simulation environment is built on the UUV Simulator and Gazebo, enabling high-fidelity dynamic interactions among multiple AUVs in underwater environments. Compared with traditional two-dimensional or simplified dynamic simulation environments based on MATLAB, the simulation platform developed in this paper provides more realistic dynamic models as well as higher scalability and transferability. Therefore, the constructed simulation environment not only effectively verifies the performance of multi-agent collaborative pursuit strategies but also provides a more reliable experimental foundation for transferring the proposed algorithms to real ocean applications.
During the training process, as shown in Figure 5, the pursuit strategy exhibits clear stage-wise evolution characteristics. In the strategy exploration stage (a), each pursuer operates in an independent exploration state. Since the strategy has not yet converged, different agents exhibit significant behavioral differences. Some agents tend to move toward the evader, while others perform ineffective actions (such as hovering or circling), resulting in a lack of overall coordination. As training progresses, the system gradually enters the strategy stabilization stage (b). At this stage, most pursuers have learned effective goal-approaching strategies, and their behaviors become increasingly consistent, with the overall strategy showing a preliminary convergence trend. In the strategy convergence stage (c), multiple pursuers further develop collaborative behaviors. Through spatial cooperation, they jointly capture the evader, which significantly improves the task success rate. Eventually, the system strategy converges to a stable cooperative pursuit policy.
Evolution of multi-agent pursuit strategies during training. (a) Strategy exploration stage; (b) Strategy stabilization stage; (c) Strategy convergence stage.
Figure 5. Long description
The image contains six line graphs showing the evolution of multi-agent pursuit strategies during training. Each graph represents a different stage: (a) Strategy exploration stage, (b) Strategy stabilization stage, and (c) Strategy convergence stage. The graphs are arranged in two rows and three columns. Each graph has a legend indicating the paths of three agents and their targets. The x-axis represents the position coordinates ranging from 0 to 50, and the y-axis represents the position coordinates ranging from 0 to 50. The graphs show the paths taken by the agents and targets, with different colors representing different agents and targets. The agents start at different positions and move towards their targets, navigating around obstacles represented by large white circles.
The effectiveness of the proposed SVRN-MADDPG algorithm and the ASOA is verified through multiple comparative experiments. The evaluation focuses on four aspects: the effectiveness of the obstacle avoidance strategy, the structure of the critic network, algorithmic performance comparison, and the weight ratio between the global and local critics.
5.2. Algorithm ablation experiment
5.2.1. Verification of the effectiveness of active sonar obstacle avoidance strategy
Without introducing ASOA during training, the neural network must simultaneously learn the pursuit policy while avoiding fixed obstacles and other pursuers in the environment. Once a collision occurs, the current training episode is terminated. In the early stage of training, the policy has not yet converged, and the pursuers are still in the exploration phase, which makes collisions more likely. Consequently, most episodes terminate before reaching the maximum step limit in the early training stage, resulting in low training efficiency.
To address the above issues, this paper introduces the ASOA mechanism, which assists obstacle avoidance through a rule-based strategy, thereby significantly reducing collisions in the early stage of training. To evaluate the impact of the ASOA mechanism on algorithm performance, comparative experiments were conducted between algorithms with and without the ASOA mechanism (NO-ASOA) while maintaining identical network structures and training parameters. The experimental results are presented in Figure 6. From the perspective of convergence speed, the algorithm incorporating ASOA achieves stable convergence after approximately 400 training episodes. In contrast, the algorithm without ASOA exhibits significant fluctuations throughout the entire training process and fails to demonstrate a clear convergence trend, indicating that the ASOA mechanism effectively accelerates the policy learning process. When no obstacle avoidance mechanism is applied, the pursuit success rate of the algorithm is only 14%. Although the reward value converges to around 10, the learning process is frequently interrupted due to collisions between pursuers and obstacles or among multiple pursuers themselves, which significantly reduces the number of effective learning steps. After introducing the ASOA strategy, the pursuit success rate increases to 65%, and the reward value achieves stable positive convergence. These results demonstrate that the ASOA strategy effectively prevents training interruptions caused by collisions through a low-level rule-based obstacle-avoidance safety constraint mechanism. As a result, the reinforcement learning pursuers can focus on learning higher-level task strategies, such as multi-agent coordination and target encirclement, thereby significantly improving both the training stability and the task success rate of the algorithm.
Comparison of training performance with and without the ASOA strategy.
From a mechanistic perspective, the ASOA mechanism introduces a guidance term during the policy update process, enabling pursuers to maintain a safe distance from obstacles during the early stages of training. This effectively avoids the training overhead caused by collisions with obstacles that would otherwise terminate the entire training episode. Meanwhile, ASOA reduces the randomness of the initial exploration policy and decreases the probability that pursuers move toward obstacles, thereby making the training process smoother and more stable and effectively alleviating the non-stationarity problem commonly encountered in multi-agent environments. In addition, ASOA reduces the learning burden of the neural network, allowing the pursuers to focus primarily on target pursuit rather than learning obstacle-avoidance behaviors from scratch. As a result, the overall convergence speed of the algorithm is significantly improved.
5.3. Comparison of SVRN dual-critic and original MADDPG
The original MADDPG algorithm employs a single centralized critic network to evaluate value based on global state-action pairs. Although this architecture can model interactions among multiple agents, it exhibits significant limitations in multi-agent cooperative tasks. Since the value function relies entirely on global information, it is difficult to accurately distinguish the contribution of each individual agent’s action to the overall reward, which leads to severe credit assignment problems during policy updates. Particularly in the early stages of training, when agents’ policies have not yet converged, the global critic often struggles to provide effective feedback at the individual level. As a result, the learning direction of each agent may become unstable, thereby reducing the overall training efficiency.
Comparison of SVRN-MADDPG and original MADDPG.
To address the aforementioned issues, this paper introduces a SVRN dual-critic architecture that combines a local critic with a global critic (with a weight ratio of 0.2:0.8), enabling action-value evaluation at both the individual and team levels. As shown in Figure 7, the dual-critic network significantly improves both the task success rate and the reward convergence performance of the algorithm. Under the single critic network, the algorithm achieves only a 27% pursuit success rate, and the 100-round sliding average reward score converges around −30, remaining in the negative range. In contrast, the SVRN-based dual-critic architecture increases the success rate to 65%, with the reward score converging to approximately 25, achieving stable positive convergence. During training, the single critic method exhibits substantial fluctuations in the mid-to-late stages, indicating unstable policy updates. Conversely, the SVRN-based dual-critic method produces smoother reward curves and more stable convergence, suggesting that it can provide more accurate value evaluation signals for policy learning.
From a mechanistic perspective, the SVRN-based dual-critic structure effectively mitigates the problem of individual contributions being “overwhelmed” in the global critic by introducing local value evaluation. This allows the policy gradient to more accurately reflect the actual impact of individual agent actions, thereby improving the effectiveness of policy update directions. At the same time, the global critic constrains joint behaviors, ensuring that each agent considers not only its own policy optimization but also the overall cooperative strategy. Through the complementary roles of the local and global critic networks, the framework achieves a unified balance between individual optimality and team optimality.
It can be observed that the SVRN-based dual-critic network effectively addresses the credit assignment and collaborative learning problems in multi-agent reinforcement learning by simultaneously modeling both local and global value information. Consequently, the proposed architecture significantly improves the convergence performance of the algorithm and enhances its capability to accomplish the pursuit task.
5.4. Performance comparison of SVRN-MADDPG
This section evaluates the performance of the proposed SVRN-MADDPG method through simulation experiments in multi-AUV cooperative pursuit tasks. The analysis focuses on two key performance metrics: the average reward and the task success rate. To demonstrate the effectiveness of the proposed approach, SVRN-MADDPG is compared with several representative algorithms with different methodological characteristics, including MAPPO, MADDPG, MATD3, TP-MADDPG, and MPC + APF. Both MAPPO and MADDPG are benchmark algorithms based on the centralized training and decentralized execution (CTDE) framework. MAPPO improves policy optimization through a clipped surrogate objective, which stabilizes policy updates and makes it suitable for fused action spaces, although it may suffer from relatively high gradient variance. In contrast, MADDPG employs deterministic policy gradients and utilizes the actor-critic architecture to achieve low-variance gradient estimation, demonstrating strong convergence stability and sample efficiency in continuous control tasks.
In addition to these two general multi-agent reinforcement learning baselines, MATD3 is introduced as a representative dual-critic reinforcement learning method, in which two homogeneous critics are used to reduce value overestimation. TP-MADDPG is selected as a pursuit-evasion-oriented multi-agent reinforcement learning algorithm to evaluate the performance of task-specific policy learning. MPC + APF is adopted as a traditional model-based planning and obstacle avoidance method, providing a non-learning baseline for comparison. Through these baselines, SVRN-MADDPG is evaluated against general MARL methods, dual-critic reinforcement learning, pursuit-evasion-oriented learning, and traditional model-based planning methods.
In this study, the 100-episode sliding average reward curves of five reinforcement learning algorithms over 1200 training episodes are compared in Figure 8. SVRN-MADDPG (the SVRN-MADDPG proposed in this paper, blue curve) demonstrates the best performance, surpassing the zero-reward threshold at approximately 350 episodes and stabilizing within the range of + 10 to + 30 after around 600 episodes, exhibiting fast convergence and low variance. MADDPG (green curve) converges more slowly, with its final reward remaining around −30 and never entering the positive reward region. MAPPO (red curve) shows the poorest performance, with rewards fluctuating between −50 and −80, indicating significant stagnation in policy optimization. MATD3 (orange curve) shows a significant reward decrease during the early training stage and gradually converges to a relatively low reward level around −40, indicating that the homogeneous dual-critic mechanism can improve value estimation stability to some extent but provides limited support for cooperative pursuit learning in multi-AUV tasks. TP-MADDPG (purple curve) performs better than MADDPG, MAPPO, and MATD3, with its reward gradually increasing and stabilizing around −20 to −10 after convergence. This result suggests that pursuit-evasion-oriented policy learning can improve task performance, but its final performance remains lower than that of SVRN-MADDPG. These results indicate that the SVRN-based dual-critic architecture significantly improves both the convergence speed and the final performance of MADDPG. In contrast, MAPPO struggles to establish an effective value function estimation in high-dimensional continuous underwater control tasks due to the stochastic nature of its policy updates. Since MPC + APF is a traditional model-based planning and obstacle avoidance method rather than a reinforcement learning algorithm, it is not included in the training reward curve comparison. In terms of task success rate, SVRN-MADDPG achieves 65% during the 0–1200 training episodes, which is significantly higher than MADDPG (23%) and MAPPO (41%), and also outperforms TP-MADDPG (46%) and MATD3 (38%). In the subsequent 100-episode evaluation test, the success rate of SVRN-MADDPG further increases to 83%, while MAPPO improves to 59%, MADDPG reaches 42%, MATD3 reaches 42%, TP-MADDPG reaches 56%, and MPC + APF achieves 79%. Although MPC + APF shows competitive performance in this fixed evaluation setting, it relies on model-based planning and predefined potential-field obstacle avoidance rules rather than learned cooperative policies. In comparison, SVRN-MADDPG maintains the highest success rate among all reinforcement learning methods, demonstrating better cooperative pursuit capability after training.
Performance comparison results of SVRN-MADDPG.
The improvement in performance is primarily attributed to the proposed SVRN-based dual-critic mechanism. On the one hand, the local critic effectively guides each pursuer to learn target-approaching behaviors during the early training stage, thereby improving exploration efficiency. On the other hand, the global critic evaluates joint actions from a team-level perspective, facilitating the formation of effective cooperative strategies. By fusing the two value functions through weighted integration, SVRN-MADDPG enables a smooth and efficient transition from individual pursuit to cooperative pursuit during training, thereby significantly improving the overall pursuit performance.
To further evaluate the generalization ability and stability of each algorithm under varying initial conditions, this paper conducted 100 independent trials at five different initial positions and computed the corresponding task success rates. The results are presented in Figure 9. In addition, experiments were performed under ten randomly initialized conditions to calculate the average success rate and average reward score. The corresponding results are summarized in Table II.
Comparison of average performance of various algorithms in multiple random scenarios. Boldface highlights the proposed method and the best results for each metric.
Table II. Long description
The table presents a comparison of the average performance of various algorithms in multiple random scenarios. It has four columns and six rows. The columns are labeled Algorithm, Average success rate (%), and Average reward score. The row labels are the names of the algorithms: SVRN-MADDPG, MADDPG, MAPPO, MATD3, TP-MADDPG, and MPC + APF. The values in the table are as follows: Row 1: SVRN-MADDPG, 74.20, 20.791. Row 2: MADDPG, 23.60, 14.036. Row 3: MAPPO, 46.40, 4.437. Row 4: MATD3, 38.80, 10.241. Row 5: TP-MADDPG, 50.00, 16.582. Row 6: MPC + APF, 59.10, -. The boldface in the table highlights the highest values in the Average success rate (%) and Average reward score columns, indicating the best-performing algorithm.
Comparison of success rates of various algorithms at different initial positions.
As shown in Figure 9, the SVRN-MADDPG algorithm achieves higher task success rates across various test scenarios with different initial positions while exhibiting smaller performance fluctuations, indicating stronger environmental adaptability and policy robustness. In contrast, the success rates of MADDPG and MAPPO decrease significantly in certain scenarios, suggesting that these methods are more sensitive to initial conditions. MATD3 and TP-MADDPG achieve certain improvements in several scenarios, but their success rates still fluctuate under different initial positions, indicating that their learned policies remain sensitive to changes in initial spatial configurations. MPC + APF also obtains competitive performance in some scenarios, which confirms that traditional model-based planning combined with potential-field obstacle avoidance can accomplish the pursuit task to some extent under the fixed-evader setting. However, MPC + APF relies on short-horizon model prediction and predefined repulsive-field rules, rather than learning cooperative pursuit behaviors from multi-agent interactions. Therefore, its performance is more easily affected by the initial geometric relationship between pursuers and the evader, as well as the local obstacle distribution. Table II shows that SVRN-MADDPG outperforms the comparison algorithms in terms of both average success rate and average reward score. From the quantitative results, the average success rate of SVRN-MADDPG reaches 74.20%, which is significantly higher than that of MADDPG (23.60%), MAPPO (46.40%), MATD3 (38.80%), TP-MADDPG (50.00%), and MPC + APF (59.10%). Although MPC + APF achieves the highest success rate among the baseline methods, its average success rate is still 15.10 percentage points lower than that of SVRN-MADDPG. This result indicates that, even when the evader follows a fixed rule-based strategy, traditional control methods can provide effective local tracking and obstacle avoidance, but they are still less effective than the learned cooperative pursuit policy in multi-AUV encirclement tasks. Since MPC + APF is not trained through reinforcement learning, its average reward score is not reported in Table II. SVRN-MADDPG also achieves the highest reward score. These results indicate that the proposed method not only improves the probability of task completion but also yields better policy performance during the pursuit process. Overall, SVRN-MADDPG demonstrates clear advantages in both training and testing performance, thereby validating the effectiveness and generalization capability of the proposed SVRN-based dual-critic mechanism in multi-AUV cooperative pursuit tasks.
5.5. Analysis of the impact of global local critic weight ratio
To investigate the impact of the weight ratio between the global and local critic networks on algorithm performance, four comparative experiments were conducted with initial weight ratios of 0.2:0.8, 0.4:0.6, 0.6:0.4, and 0.8:0.2 for the local and global critic networks, respectively. The experiments evaluated the performance of each configuration in terms of task success rate, reward convergence, and training stability. As shown in Figure 10, the experimental results indicate that different weight ratios have a significant influence on the algorithm’s performance. The overall trend exhibits a clear pattern of initial improvement followed by performance degradation, suggesting the existence of an optimal balance between global and local value information.
Comparison of global local critic weight ratios.
Figure 10. Long description
A line graph titled ‘Comparison of 100-episode Moving Average Score’ displays the average score over the last 100 episodes on the vertical axis and the episode number on the horizontal axis. The graph includes four data lines representing different critic weight ratios: SVRN Local Critic Ratio 0.2 in blue, SVRN Local Critic Ratio 0.4 in orange, SVRN Local Critic Ratio 0.6 in green, and SVRN Local Critic Ratio 0.8 in red. The blue line (Ratio 0.2) generally shows the highest average scores, maintaining a positive trend throughout the episodes. The orange line (Ratio 0.4) and the green line (Ratio 0.6) exhibit moderate performance with fluctuating trends. The red line (Ratio 0.8) shows the lowest average scores, with a downward trend in the later episodes. The graph includes a legend identifying each line by color and critic ratio.
When the local-to-global weight ratio is 0.2:0.8, the algorithm achieves the best performance, with a success rate of 65%, the reward score converging to approximately 25, and the training curve exhibiting the smoothest and most stable behavior. This configuration indicates that, under this setting, the global critic dominates, providing consistent and stable value guidance during the early training stage and effectively alleviating the non-stationarity problem in multi-agent environments. Meanwhile, the introduction of the local critic allows individual agents to receive a certain degree of independent value feedback, thereby improving the refinement of policy updates while maintaining cooperative optimization. As the weight of the local critic increases (0.4:0.6 and 0.6:0.4), the success rate of the algorithm gradually decreases (57% and 60%, respectively), the convergence value of the reward score declines, and training fluctuations become significantly larger. This indicates that when the proportion of the global critic weight becomes insufficient, the global guidance for multi-agent cooperation weakens. As a result, individual behaviors may become misaligned with team objectives, leading to inconsistencies between individual optimality and team optimality, thereby weakening the formation of cooperative behaviors. When the local weight is further increased to 0.8:0.2, the algorithm performance deteriorates significantly, with the success rate dropping to 40% and the reward score converging to approximately −70, accompanied by evident instability. This phenomenon can be attributed to the excessive reliance on the local critic, which amplifies the non-stationarity of the multi-agent environment. Consequently, agents may experience stronger mutual interference during policy updates, leading to the accumulation of estimation bias in the value function and ultimately resulting in training non-convergence or even collapse.
From a mechanistic perspective, the weighting between the global and local critic networks reflects a balance between collaborative guidance capability and individual decision-making capability. The global critic provides consistent optimization directions and promotes the formation of cooperative pursuit behaviors; however, excessively strong global constraints may weaken the expressiveness of individual policies. In contrast, the local critic improves the accuracy of individual action-value evaluation, but assigning it excessively large weights may cause agents to neglect overall cooperative objectives, thereby affecting the stability of the multi-agent system. Therefore, an appropriate weight allocation between the global and local critic networks can maintain cooperative consistency while preserving individual learning capabilities, ultimately achieving improved overall performance.
In summary, the experimental results verify the critical impact of the weight ratio between the global and local Critic networks on the performance of multi-agent reinforcement learning. The results further indicate that a 0.2:0.8 local-to-global weight ratio achieves the best performance in the considered task. This finding provides an important reference for the parameter design of SVRN-based dual-critic architectures.
6. Conclusion
This paper proposed a SVRN-based dual-critic multi-agent deep reinforcement learning method, termed SVRN-MADDPG, to enable low-speed trackers to cooperatively capture high-speed escaping targets in constrained underwater environments. It innovatively adopts a dual-critic architecture based on the SVRN, which balances individual exploration and team cooperation, and effectively alleviates common challenges in multi-agent reinforcement learning, such as reward assignment difficulty and training non-stationarity. Furthermore, an ASOA strategy is integrated as a low-level safety constraint, which significantly reduces collision-induced episode terminations during the early training stage and accelerates policy convergence. The proposed approach was evaluated in a high-fidelity underwater simulation environment built on Gazebo and the UUV Simulator. Experimental results demonstrate that SVRN-MADDPG with ASOA outperforms benchmark algorithms (MADDPG and MAPPO) in terms of convergence speed, average reward, and task success rate, achieving efficient and safe cooperative pursuit control for multiple AUV. However, the current method assumes ideal sonar accuracy and does not account for time-varying communication delays or ocean current disturbances, which may affect its performance in real-world deployments. Future work will incorporate ocean current models and communication delay into the simulation environment and develop a delay-resilient training framework to enhance robustness. The scalability of the proposed algorithm will also be evaluated in larger-scale multi-AUV systems.
Data availability statement
The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.
Author contributions
Conceptualization: Zhenkun Lei; Methodology: Zhenkun Lei, Mingzhi Chen; Software: Zhenkun Lei; Writing – original draft: Zhenkun Lei; Writing – review editing: Mingzhi Chen; Supervision: Mingzhi Chen. All authors have read and agreed to the published version of the manuscript.
Financial support
This work was supported in part by the National Natural Science Foundation of China under Grants 52371331.
Competing interests
The authors declare no competing interests.
Ethical standards
Not applicable.
Use of AI
The authors used ChatGPT 5.6 Sol (OpenAI; https://chatgpt.com/) on 15 August 2026 to improve the English language, grammar, and readability of the manuscript. The authors reviewed and edited the AI-assisted text and take full responsibility for the final content. No proprietary or confidential information was provided to the tool. The authors did not modify the tool or add data to its public corpus. The authors declare no competing interests or potential biases arising from this use.
