2.1 UAV kinematic model

In the air combat game environment, our UAV and the target UAV compete in a three-dimensional space to achieve the battle goal of destroying the adversary while saving itself. The motion information of the UAV is described in the east-north-up (ENU) coordinate system OXYZ, whose origin O is located at a point on the ground, with the OX-axis in the due-east direction, the OY-axis in the due-north direction, and the OZ-axis pointing vertically to the sky.

A scalar representation of the dynamics and kinematics equations must be developed in order to study the UAV’s laws of motion. The flight characteristics of the UAV are investigated using the centre-of-mass dynamics equations within the flightpath axis system. By projecting the velocity, angular velocity of rotation, gravitational force, aerodynamic force and thrust force onto the coordinate system, the scalar dynamics equations of the UAV can be expressed as follows [Reference Stevens, Lewis and Johnson21]:

(1) \begin{align}\left\{ {\begin{array}{*{20}{l}}{m\dfrac{{dV}}{{dt}} = T{\rm{cos}}\left( {\varphi + \rho } \right){\rm{cos}}\chi - D - mg{\rm{sin}}\theta }\\[8pt]{mV{\rm{cos}}\theta \dfrac{{d\psi }}{{dt}} = T\left[ {{\rm{sin}}\left( {\varphi + \rho } \right){\rm{sin}}\phi - {\rm{cos}}\left( {\varphi + \rho } \right){\rm{sin}}\chi {\rm{cos}}\phi } \right] + C{\rm{cos}}\phi + L{\rm{sin}}\phi }\\[8pt]{ - mV\dfrac{{d\theta }}{{dt}} = T\left[ { - {\rm{sin}}\left( {\varphi + \rho } \right){\rm{cos}}\phi - {\rm{cos}}\left( {\varphi + \rho } \right){\rm{sin}}\chi {\rm{cos}}\phi } \right] + C{\rm{sin}}\phi - L{\rm{cos}}\phi + mg{\rm{cos}}\theta }\end{array}} \right.\end{align}

where $V$ , $\varphi $ , and $\chi $ are the speed, angle-of-attack, and sideslip angle of the UAV. $\rho $ is the engine mounting angle, $\theta $ and $\psi $ are track inclination and track declination, respectively. $\phi $ is the roll angle. D, C and L are the drag, side force and lift of the UAV, respectively.

The focus of the manoeuvre decision-making algorithm is to update the position and attitude of the UAV. Therefore, in this paper, the UAV is further considered as a prime and the simplified dynamics equations can be derived as follows:

(2) \begin{align}\left\{ {\begin{array}{*{20}{l}}{\dfrac{{dV}}{{dt}} = g\left( {{n_x} - {\rm{sin}}\theta } \right)}\\[8pt]{\dfrac{{d\theta }}{{dt}} = \dfrac{g}{V}\left( {{n_z}{\rm{cos}}\phi - {\rm{cos}}\theta } \right)}\\[8pt]{\dfrac{d}{{dt}} = \dfrac{{g{n_z}{\rm{sin}}\phi }}{{V{\rm{cos}}\theta }}}\end{array}} \right.\end{align}

where ${n_x}$ is the tangential overload, which represents the acceleration of the UAV along the fuselage direction; ${n_z}$ is the normal overload, which represents the acceleration of the UAV along the roof direction; and $\phi $ is the roll angle, which represents the control amount of the roll (fuselage to the right is positive). The control quantity of UAV is set to $\left[ {{n_x},{n_z},\phi } \right]$ , through which the flight attitude of the UAV can be changed.

As shown in Fig. 1, $\theta $ is the angle between the velocity vector $V$ of the UAV and the plane OXY, and $\psi $ is the angle between the projection $V{\rm{'}}$ of the velocity vector $V$ in the plane OXY and the OX axis. The three-dimensional spatial kinematic equations of the UAV in the OXYZ coordinate system are defined as follows:

(3) \begin{align}\left\{ {\begin{array}{*{20}{l}}{\dfrac{{dx}}{{dt}} = V{\rm{cos}}\theta {\rm{cos}}\psi }\\[8pt]{\dfrac{{dy}}{{dt}} = V{\rm{cos}}\theta {\rm{sin}}\psi }\\[8pt]{\dfrac{{dz}}{{dt}} = V{\rm{sin}}\theta }\end{array}} \right.\end{align}

where x, y and z indicate the coordinates of the UAV in the ENU coordinate system in three-dimensional space, respectively.

Figure 1.

UAV kinematics model.

2.2 Attack decision model

The attack decision model describes the ultimate goal of the UAV and defines in detail the conditions under which the UAV will fulfil its air combat mission. The mission objective of UAV close air combat is to quickly manoeuvre to a position where it can attack the target while ensuring safety. Figure 2 defines the attack range of the UAV, where ${\theta _A}$ represents the attack angle and ${D_A}$ represents the attack distance. Equation (4) is the decision condition to complete the mission objective, when the target enters the attack range of the UAV, it means the attack is successful.

(4) \begin{align}\left\{ {\begin{array}{*{20}{l}}{{\theta _{{\rm{attack}}}} \lt {\theta _A}}\\{{D_{{\rm{attack}}}} \lt {D_A}}\end{array}} \right.,\end{align}

where ${\theta _{{\rm{attack}}}}$ and ${D_{{\rm{attack}}}}$ are the actual attack angle and attack distance of the UAV, respectively.

Figure 2.

UAV attack range.

3.1 Markov decision process

Markov decision process theory, which began to mature gradually in the 1960s, is the theoretical basis of reinforcement learning for modelling decision-making problems in uncertain environments. The UAV air combat autonomous manoeuvre decision-making task can be characterised and solved by a Markov decision process $M = \left( {S,A,P,R,\gamma } \right)$ [Reference Masson, Ranchod and Konidaris22], which contains state space $S$ , action space $A$ , reward function $R$ , transition probability function $P$ and discount factor $\gamma $ . The basic structure of reinforcement learning is shown in Fig. 3. At each discrete time, the agent performs an action on the environment based on the current strategy. The state shifts from and the agent receives a reward at the next moment.

Figure 3.

Basic structure of reinforcement learning.

3.1.1 State space

In order to plan strategies and manoeuvre decisions effectively, environmental information needs to be filtered and extracted to accurately obtain situational information about the UAV and the target. Air combat situational information is described by the state space of the UAV manoeuvre decision model, which contains distance $D$ , attack information $\left( {{q_U},{q_T}} \right)$ , position information $\left( {{x_U},{y_U},{z_U},{x_T},{y_T},{z_T}} \right)$ , speed information $\left( {{V_U},{V_T}} \right)$ and attitude information $\left( {{\theta _U},{\psi _U},{\phi _U},{\theta _T},{\psi _T},{\phi _T}} \right)$ . The normalised state space is shown in Table 1, where ${d_0}$ and ${v_0}$ are the normalisation constants.

Table 1.

State space

3.1.2 Action space

In the majority of prior research, the action space of UAV was discrete. Although they cover the common manoeuvres of UAV, the parameters of these manoeuvres are fixed and cannot be quantitatively changed or combined. Directly discretising the continuous action space can lead to the curse of dimensionality, making it challenging for the algorithm to converge. In this paper, we choose $\left[ {{n_x},{n_z},\phi } \right]$ as the control quantity, indicating that the action space of the UAV agent is continuous. The design of continuous action space is shown in Table 2.

Table 2.

Action space

3.1.3 Reward reshaping function

In the close air combat game problem, the opposing sides have intense confrontations and rapid manoeuvre transitions. The reward function defines the goal and learning direction of the agent, directly affecting its learning process and final performance. The main objective of the UAV agent is to get rid of enemy targeting and seize a favourable attack position to shoot down the target. As shown in Equation (5), a task reward ${R_0}$ is designed to measure the completion of agent for the air combat task. When the target is within range of the UAV, it is considered a successful attack, giving a positive reward ${R_s}$ . When the UAV is within range of a target attack, it is considered destroyed, giving a negative reward ${R_f}$ . A certain out-of-bounds penalty ${R_o}$ is given when the distance between the UAV and the target exceeds a given value or when the step length reaches the maximum time step. To encourage completing the task as quickly as possible, UAV receives a negative time reward ${R_t}$ for making action selections.

(5) \begin{align}{R_0} = \left\{ {\begin{array}{l@{\quad}l}{{R_s} = 200\,{\rm{*}}\,{{\left( {\dfrac{{maxstep - step}}{{maxstep}}} \right)}^2}} & {}{{\rm{successful\;attack}}}\\[10pt]{{R_f} = - 50}& {}{{\rm{wreck}}}\\[8pt]{{R_o} = - 30} {}& {{\rm{out}} - {\rm{of}} - {\rm{bounds\;penalty}}}\\[8pt]{{R_t} = - 0.1\,{\rm{*}}\,step} {}& {{\rm{time\;penalty}}}\end{array}} \right.\end{align}

where $maxstep$ is the maximum time step and $step$ is the current round step length.

Autonomous air combat manoeuvre decision-making problem is a typically sparse reward problem, which can lead to slow model learning. To avoid sparse results caused by excessive trial and error in the initial phase, the prior knowledge is introduced indirectly using reward shaping. In order to guide the UAV to learn how to choose a manoeuvre strategy, a reward function with reward shaping based on air combat situation assessment is designed in this section. Considering several factors, such as the influence of angle, distance, speed and altitude on the air combat posture, the integrated reward function combining task reward and auxiliary guidance reward is designed as follows:

(6) \begin{align}R = {R_0} + {\omega _A}{R_A} + {\omega _D}{R_D} + {\omega _V}{R_V} + {\omega _H}{R_H}\end{align}

where ${R_0}$ is the task reward of Equation (5). ${R_A}$ , ${R_D}$ , ${R_V}$ and ${R_H}$ are the angle reward, distance reward, speed reward and height reward, respectively. ${\omega _A}$ , ${\omega _D}$ , ${\omega _V}$ and ${\omega _H}$ are their weights, respectively.

Table 3.

Weights of the auxiliary guidance reward

In different air combat game situations, each posture factor has different impacts on the overall air combat situation, and the weights of the auxiliary guidance reward function are designed as shown in Table 3. When the two UAVs are far away from each other, set the distance reward weight higher to guide the UAV to approach the target and reach the attack range quickly. When the two UAVs are in close proximity, set the angle reward weight higher to encourage the UAV to maintain an offensive posture while ensuring its safety.

Angle reward

Angle is the most important factor in air combat, and attacks with combat weapons must first fulfil the angular attack conditions. The Tail Bite strategy is used in close air combat to attack or defend, encouraging the UAV to point at the target fuselage while avoiding the target pointing at us. The angle reward function is designed as follows:

(7) \begin{align}{R_A} = 1 - \frac{{{q_U} + {q_T}}}{\pi }\end{align}

where $q_U$ and ${q_T}$ are the angles of deviation and detachment, taking values in the range $\;\left[ {0,\pi } \right]$ . As shown in Fig. 4, when ${q_U}$ tends to 0, the accuracy of the attack aiming at the target is high, indicating that the UAV has a high chance of firing. When it ${q_T}$ tends to 0, the probability of the target dodging the attack is low, indicating that the target has a low chance of survival.

Figure 4.

Air combat situational information diagram.

Distance reward

Distance is an important factor in air combat situation, which is largely influenced by the range of the weapon. As shown in Fig. 5, in order to guide the UAV to maintain an appropriate distance, the distance reward function is designed to gain a better air combat advantage.

(8) \begin{align}{R_D} = {e^{ - \frac{{{{(d - {D_{{\rm{opt}}}})}^2}}}{{{D_0}}}}}\end{align}

where $d$ is the actual distance and ${D_{{\rm{opt}}}}$ is the optimal attack distance. ${D_0}$ is used to adjust the gradient of the distance reward function.

Speed reward

When UAV has a suitable speed advantage, it can enter the attack advantage zone with a faster flight speed and quickly avoid the risk of being attacked by the target. However, when the speed difference is too large it is not favourable for firing weapons and may consume more fuel to maintain speed, reducing operational effectiveness. As shown in Fig. 6, the speed difference reward function is designed as follows:

(9) \begin{align}{r_{{\rm{\Delta }}v}} = {e^{ - \frac{{{{({\rm{\Delta }}v - {V_{{\rm{opt}}}})}^2}}}{{{V_0}}}}}\end{align}

where ${\rm{\Delta }}v$ is the actual speed difference and ${V_{{\rm{opt}}}}$ is the optimal speed difference. ${V_0}$ is used to adjust the gradient of the speed difference reward function.

Figure 5.

Distance reward curve.

At the same time, in order to ensure that the UAV can fly safely and avoid the problem of flight stall, the self speed constraint reward is designed as follows:

(10) \begin{align}r_{hself} = \left\{ \begin{array}{l@{\quad}l} \frac{h}{100} - 3 & h \lt 300\;\mathrm{m} \\ 0 & h \ge 300\;\mathrm{m} \end{array}\right.\end{align}

The speed reward function is defined as:

(11) \begin{align}{R_V} = {r_{{\rm{\Delta }}v}} + {r_{vself}}.\end{align}

Height reward

The suitable height advantage can help the UAV to adjust its spatial position and attitude. Converting gravitational potential energy into kinetic energy can obtain a higher manoeuvre rate and achieve a combat mission engagement the target. As shown in Fig. 7, the height difference reward function is designed as follows:

(12) \begin{align}{r_{{\rm{\Delta }}h}} = {e^{ - \frac{{{{({\rm{\Delta }}h - {H_{{\rm{opt}}}})}^2}}}{{{H_0}}}}}\end{align}

where ${\rm{\Delta }}h$ is the actual height difference and ${V_{{\rm{opt}}}}$ is the optimal height difference. ${H_0}$ is used to adjust the gradient of the height difference reward function.

During air combat pursuit, the UAV may be induced to crash land by the target. In order to improve the UAV’s ability to correct its altitude autonomously and increase operational safety, the self height constraint reward is designed as follows:

(13) \begin{align}{r_{hself}} = \left\{ {\begin{array}{l@{\quad}l}{\frac{h}{{100}} - 3} {}& {h \lt 300\;{\rm{m}}}.\\0 & {}{h \ge 300{\rm{\;m}}}\end{array}} \right.\end{align}

The height reward function is defined as:

(14) \begin{align}{R_H} = {r_{{\rm{\Delta }}h}} + {r_{hself}}.\end{align}

3.2 Principles of the DDPG algorithm

DDPG [Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra23] algorithm is a continuous control method that combines deep Q-network and actor-critic network. Drawing on the techniques of target network and experience replay, DDPG can handle the task of spatial decision-making for continuous manoeuvres in air combat.

Figure 6.

Speed reward curve.

Figure 7.

Height reward curve.

The traditional policy function is a probability density function and the output is a probability value. In contrast, the output of DDPG’s deterministic policy network $\mu\!\left({s;\vartheta } \right)$ is a three-dimensional action vector $a = \left[ {{n_x},{n_z},\phi } \right]$ , representing the control quantity of the UAV. The value network $q\!\left( {s,a;\,\omega } \right)$ is an approximation of the action value function ${Q_\pi}\!\left({s,a} \right)$ with inputs state $s$ and action $a$ . The output q-value is used to evaluate the performance of the policy network and help train the policy network.

Traditional reinforcement learning algorithms such as SARSA and REINFORCE are typically on-policy, discarding experience after a single learning iteration. The strong correlation between neighbouring experiences can also reduce learning efficiency and lead to overfitting. In contrast, as shown in Fig. 8, the off-policy algorithm allows behaviour policy to be different from the actual policy, enabling the reuse of experience gathered from behaviour policy. The policy network $\mu\!\left({s;\,\theta } \right)$ of DDPG is an off-policy method, which utilises the behaviour policy interacts with the environment to collect experiences into the experience replay buffer. Subsequently, the actual policy is trained by experience samples from the replay buffer.

Figure 8.

Off-policy method.

Figure 9.

The framework of the DDPG algorithm.

The framework of the DDPG algorithm is shown in Fig. 9, where $N$ training samples $\left( {{s_t},{a_t},{r_t},{s_{t + 1}}} \right)$ are randomly selected from the experience replay buffer, representing the state, action, reward and next state of the agent. These samples are used to compute the temporal-difference (TD) target, a key concept in reinforcement learning where the value estimate of the current state is updated based on the immediate reward and the discounted value of the next state. The training process for DDPG is as follows:

1. 1. The target policy network predicts the output manoeuvre control quantity $\left[ {{n_x},{n_z},\phi } \right]$ :

   (15) \begin{align}a_{t + 1}^{\rm{'}} = \mu \left( {{s_{t + 1}};\,\boldsymbol{\vartheta} '} \right)\end{align}

2. 2. The target value network performs predictions to compute the TD target:

   (16) \begin{align}{\hat y_t} = {r_t} + q\left( {{s_{t + 1}},a_{t + 1}^{\rm{'}};\,\boldsymbol{\omega} '} \right)\end{align}

3. 3. The value network performs predictions to compute the TD error:

   (17) \begin{align}{\delta _t} = q\left( {{s_t},{a_t};\,\boldsymbol{\omega}} \right) - {\hat y_t}\end{align}

4. 4. Update the value network according to the gradient descent method:

   (18) \begin{align}\boldsymbol{\omega} \leftarrow \boldsymbol{\omega} - \alpha \,{\rm{*}}\,{1 \over N}\mathop \sum \limits_{t = 1}^N {\delta _t}\,{\rm{*}}\,{\nabla _\omega }q\left( {{s_t},{a_t};\,\boldsymbol{\omega}} \right)\end{align}

5. 5. Update the policy network according to the gradient ascent method:

   (19) \begin{align}\vartheta \leftarrow \vartheta + \beta \,{\rm{*}}\,{1 \over N}\mathop \sum \limits_{t = 1}^N {\nabla _\vartheta }\mu \left( {{s_t},{a_t};\,\boldsymbol{\vartheta}} \right)\,{\rm{*}}\,{\nabla _a}q\left( {{s_t},{a_t};\, \boldsymbol{\omega}} \right)\end{align}

6. 6. Update the target network according to soft update:

   (20) \begin{align}\left\{ {\begin{array}{*{20}{l}}{\boldsymbol{\omega} ' = \tau \boldsymbol{\omega} + \left( {1 - \tau } \right)\boldsymbol{\omega} '}\\[4pt]{\vartheta ' = \tau \vartheta + \left( {1 - \tau } \right)\vartheta '}\end{array}} \right.\end{align}

3.3 Principles of the TD3 algorithm

DDPG is an off-policy algorithm with its evolved overestimation problem. It exploits the error in the value network so that the value network significantly overestimates the q-value of the state-action pair, which leads to the destruction of the policy network. Consequently, the practical performance of DDPG may be less effective, and it is relatively sensitive to hyperparameters. The twin delayed deep deterministic policy gradient (TD3) [Reference Fujimoto, Hoof and Meger24], building upon DDPG, optimises both the policy and value network simultaneously, effectively addressing the issues of overestimation and high variance.

1. 1. Clipped double Q-learning

   To address the overestimation issue, TD3 employs two independent value network and a policy network. Corresponding to each neural network are target networks with the same structure but different parameters. As shown in Equation (21), the smaller value calculated from the two target value networks is utilised as the TD target. Subsequently, the two value networks are trained by minimising the mean square deviation.

   (21) \begin{align}{\hat y_t} = {r_t} + \gamma {\rm{min}}\left( {q\left( {{s_{t + 1}},a_{t + 1}^{\rm{'}};\,\boldsymbol{\omega _1^{\rm{'}}}} \right),q\left( {{s_{t + 1}},a_{t + 1}^{\rm{'}};\,\boldsymbol{\omega_2^{\rm{'}}}} \right)} \right)\end{align}

2. 2. Delayed policy updates

   The guidance of the policy network update by the value network is crucial. If the value network is unreliable, it can negatively impact the performance of the algorithm. Omitting the use of target networks may lead to instability in the training process. Therefore, updating the policy and target networks at a slower frequency than the value network can lead to convergence to the correct solution.

3. 3. Target policy smoothing

   TD3 introduces a regularisation strategy for deep value learning, target policy smoothing, which adds noise $\xi $ to the output of the target policy network. By smoothing the variation of the target value network along the action, the policy network is prevented from exploiting the errors of the value network.

   (22) \begin{align}\left\{ {\begin{array}{*{20}{l}}{a_{t + 1}^{\rm{'}} = \mu \left( {{s_{t + 1}};\,\boldsymbol{\vartheta '}} \right) + \xi }\\{\xi \sim clip\left( {{\mathcal N}\left( {0,{\sigma ^2}} \right), - c,c} \right)}\end{array}} \right.\end{align}
   where $\xi $ is a randomly generated action vector. $clip\left( {{\mathcal N}\left( {0,{\sigma ^2}} \right), - c,c} \right)$ denotes a normal distribution with mean 0 and standard deviation $\sigma $ , clipped in the interval $\left[ { - c,c} \right]$ . Each element of $\xi $ is randomly and independently drawn from the clipped normal distribution.

3.4 Framework of autonomous decision-making algorithm based on HTRPER-TD3

To mitigate the issues of data correlation and wastage of experiences, traditional experience replay algorithm stores transitions $\left[ {{s_t},{a_t},{r_t},{s_{t + 1}}} \right]$ in the replay buffer and randomly samples transitions from the buffer during training. When new experiences are stored and the buffer reaches capacity, the oldest data is discarded. Nevertheless, not all experiences are equally important, and some may be more critical to learning.

Prioritised experience replay (PER) [Reference Schaul, Quan, Antonoglou and Silver25] is a variant of the traditional experience replay algorithm used in reinforcement learning. PER assigns different weights to experiences based on their importance and ranks them accordingly. Non-uniform sampling of the experiences in the buffer allows for a higher probability of selecting those with greater importance, thereby enhancing learning efficiency. During the initial phase of air combat training, critical tactical decisions (e.g. evasive manoeuvres and sustained pursuit) play a decisive role in shaping combat situations and constitute primary learning objectives. The PER mechanism, relying solely on the TD error for experience prioritisation, demonstrates limited efficacy in identifying the experiential value of such critical decisions. To address this limitation, we propose a novel algorithm that constructs a hybrid priority metric through weighted integration of TD error and immediate reward in experience prioritisation. Furthermore, a dynamic decay ratio is designed to progressively regulate reward weights, enabling the algorithm to leverage the strong guidance of reward signals for accelerated strategy learning during early training stages while gradually enhancing the dominance of the TD error to ensure stable policy convergence as training progresses.

The success of air combat fundamentally relies on the coordinated progression of consecutive tactical manoeuvres (for example, securing advantageous positions, adjusting attack postures and executing strike missions). However, traditional sparse reward schemes only assign an episode-terminating reward to the final state, resulting in diluted contributions of critical intermediate decisions. To address this issue, we introduce a reward backpropagation mechanism that retroactively propagates the task reward in a decaying manner along the decision sequence, thereby amplifying the contribution of critical decisions preceding episode completion and consequently enhancing the efficiency of policy learning. Importantly, the reward backpropagation is applied only during the priority calculation, while the actual rewards obtained during the learning process remain unchanged.

(23) \begin{align}r_{t - k}^{trace} + = 0.01 \cdot {r_{end}} \cdot \gamma _r^k{\rm{\;\;\;\;for\;\;}}k = 0,1, . . . ,L\end{align}

where $r_{t - k}^{trace}$ is the reward after the backpropagation, and ${r_{end}}$ is the task reward for each episode. ${\gamma _r}$ is the backpropagation discount factor. $L$ is the number of steps in reward backpropagation.

To effectively leverage massive experience, we implement a TD error filtering technique. Specifically, for each transition, we compute its TD error and employ an adaptive threshold set at 1% of the current maximum TD error to automatically identify well-trained samples, decreasing its contribution during priority calculation.

(24) \begin{align}\delta _t^{filter} = \left( {\begin{array}{l@{\quad}l}{{\delta _t}/n} {}& {{\rm{if}}\left| {{\delta _t}} \right|\left\langle {0.01 \cdot } \right|\delta {|_{{\rm{max}}}},}\\{{\delta _t}} & {}{{\rm{otherwise}}.}\end{array}} \right.\end{align}

The transition priority is defined as

(25) \begin{align}{p_t} = \left( {1 - \eta } \right) \cdot \left| {\delta _t^{filter}\left| { + \eta \cdot } \right|r_t^{trace}} \right| + \varepsilon \end{align}

where $\delta _t^{filter}$ is the filtered TD error. $r_t^{trace}$ is the reward for backpropagation of the transition. $\eta $ is the dynamic decay ratio. $\varepsilon $ is a small positive constant that prevents the sampling probability from being close to zero and ensures that all samples are likely to be drawn.

Direct sampling with Equation (25) is pure greedy sampling, which leads to the reuse of experiences with relatively high priorities. In contrast, experiences with relatively small initial priorities are rarely used, resulting in a lack of sample diversity. To overcome these issues, a combination of uniform and greedy sampling is used and the probability of sampling transition $t$ is defined as:

(26) \begin{align}{P_t} = {{{p_t}^a} \over {\mathop \sum \nolimits_k {p_k}^a}}\end{align}

where ${p_t}$ is the priority of transition. The exponent $a$ is used to regulate the proportion of combinations of greedy and uniform sampling, with $a = 0$ corresponding to the uniform sampling.

Non-uniform sampling alters the distribution of samples, with different probabilities for sampling different transitions, which leads to bias in the model’s predictions. To ensure both an increase in training speed and convergence to the same results, we introduced a method of importance sampling. The importance sampling weights are designed as follows:

(27) \begin{align}{w_t} = {\left( {\frac{1}{{M\,{\rm{*}}\,{P_t}}}} \right)^b}\end{align}

where $M$ is the number of samples in the buffer. $b$ is the importance sampling weight parameter, which denotes the extent to which offsetting non-uniform sampling affects the convergence results. After normalisation ${w_t}$ become

(28) \begin{align}{w_t} = \frac{{{{\left( {M\,{\rm{*}}\,{P_t}} \right)}^{ - b}}}}{{\mathop {{\rm{max}}}\limits_t {w_t}}} = \frac{{{{\left( {{P_t}} \right)}^{ - b}}}}{{\mathop {{\rm{max}}}\limits_t \left[ {{{\left( {{P_t}} \right)}^{ - b}}} \right]}} = {\left( {\frac{{{P_t}}}{{\mathop {{\rm{min}}}\limits_t \left( {{P_t}} \right)}}} \right)^{ - b}}\end{align}

The proposed autonomous air combat manoeuvre decision-making training framework centres on the hybrid td error-reward prioritised experience play with TD3 (HTRPER-TD3). To address limitations in traditional methods, including insufficient identification of critical tactical decisions and the diluted contributions of critical intermediate decisions under sparse rewards, we establish a closed-loop learning system spanning from environment interaction to policy optimisation. As illustrated in Fig. 10, within this framework, the UAV agent collects experience in the air combat environment using behavioural policy, storing the transitions $\left( {{s_t},{a_t},{r_t},{s_{t + 1}}} \right)$ into the replay buffer. During the experience replay phase, we construct a hybrid priority by integrating reward backpropagation with TD error filter, enabling precise identification of high-value experiences. This mechanism is further enhanced with importance sampling to balance sample diversity with critical experience reuse. Through continuous environmental interaction, the agent progressively optimises network parameters, ultimately converging to a robust air combat manoeuvre decision-making policy. The implementation details are shown in Algorithm 1.

Figure 10.

The framework of the autonomous decision-making algorithm based on HTRPER-TD3.

Algorithm 1.

HTRPER-TD3 Algorithm

4.2 Comparison of simulation results

4.2.1 Basic manoeuvre engagement scenario

In basic manoeuvre engagement scenarios, our UAV is autonomously controlled by an air combat decision-making algorithm, while the target UAV executes three distinct kinematic patterns: constant speed straight flight, serpentine manoeuvre and horizontal hover. During each round of training, when the UAV reaches the mission judgement condition of Equation (4), the round is terminated and initialised to restart the next training round.

Figure 11.

Constant speed straight engagement training curves.

Figure 11 shows the sliding average curves for reward, step length and success rate within the constant speed straight flight scenario for the target UAV. The sliding average curve, calculated from the average value over every hundred training rounds, effectively filters out noise and volatility from the raw data, thereby enhancing the graph’s legibility. In the initial phase of training, the UAV agent was unexperienced and chose poor strategy during manoeuvre decision-making. Even in the case of step length growth, it did not obtain a high reward and success rate. As the training progressed, the average reward and success rate gradually increased and converged, indicating that the agent’s decision-making ability and task completion rate were continuously improving. Additionally, the decreasing trend in the step length curve indicates that the agent has learned a more rational manoeuvre strategy.

Figure 12 shows the manoeuvre trajectory and posture value change of the UAVs after raining in the constant speed straight engagement scenario. Initially, our UAV is behind the target UAV and in a dominant position, then the target UAV tries to manoeuvre away from our UAV. Under the condition of ensuring its safety, our UAV maintains an advantageous position and reaches the attack range quickly to accomplish the objective of shooting down the target UAV. Figure 13 shows the manoeuvre trajectory and posture value change of the UAVs in the serpentine manoeuvre and horizontal hover engagement scenario.

Figure 12.

Constant speed straight engagement manoeuvre trajectory.

Figure 13.

Serpentine manoeuvre and horizontal hover engagement manoeuvre trajectory.

In the simulation experiment, we compare the HTRPER-TD3-based air combat manoeuvre algorithm with other algorithms, demonstrating the advantages of the algorithm in air combat decision-making. Table 6 records the convergence results of different algorithms with the same parameters during the constant speed straight.

Among algorithms applicable to continuous action space, HTRPER-TD3 demonstrates superior convergence performance. It achieves convergence in just 1,000 episodes, which is faster than TD3-PER (1,350), TD3 (4,000) and DDPG (5,400). HTRPER-TD3 not only converges more quickly but also achieves the highest reward peak (157.2) and reward convergence value (155.9), outperforming all other algorithms. In comparison, TD3-PER, while already faster than TD3 and DDPG, still requires more episodes to converge and achieves lower reward values. This indicates the HTRPER-TD3 further improves the training efficiency and reward performance of the algorithm, effectively identifying and utilising key experience samples. Compared to TD3-PER, which also demonstrates good stability, HTRPER-TD3 further reduces jitter during convergence, resulting in smoother and more reliable training outcomes. DDPG, on the other hand, remains significantly less stable, with high jitter and less satisfactory convergence results even after extensive training. HTRPER-TD3 not only converges faster and more stably but also achieves superior task efficiency. It attains the smallest step length (43.0), indicating that it can complete missions more efficiently with fewer steps. This efficiency is critical in real-world applications, where minimising the number of steps can directly impact mission success and resource utilisation.

4.2.2 Expert system-based engagement scenario

In the air combat scenario against the expert system-based target UAV, our UAV is controlled by the air combat autonomous decision-making algorithm, while the target UAV is governed by an expert system evolved through adversarial training against randomised strategies using the TD3 algorithm. During close-range combat engagements, our UAV is randomly initialised within a spherical region, whereas the target UAV is deployed in a spherical shell space centred on our UAV’s position, maintaining a minimum horizontal safety distance of 500 m and spanning between 2,000 and 3,000 m.

Table 6.

Performance of constant speed straight engagement

Convergence reward, step length and success rate are the averages computed over the last 2,000 episodes. Convergence episode is the episode when the reward reaches 90% of the convergence reward.

Table 7.

Adversarial performance against expert system

Figure 14 shows the air combat training results against the expert system, and Table 7 records the convergence results of different algorithms with the same parameters during the training process. It can be observed that, compared to basic manoeuvre engagement scenarios, the expert system possesses autonomous manoeuvre capabilities, which significantly increases the convergence difficulty for the algorithms. From the analysis of the reward curve and success rate curve, the HTRPER-TD3 algorithm demonstrates significant advantages throughout the training process. Its reward values and success rates consistently surpass those of other algorithms, achieving rapid convergence to stable high values in later stages (convergence reward: 120.7, success rate: 81.7%), indicating exceptional policy stability. In contrast, DDPG maintains a low reward value (56.9) with severe fluctuations, reflecting inefficient exploration and policy fragility. While TD3 and TD3-PER show improvements in peak rewards and convergence speed compared to DDPG, they still fall short of HTRPER-TD3’s superior performance.

Figure 14.

Expert system engagement training curves.

Notably, the HTRPER-TD3 algorithm achieves a higher average step length (75.7) compared to TD3-PER (72.7) and TD3 (68.5), while step statistics include situations of air combat failure and draw. Combined with Fig. 15, Unlike engagements against basic manoeuvre strategies, this increased step length reflects the agent’s game resilience in disadvantaged situations. By dynamically adjusting its manoeuvre strategies, HTRPER-TD3 prolongs engagement rounds, avoiding rapid defeat by the expert system. This approach not only enables HTRPER-TD3 to secure victories in scenarios that might otherwise result in stalemates but also facilitates the accumulation of richer adversarial experience. This capability is directly linked to the proposed hybrid TD error-reward prioritised experience play mechanism, ultimately manifesting as a higher success rate (81.7%) and lower convergence episode (3,700), indicating superior policy generalisation within shorter training cycles. In contrast, DDPG exhibits insufficient policy stability due to its high convergence episode (7,900), low convergence reward (56.9) and poor success rate (54.5%). While TD3 and TD3-PER show improvements, their policy stability and task generalisation remain inferior to HTRPER-TD3.

Figure 15.

Dynamic changes in the distribution of results during the training process.

Figure 16 shows the manoeuvre trajectory and posture value change of the UAVs in the air combat scenario against the expert system. In Fig. 16(a), our UAV is in a disadvantaged posture, with the target UAV holding both altitude and angular advantages and approaching aggressively. Our UAV first accelerates and climbs to a certain altitude before the target UAV approaches, overcoming the initial disadvantage and gradually gaining angular attack advantage. Subsequently, our UAV waits for the target UAV to come within proximity and then executes a strike from below. In Fig. 16(b), upon detecting the target UAV, our UAV rapidly manoeuvres to close in on the target. When the target UAV attempts to leave the engagement zone, our UAV persistently pursues it and ultimately complete the strike before the target UAV can readjust its attack posture.

Figure 16.

Expert system engagement manoeuvre trajectory.

The comprehensive evaluation demonstrates that the HTRPER-TD3 algorithm exhibits outstanding performance in the field of autonomous decision-making for air combat. The algorithm significantly outperforms comparative algorithms in key metrics such as reward acquisition and mission success rate, while also featuring rapid convergence and excellent training stability, achieving a steady state within fewer training episodes. In complex air combat scenarios, the algorithm shows strong adaptability, capable of quickly learning highly reasonable and effective manoeuvre strategies, and achieving precise strikes through robust decision-making capabilities.