2.1 Form

Assume that the WIG traverses a trajectory consisting of $m$ steps and has not reached the target altitude ${h^{\rm{*}}}$ by the ${t_r}$ -th step. The whole trajectory can be divided into two parts by the step ${t_r}$ , as illustrated in Fig. 2. To evaluate an optional trajectory, two kinds of reward are calculated in each step, which are named as $r_r^{\left( t \right)}$ representing the reward of RT and $r_e^{\left( t \right)}$ representing the reward of AE, expressed as:

(1) \begin{align} r_e^{\left( t \right)} = \left\{ {\begin{array}{*{20}{l}}{0,} \quad \qquad\qquad\qquad{{\rm{if\;}}t \lt {t_r}}\\[6pt]{1 - \left| {\dfrac{{{h_i} - {h^{\rm{*}}}}}{{{h^{\rm{*}}} - {h_0}}}} \right|,} {} \quad \quad{{\rm{if\;}}t \geqslant {t_r}}\end{array}} \right. \end{align}

(2) \begin{align}r_r^{\left( t \right)} = \left\{ {\begin{array}{*{20}{l}}{0,} \qquad \quad \quad \ \qquad\quad\qquad \qquad {{\rm{if\;}}t \ne {t_r}}\\[6pt]{1 + \dfrac{{n\,{\rm{*}}\left( {n + 1} \right)}}{2}{\rm{\;}},{\rm{\;\;}}n = m - t,} \quad {{\rm{if\;}}t = {t_{}}}\end{array}} \right. \\[3pt] \nonumber \end{align}

where ${h_i}$ is the WIG’s flight altitude at the $i$ -th step. ${h_0}$ is the initial altitude at the beginning of the trajectory, where the WIG starts to operate. $n$ is the number of remaining steps after RT occurs, i.e., steps ( $t$ +1) to $m$ , so $n = m - t$ . For the whole trajectory, these two kinds of corresponding cumulative rewards are ${R_r}$ and ${R_e}$ , respectively, and their sum is the reward evaluated for the trajectory, given as:

(3) \begin{align} {{R_r} = \mathop \sum \limits_{t = 1}^m r_r^{\left( t \right)}} \nonumber \\[3pt] {{R_e} = \mathop \sum \limits_{t = 1}^m r_e^{\left( t \right)}} \\[3pt] {R = {R_e} + {R_t}} \nonumber \end{align}

Figure 2.

The division of the whole trajectory and reward for different steps.

Rewards for parts of a trajectory are also shown in Fig. 2. $r_e^{\left( t \right)}$ are assigned as 0 for steps from the first to the ( ${t_r}$ -1)-th, where RT has not occurred. With the exception of the ${t_r}$ -th step, $r_t^{\left( t \right)}$ of all other steps are set to 0.

The process by which the reward function assesses the reward of an available trajectory where the WIG adjusts its altitude to a higher one is as follows, and the algorithm operates as 9(‘)@, whose flowchart is shown in Fig. 3. It proceeds in the forward direction from the initial state to the end of the trajectory, with the RT initially assumed to occur at the final step, i.e. $m$ -th step. When the WIG’s flight altitude exceeds the target altitude for the first time at an intermediate step, the occurrence of the RT is identified, and the corresponding ${t_r}$ for that trajectory is thereby determined. In the current study, which considers the case of the WIG reducing its flight altitude, the condition $h\,{\rm{*}} \gt {h_t}$ is used to determine whether the RT has occurred.

1. (1) Give the trajectory with the altitude of each trajectory point ${h_t}$ , $t\,$ = $\,1,2 \ldots m$ .

2. (2) Initially, the order of the current step, $t$ =1. Initially, the total RT-related and total IAE-related reward, ${R_r}\,$ = 0 and ${R_e}\,$ = 0.

3. (3) Assume the rise time occurs at the last step, as ${t_r}\,$ = $\,m$ .

4. (4) If the current step $t$ is after ${t_r}$ , meaning that $t \gt {t_r}$ , the current step RT reward $r_r^{\left( t \right)}$ is assigned 0, and the current step AE reward $r_e^{\left( t \right)}$ is calculated by Equation (1) based on the current flight altitude ${h_t}$ .

5. (5) If $t \lt {t_r}$ and ${h_t}$ is higher than the target altitude ${h^{\rm{*}}}$ , ${t_r}$ is replaced by $t$ . $r_r^{\left( t \right)}$ and $r_e^{\left( t \right)}$ are calculated by Equations (1) and (2), respectively.

6. (6) If $t \lt {t_r}$ and ${h_t} \lt {h^{\rm{*}}}$ , $r_r^{\left( t \right)}$ and $r_e^{\left( t \right)}$ are both assigned 0.

7. (7) Accumulate $r_r^{\left( t \right)}$ and $r_e^{\left( t \right)}$ to the total RT-related reward ${R_r}$ and total IAE-related reward ${R_e},$ respectively.

8. (8) Proceed one step forward, and loop through steps (4) to (7), until the current step reaches the end.

9. (9) Summarise ${R_r}$ and ${R_e}$ as the total reward $R$ of the trajectory.

Figure 3.

Workflow of reward evaluation on the trajectory.

2.2 Proof

Assuming there are multiple trajectories for the WIG to manoeuvre during altitude change, consider one trajectory $T$ with RT as ${t_r}$ . For comparison, there exists another trajectory $T{\rm{'}}$ with a smaller RT denoted as ${t'_{r}}$ , assuming ${t'_{r}} = {t_r} - \delta $ . The respective rewards for the two trajectories are labeled as $R$ and $R{\rm{'}}$ , which can be combined using Equations (1), (2) and (3), then expressed as:

(4) \begin{align} R = \mathop \sum \limits_{t = {t_r}}^m r_e^{\left( t \right)} + {{n\,{\rm{*}}\left( {n + 1} \right)} \over 2} + 1{\rm{\;}},{\rm{\;}}n = m - t \\[-10pt] \nonumber \end{align}

(5) \begin{align} R{\rm{'}} = \mathop \sum \limits_{t = {t'_{r}}}^m {r'_{e}}^{\left( t \right)} + {{n{\rm{'*}}\left( {n{\rm{'}} + 1} \right)} \over 2} + 1{\rm{\;}},{\rm{\;}}n{\rm{'}} = m - t' \\[10pt] \nonumber \end{align}

where ${r'_{e}}^{\left( t \right)}$ and $n{\rm{'}}$ are the AE reward of each step and the number of subsequent steps after RT for $T{\rm{'}}$ , $n{\rm{'}} = n + \delta $ . $R{\rm{'}}$ minus $R$ gives:

(6) \begin{align}{\rm{\Delta }}R = R{\rm{'}} - R = \mathop \sum \limits_{t = {t'_{r}}}^m {r'_{e}}^{\left( t \right)} - \mathop \sum \limits_{t = {t_r}}^m r_e^{\left( t \right)} + \delta \,{\rm{*}}\left(n + {{\delta + 1} \over 2}\right)\\[-25pt] \nonumber \end{align}

Algorithm 1

Algorithm 1 Long description

The algorithm starts by inputting the altitude of each trajectory point. It initializes variables R subscript r and R subscript e to 0, and t to 0. The time step of rise time is assumed as t subscript r equals m. A while loop runs as long as t is less than or equal to m. Inside the loop, if t is greater than t subscript r, r subscript r superscript t and r subscript e superscript t are set to 0 and 1 minus the absolute value of h subscript t minus h asterisk divided by h asterisk minus h subscript 0, respectively. If h subscript t is greater than h asterisk, t subscript r is set to t, r subscript r superscript t is set to 1 plus n times n plus 1 divided by 2, where n equals m minus t, and r subscript e superscript t is set to 1 minus the absolute value of h subscript t minus h asterisk divided by h asterisk minus h subscript 0. If neither condition is met, r subscript r superscript t and r subscript e superscript t are set to 0. R subscript r and R subscript e are updated by adding r subscript r superscript t and r subscript e superscript t, respectively, and t is incremented by 1. The loop continues until t exceeds m. Finally, R is calculated as the sum of R subscript r and R subscript e, and R is outputted.

To explore the minimum value of ${\rm{\Delta }}R$ , note that the per-step AE reward lies within [0,1] for $r_e^{\left( t \right)}$ and ${r'_{e}}^{\left( t \right)}$ . Since $\mathop \sum \nolimits_{t = {t'_{r}}}^m {r'_{e}}^{\left( t \right)}$ appears as an additive term, its minimum value should be taken as 0. In contrast, $\mathop \sum \nolimits_{t = {t_r}}^m r_e^{\left( t \right)}$ appears as a subtractive term, and thus its maximum, $n + 1$ , should be used. Consequently, ${\rm{\Delta }}R$ can be derived as:

(7) \begin{align}{\rm{\Delta }}R = \delta\, {\rm{*}}\left( {n + \frac{{\delta + 1}}{2}} \right) - \left( {n + 1} \right) \\[-30pt] \nonumber \end{align}

Given that $\delta $ is a positive integer, $\delta \geqslant 1$ , the value of ${\rm{\Delta }}R$ is therefore never less than 0, ${\rm{\Delta }}R \geqslant 0$ . Moreover, since the boundary value cannot be reached in practice, ${\rm{\Delta }}R$ is greater than 0, ${\rm{\Delta }}R \gt 0$ . As the optimisation algorithm seeks trajectories with larger reward, it will prefer $T{\rm{'}}$ over due to its higher reward, thereby driving RT to decrease, as:

(8) \begin{align}R \lt R{\rm{'}} \Rightarrow T \to T' \Rightarrow {t_r} \to {t'_{r}} \end{align}

Meanwhile, the smaller the deviation, the larger the IAE reward becomes.

2.3 Interpretation

For comparison, trajectories where RT occurs in the ${t_r}$ -th and ${t'_{r}}$ -th step is selected, assuming ${t'_{r}}$ = ${t_r}$ -1. Figure 4 illustrates both typical and extreme cases for these two types of trajectories. ${T_1}$ represents a typical possible trajectory. After RT, if there is some deviation from the target altitude that does not reach the maximum, the reward ${R_e}$ will not be maximised. If no deviation from the target altitude occurs after RT, as in trajectory ${T_2}$ , ${R_e}$ reaches its maximum value, and this trajectory achieves the highest possible reward for RT= $t$ . When RT decreases, if the deviation after RT reaches its maximum, as in trajectory ${T_3}$ , the reward ${R_{e{\rm{'}}}}$ drops to its minimum of zero, representing the lowest possible reward for RT= $t{\rm{'}}$ . A more common case with sub-maximum deviation is trajectory ${T_4}$ , where ${R_{e{\rm{'}}}}$ is greater than zero. To drive the transition from trajectories with RT= ${t_r}$ to those with RT= $t{{\rm{'}}_r}$ feasibly through reward shaping, it must be ensured that the lower bound of the reward evaluation for RT= $t{{\rm{'}}_r}$ trajectories is no less than the reward value for RT= ${t_r}$ trajectories. Therefore, the following equation must be satisfied:

(9) \begin{align}{R_{r{\rm{'}}}} = {R_r} + {\rm{max}}\,\left( {{R_e}} \right) \end{align}

Figure 4.

The comparison of the reward for trajectories whose RT occurs at $t$ -th step and ( $t{\rm{'}}$ )-th step, respectively.

The RT reward after a one-step reduction must equal the sum of the RT reward and the maximum achievable IAE reward before the reduction. This condition enables the transition from trajectory ${T_2}$ to ${T_3}$ and makes it possible to drive the reduction in RT. This relationship can be formally expressed as

(10) \begin{align}{R_{r{\rm{'}}}} - {R_r} \geqslant {R_e} \end{align}

Physically, this equation implies that the incremental gain in RT reward resulting from a one-step decrease in RT should be greater than the loss of IAE reward caused by the RT reduction. In more general cases, such as trajectories ${T_1}$ and ${T_4}$ , the RT reward gain exceeds the corresponding IAE loss. Moreover, the trajectory after RT reduction still retains a residual IAE reward. These effects collectively promote the transition from trajectories like ${T_1}$ toward those like ${T_4}$ .

The above reasoning applies to adjacent RT differing by exactly one time step. This concept can be generalised to multi-step RT differences through iterative computation, allowing derivation of the RT reward for RT occurring at any arbitrary step. For notational clarity, $R_r^{\left( t \right)}$ and $R_e^{\left( t \right)}$ represent the RT reward and IAE reward, respectively, when RT= $t$ , that can be derived step-by-step backwards, as:

(11) \begin{equation}\begin{aligned}{R_r^{\left( t \right)}} {}& { = R_r^{\left( {t + 1} \right)} + {\rm{max}}\left( {R_e^{\left( {t + 1} \right)}} \right)}\\[5pt]{} {}& { = R_r^{\left( {t + 2} \right)} + {\rm{max}}\left( {R_e^{\left( {t + 2} \right)}} \right) + {\rm{max}}\left( {R_e^{\left( {t + 1} \right)}} \right)}\\[5pt]{} {}& {\qquad\qquad\qquad \vdots }\\[5pt]{} {}& { = R_r^{\left( m \right)} + {\rm{max}}\left( {R_e^{\left( m \right)}} \right) + {\rm{max}}\left( {R_e^{\left( {m - 1} \right)}} \right) \cdots + {\rm{max}}\left( {R_e^{\left( {t + 2} \right)}} \right) + {\rm{max}}\left( {R_e^{\left( {t + 1} \right)}} \right)}\\[5pt]{} {}& { = R_r^{\left( m \right)} + 1 + 2 + 3 + \cdots + n}\\[5pt]{} {}& { = 1 + \dfrac{{n\left( {n + 1} \right)}}{2}}\end{aligned} \end{equation}

where $R_r^{\left( m \right)}$ is defined as the initial RT reward for the situation where RT occurs at the last step, $m$ -th step. To prioritise achieving RT in the final step of the trajectory, the RT reward value should not exceed the maximum AE, i.e. 1. From the end to the beginning, the reverse analysis of RT reward for the step where RT happens is conducted to define its magnitude, illustrated in Fig. 5.

Table 1 shows the upper and lower bounds of the reward for trajectories whose RT occurs at different steps. Moving from left to right, the RT becomes progressively smaller. Although ${R_r}$ and ${R_e}$ is greater and greater, the maximum of the total reward is the same as the minimum of that when RT occurs in one earlier step. The staircase-like pattern strictly drives the trajectory to shift towards the direction of RT’s decrease.

Table 1.

Bounds of reward for the trajectory whose rise time occurs in different steps

Table 1. Long description

The table is labeled as Table 1 and titled Bounds of reward for the trajectory whose rise time occurs in different steps. It shows the relationship between rise time in different steps and the corresponding reward magnitude and total reward. The table has two columns: Rise time in -th step and Reward Magnitude. The rise time steps are labeled as equals, equals minus 1, equals minus 2, and equals minus 3. The reward magnitude values are 1, 1, 2, 2, 4, 3, 7, and 4. The total reward is given as 1 plus, open parenthesis 1 plus n times open parenthesis n plus 1 close parenthesis divided by 2 close parenthesis.

Figure 5.

RT reward ${R_r}$ and the bounds of IAE reward ${R_e}$ if rise time occurs in the last five steps.

The RT reward value exhibits a form similar to the sum of an arithmetic sequence, i.e. $n$ ^*( $n$ +1)/2 concerning the remaining steps, which is the triangular number shown in Fig. 6.

Figure 6.

The form of a triangular number for the construction of the RT reward ${R_r}$ based on IAE reward ${R_e}$ .

3.1 Flight simulation

Validation is carried out using a dynamic and kinematic model of a small WIG operating in the longitudinal plane. The model has three degrees of freedom, with control surfaces consisting of an elevator and a propulsion [Reference Hu, Zhang, Ding, Jiao, Zhang and Zhang4]. Its geometrical shape and coordinate frames are shown in Fig. 7. The inertial frame ${F^i}$ is a fixed Earth-based reference frame, with its origin located at a specified home position. Its unit vector ${i^i}$ points forward, and ${k^i}$ points downward. The WIG-fixed frame ${F^f}$ is attached to the vehicle’s centre of gravity ( $CG$ ), and its axes are aligned with those of the inertial frame. The body frame ${F^b}$ also originates at $CG$ , but its axes move with the vehicle: ${i^b}$ points toward the nose, and ${k^b}$ points downward along the belly direction [Reference Beard and McLain34]. The WIG’s state quantities are displacements and angles along with their first and second derivatives, and the flight altitude can be extracted from the state. Aerodynamic forces applied to the WIG are computed using the vortex lattice method and image method [Reference Hu, Zhang, Ding, Jiao, Zhang and Zhang4], and then fed into the kinematic model to calculate the WIG’s motion response.

(12) \begin{align} \dot {\boldsymbol{u}} = {\boldsymbol{u}} \times {\omega} + {\boldsymbol{C}}_{\boldsymbol{g}}^{\boldsymbol{b}} \cdot \left( {\frac{{\boldsymbol{F}}}{{{m_w}}} + {\boldsymbol{g}}^{\boldsymbol{n}}} \right) \\[-10pt] \nonumber \end{align}

(13) \begin{align} \dot \omega = {{\boldsymbol{I}}^{ - 1}}\left( { - \omega \times {\boldsymbol{I}}\omega + {\boldsymbol{M}}} \right) \\[10pt] \nonumber\end{align}

where ${m_w}$ is mass of the WIG. $\boldsymbol{u} = {\left[ {u,0,w} \right]^T}$ and $\omega = {\left[ {0,q,0} \right]^T}$ denote the translational and angular velocity vectors in the body frame ${F^b}$ . The aerodynamic force and moment vectors are given by $\boldsymbol{F} = {\left[ {{F_x},0,{F_z}} \right]^T}$ and $\boldsymbol{M} = {\left[ {0,{M_y},0} \right]^T}$ , respectively. Gravity in the inertial frame is represented as ${\boldsymbol{g}^\boldsymbol{n}} = {\left[ {0,0,g} \right]^T}$ , and $m$ is the vehicle mass. Velocities $u$ and $w$ are along the body axes ${i^b}$ and ${k^b}$ , with their derivatives $\dot u$ and $\dot w$ indicating linear accelerations. The pitch rate $q$ is the angular velocity about ${j^b}$ . Forces ${F_x}$ and ${F_z}$ act along ${i^f}$ and ${k^f}$ in the WIG-fixed frame ${F^f}$ , while $g$ denotes gravitational acceleration. The transformation matrix $\boldsymbol{C}_\boldsymbol{g}^\boldsymbol{b}$ converts vectors from ${F^f}$ to ${F^b}$ . The inertial velocity $\boldsymbol{V} = {\left[ {{V_x},0,{V_z}} \right]^T}$ is obtained via $\boldsymbol{V} = \boldsymbol{C}_\boldsymbol{b}^\boldsymbol{g} \cdot \boldsymbol{u}$ , where ${V_x}$ and ${V_z}$ are velocity components along ${i^i}$ and ${k^i}$ , and $\boldsymbol{C}_\boldsymbol{b}^\boldsymbol{g} = {\left( {\boldsymbol{C}_\boldsymbol{g}^\boldsymbol{b}} \right)^T}$ . The inertia tensor is denoted by $\boldsymbol{I}$ .

Figure 7.

The WIG’s geometry and coordinate frames for describing the WIG’s motion.

3.2 Optimisation method

The reward function directs the trajectory towards the extreme value, and therefore, optimisation algorithms are employed to identify this trajectory. Consequently, the identified trajectory is analysed to determine whether it aligns with the design expectations. Deep reinforcement learning (DRL) excels in solving complex, high-dimensional and dynamic decision-making problems by autonomously learning sequential policies. Thus, DRL is utilised as an optimisation tool in this research to search for the trajectory corresponding to the reward function. The basis structure of deep reinforcement learning for WIG’s trajectory planning is shown in Fig. 8. Specifically, the PPO algorithm is employed, and details such as hyperparameters can be found at the link mentioned in the final section Data availability statement . PPO is a policy gradient method that improves stability by limiting the policy update step size. The mathematical formulation of PPO and the corresponding importance sampling mechanism are given in [Reference Schulman, Wolski, Dhariwal, Radford and Klimov35]:

(14) \begin{align} {L^{CLIP}}\left( \theta \right) &= {\mathbb{\hat{E}}_{t}} [ {{\rm{min}}( {{r_t}\left( \theta \right){{\hat A}_t},{\rm{clip}}\left( {{r_t}\left( \theta \right),1 - \varepsilon ,1 + \varepsilon } \right){{\hat A}_t}} )} ] \\[-10pt] \nonumber\end{align}

(15) \begin{align} & \qquad \qquad {r_t}\left( \theta \right) = \frac{{{\pi _\theta }\left( {{a_t}|{s_t}} \right)}}{{{\pi _{\theta {\rm{old}}}}\left( {{a_t}|{s_t}} \right)}} \\[10pt] \nonumber \end{align}

where $\theta $ denotes the parameters of the policy ${\pi _\theta }$ , which defines the probability of taking action ${a_t}$ given state ${s_t}$ at timestep $t$ . The term ${\hat A_t}$ represents the estimated advantage at time $t$ , quantifying how favourable action ${a_t}$ is in state ${s_t}$ compared to the expected baseline. It is defined as ${\hat A_t}\left( {s,a} \right) = \mathbb{E}\left[ {r\left( {s,a} \right) - r\left( s \right)} \right]$ , where $r\left( {s,a} \right)$ denotes the expected reward for taking action $a$ in state $s$ , and $r\left( s \right)$ is the expected reward for state $s$ regardless of action. The objective $L$ is used to guide gradient ascent updates. The expectation operator ${{\hat {\mathbb E}}_t} $ indicates an empirical average over a batch of collected data in the alternating sampling and optimisation process. The ratio ${r_t}$ is the importance sampling weight, representing the relative likelihood of the action under the new policy compared to the old one and is essential for stable updates. The function ${\rm{clip}}\left( \cdot \right)$ constrains the ratio within a predefined interval to prevent overly large updates; for instance, values below 0 become 0, and those above 1 become 1 when clipped to $\left[ {0,1} \right]$ . The hyperparameter $\varepsilon $ controls the clipping threshold in the objective function, limiting how far the policy is allowed to deviate during each update. This work implements PPO using ElegantRL [Reference Xiaoyang, Zechu, Zhaoran and Jiahao36], an open-source DRL framework.

Each DRL case is trained with eight different random seeds, and the training result with the highest reward value is selected as the final outcome for that case. The training hyperparameters are set as follows: each policy iteration collects 1,024 complete trajectory steps of data, with the clip ratio set to 0.15. During training, when the reward showed no improvement after 200 consecutive iterations, the learning rate is reduced from the initial ${10^{ - 4}}$ to ${10^{ - 5}}$ , and the entropy coefficient is decreased from 0.01 to 0.005. Training then continued until the reward again showed no improvement after another 200 iterations, at which point the learning rate is further reduced to ${10^{ - 6}}$ and the entropy coefficient to 0.0025. The training is terminated when no reward improvement is observed for a final 200-iteration period. Table 2 lists hyperparameters of the DRL algorithm.

Table 2.

Hyperparameters of DRL for the adjustment of altitude

Table 2. Long description

The table lists hyperparameters for a deep reinforcement learning algorithm used for altitude adjustment. It has six rows and two columns. The columns are labeled Hyperparameters and Magnitude. The rows are as follows: Row 1: Learning rate for policy optimiser and value function optimiser, 0.0003; Row 2: Discount factor, 0.99; Row 3: Number of steps of interaction per epoch/replay buffer, 1024; Row 4: Batch size, 64; Row 5: Repeating times for each batch to be trained on per training iteration, 16; Row 6: Clipping ratio, 0.15; Row 7: Entropy coefficient, 0.01, 0.005, 0.0025; Row 8: Consecutive epochs if no growth in episode reward for the next and, 200.

Figure 8.

The structure of deep reinforcement learning for WIG’s trajectory planning.

4.1 Effectiveness

To validate the effectiveness of the proposed method, we employed a traditional multi-objective function configured with a sequence of seven continuously increasing weight values. This setup simulates the practical process of iteratively tuning weights in multi-objective optimisation until convergence to the target trajectory is achieved. The proposed method is implemented under the same conditions for comparison. The traditional approach uses seven distinct weight values: $k = 0$ , $k = {10^{ - 2}}$ , $k = {10^{ - 1}}$ , $k = {10^0}$ , $k = {10^1}$ , $k = {10^2}$ and $k = {10^3}$ , while the proposed method is represented by ‘Proposed’.

Figure 10 shows the results of the multi-objective reward function with different weight schemes. Figure 10(c) demonstrates that as the RT-related weight increases from $k = 0$ to $k = {10^1}$ , the moment when the WIG craft reaches the target altitude gradually advances, until the rise time is minimised at the 11-th step. In the latter part of the trajectory, Fig. 10(a) shows that after reaching the target altitude, the concavity of the trajectory becomes increasingly pronounced, accompanied by a corresponding growth in deviation from the target altitude. The degree of concavity reaches its maximum when the RT-related weight increases to $k = {10^1}$ . Figure 10(b) further reveals that although trajectory deviation increases with the RT-related weight, this trend saturates once the weight exceeds $k = {10^1}$ , beyond which further increases yield no significant change in deviation. The trajectory corresponding to the proposed reward function directly achieves the minimum RT and subsequent approach to the target altitude. The proposed reward function does not introduce any part that needs iterative adjustments of weights through trial and error, and it directly results in the expected trajectory with the maximum concavity.

Figure 10.

Trajectories of multi-objective reward functions assigned with different weight $k$ and the proposed reward function.

Figure 11 displays the RT and IAE values for the aforementioned cases. It can be observed that as the RT-related weight in the multi-objective function increases exponentially, the trajectory’s RT decreases while the IAE increases. Beyond $k = {10^1}$ , both RT and IAE have plateaued, and the proposed method directly achieves identical RT and IAE values. The observed behaviour aligns with the design intent: a relatively large RT-related weight in the reward function effectively reduces RT. The proposed method achieves the same RT and IAE as the multi-objective approach after full convergence, demonstrating that it successfully minimises RT while preventing further increase in IAE, thereby reaching the minimum IAE achievable under the constraint of minimised RT. With increasing RT-related weight, its proportion in the total reward becomes more substantial. Since the WIG tends to maximise reward, its trajectory shows a tendency toward reduced RT, though this comes at the cost of greater deviation from the target altitude. When the RT-related weight is sufficiently large, the reward gain from reducing RT exceeds the loss caused by trajectory deviation, thereby promoting a faster response. This indicates that the earlier arrival at the target altitude is achieved at the expense of trajectory precision during the approach.

Figure 11.

Performance of RT and IAE for the multi-objective reward function with different weight $k$ and the proposed reward function

4.2 Invariance

The proposed reward function is highly effective in achieving both reduced RT and lower deviation, and its performance is unlikely to be significantly improved through further modifications. To demonstrate the advantage of the proposed reward function over traditional multi-objective approaches that require parameter tuning, an additional coefficient $\lambda $ is introduced to examine its invariance, as:

(19) \begin{align} {R_r} &= 1 + \frac{{n\,{\rm{*}}\left( {n + 1} \right)}}{2}{\rm{\;}},{\rm{\;\;}}n = m - {t_r} \\[-10pt] \nonumber\end{align}

(20) \begin{align} {R_e} &= \mathop \sum \limits_{i = {t_r}}^m \left( {1 - \left| {{{{h_i} - {h^{\rm{*}}}} \over {{h^{\rm{*}}} - {h_0}}}} \right|} \right) \\[-10pt] \nonumber\end{align}

(21) \begin{align} R &= {R_e} + \lambda \cdot {R_r} \\[10pt] \nonumber \end{align}

Four coefficients – $\lambda = 1$ , $\lambda = 10$ , $\lambda = {10^1}$ and $\lambda = {10^2}$ – are selected to compare the resulting trajectories, as shown in Fig. 12. It can be observed that the trajectories are similar. Figure 13 presents the corresponding RT and IAE values, which show only minor differences. Slight variations can be seen in the concavity of the latter trajectory segments, which may be attributed to optimisation errors. This consistency occurs because when $\lambda \geqslant 1$ , the reward gain from reducing RT is guaranteed to be sufficiently large to outweigh the penalty caused by increased deviation. Since the quantitative condition of the reward function is satisfied, essentially identical trajectories are obtained.

Figure 12.

Trajectories of the proposed reward functions assigned with different coefficient $\alpha $ .

Figure 13.

Performance of RT and IAE for the proposed reward function with different coefficient $\lambda $ .

Validation is performed by attaching a coefficient greater than 1 to the RT-related term and comparatively evaluating a series of coefficient values. The results confirm that the proposed reward function requires no custom development for performance adjustment. Unlike conventional multi-objective reward functions, this method eliminates the need for iterative coefficient tuning and directly generates the ideal trajectory that achieves minimum rise time while maintaining maximum proximity to the target altitude.