3.1 Frame

This paper develops a 3D dynamic environment for investigating real-time UAV obstacle avoidance, in which dynamic obstacles are modeled as spheres of varying radii moving along predefined trajectories at different velocities. To enhance motion realism, we implemented continuous UAV control through variable manoeuver commands, with the reinforcement learning framework generating continuous 3D velocity adjustments. Figure 1 illustrates the proposed model architecture.

Figure 1.

The framework of the RL-based obstacle avoidance method.

The UAV navigation task requires reaching a randomly positioned target point $goal = [{x_g},{y_g},{z_g}]$ from a random start point $start = [{x_s},{y_s},{z_s}]$ near the environment boundary in 3D space, while avoiding collisions with dynamic obstacles and maintaining safe distance from boundaries. The UAV operates under partial observability, relying solely on limited detection sectors for environmental perception rather than complete obstacle position and velocity information. This observation scheme introduces perception uncertainty, leading researchers to model the DOA problem as a Partially Observable Markov Decision Process (POMDP). The POMDP framework formally describes the problem using the tuple $\left( {S,A,T,R,\Omega ,{\rm O}} \right)$ , where: $S = \left\{ {{s_1}, \cdots {s_n}} \right\}$ represents the state space, $A = \left\{ {{a_1}, \cdots {a_n}} \right\}$ denotes the action space, the state transition function $T\left( {s,a,s'} \right) = P\left( {s'|s,a} \right)$ specifies the probability of moving to state $s'$ when taking action $a$ in state $s$ . $R = r(s,a)$ indicates the reward function. The agent directly optimises policy ${\pi _\theta }\left( {a|s} \right)$ for obstacle avoidance decisions, with cumulative return is defined as

(1) \begin{align}J\left( \theta \right) = {{\mathbb E}_{\tau \sim {\pi _\theta }}}\left[ {\sum\limits_{t = 0}^\infty {{\gamma ^t}R\left( {{s_t},{a_t}} \right)} } \right]\end{align}

Under policy ${\pi _\theta }$ , the state-value function represents the expected cumulative discounted reward starting from state $s$ .

(2) \begin{align}{V^\pi }\left( s \right) = {{\mathbb E}_\pi }\left[ {J\left( \theta \right)|{s_t} = s} \right] = {{\mathbb E}_\pi }\left[ {\sum\limits_{t = 0}^\infty {{\gamma ^t}R\left( {{s_t},{a_t}} \right)|{s_t} = s} } \right]\end{align}

To evaluate the expected return when taking action $a$ in state $s$ , we define the action-value function ${Q^\pi }\left( {s,a} \right)$ as:

(3) \begin{align}{Q^\pi }\left( {s,a} \right) = {{\mathbb E}_\pi }\left[ {J\left( \theta \right)|{s_t} = s,{a_t} = a} \right] = {{\mathbb E}_\pi }\left[ {\sum\limits_{t = 0}^\infty {{\gamma ^t}R\left( {{s_t},{a_t}} \right)|{s_t} = s} ,{a_t} = a} \right]\end{align}

The optimal policy maximises the action-value function. For any state-action pair $\left( {s,a} \right)$ , we obtain ${\pi _\theta }$ by solving the Bellman Optimality Equation (BOE):

(4) \begin{align}{\pi ^*}\left( s \right) = \mathop {\arg \max }\limits_a {Q^\pi }\left( {s,a} \right),\forall s \in S,\forall a \in {\rm A} = \mathop {\arg \max }\limits_a {{\mathbb E}_\pi }\left[ {{r_t} + \gamma {Q^\pi }\left( {s',a'} \right)|{s_t} = s,{a_t} = a} \right]\end{align}

3.2 Motion model of UAV

Our research proposes a direct control method based on 3D-axis velocity to achieve continuous altitude and speed variations for UAVs in three-dimensional environments. Precise attitude control is enabled through dynamic adjustment of velocity components along three spatial axes. The UAV’s heading is automatically aligned with the resultant velocity vector to ensure motion continuity and flight stability.

To better simulate realistic tracking and avoidance manoeuvers, constraints were imposed on climbing, descending and turning operations. Based on UAV kinematic characteristics, a kinematic model incorporating dual constraints on both azimuth and climbing angles was developed, as shown in Fig. 2. Firstly, the UAV’s position at current time step is ${q_t} = ({x_t},{y_t},{z_t})$ , with its previous position is ${q_{before}} = ({x_{t - 1}},{y_{t - 1}},{z_{t - 1}})$ and the next position is ${q_{next}} = ({x_{t + 1}},{y_{t + 1}},{z_{t + 1}})$ . The azimuth angle $\phi $ calculated from the horizontal projection of position vectors, quantifies the heading change in horizontal plane, as shown in Equations (5) and (6).

(5) \begin{align}{\phi _0} = \arctan \left( {\frac{{\Delta {q_y}}}{{\Delta {q_x}}}} \right) = \arctan \left( {\frac{{{y_t} - {y_{t - 1}}}}{{{x_t} - {x_{t - 1}}}}} \right)\end{align}

(6) \begin{align}{\phi _1} = \arctan \left( {\frac{{\Delta {q_{next}}_y}}{{\Delta {q_{next}}_x}}} \right) = \arctan \left( {\frac{{{y_{t + 1}} - {y_t}}}{{{x_{t + 1}} - {x_t}}}} \right)\end{align}

Figure 2.

UAV kinematic model.

Climbing angle characterises the movement trend of the UAV in the vertical direction, as shown in Equations (7) and (8).

(7) \begin{align}{\varphi _0} = \arctan \left( {\frac{{\Delta {q_{\rm{z}}}}}{{\Delta {q_{xy}}}}} \right) = \arctan \left( {\frac{{{z_t} - {z_{t - 1}}}}{{\sqrt {{{\left( {{x_t} - {x_{t - 1}}} \right)}^2} + {{\left( {{y_t} - {y_{t - 1}}} \right)}^2}} }}} \right)\end{align}

(8) \begin{align}{\varphi _1} = \arctan \left( {\frac{{\Delta {q_{next}}_{\rm{z}}}}{{\Delta {q_{next}}_{xy}}}} \right) = \arctan \left( {\frac{{{z_{t + 1}} - {z_t}}}{{\sqrt {{{\left( {{x_{t + 1}} - {x_t}} \right)}^2} + {{\left( {{y_{t + 1}} - {y_t}} \right)}^2}} }}} \right)\end{align}

The maximum azimuth angle $\Delta \phi $ limits abrupt heading changes within unit time. The adjusted yaw angle ${\phi _{res}}$ follows Equation (9).

(9) \begin{align}\left\{ \begin{array}{l}{\phi _{res}} = {\phi _1},\quad if\left| {{\phi _1} - {\phi _0}} \right| \le \Delta \phi \\[5pt]{\phi _{res}} = {\phi _0} + sign\left( {{\phi _1} - {\phi _0}} \right) * \Delta \phi ,\quad else\end{array} \right.\end{align}

The climbing angle constraints comprise rate constraints and physical limits. The rate constraint (Equation 10) maintains smooth vertical motion by restricting the climbing angle’s change rate. The physical constraint (Equation 11) enforces UAV performance limits through bounded climbing angle and diving angle.

(10) \begin{align}{\varphi _{res}} = \left\{ \begin{array}{l}{\varphi _1},\quad \left| {{\varphi _1} - {\varphi _0}} \right| \le \Delta \varphi \\[5pt]{\varphi _0} + sign\left( {{\varphi _1} - {\varphi _0}} \right) * \Delta \varphi ,\quad \left| {{\varphi _1} - {\varphi _0}} \right| \gt \Delta \varphi \end{array} \right.\end{align}

(11) \begin{align}{\varphi _{res}} = \max \left( {{\varphi _{\min }},\min \left( {{\varphi _{res}},{\varphi _{\max }}} \right)} \right)\end{align}

Where ${\varphi _{\max }}$ denotes the maximum climbing angle, ${\varphi _{\min }}$ represents the maximum diving angle.

3.3 Target tracking and dynamic obstacles avoidance model based On DRL

The UAV DOA problem is formulated as a Markov process, as illustrated in Fig 3. The environment is modeled in three-dimensional space and populated with multiple dynamic obstacles of different radii and motion patterns, providing diverse conditions for evaluating strategies. At each decision step, obstacle intrusions and target status are detected within the UAV’s sensor range, and the current state is updated accordingly. Conditioned on this observed state and constrained by UAV kinematics, the policy network trained with reinforcement learning selects avoidance actions in the form of incremental velocity adjustments. These actions continuously update the UAV trajectory in a stepwise manner, enabling the agent to react adaptively to dynamic obstacles until the mission is successfully accomplished at the target location.

Figure 3.

Autonomous UAV obstacle avoidance model.

3.3.1 State space

The UAV obstacle avoidance task fundamentally involves searching for an optimal collision-free path. Our system processes three primary perception data types: the UAV internal state, intruding dynamic obstacle information and target-relative positioning. UAV internal state is characterised by triaxial velocity components $v = [{v_x},{v_y},{v_z}]$ , while its target-relative position is denoted by vector $\theta = {p_{goal}} - p$ . For three-dimensional environment perception, we propose a modular partial observation model [Reference Chen27] as shown in Fig. 4. The UAV observes a cylindrical region centred at its current position with detection radii ${R^D}$ and height $2{R^H}$ . When ${z_{obstacle}} - z \le {R^H}$ , the system registers obstacles occupying higher altitude layers ${k^u}$ , otherwise marked as ${k^l}$ . The detection zone is partitioned into n sectorial regions ${N_D}$ ordered relative to the UAV’s horizontal heading, yielding the obstacle observation matrix at timestep $t$ follows Equation (12).

(12) \begin{align}{o_t} = \left[ \begin{array}{l}o_t^{{k^u},1} \cdots o_t^{{k^u},{N_D}}\\[5pt]o_t^{{k^l},1} \cdots o_t^{{k^l},{N_D}}\end{array} \right]\end{align}

Figure 4.

UAV obstacle detection model.

The intrusion distance is defined as the distance from the UAV to an obstacle’s surface. We calculate vertical $h$ and horizontal $d$ projections of through sine and cosine operations of their relative angles. The system registers an obstacle intrusion only when $h \le {R^H}\ \textrm{and}\ d \le {R^D}$ , enabling UAV perception. The classification of obstacles based on UAV detection states divides them into three categories. If an obstacle does not intrude, it is marked as undetected. When two obstacles appear in the same detection sector, the UAV only observes the one with the closest intrusion distance. Specifically, if an obstacle intrudes into two detection sectors, it is only counted in the sector containing its closest surface point.

3.3.2 Action space

To enable continuous variations in UAV altitude and speed within three-dimensional environments, this work directly controls the UAV’s 3D-axis velocity. The UAV’s heading always aligning with its velocity direction. The proposed DRL framework learns the 3D-axis velocity increments, which represent the action output $a = [\Delta {v_x},\Delta {v_y},\Delta {v_z}]$ at each timestep, allowing the UAV’s velocity and position updates to be expressed by Equation (13).

(13) \begin{align}\left\{ \begin{array}{l}v_x^{t + 1} = v_x^t + \Delta {v_x}\\[5pt]v_y^{t + 1} = v_y^t + \Delta {v_y}\\[5pt]v_z^{t + 1} = v_z^t + \Delta {v_z}\\[5pt]{x_{t + 1}} = {x_t} + v_x^{t + 1}\Delta t\\[5pt]{y_{t + 1}} = {y_t} + v_y^{t + 1}\Delta t\\[5pt]{z_{t + 1}} = {z_t} + v_z^{t + 1}\Delta t\end{array} \right.\end{align}

The variable $({x_t},{y_t},{z_t})$ represents the UAV’s position at timestep $t$ , while $\Delta t$ denotes the time interval between consecutive velocity control commands sent to the UAV, with a default value of 0.1 seconds. This short timestep interval enables continuous and smooth UAV control.

3.3.3 Reward

The reward value serves as the key mechanism for guiding UAV obstacle avoidance and target approach. We design a reward function to continuously incentivise UAV behaviour, consisting of four components: a sparse reward function, distance function, yaw function and speed function. The sparse reward function incorporates obstacle avoidance rewards, target rewards and boundary rewards as specified in Equation (14). During operation, the UAV receives penalties for colliding with obstacles or exiting the operational area. Additionally, the system imposes minor penalties when the UAV-obstacle distance falls below a buffer threshold to encourage timely evasion. When the UAV completes an action at each timestep, it obtains the full reward if reaching within the target threshold distance, thereby completing the episode. Otherwise, penalties are assessed based on the remaining distance to target when maximum episode steps are reached.

(14) \begin{align}{R_{sparse}} = {R_{avoid}} + {R_{goal}} + {R_{boundary}}\end{align}

(15) \begin{align}{R_{avoid}} = \left\{ \begin{array}{l}\frac{{\left\| {{P_{uav}} - {P_{obs}}} \right\|}}{{{r_{obs}}}} - {C_0},\quad 0 \le \left\| {{P_{uav}} - {P_{obs}}} \right\| \le {r_{obs}}\\[5pt]\frac{{\left\| {{P_{uav}} - {P_{obs}}} \right\|}}{{{r_{obs}} + {r_{buffer}}}} - {C_1},\quad {r_{obs}} \le \left\| {{P_{uav}} - {P_{obs}}} \right\| \le {r_{buffer}}\end{array} \right.\end{align}

(16) \begin{align}{R_{goal}} = \left\{ \begin{array}{l}C,{\rm{ 0}} \le \left\| {{P_{goal}} - {P_{uav}}} \right\| \le threshold\\[5pt]\max \left(c, - \frac{{\left\| {{P_{goal}} - {P_{uav}}} \right\|}}{{{d_{boundary}}}}\right),\quad else\end{array} \right.\end{align}

While sparse rewards prove insufficient for effective UAV mission completion, we augment the reward system with dense rewards comprising distance and yaw functions. These functions operate at every timestep to progressively guide the UAV toward the target point, as specified in Equations (17) and (18). The distance function combines two key components: it simultaneously encourages target approach while maintaining altitude constraints. Specifically, the UAV receives incentives for reducing target distance at each step, but is penalised for exceeding permissible altitude boundaries.

(17) \begin{align}{R_d} = \left(1 - \frac{{\left\| {P_{goal}^{} - P_{uav}^{t + 1}} \right\|}}{{\left\| {P_{goal}^{} - P_{uav}^t} \right\|}}\right) + \lambda \left\| {z_{goal}^t - z_{uav}^t} \right\|\end{align}

(18) \begin{align}{R_{{\rm{yaw}}}} = \lambda \cdot \frac{{\left| \phi \right|}}{\pi }\end{align}

The speed function constrains the UAV’s velocity within performance limits, imposing penalties when the resultant speed exceeds the maximum threshold as specified in Equation (19).

(19) \begin{align}{R_{speed}} = \lambda \cdot \frac{{v - {v_{\max }}}}{{{v_{\max }}}}\end{align}

Integrating all aforementioned reward components yields the complete reward function shown in Equation (20).

(20) \begin{align}R = {R_{sparse}} + {R_d} + {R_{{\rm{yaw}}}} + {R_{speed}}\end{align}

3.4 Soft Actor-Critic

This study employs the Soft Actor–Critic (SAC) algorithm, which integrates the Actor–Critic framework with policy gradient methods. In the conventional Actor–Critic architecture, the actor network generates candidate actions according to the current policy, while the critic network evaluates these actions by estimating their expected return through a value function. SAC extends this architecture by introducing the maximum entropy reinforcement learning framework, where the optimisation objective consists not only of the cumulative return but also an additional entropy term, thereby explicitly encouraging stochasticity in action selection. By balancing reward maximisation with entropy maximisation, SAC facilitates more effective exploration, prevents premature convergence to suboptimal deterministic solutions and enhances robustness in highly dynamic or uncertain environments. Information entropy, which serves as a measure of uncertainty in action selection, is formally defined in Equation (21).

(21) \begin{align}{\rm H}\left( x \right) = {{\mathbb E}_{x \sim P}}\left( { - \log P\left( x \right)} \right)\end{align}

The entropy-maximised policy objective becomes Equation (22), where ${\rho _\pi }$ represents the state-action pair distribution encountered by the agent under policy $\pi $ . The temperature coefficient serves as a hyperparameter that regulates the emphasis on entropy maximisation

(22) \begin{align}{\pi ^*} = \mathop {\mathop {\arg \max }\limits_\pi } \,\mathbb{E}_{\left( {s_t},{a_t} \right) \sim {\rho _\pi }} \left[ {\sum\limits_t {r\left( {{s_t},{a_t}} \right) + \alpha {\rm H}\left( {\pi \left( {\left. \cdot \right|{s_t}} \right)} \right)} } \right]\end{align}

Traditional reinforcement learning aims to maximise the expected cumulative reward as shown in Equation (1). In contrast, SAC introduces an entropy term that incorporates a policy randomness weight into the reward function, as specified in Equation (23).

(23) \begin{align}J\left( \theta \right) = {{\mathbb E}_{\tau \sim {\pi _\theta }}}\left[ {\sum\limits_{t = 0}^\infty {{\gamma ^t}R\left( {{s_t},{a_t}} \right) + \alpha H\left( {\pi \left( {\left. \cdot \right|{s_t}} \right)} \right)} } \right]\end{align}

$H\left( {\pi \left( {\left. \cdot \right|{s_t}} \right)} \right)$ quantifies the randomness level of policy $\pi $ in state $s$ .

The pseudocode of the SAC algorithm 1 is presented above.

Algorithm 1

Soft Actor-Critic