2.1. Reinforcement learning

The goal of RL is to learn a policy to maximize the expected cumulative reward in a Markov decision process (MDP) framework [Reference Puterman41]. MDP is defined by the tuple $(S, A, P, R)$ , which consists of a set of states $s \in S$ , a set of actions $a \in A$ , transition dynamics $P(s' \mid s, a)$ , and a reward function $r = R(s, a)$ [Reference Sutton and Barto30, Reference Puterman41]. In online RL, the agent interacts with the environment to collect trajectory data and improve policy. A trajectory is denoted as $(s_t, a_t, r_t)_{t=0}^{T}$ , where $s_t, a_t,$ , and $t$ represent the state, action, and reward at time step t, respectively, and $T$ denotes the episode length.

2.2. The exploration-exploitation dilemma

Balancing exploration and exploitation is a challenge in RL [Reference Thrun42]. In uncertain environments, agents interactions with the environment and gather information. The dilemma lies in whether the agent should select actions that it believes to be optimal based on existing experience or choose actions with high uncertainty. Exploration refers to the agent trying novel actions to discover new states and rewards, while exploitation involves selecting actions known to yield high rewards [Reference Kober, Bagnell and Peters43].

If an agent focuses solely on exploitation, it risks converging to suboptimal policies due to incomplete information, potentially missing better strategies. Conversely, if the agent performs uniform random exploration, it may repeatedly sample low-reward state-action pairs, increasing the number of low-quality samples and wasting time and resources without effectively accumulating rewards. Therefore, RL algorithms must balance exploration and exploitation to efficiently explore the environment and reduce uncertainty about it.

2.3. Detailed explanation of the PPO objective

Our model here based on the traditional and efficient model PPO framework. At the same time, we improve the stability and sample efficiency by incorporating a clipping mechanism into the standard policy gradient objective [Reference Sutton, McAllester, Singh and Mansour44].

At each time step $t$ , the model is given the environment state $s_t$ , the policy network with parameters $\theta$ outputs a distribution results over actions $\pi _{\theta }(\cdot \mid s_t)$ , from which the agent samples $a_t\sim \pi _{\theta }(\cdot \mid s_t)$ .

2.3.1. Importance sampling ratio

Define the importance sampling ratio as follows:

(1) \begin{align} r_t(\theta ) \;=\; \frac {\pi _{\theta }(a_t\mid s_t)} {\pi _{\theta _{\mathrm{old}}}(a_t\mid s_t)}\,. \end{align}

This ratio measures how the probability assigned to $(s_t,a_t)$ changes under the new policy compared to the old one. If $r_t(\theta )$ is differentiated too far from $1$ , it shows an overly large policy update, which can destabilize training.

2.3.3. Clipped surrogate objective

Rather than use the unconstrained objective $[r_t(\theta )\,\hat A_t]$ , PPO introduces clipping to limit policy updates:

(2) \begin{align} L^{\mathrm{CLP}}(\theta ) \;=\;\min \big (r_t(\theta )\,\hat A_t,\; \mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )\,\hat A_t\big ). \end{align}

If $r_t(\theta )$ stays within $[\,1-\epsilon ,1+\epsilon \,]$ , the true ratio is used; once it exits this interval, it is clipped to the boundary. This preserves the gradient direction for small updates and prevents excessively large policy shifts.

2.3.4. Value function loss

PPO jointly learns a state-value function $V_\theta (s)$ by minimizing the mean-squared error:

(3) \begin{align} L^{\mathrm{VF}}(\theta ) \;=\; \Bigl [\bigl (V_\theta (s_t)-V_{\mathrm{target}}(s_t)\bigr )^2\Bigr ]. \end{align}

2.3.5. Full PPO objective

Putting these components together, the PPO loss is

(4) \begin{align} L^{\mathrm{PPO}}_t(\theta ) = L^{\mathrm{CLP}}(\theta ) \;+\; c_1\,L^{\mathrm{VF}}(\theta ). \end{align}

In our quadruped gait learning experiments [Reference Hong, Lee, Yi and Lee46], the standard PPO-trained policy frequently converges to suboptimal locomotion patterns, especially in complex environments [Reference Liu, Wang, Wang and Fei47]. We propose ECIM to overcome these.

3.1. Adaptive entropy regularization: achieving a balance between exploration and exploitation

To enhance exploration efficiency and balance the trade-off between exploration and exploitation, we introduce an adaptive entropy weighting coefficient, $\beta _t$ . Unlike a static decay schedule, $\beta _t$ is dynamically adjusted at the end of each training episode based on the agent’s recent performance. Specifically, the learning state is evaluated using the cumulative return of the completed episode: if the policy exhibits stagnation or performance degradation, $\beta _t$ is moderately increased to re-invigorate exploration and help escape local optima. Conversely, if performance shows consistent improvement, $\beta _t$ is further decreased to attenuate the influence of the entropy bonus, thereby facilitating convergence to a deterministic policy. This dual-regulation mechanism ensures that the robot maintains sufficient behavioral diversity in complex terrains while promptly locking onto efficient locomotion patterns as learning progresses.

(5) \begin{align} R_t &= \frac {1}{\tau }\sum _{k=0}^{\tau -1} G_{t-k}, \end{align}

(6) \begin{align} \beta _t &= \beta _{\max }\!\left(1-\frac {R_t}{R_{\max }}\right)\!. \end{align}

Specifically, the calculation of $\beta _t$ depends on the cumulative return at time $t$ , denoted as $G_t$ , a sliding average window length $\tau$ , and a predefined maximum achievable return per episode, $R_{\max }$ . During the early stages of training, since the obtained return is significantly lower than the theoretical maximum (i.e., $G_t \ll R_{\max }$ ), $\beta _t$ approaches its upper bound $\beta _{\max }$ , thereby strongly encouraging exploration. As training progresses and the cumulative return converges toward the maximum (i.e., $G_t \approx R_{\max }$ ), $\beta _t$ decays to zero or a negligible value, significantly reducing exploratory pressure and shifting the focus toward exploitation. The per-step objective thus becomes:

(7) \begin{equation} L_t(\theta ) \;=\; L^{\mathrm{PPO}}_t(\theta ) \;-\; \beta _t\,\mathbb{E}\bigl [\mathcal{H}(\pi _{\theta }(\cdot \mid s_t))\bigr ]. \end{equation}

We design the dynamic scheduling to initiate policy exploration in the early stages, then enhance the policy.

3.2. Motion continuity regularization for quadrupeds: ensuring smoothness and stability

The motion of a robot is continuous. To ensure system stability, its dynamic behavior must satisfy constraints [Reference Spyrakos-Papastavridis, Medrano-Cerda, Tsagarakis, Dai and Caldwell35]. In our experiments, we observed that action sequences containing sharp transitions or high-frequency components frequently led to loss of balance and unstable locomotion.

To address these challenges, we introduced two complementary regularization functions to enhance the stability of the learning strategy:

(8) \begin{align} L_T &= \mathbb{E}_{s_t\sim \mathcal{D}}\bigl [\|\pi _{\theta }(s_{t+1}) - \pi _{\theta }(s_t)\|^2\bigr ], \end{align}

(9) \begin{align} L_S &= \mathbb{E}_{s\sim \mathcal{D},\,s'\sim \mathcal{N}(s,\sigma )}\bigl [\|\pi _{\theta }(s') - \pi _{\theta }(s)\|^2\bigr ]. \end{align}

$L_T$ prevents the robot from sudden changes. We design $L_S$ to enhance robustness against input noise. We incorporate these as:

(10) \begin{align} L_{\mathrm{smooth}} \;=\; \lambda _T\,L_T \;+\; \lambda _S\,L_S, \end{align}

$\lambda _T$ and $\lambda _S$ balance time smoothness and space smoothness. Robot generates smoother control trajectories and improve system stability by incorporating $L_{\mathrm{smooth}}$ .

Figure 2.

Data update flowchart.

Figure 2. Long description

Three columns represent consecutive time steps. In each, the intrinsic exploration mechanism and the environment reward are combined into a shaped reward, which flows down to the state, the policy, the action and the environment. The environment output of each step feeds the next step. All steps feed an episode collection bar and then an experience buffer.

Figure 3.

Intrinsic motivation-driven exploration enhancement mechanism.

Figure 3. Long description

The current state feeds three blocks inside a dashed box: random network distillation (target and predictor networks), count-based pseudo-counts, and an intrinsic curiosity module (inverse and forward models). Each block outputs an intrinsic reward term, and the three terms merge into one intrinsic reward. On the left, the policy, action and environment produce the extrinsic reward. The intrinsic and extrinsic rewards combine into the shaped reward, leading to the next state.

3.3. Intrinsic rewards for improved exploration–exploitation and policy diversity

The data update process of the ECIM algorithm is shown in Figure 2. We introduce an intrinsic reward-guided mechanism to address the inherent exploration-exploitation dilemma in RL. It motivates the robot to explore a broader state space. It directs the robot to visit under-explored or information-rich areas in the environment. The intrinsic reward is dynamically adjusted. By prompting the agent to explore in more varied ways, our approach not only enriches policy diversity but also keeps learning stable and convergent. As shown in Figure 3, the intrinsic reward-guided mechanism consists of three components:

3.3.1. Intrinsic curiosity module

Standard extrinsic rewards may be sparse or delayed, which hinders efficient exploration. We equip an intrinsic motivation signal based on prediction error to address this. Specifically, we learn:

• • Inverse model $g_{\phi }\!\bigl (\phi (s_t),\phi (s_{t+1})\bigr )\to a_t$ , which encourages the encoder $\phi$ to capture controllable aspects of the environment.

• • Forward model $f_{\psi }\!\bigl (\phi (s_t),a_t\bigr )\to \phi (s_{t+1})$ , whose prediction error reflects the novelty of the transition.

The intrinsic reward is defined as:

(11) \begin{align} r_t^{\mathrm{icm}} \;=\; \eta _{\mathrm{icm}}\,\bigl \|\phi (s_{t+1}) - f_{\psi }(\phi (s_t),a_t)\bigr \|^2. \end{align}

By jointly training $\phi$ , $f_{\psi }$ , and $g_{\phi }$ , the agent is driven to seek out state transitions that are difficult to predict, thereby improving coverage of the state space.

3.3.2. Random network distillation

As an alternative source of intrinsic motivation that does not require explicit inverse modeling, we adopt random network distillation. A fixed randomly initialized “target” network $f_{\mathrm{tgt}}$ provides embeddings of states, and a trainable “predictor” network $f_{\mathrm{pred}}$ attempts to match these embeddings:

(12) \begin{align} r_t^{\mathrm{rnd}} \;=\; \eta _{\mathrm{rnd}}\,\bigl \|\,f_{\mathrm{tgt}}(s_t) - f_{\mathrm{pred}}(s_t)\bigr \|^2. \end{align}

Since $f_{\mathrm{pred}}$ gradually learns to mimic $f_{\mathrm{tgt}}$ , the prediction error naturally decays for familiar states and remains high for novel ones, guiding exploration without explicit density modeling.

3.3.3. Count-based pseudo-counts

To further encourage rare-state visitation in high-dimensional settings, we estimate a density model $\rho (s)$ and derive pseudo-counts $\,\hat{\!N}(s)$ :

(13) \begin{align} \,\hat{\!N}(s) \;=\; \frac {\rho (s)\bigl (1-\rho '(s)\bigr )}{\rho '(s)\bigl (1-\rho (s)\bigr )}, \end{align}

where $\rho '$ is the density after updating on $s$ . The corresponding bonus reward is:

(14) \begin{align} r_t^{\mathrm{cnt}} \;=\; \frac {\eta _{\mathrm{cnt}}}{\sqrt {\,\hat{\!N}(s_t)}}. \end{align}

This mechanism systematically boosts the reward for under-visited regions, complementing the above curiosity signals.

3.4. Reinforcement learning integrated optimization architecture

Finally, we merge all extrinsic and intrinsic signals into a unified per-step return:

(15) \begin{align} \,\tilde{\!r}_t \;=\; r_t + r_t^{\mathrm{icm}} + r_t^{\mathrm{rnd}} + r_t^{\mathrm{cnt}}. \end{align}

and optimize the overall loss:

(16) \begin{align} \mathcal{L}(\theta ) = \mathbb{E}\bigl [L^{\mathrm{PPO}}_t(\theta ;\,\tilde{\!r}_t)\bigr ] + \lambda _T L_T + \lambda _S L_S - \beta _t\,\mathbb{E} [\mathcal{H}(\pi _{\theta }(\cdot \mid s_t))]. \end{align}

All hyperparameters $\{\beta _{\max },\tau ,\eta _{\cdot },\lambda _{\cdot }\}$ were optimized via grid search to achieve an optimal balance between efficient exploration, smooth control, and stable convergence. The experimental results section below provide a systematic evaluation of the model variants.

4.1. Quadruped motion in six terrains

For our experiment here, we chose the platform Isaac Gym [Reference Makoviychuk, Wawrzyniak, Guo, Lu, Storey, Macklin, Hoeller, Rudin, Allshire, Handa and State48], which is an advanced physics simulation framework developed by NVIDIA Corporation. It allows large-scale simulation of parallel agents, which provides unprecedented computational capabilities for advancing research in robotics and RL. Then, we introduce the ANYmal C as the benchmark platform [Reference Rudin, Hoeller, Reist and Hutter40]. This benchmark platform is a 12-DOF quadruped with three actuated joints per leg, which is widely adopted in prior locomotion work. As shown in Figure 4, we designed six terrains to probe robust locomotion under diverse conditions that approximate real-world scenarios.

In the six terrains of our experiments, these six terrains are chosen for their diversity. The terrains span a range from relatively regular surfaces to highly irregular, rough, or discontinuous structures, covering challenges like uneven ground, height changes, and small-area disturbances. All these changes help our model test its ability in adaptive entropy control, motion regularization, and intrinsic motivation output stabilization. In combination, these environments provide a systematic testing range for assessing both robustness and adaptability of learned quadruped policies.

Our experiments include 4096 parallel robot instances in total for 2000 iterations, and the performance is evaluated every 50 iterations. The reason we include this large-scale parallelism is to increase the rate of policy sample collection and thus accelerate convergence. Also, our model introduces additional learning signals, which were illustrated previously in the method section, namely entropy control, continuity regularization, and intrinsic rewards, and all of these benefit from high-throughput sampling to stabilize the whole interplay. In order to keep its robustness and generalization, this parallel training regime improves sample efficiency and helps ECIM generalize its learned behaviors across the six terrain types while mitigating overfitting.

We adapt the original PPO configuration of Rudin et al. for ANYmal in Isaac Gym as our baseline. For comparison, we use the benchmark source PPO with the same architecture, observation, and action spaces. For controlled reasons, we compare the series of benchmarks based on the same pipeline, and our model differs only by the addition of adaptive entropy control, motion continuity regularization, and intrinsic motivation components. Under this design, any systematic difference in performance between PPO and ECIM can be attributed to the proposed pipeline rather than differences in architecture, training budget, or simulator settings.

Table I summarizes the key PPO hyperparameters used for both PPO and ECIM, including batch size, mini-batch size, number of epochs, clipping range, discount factors, and the target KL-divergence. This table follows the widely used configuration from Rudin et al. [Reference Rudin, Hoeller, Reist and Hutter40] to ensure the comparability with prior work. At the beginning of each episode, all robot instances are initialized at the geometric center of the corresponding terrain, with identical initial poses, to avoid positional bias and to ensure a fair comparison between methods. Overall, this setup provides a standardized and reproducible experimental environment in which the incremental benefits of the ECIM pipeline over PPO can be rigorously assessed.

Table I.

Hyper-parameters used for the training of the tested policy.

Table I. Long description

Two columns, Hyper-parameter and Value: Batch size, 98304; Mini-batch size, 24576; Clip range, 0.2; Discount factor, 0.99; GAE discount factor, 0.95; Desired KL-divergence, 0.01.

4.2. Metrics for entropy control

In order to evaluate ECIM in a systematic way, we use a set of metrics that follow the three main design goals of our framework. First, adaptive entropy control (AECPOM) aims to speed up learning and improve task performance. Second, motion continuity regularization (MCRF) targets stable and smooth locomotion. Third, intrinsic motivation for efficient exploration (IMDEEM) focuses on reducing energy usage while still exploring the environment effectively. Besides the standard episode reward, we include several additional metrics that describe learning speed, body stability, motion smoothness, and energy-related actuation cost. These metrics are later used both for comparing ECIM with PPO and for understanding how each ECIM module contributes to the final behavior.

Reward: Reward is the main task-performance metric and measures the total return obtained by the agent during an episode. It is computed from the environment-defined reward function, which typically combines terms for forward velocity, energy consumption, posture stability, and other factors. Normally, a higher reward indicates better overall performance in task completion, motion efficiency, and stability. The total reward is calculated by:

(17) \begin{align} R = \sum _{t=1}^{T} r_t, \end{align}

where $ r_t$ denotes the instantaneous reward at time step $ t$ , and $ T$ is the episode length. In our experiments, reward is used as the primary measure to compare PPO and ECIM, and also as one of the metrics for attribution gain (AG).

Angular velocity loss ( $\bf{L}_{\text{ang-xy}}$ ): The pitch angle serves as a fundamental attitude parameter that characterizes the longitudinal stability of a robotic system. Variations in pitch angle reflect the dynamic tilting behavior of the robot’s body in the sagittal plane and directly influence motion smoothness, balance control, load-bearing capability, and adaptability to complex terrains. We describe this behavior using the angular velocity vector of the robot base at time step $t$ ,

(18) \begin{equation} \boldsymbol{\omega }_t = \begin{bmatrix} \omega _{x,t} \\ \omega _{y,t} \\ \omega _{z,t} \end{bmatrix}\!, \end{equation}

where $\omega _{x,t}$ and $\omega _{y,t}$ represent the roll and pitch angular velocity components, respectively. The angular velocity energy in the $x$ – $y$ plane is computed as

(19) \begin{equation} \mathcal{L}_{\text{ang-xy}} = \sum _{t=1}^{N} \left ( \omega _{x,t}^2 + \omega _{y,t}^2 \right )\!. \end{equation}

where $N$ is the number of time steps. Excessive oscillations in roll and pitch can induce undesired body tilting, shift center of the total mass, and increase the likelihood of instability or falling. A lower $\mathcal{L}_{\text{ang-xy}}$ therefore indicates a more stable base motion, which is one of the main targets of MCRF.

Joint acceleration: Joint acceleration is the change of joint velocity over time. It is used to assess how smooth and continuous the robot’s movements are. In general, lower joint acceleration means smoother trajectories without sudden start or stop, and often better energy usage. At each simulation step, we compute the joint acceleration as the difference between the current and previous joint velocities divided by the time interval $\Delta t$ :

(20) \begin{equation} \boldsymbol{a}_{t} = \frac {\boldsymbol{\dot {q}}_{t} - \boldsymbol{\dot {q}}_{t-1}}{\Delta t}, \end{equation}

where $\boldsymbol{\dot {q}}_{t}$ and $\boldsymbol{\dot {q}}_{t-1}$ are the joint velocity vectors at the current and previous time steps, and $\Delta t$ is the simulation time step. Since MCRF penalizes sudden changes in motion, we expect ECIM policies to show lower joint acceleration than the PPO baseline.

Joint torque ( ${\tau }_t$ ): Joint torque represents the actuator effort at each joint during motion. It is an important indicator for both the naturalness of movement and the efficiency of energy use. We collect the joint torques at time step $t$ in the vector

(21) \begin{equation} \boldsymbol{\tau }_t = [\tau _{1,t}, \tau _{2,t}, \ldots , \tau _{n,t}]^{T}, \end{equation}

where $\tau _{i,t}$ is the torque of the $i$ -th joint at time step $t$ , and $n$ is the number of actuated joints of the robot. Torque magnitude serves as a proxy for control intensity: higher values frequently indicate rapid actuator responses or inefficient force application, whereas lower values are associated with smoother motion profiles and greater energetic economy.

Steps-to- $\bf R^*$ : Steps-to- $R^*$ measures the number of environment interaction steps required for the agent to first achieve a target performance threshold $R^*$ (e.g., 90% of the optimal policy’s average return). We compute it as

(22) \begin{align} \text{Steps-to-R}^* = \min \left \{ N \,\left |\, \frac {1}{K}\sum _{i=N-K+1}^{N} R_i \geq R^* \right .\right \}\!, \end{align}

where $R_i$ is the cumulative reward of the $i$ -th training episode, $K$ is the smoothing window size, and $R^*$ is the target performance threshold.

Pitch RMS: The root mean square of the body pitch angle (Pitch RMS) quantifies longitudinal trunk oscillations during locomotion. We compute it as

(23) \begin{align} \text{Pitch RMS} = \sqrt {\frac {1}{T} \sum _{t=1}^{T} \theta _t^2}, \end{align}

where $\theta _t$ is the pitch angle of the body relative to the horizontal plane at time $t$ , and $T$ is the test time duration.

Acc RMS: The root mean square of the body’s linear acceleration (Acc RMS) quantifies motion smoothness by reflecting dynamic impacts and high-frequency vibrations during locomotion. It is computed as

(24) \begin{align} \text{Acc RMS} = \sqrt {\frac {1}{T} \sum _{t=1}^{T} \|\mathbf{a}_t\|^2}, \end{align}

where $\mathbf{a}_t$ is the linear acceleration vector of the body at time $t$ , and $T$ is the test duration. As with the other stability measures, MCRF is expected to reduce Acc RMS by promoting temporally consistent motion.

Torque RMS: The root mean square of joint torques (Torque RMS) represents the average magnitude of actuator torques across all joints, averaged over both time and joint index. It reflects energy consumption and actuation load. A lower value indicates more energy-efficient control, which is important for extending battery life and reducing hardware wear. We define it as

(25) \begin{align} \text{Torque RMS} = \sqrt {\frac {1}{T \cdot J} \sum _{t=1}^{T} \sum _{j=1}^{J} \tau _{t,j}^2}, \end{align}

where $\tau _{t,j}$ is the torque of joint $j$ at time $t$ , $J$ is the number of joints, and $T$ is the test duration. This metric is mainly used to quantify the effect of IMDEEM on energy-related behavior.

AG: To quantify the contribution of each ECIM module in the ablation study, we define AG as

(26) \begin{align} \text{AG} = \frac {1}{E} \sum _{e=1}^{E} \left ( M_{\text{full},e} - M_{\text{minus-}m,e} \right )\!, \end{align}

where $ M$ denotes a specific performance metric, like Reward and Steps-to-R^*, $ e$ ranges over all $ E=6$ environments, and $ m$ represents the ablated module. A higher AG value indicates a greater influence of the module to that metric.

4.3. Entropy-controlled intrinsic-motivation in complex terrains

To comprehensively evaluate the proposed ECIM algorithm as a whole framework, we first compare it against the PPO baseline across the six terrain types across varying complexity as well: flat, sloped, rough, stair-climbing, stair-descending, and stepping-stones.

4.3.1. Learning efficiency and task reward under entropy-controlled exploration

Figure 5.

Robot average reward variations under six different terrain environments for PPO and ECIM.

Figure 5. Long description

Each panel plots average reward against training step. Both curves rise steeply at the start of training. On flat, sloped and rough terrain they level off quickly, with ECIM slightly above PPO. On stair-climbing, stair-descending and stepping-stone terrain both keep rising more slowly and fluctuate more, and the gap in favour of ECIM is wider.

Figure 6.

Comparison of reward heatmaps for the PPO and ECIM algorithms across different terrains. The vertical axis represents training steps, with color intensity reflecting the magnitude of rewards obtained by the two algorithms at corresponding steps: different colors denote the reward levels of the ECIM and PPO algorithms, respectively.

Figure 6. Long description

Each panel has two columns, PPO left and ECIM right, with training step increasing upward and reward shown by colour from purple (low) through green to yellow and red (high). All columns start cool at the bottom and warm quickly. On flat, sloped and rough terrain the columns reach yellow to red; on the stair and stepping-stone terrains they stay green to yellow-green. In every panel the ECIM column is a warmer shade than PPO at the same step.

The evolution of cumulative rewards is the main indicator we use to assess learning efficiency and task performance in our setting. It reflects how quickly the robot learns a useful policy and how well it can stabilize and coordinate its motion on each terrain. Figure 5 shows the average reward curves of PPO and ECIM across the six terrain environments. Figure 6 further visualizes the reward evolution as heatmaps over training steps, making it easier to compare learning speed and final performance for the two algorithms.

At the beginning of training, PPO and ECIM show similar reward growth and reach basic locomotion skills at roughly the same pace. This shows that ECIM keeps the strong exploration ability of original PPO and can quickly learn fundamental gait patterns. As training continues, however, the differences between the two methods become more obvious, especially on the more complex terrains. On these terrains, PPO’s reward curves tend to grow more slowly and enter a plain earlier, which suggests that the policy may stuck in local optimal values. In contrast, ECIM maintains a higher learning rate in this critical phase, and the rewards continue to increase over a longer period. This behavior shows consistent output with the design of our pipeline, where AECPOM and IMDEEM work together to adapt exploration and provide additional learning signals beyond the standard PPO objective to give a bigger picture hint of exploration for the model.

Table II summarizes the quantitative improvements of ECIM over benchmarks across all mentioned metrics and terrain complexity levels. For cumulative reward, ECIM achieves approximately $+10.5\%$ improvement on the simple terrain (flat), $+4.3\%$ on the medium-complexity terrains (sloped, rough), and $+11.1$ – $11.8\%$ on the complex terrains (stairs and stepping-stones), leading to an overall gain of $+8.6\%$ . At the same time, Pitch Angle RMS and Joint Acceleration RMS are reduced by around $-25\%$ and $-27\%$ on average, and Torque RMS is reduced by about $-16\%$ . These results show that the full ECIM pipeline not only improves reward but also produces more stable and smoother motion with lower torque usage. In combination, the reward curves, heatmaps, and aggregated metrics confirm that ECIM supports faster and more reliable policy learning than PPO and that the benefits are most pronounced on the more challenging terrains.

Table II.

Performance improvements of ECIM over PPO across six terrains.

Table II. Long description

Columns: Metric, Simple, Medium, Complex, Overall. Cumulative Reward (%): +10.5, +4.3, +11.1 to +11.8, +8.6. Pitch Angle RMS (%): -25.2, -23.1, -27.6 to -29.4, -26.3. Joint Acceleration RMS (%): -20.3, -24.7, -28.9 to -32.4, -26.6. Torque RMS (%): -11.3, -14.8, -17.2 to -20.0, -15.8. Simple = flat; Medium = sloped and rough; Complex = stairs up, stairs down and stepping stones.

Simple: Flat; Medium: Sloped, Rough; Complex: Stairs Up/Down, Stepping Stones.

4.3.2. Locomotion stability and smoothness in intrinsically motivated gaits

Figure 7.

Pitch angle variations under six different terrain environments for PPO and ECIM.

Figure 7. Long description

Each panel plots the pitch oscillation metric against training step. Both curves spike early and then decrease. On flat, sloped and rough terrain they flatten quickly; on the stair terrains they decline gradually; on stepping stones they fluctuate strongly. The ECIM curve lies below the PPO curve and is smoother in every terrain.

Figure 8.

Joint acceleration variations under six different terrain environments for PPO and ECIM.

Figure 8. Long description

Each panel plots joint acceleration against training step. On flat, sloped and rough terrain both curves peak early and then drop to a steady level. On stair-climbing and stair-descending terrain they rise to a broad peak and decline slowly; on stepping stones they stay roughly level. The ECIM curve lies below PPO after the initial transient in every terrain, with the largest gap on the stair terrains.

The above Figure 7 shows the pitch angle evolution of PPO and ECIM on all six terrains. Overall, ECIM improves posture stability by about $23.1$ – $29.4\%$ compared with PPO (see Table II). In all environments, both algorithms reduce the initially large pitch deviations and converge towards a stable range. However, the ECIM curves stay consistently below those of PPO and converge to smaller mean pitch values with narrower fluctuation ranges. On the more complex terrains, such as stair climbing, stair descending, and stepping stones, PPO shows larger oscillations and sometimes stagnation in the later training stages, while ECIM maintains smooth, low-amplitude pitch trajectories. This indicates that the proposed motion continuity regularization, together with the intrinsic reward design, effectively suppresses unstable body tilting and helps the robot keep a more stable trunk posture throughout training.

The joint acceleration results in Figure 8 further highlight the improvement in motion smoothness. Across all terrains, ECIM reduces the joint acceleration RMS by approximately $20.3$ – $32.4\%$ relative to PPO (see Table II). Both methods start with high joint accelerations and then quickly move into a more stable range during early training. In complex terrains, however, PPO often shows slower convergence or oscillations in joint acceleration curves, while ECIM continues to optimize and eventually converges to lower average acceleration with smooth, low-oscillation trajectories. These results confirm that the ECIM framework not only stabilizes the robot’s body attitude, but also produces smoother and more natural gait transitions. In addition, smoother accelerations are consistent with the observed reduction in torque usage, which shows that ECIM improves motion quality without relying on overly actuation effort.

4.3.3. Energy efficiency of entropy-controlled quadruped policies

Figure 9 shows the joint torque evolution of PPO and ECIM across the six terrain environments. Joint torque reflects the actuation effort of the robot and is closely related to energy usage and hardware load. As summarized in Table II, ECIM achieves a reduction of approximately $11.3$ – $20.0\%$ in Torque RMS across all terrains, with an overall reduction of about $15.8\%$ . This means that, for similar or better task performance, ECIM needs less torque on average than PPO. In the early stage of training, both methods quickly reduce torque from high initial values as basic locomotion skills are learned. After this phase, the PPO torque curves tend to flatten and still show noticeable fluctuations, indicating that the policy remains more aggressive and less consistent in its torque usage. In contrast, ECIM maintains a steady downward trend and eventually converges to lower and more stable torque levels on all terrains. This behavior is consistent with the design of IMDEEM, which provides an intrinsic reward signal that encourages efficient exploration and discourages unnecessarily strong control actions. On the complex terrains, such as stairs and stepping stones, this difference becomes especially important. ECIM is able to keep torque levels low at the same time still achieving robust and stable locomotion, but PPO often relies on higher and more variable torque to cope with the same challenges. Combined with the improvements in reward, pitch stability, and joint acceleration discussed above, these results show that the proposed ECIM framework achieves more energy-efficient and hardware-friendly control than PPO, which is crucial for long-term deployment of quadruped robots in real-world environments.

Table III.

Experimental results of PPO.

Table III. Long description

Columns: Flat, Slope, Rough, Stairs Up, Stairs Down, Stepping Stones. Steps-to-R*: 1241.42, 1207.90, 1389.98, 1397.47, 1207.59, 1372.31. Pitch RMS: 0.0862, 0.0905, 0.0840, 0.0967, 0.0847, 0.0877. Acc RMS: 0.0697, 0.0871, 0.0874, 0.0762, 0.0760, 0.0649. Torque RMS: 0.1020, 0.0930, 0.0827, 0.1151, 0.1010, 0.1116.

Table IV.

Experimental results of ECIM (full model).

Table IV. Long description

Columns: Flat, Slope, Rough, Stairs Up, Stairs Down, Stepping Stones. Steps-to-R*: 876.45, 839.21, 836.39, 838.03, 871.97, 859.18. Pitch RMS: 0.0536, 0.0520, 0.0546, 0.0540, 0.0511, 0.0584. Acc RMS: 0.0593, 0.0572, 0.0571, 0.0562, 0.0574, 0.0532. Torque RMS: 0.0612, 0.0669, 0.0720, 0.0679, 0.0788, 0.0608.

Figure 9.

Torque variations under six different terrain environments for PPO and ECIM.

Figure 9. Long description

Each panel plots joint torque against training step. Flat terrain shows a large early transient followed by a gradual decline; sloped and rough terrain show a brief early peak and then flat curves; the stair and stepping-stone terrains rise to a plateau. In every terrain the ECIM curve lies below the PPO curve after the initial transient.

5.1. Baseline comparison: PPO vs. full entropy-controlled intrinsic motivation

As shown in Table III, the vanilla PPO baseline performs poorly across all environments. It is worse than other algorithms. The average Steps-to-R^* exceeds 1300 steps, and both Pitch RMS and Torque RMS are high, indicating inefficient exploration, unstable locomotion, and higher actuation effort. In contrast, the full ECIM model in Table IV achieves clear improvements:

• • the average Steps-to-R^* is reduced to $853.5$ , a $35.2\%$ decrease compared to PPO;

• • Pitch RMS drops from $0.0883$ (PPO average) to $0.0548$ , a $37.9\%$ reduction, indicating much better posture stability;

• • Torque RMS decreases from $0.1011$ (PPO average) to $0.0679$ , a $32.8\%$ reduction, reflecting more energy-efficient actuation.

These results confirm that the full ECIM framework, which combines AECPOM, MCRF and IMDEEM, enables faster learning, more stable motion and lower energy usage across all terrains.

5.2. Removing adaptive entropy control (AECPOM): effects on exploration and generalization

Removing the AECPOM leads to a clear drop in learning efficiency, as shown in Table V. The average Steps-to-R^* increases to $1068.5$ , which is about $+215.0$ steps compared to the full ECIM model, and the degradation is most pronounced on complex terrains such as rough and stepping-stones. In contrast, stability- and energy-related metrics (Pitch RMS, Acc RMS, and Torque RMS) remain relatively close to those of the full model. This indicates that AECPOM mainly acts on the exploration-exploitation balance during training, helping the policy escape local optima and reach high reward levels more quickly, especially in difficult environments. Without AECPOM, the agent can still obtain reasonably stable and efficient gaits, but it takes longer to learn them and generalizes less robustly across terrains.

5.3. Removing motion continuity (MCRF): effects on locomotion stability and smoothness

The ECIM_minus_MCRF variant in Table VI shows a different pattern. The average Steps-to-R^* remains relatively low at $931.0$ , indicating that task completion speed is still good. However, Pitch RMS increases from $0.0548$ (full ECIM) to $0.0620$ , and Acc RMS increases from $0.0567$ to $0.0608$ , which points to degraded motion smoothness and dynamic stability. In practice, the robot exhibits more oscillatory and jerky movements, particularly on sloped and stair-descending terrains, even though it still reaches the target reward threshold in a reasonable number of steps. This confirms that the MCRF is the main driver for stabilizing motion dynamics and enforcing consistent locomotion patterns across terrain transitions, without sacrificing convergence speed.

Table VII.

Experimental results of ECIM_minus_IMDEEM.

Table VII. Long description

Columns: Flat, Slope, Rough, Stairs Up, Stairs Down, Stepping Stones. Steps-to-R*: 903.15, 878.44, 875.22, 880.99, 908.66, 894.33. Pitch RMS: 0.0563, 0.0541, 0.0568, 0.0562, 0.0534, 0.0610. Acc RMS: 0.0602, 0.0581, 0.0579, 0.0570, 0.0583, 0.0540. Torque RMS: 0.0785, 0.0832, 0.0891, 0.0854, 0.0963, 0.0772.

5.4. Removing intrinsic motivation-driven exploration (IMDEEM): effects on energy use

When the IMDEEM is removed, Table VII shows that Torque RMS increases sharply to an average of $0.0849$ , which is $+0.0170$ higher than the full ECIM model, corresponding to a $25.0\%$ increase. In contrast, Steps-to-R^* ( $885.0$ on average) and the stability-related metrics remain close to those of the full model. It shows that IMDEEM plays an important role in energy efficiency. The agent can still learn effective gaits and converge quickly, but it does so with higher actuator effort when IMDEEM is absent. For real-world deployment scenarios with limited power budgets and hardware wear concerns, such additional energy consumption is undesirable, highlighting the importance of IMDEEM for practical quadruped locomotion.

Table VIII.

Attribution Gain (AG) of each ECIM module on key metrics.

Table VIII. Long description

Columns: Reward, Steps-to-R*, Pitch RMS, Acc RMS, Torque RMS. AECPOM: +1.1008, -213.3833, -0.0016, -0.0017, -0.0020. MCRF: +0.7048, -75.7850, -0.0077, -0.0040, -0.0036. IMDEEM: +0.3832, -36.5933, -0.0024, -0.0008, -0.0170.

5.5. Model attribution and cross-terrain generalization robustness

To quantify the relative contributions of AECPOM, MCRF, and IMDEEM on the key performance metrics, we compute the AG for each module. Also, the added AG values across five core metrics are summarized in Table VIII. The results show that:

• • AECPOM achieves the highest AG values in Reward (+1.1008) and Steps-to-R^* ( $-213.3833$ ), indicating its dominant role in improving task performance and accelerating policy convergence;

• • MCRF contributes most strongly to Pitch RMS ( $-0.0077$ ) and Acc RMS ( $-0.0040$ ), highlighting its impact on motion smoothness and dynamic stability;

• • IMDEEM shows the largest AG in Torque RMS ( $-0.0170$ ), far exceeding the other modules, which confirms its leading role in energy efficiency.

These quantitative results make the specialized effect of each component explicit for design adjustments. Also, it supports the ECIM functional decoupling and role specialization of each module inside.

Table IX.

Standard deviation of performance across environments (lower is better).

Table IX. Long description

Columns: Steps-to-R*, Pitch RMS, Acc RMS, Torque RMS, Reward. PPO: 84.8851, 0.0043, 0.0083, 0.0109, 2.1232. ECIM: 16.5145, 0.0023, 0.0018, 0.0062, 0.5518. ECIM_minus_AECPOM: 20.6435, 0.0024, 0.0019, 0.0064, 0.5242. ECIM_minus_MCRF: 14.5766, 0.0025, 0.0019, 0.0064, 0.5247. ECIM_minus_IMDEEM: 12.7346, 0.0024, 0.0019, 0.0065, 0.4922.

We further evaluate the consistency of each model’s performance across terrains by computing the standard deviation (Std) of each metric over the six environments, as shown in Table IX. ECIM attains the lowest standard deviations on all metrics (e.g., Steps-to-R^* : 16.51, Reward: 0.55), indicating better cross-terrain stability and generalization. The ablated variants show slightly higher variance: ECIM_minus_AECPOM has larger fluctuations in Steps-to-R^* , ECIM_minus_MCRF has marginally higher Pitch RMS Std, and ECIM_minus_IMDEEM exhibits increased Torque RMS Std. Although the absolute differences are rather small, the full ECIM model is consistently more stable than all ablated variants across all metrics.

Overall, the ablation studies show that AECPOM, MCRF, and IMDEEM play complementary roles: AECPOM mainly enhances learning efficiency and task performance, MCRF improves motion stability and smoothness, and IMDEEM reduces energy consumption. At the same time, the three modules interact positively and form a coherent pipeline that supports high-performance and robust quadruped locomotion across a wide range of terrains.