2. Preliminaries and methods

2.1. Twin-delayed deep deterministic policy gradient

Twin-delayed DDPG is an offline deterministic policy reinforcement learning algorithm specifically designed for high-dimensional continuous action spaces. TD3 algorithm enhances the DDPG algorithm by introducing an additional Critic network, resulting in two Critic networks and one Actor network in the main structure. The actor-critic framework of TD3 is illustrated in Figure 1. Additionally, the target network mirrors the main network, leading to a total of six networks. In this subsection, the TD3 will be introduced in detail.

One of the key improvements in the TD3 algorithm is the addition of a second Critic network, which helps mitigate the risk of persistent overestimation of Q-values by selecting the minimum Q-value between the two networks. Additionally, TD3 adopts a delayed update strategy, where the Critic networks are updated multiple times before the Actor network is updated once. This delay helps maintain the Q-value of the policy network at its current optimal level, preventing its degradation. Together, these strategies enable the TD3 algorithm to better estimate Q-values by incorporating three main techniques:

1. 1. Clipped Double-Q Learning: Reduces the overestimation bias by using the smaller Q-value from two Critic networks.

2. 2. Delayed Policy Updates: Ensures the Critic network stabilizes before the Actor network is updated, enhancing overall stability.

3. 3. Target Policy Smoothing: Adds noise to the target action to smooth policy updates and prevent sharp changes in the learned policy.

Together, these methods allow TD3 to achieve better performance in tasks with continuous action spaces by providing more accurate Q-value estimates and improving the stability of the learning process.

To prevent overestimation in Q-values, the TD3 algorithm employs two Q-networks, $ Q_1$ and $ Q_2$ . During the update process, it selects the smaller value between $ Q_1'$ and $ Q_2'$ as the target value. The update equation is

(1) \begin{equation} y = r + \lambda \min _{i=1,2} Q_{\theta _i}(s', \pi _{\phi }(s')) , \end{equation}

where $ y$ represents the target value, which is used to update the Q-networks. $ r$ denotes the immediate reward received from the current state-action pair. $\lambda$ is the discount factor that quantifies the importance of future rewards, taking values between 0 and 1. It reflects the present value of future rewards, and $\min _{i=1,2} Q_{\theta _i}(s', \pi _{\phi }(s'))$ indicates the selection of the minimum Q-value from two Q-networks ( $Q_1$ and $Q_2$ ), a strategy employed to mitigate overestimation bias that can arise from function approximation. The parameters for the value networks and policy networks are denoted as $ \theta$ for the value network and $ \phi$ for the policy network.

Delayed updates in TD3 mean the policy network is updated less frequently than Q-networks, specifically once for every two Q-network updates, which was realized by the soft update mechanism. The policy gradient is calculated as

(2) \begin{equation} \nabla _{\phi }J(\phi ) = N^{-1} \sum \nabla _a Q_{\theta _i}(s, a)|_{a = \pi _{\phi }(s)} \nabla _{\phi }\pi _{\phi }(s) , \end{equation}

and the soft update rules are

(3) \begin{equation} \begin{aligned} \theta _i' &\leftarrow \boldsymbol{\tau } \theta _i + (1 - \boldsymbol{\tau })\theta _i' ,\\ \phi _i' &\leftarrow \boldsymbol{\tau } \phi _i + (1 - \boldsymbol{\tau })\phi _i' , \end{aligned} \end{equation}

where $ \nabla _{\phi }J(\phi )$ represents the policy gradient and $ \boldsymbol{\tau }$ is the delayed update factor, which was set to 0.005.

To achieve smoother results, avoid outliers, and enhance exploration, random noise is added to the target policy’s output, producing stochastic actions based on a normal distribution. The action update is defined as

(4) \begin{equation} \begin{aligned} \tilde {a} &\leftarrow \pi _{\phi }(s') + \epsilon , \\ \epsilon &\leftarrow clip(N(0, \sigma ), -c, c), \end{aligned} \end{equation}

where $ \epsilon$ is the random noise applied to the action, $ clip(\cdot )$ is the clipping function, and $ N(0, \sigma )$ represents the normal distribution.

Figure 2.

Image input-based actor-critic network.

2.2. Reinforcement learning methods based on real-time natural inputs

To ensure robustness and accuracy in the assembly process, it is crucial to incorporate visual or force feedback for safety and stability. In this section, an end-to-end approach that utilizes images as natural inputs for policy learning was presented, where images act as both state representations and target guidance. Using reference images to specify targets simplifies real-world applications, while real-time image acquisition allows robots to adjust their behavior based on direct visual feedback, facilitating real-time control. Meanwhile, the robot learns to perform assembly operations directly from visual images, without accessing internal state information. To support this, the network architecture was specifically designed with image processing components to extract relevant features from the visual input, as illustrated in Figure 2. In this work, RGB images serve as state inputs, and various augmentation techniques are applied. These image augmentation methods for reinforcement learning – including grayscale conversion, sharpening, brightness randomization, contrast randomization, and motion blur – enhance the model’s adaptability to inputs by introducing noise, thereby improving the transferability of the reinforcement learning model. This is illustrated in Figure 3, which demonstrates the outstanding performance of the trained model.

A well-designed reward mechanism effectively guides an agent’s learning by balancing exploration and exploitation, enhancing decision-making in complex environments. Sparse rewards are given only at specific moments, making learning difficult to guide. Dense rewards provide continuous feedback, aiding learning, but are harder to design. And natural rewards, which are based on intrinsic task feedback, can better align with human intuition and guide behavior more effectively. In this paper, a continuous reward function is designed to improve sample efficiency and achieve faster convergence, instead of using a sparse reward function (as shown in 5). Sparse rewards are used to guide exploration, while dense rewards provide more timely feedback. The sparse reward is defined as follows:

(5) \begin{equation} R_{\text{success}} = \begin{cases} 0 & \text{done = true} ,\\ 1 & \text{done = false} , \end{cases} \end{equation}

The distance-based reward function is defined as follows:

(6) \begin{equation} R_{\text{dis}} = d_t - \log (d_t^2 + \epsilon ) + \nu , \end{equation}

(7) \begin{equation} d_t = \sqrt {(x_p - x_h)^2 + (y_p - y_h)^2 + (z_p - z_h)^2}. \end{equation}

Here, $(x_p, y_p, z_p)$ represents the position of the peg, and $(x_h, y_h, z_h)$ represents the position of the hole. $d_t$ denotes the distance between the current position and the target position. $d_t$ is the current instantaneous reward. $\epsilon$ is a constant that is set to 0.005, to prevent the logarithmic function of the reward from failing to reach the zero distance value. $\nu$ is the reward offset, which is set to 5. The task is considered successful when the robot’s distance to the target alignment is less than 0.03 m. When the task is completed, the environment generates a reward $R_{\text{success}} = 1$ . Consider the strengths and limits of the single reward method, a reward acquisition mechanism that combines sparse, dense, and natural rewards is proposed. For visual inputs, a domain-specific visual classifier is incorporated to provide binary feedback on task success or failure, enabling interaction with the environment for reward learning based on classification.

Figure 3.

Image augmented results: (1) RGB image, (2) grayscale image, (3) sharpened image, (4) brightness randomized image, (5) contrast randomized image, (6) Gaussian blurred image.

To address the strengths and limitations of using a single reward method, this paper proposes a reward acquisition mechanism that combines sparse, dense, and natural rewards in a targeted manner. Each type of reward is designed to serve a specific purpose in guiding the learning process. The sparse reward, $R_{\text{sparse}}$ , is used to signal task success or failure, providing clear and direct feedback. The dense reward, $R_{\text{dense}}$ , incorporates a distance-based component, $R_{\text{dis}}$ , which encourages the robot to move toward the target, while also including a success-based reward, $R_{\text{success}}$ , to reinforce successful task completion. The natural reward, $R_{\text{natural}}$ , is derived from the robot’s interaction with the environment, specifically through the feedback of an image classifier that provides a binary signal based on task success, $R_{\text{image}}$ . This multi-tiered reward system helps balance the trade-off between providing immediate feedback and encouraging longer-term task learning. The reward acquisition mechanism is defined as

(8) \begin{equation} \begin{aligned} & R_{\text{sparse}} = R_{\text{success}}, \\ & R_{\text{dense}} = R_{\text{dis}} + R_{\text{success}}, \\ & R_{\text{natural}} = R_{\text{image}}, \end{aligned} \end{equation}

where $ R_{\text{sparse}}$ , $ R_{\text{dense}}$ , and $ R_{\text{natural}}$ represent sparse, dense, and natural rewards, respectively. $ R_{\text{success}}$ refers to a binary reward based on task success, $ R_{\text{dis}}$ is a continuous distance-based reward, and $ R_{\text{image}}$ is a binary reward provided by the image classifier.

3. Admittance control-based residual reinforcement learning algorithm

To ensure robustness and accuracy in the assembly process, RRL is extended to vision-based learning. By leveraging visual inputs, correct assembly actions are learned, and residual policies based on image data are further optimized on top of a pre-designed controller. Real-time detection during the assembly process is used to adjust the current actions, seamlessly integrating visual correction with control to achieve a balanced approach. The control framework is shown in Figure 4.

Figure 4.

Residual reinforcement learning algorithm based on admittance control.

The admittance controller uses force sensors to capture real-time force information from the robot’s end-effector in the external environment, while also obtaining the robot’s current position and velocity. The control algorithm feeds the force information back to the robot controller, adjusting the robot’s posture and motion based on changes in external forces or position. The formulation is given as

(9) \begin{equation} \mathbf{F}_{\text{ext}} = \mathbf{K}_e (\mathbf{x}_d - \mathbf{x}) + \mathbf{D}_e (\dot {\mathbf{x}}_d - \dot {\mathbf{x}}), \end{equation}

where $\mathbf{F}_{\text{ext}}$ represents the force exerted by the environment on the robot’s end-effector, $\mathbf{x}_d$ is the desired end-effector position, and $\mathbf{x}$ is the actual end-effector position. $\dot {\mathbf{x}}_d$ and $\dot {\mathbf{x}}$ are the desired and actual end-effector velocities, respectively. $\mathbf{K}_e\succ 0$ and $\mathbf{D}_e\succ 0$ are symmetric positive definite stiffness matrix and damping matrices, respectively. Through inverse kinematics, the end-effector’s position and velocity data are converted into joint control commands, enabling the robot’s control system to implement compliant control.

In the proposed framework, the admittance policy $\pi _A$ is implemented as a neural network that adaptively modulates the impedance behavior based on the robot’s motion state and task context. Specifically, $\pi _A$ takes the current state vector as input and outputs the diagonal elements of the stiffness matrix $\mathbf{K}_e$ and damping matrix $\mathbf{D}_e$ used in the admittance dynamics. To ensure physical plausibility and stability, the outputs of $\pi _A$ are constrained to be strictly positive using softplus activation functions, such that

(10) \begin{equation} \mathbf{K}_e = \operatorname {diag}(\text{softplus}(w_K)), \quad \mathbf{D}_e = \operatorname {diag}(\text{softplus}(w_D)), \end{equation}

where $w_K, w_D \in \mathbb{R}^3$ are raw outputs of the network and $\mathrm{Softplus}(x) = \log (1 + e^x)$ is a smooth and continuously differentiable function that guarantees positive values. This design guarantees that both $\mathbf{K}_e$ and $\mathbf{D}_e$ remain symmetric and positive definite throughout learning and execution. It enables the system to continuously adapt its compliance properties during interaction, facilitating flexible and robust task execution under varying contact conditions.

RRL interacts with the environment to generate transitions $e = (s, a, r = f(s, a), s')$ , stored in the experience replay buffer for training. The control policy is expressed as

(11) \begin{equation} s' = \pi _A(f) + k \pi _{\phi }(s), \end{equation}

here, $k$ is the reinforcement learning weight, when $k = 0$ , the control strategy is pure admittance control. By adjusting $k$ , the magnitude of the output from the reinforcement learning model can be controlled. $\mathbf{s}$ is the state, $\mathbf{a}$ the action, $\mathbf{r}$ the reward, and $\mathbf{s}'$ the next state. In admittance control, the residual policy refines the traditional controller. For RRL in admittance control, the residual policy is learned via reinforcement learning. The RL network structure uses a convolutional neural network to process image inputs, with output actions $\mathbf{a}_t = [\delta _x, \delta _y, \delta _z]$ , which are added to the outputs of the admittance control system. The admittance controller’s target is the desired assembly position, while the feedback signal consists of external forces applied to the robot’s end-effector from the force sensors $[f_x, f_y, f_z]$ . The desired end-effector position is $[x, y, z]$ , and the actual end-effector position after control is $[x + \delta _x, y + \delta _y, z + \delta _z]$ . The complete algorithm flow is shown in Table I below as Algorithm1. The superiority of our proposed method compared with other existing methods is listed in Table I.

Table I.

Comparison of related work and our proposed method.

Algorithm 1

$\rm{Admittance\, Control\, RRL\, Algorithm\, Based\, on\, TD3\, and\, Natural\, Inputs}$

To analyze the convergence behavior of the system, we define the tracking error as $e = x - \mathbf{x}_d$ . The operational space dynamics consider the centripetal and Coriolis forces, and the gravitational torque is expressed as

(16) \begin{equation} M(q) \ddot x + \tilde C(q,\dot q) \dot x + p(q) = F_{\text{ext}}. \end{equation}

where $M(q) = (J M_r^{-1} J^\top )^{-1}$ is symmetric positive definite and $J$ is assumed full rank within the workspace, $\tilde C(q,\dot q)$ is the operational space Coriolis/centrifugal mapping, and $p(q)$ denotes the operational space gravity term. The force

(17) \begin{equation} F_{\text{ext}} = M \ddot x_d + \tilde C \dot x_d + p + K_e (x_d - x) + D_e (\dot x_d - \dot x). \end{equation}

Then the error dynamics becomes

(18) \begin{equation} M \ddot e + (\tilde C + D + D_e) \dot e + K_e e = 0. \end{equation}

Define $S \;:\!=\; D + D_e$ as the symmetric positive definite damping part. Choosing candidate Lyapunov function as

(19) \begin{equation} V(e,\dot e) = \frac {1}{2} \dot e^\top M(q) \dot e + \frac {1}{2} e^\top K_e e + \varepsilon e^\top M(q) \dot e, \end{equation}

where $\varepsilon \gt 0$ is a small cross-term coefficient. Assuming there exist $\underline M, \overline M \gt 0$ such that $\underline M I \preceq M(q) \preceq \overline M I,$ choosing $\varepsilon$ sufficiently small ensures that $V$ is positive definite and radially unbounded. The time derivative of $V$ yields

(20) \begin{equation} \begin{aligned} \dot V &= \dot e^\top M \ddot e + \frac {1}{2} \dot e^\top \dot M \dot e + \dot e^\top K_e e + \varepsilon (\dot e^\top M \dot e + e^\top \dot M \dot e + e^\top M \ddot e)\\ &= -\dot e^\top S \dot e + \dot e^\top \tilde C \dot e + \varepsilon \dot e^\top M \dot e + \varepsilon e^\top \dot M \dot e - \varepsilon e^\top S \dot e - \varepsilon e^\top K_e e, \end{aligned} \end{equation}

under $\frac {1}{2} \dot e^\top \dot M \dot e = \dot e^\top \tilde C \dot e.$ Assuming constants $\|M(q)\| \le L_M$ , $\|\dot M(q)\| \le L_{\dot M}$ , $\|\tilde C(q,\dot q)\| \le L_C$ , $\|S\| \le L_S$ and $s_{\min } \;:\!=\; M_{\min }(S) \gt 0$ , $\kappa \;:\!=\; M_{\min }(K_e) \gt 0$ , then utilizing Young’s inequality, $\dot V$ can be reformed as

(21) \begin{equation} \dot V \le -\bigg ( s_{\min } - L_C - \varepsilon L_M - \frac {\varepsilon C_1}{2\delta } - \frac {\varepsilon L_S}{2} \bigg ) \|\dot e\|^2 - \bigg ( \varepsilon \kappa - \frac {\varepsilon \delta }{2} - \frac {\varepsilon L_S}{2} \bigg ) \|e\|^2. \end{equation}

Choosing proper parameters according to actual scenarios to ensure $\alpha =s_{\min } - L_C - \varepsilon L_M - \frac {\varepsilon C_1}{2\delta } - \frac {\varepsilon L_S}{2}\gt 0, \beta = \varepsilon \kappa - \frac {\varepsilon \delta }{2} - \frac {\varepsilon L_S}{2}\gt 0$ , one has

(22) \begin{equation} \dot V \le -\alpha \|e\|^2 - \beta \|\dot e\|^2 \end{equation}

Therefore, by strict Lyapunov theory, we have asymptotic convergence

(23) \begin{equation} e(t) \to 0, \quad \dot e(t) \to 0, \quad \text{as } t \to \infty . \end{equation}

It should be emphasized that, in our experimental setup, the robotic arm is restricted to end-effector position-level control, with a maximum operational speed of only about 0.1 m/s. Under such low-speed conditions, the influence of Coriolis and centripetal forces becomes negligible, which allows us to simplify the dynamic model without sacrificing accuracy. For the sake of theoretical rigor, the terms $\tilde {C}$ and $p$ are retained in the proof, but in practice, their effects are insignificant. In addition, the gravitational torques are fully compensated by the low-level controller, ensuring that they do not interfere with the control performance. Furthermore, throughout the experiments, the manipulator operates in a region of the workspace that is far from singular configurations, which eliminates potential issues of ill-conditioning or instability due to singularities. Taken together, these conditions validate the assumptions used in the analysis and guarantee that the theoretical results remain consistent with the actual experimental scenario.

Figure 5.

The real-world experiment platform.

Figure 6.

Simulation experiment result.

4. Experiments and results

We conducted the peg-in-hole experiments to facilitate the effectiveness of the proposed framework. The real-world experiment platform is shown in Figure 5. The Elite manipulator has six degrees of freedom, with the RealSense camera mounted on the end of the manipulator and the ATI force sensor mounted on the end-effector. The peg and hole were 3D-printed with radii of 48 mm and 50 mm, respectively. Before the real-world validation, simulation experiments were carried out in MuJoCo. Ablation experiments demonstrate that the controller trained solely on image input can achieve millimeter-level precision in assembly tasks, with the image input shown in Figure 6; it can be seen that from subfigure $a$ to $g$ , the assembly task was completed successfully. The key reinforcement learning parameters include a learning rate of 0.0003, a sample size of 4000 per iteration, a reward discount factor of 0.99, an action Gaussian noise variance of 0.1, and a storage update period of 50.

Due to the action policy incorporates knowledge from the admittance control strategy, training the agent on this basis significantly reduces the overall training time. As illustrated in Figure 7, when the admittance control strategy is combined with RRL, the agent learns a successful assembly strategy around the 100th iteration. In contrast, without the residual learning, it takes more than 400 iterations to achieve the same result. This reduction in training time makes the method much more practical for real-world robotic applications.

Figure 7.

Training results of TD3 and TD3 with RRL.

Figure 8.

Training results of different reward functions.

Moreover, as shown in Figure 8, a comparison of different reward functions shows that our custom-designed reward function further boosts the training efficiency of RRL. Our designed method and framework have low requirements for the reward function design and high adaptability. It can achieve good training results with different reward functions, requiring only a simple reward function design.

Figure 9.

Assembly force of $x,y,z$ in simulation experiment.

Figure 10.

Assembly trajectories of $x,y,z$ in simulation experiment.

In the simulation experiment, the contact force during assembly remains within the range of −100N to 100 N. The starting point of the assembly is (0.4 m, 0.1 m, 0.28 m), and the target point is (0.3 m, 0.0 m, 0.16 m). As shown in Figure 9, the left figure shows the contact forces using the original admittance control algorithm, while the right figure illustrates the forces with RRL control. When using only the admittance control strategy, the robot’s contact time with the environment is prolonged, which is unfavorable for safe assembly. In contrast, after training with RRL, the robot significantly reduces contact time during the assembly process, allowing for quicker and safer task completion. The trajectories illustrated in Figure 10 also reached the same result. This shows that combining reinforcement learning with admittance control improves disturbance resistance, enabling the robot to better adapt to environmental changes and enhancing both the success rate and overall control performance.

In real-world experiments, disturbances were introduced to the hole’s position by adding a $1cm$ offset in both the x and y directions. A comparison was then made between the performance of the robot using only the admittance controller and that using the RRL algorithm. The admittance control parameters were set to $D = 1000$ and $K = 1000$ , with a step time of $dt = 0.5s$ . The control coefficient for reinforcement learning was set to 0.01, with the action output range between −0.01 m and 0.01 m. The robot moved from the starting point (500 mm, 120 mm, 330 mm) to the target assembly point (510 mm, 110 mm, 50 mm). The contact force experienced by the robot was within the range of −100 N to 100 N, and if the force exceeded this threshold, the robot executed a reset action to ensure safety. The results show that when only the admittance controller was used, the robot failed to align with the hole after the disturbance, leading to collisions around the hole and preventing successful assembly. In contrast, with the RRL algorithm, the robot was able to dynamically adjust its actions using visual feedback, adapting to the changes in the environment. The real-world experiment results are shown in Figure 11, and the trajectory is illustrated in Figure 12.

Figure 11.

The real-world peg-in-hole assembly process.

Figure 12.

Assembly trajectories of $x,y,z$ in real-world experiment.

Figure 13.

Assembly force of $x,y,z$ in real-world experiment.

As shown in Figure 13, without the reinforcement learning strategy, the contact force exceeded the 100 N threshold. After applying the reinforcement learning strategy, the contact force remained below 1 N in successful cases, significantly improving the robot’s operational safety and reducing the contact force duration. The experimental results demonstrate that the proposed method delivers the same training efficiency and effectiveness in real-world applications, significantly enhancing the robot’s safety and adaptability in assembly tasks.