1. Introduction
Assembly operations using robots have increasingly become a crucial aspect of industrial production. Prolonged programing time for industrial robots in personalized manufacturing frequently requires extensive code compilation, significantly reducing productivity [Reference Mnih, Badia, Mirza, Graves, Lillicrap, Harley, Silver and Kavukcuoglu20, Reference Villagomez and Ordoñez30]. Most robots in current production environments rely on preprograming, leading to poor adaptability in complex and variable conditions.
With various use cases and developments in automation technology, industrial robots must conduct complicated tasks in contact-rich, dynamic environments. Insertion is a fundamental contact-rich robotic task essential for various practical deployments, from domestic tasks to modern assembly lines [Reference Kroemer, Niekum and Konidaris15]. In the satellite component assembly industry, industrial robots encounter challenges when assembling nonstandard multiaxis hole parts. Traditional peg-in-hole methods, which depend on predefined paths or movements [Reference Stemmer, Schreiber, Arbter and Albu-Schaffer28], face challenges due to significant problem variability, limited adaptability to slight changes, and high costs.
In the past few years, the reinforcement learning (RL) method has excelled in automatically learning manipulation skills through interaction with environments, particularly for peg and hole assembly tasks [Reference Inoue, De Magistris, Munawar, Yokoya and Tachibana11, Reference Luo, Solowjow, Wen, Ojea, Agogino, Tamar and Abbeel19]. Nevertheless, in the course of the robot agent training process, numerous useless explorations occur due to redundant state dimensions and a shortage of basic geometric knowledge before the agent acquires successful skills. In view of longevity and safety considerations, direct real robot training is often impractical, prompting a shift to simulation-based learning for contact-rich tasks [Reference Zhao, Queralta and Westerlund41]. While sim-to-real algorithms are both cheap and safe, they require precise modeling to ensure effective transfer to real-world scenarios [Reference Breyer, Furrer, Novkovic, Siegwart and Nieto5]. Generally, getting an accurate model of the tangible interaction between two components is challenging [Reference Bouchard, Nesme, Tournier, Wang, Faure and Kry4], making it hard to create simulators for diverse contact-rich tasks. Setting simulation parameters to match real-world conditions is challenging, leading to visual discrepancies in camera position, lighting, rendering, and texture.
To reduce the sim-to-real difference, two approaches can be used: 1) Improve the simulation to more closely mimic reality, and 2) Develop invariant representations that align the simulation with reality, then train the robot agent with invariant representations. Domain randomization is a typical technique for studying invariant representations [Reference Tobin, Fong, Ray, Schneider, Zaremba and Abbeel29]. By introducing arbitrary noise into simulation conditions including camera viewpoint and light shade, the RL policy can be trained to be robust to such factors. As a result, the robot agent demonstrates robustness when applied to the real environment.
However, this approach increases the complexity, as the robot agent must be trained within a broader observation space. A different approach is to describe a canonical observation space and subsequently map both the simulation environment and practical observations into this space to minimize the gap between the two environments [Reference James, Wohlhart, Kalakrishnan, Kalashnikov, Irpan, Ibarz, Levine, Hadsell and Bousmalis12].
In contact-rich tasks, the minor clearance between the plug and hole results in repeated contact. Factory robots are generally rigid, position-controlled devices, where even a minor position error can cause considerable contact force during assembly, possibly leading to assembly failures or device destruction. Hence, compliance control is essential to prevent these issues and enhance assembly outcomes [Reference Beltran-Hernandez, Petit, Ramirez-Alpizar and Harada1].
Under active compliance control, the robot uses force sensor feedback and a control policy to purposefully adjust to the simulator, offering greater flexibility compared to passive compliance [Reference Kim, Song and Song14].
Therefore, a combined system should be developed to address these challenges. To tackle sample limitations and enable the shift from simulation to the real scene, this article presents a DRL-based insertion policy learning method that allows a robot to efficiently align satellite component pieces using both image feedback and exerted force. First, we developed a sim-to-real learning framework for peg-in-hole tasks that integrates vision and force control using segmented images from the simulation camera. To speed up the agent learning process, the DRL policy is initially pretrained on a small number of demonstrations. Secondly, we pretrain the segmentation model U-nets by constructing datasets in a simulation environment and then fine-tune it based on real-world data. This approach significantly reduces the demand for real image segment labels, which are laborious to acquire. In addition, impedance control safety interaction strategies are integrated into the suggested system to aid the agent handle contact-rich tasks. Finally, the peg-in-hole manipulation is assessed in both virtual and physical robot environments, showing that the suggested image segmentation-driven robot agent, trained with canonical space successfully accomplishes assembly.
To sum up, the key contributions of this paper are:
-
1) We propose a DRL-based insertion policy learning method (Seg-DQN) that combines image feedback and exerted force, enabling efficient alignment of satellite components by robots.
-
2) We develop a sim-to-real learning framework for peg-in-hole tasks, using segmented images from a simulation camera, with the DRL policy pretrained on a small number of demonstrations to expedite the learning process.
-
3) We integrate impedance control and safety interaction strategy into the system to handle contact-rich tasks.
-
4) We evaluate the peg-in-hole task in both simulation and real robot environments, demonstrating the success of the image segmentation-driven robot agent trained with canonical space.
2. Related work
2.1. RL-based assembly methods
With the fast breakthroughs in RL, RL-based methods have emerged as the most favored approach for peg-in-hole robot assembly tasks. RL-based methods eliminate the need for manually developing particular policies for different phases. Instead, they enable the robot to independently identify the optimal actions to carry out the task successfully.
Peg-in-hole assembly manipulation can be hazardous during real-world training due to the unforeseeable contact forces produced by physical interactions. Visual and force feedback are crucial modalities in peg-in-hole assembly tasks [Reference Chen, Zeng, Liang, Sun and Zhang7]. The visual modality can implicitly offer vital information like object pose and end-effector touch condition, while the force modality is primarily used for precise adjustments and compliant control.
In a prior study [Reference Inoue, De Magistris, Munawar, Yokoya and Tachibana11], a method was proposed in which a robot uses a recurrent neural network trained with RL to perform high-precision peg-in-hole tasks. This approach reduces manual parameter tuning and enhances fitting performance and robustness against errors. InsertionNet [Reference Spector and Di Castro27] addresses the insertion problem in assembly tasks as a regression problem, effectively combining visual and force inputs. This approach scales rapidly to various insertion tasks, demonstrating robustness to variations in socket position, orientation, peg color, and shape, and offers an end-to-end solution for complex multi-insertion objectives.
Given that the classic individual hole insertion manipulation has been thoroughly studied and confirmed using RL methods, tackling the more complex problem of multiple hole insertion manipulation presents a greater challenge. As the count of pegs and holes increases, the security and dependability of the pose and force predictions from the RL agent tend to decrease. Specifically when accounting for rotational dimensions, aligning one peg perfectly may cause the remaining pegs to fail to align the hole owing to errors in angle.
In recent times, several researchers have initiated using RL to tackle this task. Hou et al. [Reference Hou, Dong, Zhang, Gao, Chen and Xu10] first proposed a deep deterministic policy gradient (DDPG) algorithm augmented with PD force control and a hybrid exploration strategy to efficiently and stably learn multiple hole insertion manipulation, demonstrating significant performance improvements in both simulation and real-world experiments after only 150 training episodes. To enhance training efficiency, Xu et al. [Reference Xu, Hou, Wang, Xu, Zhang and Chen35] employed reaction exploration and an ambiguous reward mechanism, showing a notable improvement in performance. Based on the same setup, Li et al. [Reference Li, Xiao, Zhao, Liu and Wang17] proposed a learning-accelerated deep deterministic policy gradient (LADDPG) algorithm for autonomous and efficient multiple peg-in-hole assembly, using an advanced planning module and a bottom control module to derive commands and drive compliant assembly, significantly reducing contact force and torque compared to the original DDPG algorithm.
The work presented above has demonstrated promising experimental results for multiaxis hole assembly using RL. However, it is important to note that the experimental conditions employed did not entirely align with the real-world physical model. This discrepancy highlights the need for further refinement to bridge the gap between simulated environments and practical applications. Specifically, the lack of visual feedback has made it challenging for the trained assembly strategy to be directly transferred and deployed in real-world environments. To address the aforementioned issues, this article integrates visual, force, and robot body information to develop an interactive learning strategy for insertion tasks. This approach aims to enhance the adaptability and effectiveness of the assembly strategy by incorporating multiple sources of feedback. To address the data lack from the physical environment, we create a simulation scene to train the robot agent. We then apply a sim-to-real technique to seamlessly transfer the learned skill from simulation to real-world applications without refinement.
2.2. Sim-to-real methods
In robotic assembly, gathering actual interaction datasets for RL methods is costly. Specifically, RL methods require extensive data for sampling and exploration. Sim-to-real methods [Reference Wu, Zhou, Yang, Huang and Lv32, Reference Zhao, Queralta and Westerlund41] have been developed to address these issues.
While certain physics-based RL approaches [Reference Pareek, Nisar and Kesavadas21] can seamlessly transition to the physical environment, vision-based RL methods necessitate discovering an invariant representation for effective sim-to-real transfer. Invariant representations can be categorized into two main types: 1) Feature-level invariant representation, which seeks to find a low-size depiction including only work-related content and 2) Pixel-level invariant representation, which entails mapping both virtual and physical visual data into a unified canonical image observation space.
When it comes to feature-level representation learning, Puang et al. [Reference Puang, Tee and Jing23] introduces KOVIS, a keypoint-based visual servoing method for robotic manipulation using an eye-in-hand stereo camera system. KOVIS trains a deep neural network in simulation and achieves zero-shot transfer to real-world environments, including holding and peg-in-hole assembly. The method features two networks for keypoint representation and motion learning, demonstrating effectiveness in both virtual and physical experiments. Jeong et al. [Reference Jeong, Aytar, Khosid, Zhou, Kay, Lampe, Bousmalis and Nori13] utilize time contrastive learning to develop an encoder for visual observation. Instead of behavior cloning (BC), they derive the related actions from deep reinforcement learning (DRL), enhancing stability in unfamiliar environments.
To directly apply advanced visual-based control approaches, pixel-level invariant representation is a viable solution. James et al. [Reference James, Wohlhart, Kalakrishnan, Kalashnikov, Irpan, Ibarz, Levine, Hadsell and Bousmalis12] introduce randomized-to-canonical adaptation networks (RCANs), a novel method that bridges the visual reality gap without requiring real-world data by translating randomized rendered images into canonical versions. RCANs enable successful sim-to-real transfer, achieving 70
$\%$
zero-shot grasp success and 91
$\%$
performance with minimal real-world data, significantly outperforming traditional domain randomization approaches. Similarly, Rao et al. [Reference Rao, Harris, Irpan, Levine, Ibarz and Khansari24] present RL-CycleGAN, a method that integrates RL-scene consistency loss into unsupervised domain translation to ensure task-aware simulation-to-real transfer for RL, improving real-world performance in vision-based robotic grasping tasks. Nevertheless, GAN-based generators can encounter challenges such as slow convergence, instability, and high computational demands.
Alternatively, Pashevich et al. [Reference Pashevich, Strudel, Kalevatykh, Laptev and Schmid22] use depth images as the canonical space. This article introduces an optimized augmentation strategy for depth images using domain randomization to improve sim-to-real transfer and enable policy learning for robotic manipulation tasks without real-world images. Hong et al. [Reference Zhang, Chen, Yang, Su, Shann, Chang, Brian, Tu, Hsiao, Lai, Chang and Lee9] present a modular architecture that separates learning into a perception module for semantic image segmentation and a control policy module for DRL, effectively bridging the virtual-to-real discrepancy and improving achievement in both simulated and physical tasks. However, their work only demonstrates simple reaching tasks performed by the mobile robot. Similarly to [Reference Zhang, Chen, Yang, Su, Shann, Chang, Brian, Tu, Hsiao, Lai, Chang and Lee9], Xu et al. [Reference Xu, Hassan and Hussain34] introduce Seg-CURL, a low-cost, segmentation-driven unsupervised RL framework that improves sim-to-real transfer by using semantic segmentation to align simulated and real-world images, achieving effective performance with minimal real-world data.
Owing to the limitations in collision modeling and contact simulation for contact-rich tasks, there has been limited research on incorporating force feedback in simulations. As a result, transferring contact-rich skills and force from simulation to real-world applications remains a significant challenge.
2.3. Force controlled robots
Managing a robot’s interaction with its environment is crucial for effectively performing tasks that involve acting on surfaces. Robots are often required to engage with their environments in contact-intensive tasks. A robot’s workstation restricts the spatial pathways its end-effector can follow during constrained motion. In this case, using a position control method to manage interactions is ineffective.
In complex assembly tasks, the robot must achieve a specific position and address environmental opposition with the required force. Therefore, robot force control needs to dynamically output joint torque according to the desired force, speed, and position. Therefore, different kinds of robot force control algorithms and corresponding robots have been developed. As a result, various robot force control methods have been created [Reference Liu, Chen, Qiao and Gan18, Reference Wu, Liu and Wang33].
It has also been well established that impedance control is effective for various robotic assembly tasks [Reference Beltran-Hernandez, Petit, Ramirez-Alpizar, Nishi, Kikuchi, Matsubara and Harada2, Reference Cao, Laws and y Baena6, Reference Yu, He, Li, Li and Li37–Reference Zhang, Sun, Kuang and Tomizuka40]. Yu et al. [Reference Luo, Solowjow, Wen, Ojea, Agogino, Tamar and Abbeel19] combines RL with operational space force/torque information and a specialized force controller, using a neural network architecture to handle environmental variations, and evaluates the method on the Siemens Robot Learning Challenge for precise gear wheel assembly. In the context of the single hole insertion manipulation, Kulkarni et al. [Reference Kulkarni, Kober, Babuška and Santina16] suggested an architecture that integrates RL with classical control strategies using a time-varying weighted sum to adaptively handle uncertainties in real-world assembly tasks, achieving high success rates and robustness with minimal retraining. In addition, Bogdanovic et al. [Reference Bogdanovic, Khadiv and Righetti3] investigates how choosing the appropriate action space can improve robustness in the presence of contact uncertainties, proposing a policy that outputs impedance and desired joint positions, and comparing its performance with torque and position control under varying contact uncertainties, while also introducing an additional reward term to regularize these variable impedance control policies for better interpretability and transferability to real systems. According to that, Yang et al. [Reference Yang, Dürr, Topp, Stork and Stoyanov36] presented an extended skill-based RL framework with a variable impedance action space for learning contact-rich manipulation policies, utilizing both position and impedance-space information from a few suboptimal demonstrations, and shows that this method can be directly applied to real robots for peg-in-hole tasks without the need for simulation-based training. Building on the previous work, this article applies impedance control and safety protection strategies to satellite component assembly tasks.
3. Methods
In this section, we suggest a sim-to-real learning system for contact-rich peg-in-hole tasks, using a U-Net and robot force control. Figure 1 provides an outline of our suggested framework for satellite component assembly. Our system primarily consists of both simulation and real-world components. A RL-based peg-in-hole assembly policy leveraging vision and force feedback is proposed, where force feedback ensures safe assembly. To prevent excessive force output from activating an emergency stop, a force-driven safety interaction lock (SIL) is suggested to manage the robot’s pressing force. To achieve sim-to-real transfer, simulation and actual image observations are transformed into canonical space through an image segmentation model, U-Net. The assembly actions output by the RL-trained agent are sent to a compliant Cartesian space controller to enable compliant assembly.
A sim-to-real learning framework for a peg-in-hole (PiH) assembly task. The simulation component is used to train the image segmentation model and the RL policy network within a simulator. In the real pipeline, real images are mapped to segmented images to translate them from a raw image space to a canonical space. An impedance-based controller is used to execute agent actions when transferring the RL agent to the real scene.
RL policy learning with demonstration.
3.1. Framework design
Our method utilizes a dueling deep Q-network (DQN) [Reference Wang, Schaul, Hessel, Hasselt, Lanctot and Freitas31] framework. Dueling DQN is an improvement of the DQN algorithm, which effectively improves the algorithm performance. We initially train the neural network with demonstration data prior to engaging with the assembly environment. By manually controlling the robotic arm to perform the insertion task within the simulator, expert demonstration data can be collected for the preliminary training of the RL policy, as shown in Figure 1. After this pretraining phase, the robot begins interacting with the environment following its initial policy.
3.1.1. Pretraining with demonstration data
As shown in Figure 2, the robot agent utilizes demonstration data to train the initial model, followed by online interactive training. During the pretraining phase, the agent collects samples in batches from the demonstration data to optimize the model parameters. The pretraining loss function comprises a blend of three loss functions: DQN loss
$\lambda _d$
, supervised loss
$\lambda _E$
, and L2 regularization loss.
\begin{equation} {\lambda _p} = {\lambda _d} + {\alpha _1}{\lambda _E} + {\alpha _2}{\left \| w \right \|^2} \end{equation}
where
$\alpha _1$
and
$\alpha _2$
scaling factors. The DQN loss guarantees that the network adheres to the Bellman equation. The supervised loss compels the values of the other actions to have a significant margin compared to those of the demonstrator’s action, specifically,
\begin{equation} {\lambda _E} = \mathop {\max }\limits _{a \in A} \left [ {Q\left ( {s,a} \right ) + l\left ( {{a_d},a} \right )} \right ] - Q\left ( {s,{a_d}} \right ) \end{equation}
In this context,
$a_d$
represents the action performed by the teacher according to state
$s$
, and
$l\left ( {{a_d},a} \right )$
is a margin function that equals 0 when
$a = a_d$
and is positive or else. L2 regularization is implemented to the model weights to avoid overlearning on the somewhat limited demonstration dataset.
3.1.2. RL-based insertion policy learning
We model the insertion task as a potential function that maps the current state
$\left ( S \right )$
, action
$\left ( A \right )$
, and reward
$\left ( R \right )$
to the next state
$\left ( {S'} \right )$
, described by the relationships
$(S, A) \rightarrow R$
and
$(S, A) \rightarrow S'$
. After extensive training, the policy becomes more reliable and optimally effective in the long term, aiming to identify the best actions across various states and maximize expected rewards. Convolutional neural networks excel at dimensionality reduction for high-dimensional data, particularly in image processing. Our method utilizes multimodel sensor information, including segmented images, as well as force/torque, position, and posture information from the robotic arm’s end-effector. We concentrate on the ultimate stage of the peg-in-hole task, specifically the hole exploring and insertion stage once the components are positioned close to the hole, focusing on accurate assembly within a limited area.
The model takes two types of inputs: image observation (segmented image) and force/torque data. The image data has a dimension of (640, 480, 3), and the force/torque values are (6, 1).
\begin{equation} s = \left [ {{F_x},{F_y},{F_z},{M_x},{M_y},{M_z},I} \right ] \end{equation}
where
$F$
and
$M$
are the force and torque obtained from the F/T sensor,
$I$
is the image obtained from the camera.
Every action performed by the RL agent is defined as follows:
Action refers to the path chain generated by the RL agent, which can be executed in either joint space or Cartesian space. Our method uses actions defined in Cartesian space. To guarantee accuracy and consistency, the highest permissible action value is restricted to under 1 mm or 0.01
$^\circ$
.
\begin{equation} a = \left \{ {\Delta x,\Delta y,\Delta z,\Delta \alpha ,\Delta \beta ,\Delta \gamma } \right \} \end{equation}
Figure 3 illustrates an optimal dynamic peg-in-hole procedure for the trained RL agent to complete the difficult task, which includes initial positioning, nearing the hole, pose alignment, depth exploring, and achieving the goal.
Our effective robot agent exploration process for the peg-in-hole insertion task involves four stages: moving from the initial pose to nearing the goal, from nearing the goal to pose refinement, from pose refinement to depth exploring, and from depth exploring to achieving the goal.
In the course of the approaching phase, a custom reward is designed by calculating the L2 distance between the key point position
$p_g$
on the peg and their corresponding target locations
$p_h$
in the hole object.
The reward is defined as follows:
\begin{equation} R\left ( s \right ) = \left \{ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {0.01 - \tanh (||{p_g} - {p_h}||),}&{if\,||{p_g} - {p_h}|| \ge {d_1}} \end{array}}\\ {\begin{array}{*{20}{c}} {0.01 + (0.01 - \tanh (||{p_g} - {p_h}||),}&{if\,||{p_g} - {p_h}|| \lt {d_1}} \end{array}}\\ {\begin{array}{*{20}{c}} {50,}&{if\,||{p_g} - {p_h}|| \lt {d_2}} \end{array}} \end{array}} \right . \end{equation}
A value of
${d_1} = 0.03$
signifies the transition from pose alignment to depth exploring in the interactive assembly procedure as the pegs enter the holes. This incentivizes the robot agent to continue depth exploring until reaching the target position, thereby enhancing the learning process.
${d_2} = 0.01$
indicates that the robot has successfully completed the insertion task.
3.2. SIL for robot
To prevent the robot from colliding violently with the assembly part and causing issues like system breakdowns, we propose using a SIL method in each control loop, as shown in Algorithm 1. The vector containing these two sets of signals acts as the data for the SIL. As the peg makes contact with the vicinity of the hole, the F/T sensor reading rises substantially, and the logged position at this time is designated as the limited contact position
$z_t^c$
for future explorations. To accelerate finding the initial limit, we artificially introduce a minor increment
$\delta z_t^{p}$
in the insertion direction, which does not alter the entire motion path of the robot agent exploration.
SIL
As exploring the periphery of the hole significantly impacts the F/T value, we define a threshold
$\delta f$
. This threshold is triggered when the edge of the hole is detected during the initial exploration and is employed to extend the limit
$z_t^c$
. The detailed calculation is as follows.
\begin{equation} \begin{array}{l} \delta x_t^p = {w_{11}} \cdot \left ( {{F_{x\left ( t \right )}} - {F_{x\left ( {t - 1} \right )}} + {M_{x\left ( t \right )}} - {M_{x\left ( {t - 1} \right )}}} \right )\\ \delta y_t^p = {w_{12}} \cdot \left ( {{F_{y\left ( t \right )}} - {F_{y\left ( {t - 1} \right )}} + {M_{y\left ( t \right )}} - {M_{y\left ( {t - 1} \right )}}} \right )\\ \delta z_t^p = {w_{13}} \cdot \left ( {{F_{z\left ( t \right )}} - {F_{z\left ( {t - 1} \right )}} + {M_{z\left ( t \right )}} - {M_{z\left ( {t - 1} \right )}}} \right ) \end{array} \end{equation}
Here,
$F$
and
$M$
represent the last two data in the
$R_f$
log.
$w = \left [ {{w_{11}},{w_{12}},{w_{13}}} \right ]$
denotes the F/T deviation acquire vector. The small gains
$\delta x_t^p$
,
$\delta y_t^p$
, and
$\delta z_t^p$
are employed to increase the limit
$z_t^c$
.
\begin{equation} z_t^c = z_t^p + \delta x_t^p + \delta y_t^p + \delta z_t^p \end{equation}
Here,
$z_t^p$
represents the final value in the
$R_p$
record. If the threshold
$\delta f$
is not activated, indicating the hole’s edge hasn’t been touched. However, the magnitude of the last position change,
$\left \| {R_t^p - R_{t - 1}^p} \right \|$
, reflects the intensity of the downward exploration. Therefore,
$\left \| {R_t^p - R_{t - 1}^p} \right \|$
is used as a small increment to adjust the limit.
\begin{equation} z_t^c = z_t^p + {w_2} \cdot \left \| {R_t^p - R_{t - 1}^p} \right \| \end{equation}
3.3. Admittance controller
To satisfy the motion control demands of the contact-rich task, it is essential to restrict the generated action. Our goal is for the assembly process to be as compliant as human assembly. Impedance control is a versatile approach that prevents damage to pieces by managing overabundant contact forces during the insertion process. The insertion task involves translating and rotating pieces in Cartesian space, with rotation expressed by Euler angles.
Impedance control within Cartesian manipulation space can be simulated using a Mass-Spring-Damper system, outlined as follows:
\begin{equation} M\ddot x + B\dot x + Kx = {F_d} - {F_{ext}} \end{equation}
where
$F_d$
is the expected force,
$F_{ext}$
is the contact force detected by the force sensor, and
$M$
,
$B$
, and
$K$
are the diagonal matrix of inertia coefficient, damping coefficient and stiffness coefficient respectively. When the expected position is reached, the expected force
$ F$
is set to zero. That is to say, when the force achieves the expected value, the action halts to prevent surpassing the limit, confirming compliance and security during the insertion process.
3.4. Transfer to real scene
For robust skill transfer and effective use of the well-trained robot agent, real environment images must be converted into the suggested canonical space through image segmentation.
This paper selected U-Net [Reference Ronneberger, Fischer and Brox25] for the semantic segmentation of a real peg-in-hole environment. The U-Net model provides accurate and rapid image segmentation. Figure 4 illustrates the network architecture of the U-Net model, which includes max-pooling layers, up-sampling layers, and intricate cropping and concatenation layers. In the field of medical image segmentation, U-Net is widely used due to its ability to deliver reliable and accurate results with relatively few training samples. Nevertheless, training a U-Net requires a good deal of images with segmentation masks, which may require considerable time for labeling.
Train image segmentation model for real scene image by first using U-nets trained on synthetic RGB data and ground truth masks from the simulation environment. Then, fine-tune the networks with real data.
This article employs a training approach for segmentation models that involves initially training in a simulated environment and then fine-tuning in a real-world setting. This method is designed to reduce the dependency on real data, leveraging simulated data to build a robust model before adapting it to real-world conditions. This article utilizes U-Net for the actual segmentation of real visual observations. The RL agent, trained in a simulated environment, is transferred directly to execute insertion tasks in real-world scenarios without additional training, as shown in Figure 5.
The original real image is transformed into segmented image by image segmentation, and the real robot is controlled by the transfer RL strategy.
4. Experiment
The experimental section of this article is designed to rigorously evaluate the proposed methodologies in the context of satellite component peg-in-hole assembly. First, we developed a simulated environment to mimic the assembly process, allowing for controlled and repeatable experimentation. Next, we conducted comparative experiments to assess the performance of the demonstration-accelerated RL strategy and the SIL strategy introduced in this study. Finally, to validate the effectiveness of the proposed simulation-to-reality (sim-to-real) transfer strategy, we conducted experiments on a real satellite component peg-in-hole assembly platform, ensuring that our methods are applicable in real-world scenarios.
4.1. Experiment setting
The following section details the experimental setup. The simulation environment was constructed using NVIDIA’s Isaac Sim framework, which incorporates the UR5 robotic arm, a Robotiq two-finger gripper, a simulated force sensor, a satellite component assembly platform, and a camera mounted on the assembly platform and provided a forward view of the insertion task. Isaac Sim, as one of the most advanced simulators for robotic learning, enhances the accuracy of image rendering and collision force simulation. In this setup, the peg is pregripped by the gripper positioned above the hole, as the focus of this study is solely on the learning of the insertion skill.
The experiments were trained and evaluated on a computer equipped with an NVIDIA RTX 2080 Ti GPU, an Intel Xeon W-2145 CPU, and 64 GB RAM.
4.2. RL-based insertion skill learning
First, we use the demonstration data to pretrain the agent for peg-in-hole tasks. Following the demonstration, we obtained a sum of 32 episodes of records.
After pretraining, the RL-based peg-in-hole skill training begins. An episode is terminated if the insertion depth
$ z$
surpasses a predefined threshold, signaling the successful completion of the insertion process. For safety reasons, the episode is also immediately terminated if excessive force is detected or if the assembly becomes lodged in a narrow gap. These measures are crucial for preventing component damage and maintaining the integrity of the insertion process.
Both pretrained and nonpretrained agents were then subjected to the same RL training process. The training was conducted over multiple episodes, with each episode consisting of a series of steps in which the agent attempts to align and insert the peg into the hole. The primary metric for evaluating the performance of the agents was the cumulative reward, which represents the total reward accumulated over each episode. Higher cumulative rewards indicate better performance in terms of task completion and efficiency.
We employ the Dueling DQN framework in our RL model to enhance learning efficiency and decision-making quality. The dueling DQN architecture decomposes the Q-value estimation into two separate streams: the state-value function and the advantage function. This structure allows the model to better distinguish between crucial and less significant actions, thereby improving policy stability and sample efficiency. Compared to other RL algorithms such as standard DQN and Double DQN, the dueling DQN framework effectively enhances the learning dynamics by prioritizing important state-action evaluations, thereby reducing unnecessary value updates for non-critical actions. This property is particularly valuable in assembly tasks, where precise motion control and decision-making under uncertainty are essential.
The cumulative reward curves for both pretrained and nonpretrained agents are presented in Figure 6. These curves illustrate the learning efficiency and effectiveness of the agents over the course of the training episodes. Figure 6(a) shows the cumulative reward curve for the pretrained agents. As observed, the pretrained agents exhibit a positive initial reward and a significant increase in cumulative rewards after 50 episodes. This indicates that the pretraining phase effectively accelerated the learning process, allowing the agents to quickly adapt to the task and improve their performance. Figure 6 (b) displays the cumulative reward curve for the nonpretrained agents. In contrast to the pretrained agents, the nonpretrained agents start with a negative initial reward and only show a notable increase in cumulative rewards after 100 episodes. This suggests a slower learning process and greater difficulty in task adaptation without the benefit of pretraining. The comparative analysis of these curves clearly demonstrates the advantages of incorporating a pretraining phase. The pretrained RL agents achieved faster convergence and higher overall performance in the pin-in-hole assembly task, showcasing the efficacy of our proposed approach.
Results curve across RL training phases: (a) with pretraining and (b) without pretraining.
4.3. Simulation experiment
This section aims to systematically evaluate the proposed method by designing three types of sim-to-real discrepancies to assess the effectiveness of the sim-to-real algorithm, as illustrated in the accompanying Figure 7.
Evaluations in the simulation. Case 1: assess the RL policy with camera position change. The camera position offset ranges from −0.15 cm to 0.15 cm in the X, Y, and Z axis. Case 2: assess the RL policy on different light intensity. The light intensity ranges from 1 to 30,000, from dark to bright. Case 3: assess on different light colors. RL policy is tested with 8 colors: color 0 is black (RGB: 000), color 1 is blue (RGB: 001), and so forth.
Specifically, one type of discrepancy is the camera position offset, which arises from errors in the camera’s positioning. We set the camera position offset in the x, y, and z directions within a range of −0.15 cm to 0.15 cm.
The second type of discrepancy involves the illumination of the simulated environment. Illumination levels can be adjusted within the Isaac Sim simulator, with a range varying from 1 to 30,000.
The third type of discrepancy concerns the texture of the interacting objects. We introduce variations by altering the lighting color of the assembly components. Eight different colors were selected from the RGB color space for this purpose.
For comparison, we selected two baseline methods: a pure RL method and a domain randomization RL method. In the pure RL method, there is no offset in the camera position, the original RGB image is used as input, the light intensity is set to 30,000, and the assembly scene light color remains white color (RGB:111). In contrast, the domain randomization RL method randomly sets the camera position offset within −0.05 to 0.05, light intensity within 3000 to 30,000, and object color randomly chosen from the eight selected colors.
Next, the results of the simulation experiment are given. Firstly, the effect of camera offset on algorithm performance is evaluated. As shown in Figure 8, all the compared RL algorithms demonstrated a maximum success rate close to 0.95 when subjected to camera offsets in the x, y, and z directions. The DR-DQN algorithm outperformed both Vanilla DQN and Seg-DQN, as it trained across a broader range of visual perspectives through camera position randomization. Under conditions where camera errors are relatively small, our method, Seg-DQN, achieved experimental results nearly comparable to those of DR-DQN, despite being trained without camera error randomization. However, in the y and z directions, the performance of Seg-DQN significantly deteriorates when the camera offset error exceeds 0.1 cm.
Assembly success rate with varying camera position offsets.
Secondly, the effect of illumination variation on algorithm performance is evaluated. As shown in Figure 9, our Seg-DQN algorithm outperforms the compared methods significantly under low-light conditions. The DR-DQN method only achieves comparable performance to our approach when the light intensity exceeds 20,000. When the light intensity falls below 5,000, the success rate of DR-DQN drops to around 0.8. Vanilla DQN exhibits poor robustness under varying lighting conditions, performing well only when the light intensity is sufficient.
Assembly success rate with varying light intensity.
Finally, we evaluate the effect of light color change on algorithm performance. As illustrated in Figure 10, we conducted a performance evaluation of the algorithms under different lighting color conditions. Our Seg-DQN algorithm demonstrated the best performance, primarily because the influence of lighting color is minimized in the segmented image space. In contrast, the DR-DQN and Vanilla DQN algorithms exhibited significant variations in performance under different lighting conditions. Specifically, both Vanilla DQN and DR-DQN performed the worst under the color 6 condition. On the other hand, Vanilla DQN showed the best performance under color 0 and color 7 conditions.
Assembly success rate with different light colors. Sim-to-real performance is tested across eight light types.
4.4. SIL
In this subsection, we designed ablation experiments to evaluate the dynamic SIL method utilized in this article. As introduced in the second subsection of the Methods section, the SIL method differs from the traditional sliding method. The SIL method adjusts the force and motion constraints dynamically to ensure safe and precise insertion, whereas the traditional sliding method relies on consistent sliding movements to achieve insertion.
This experiment compares the effects of different action methods (SIL vs. sliding). The objective is to quantitatively assess the performance of each method in terms of reward and success rates during the peg-in-hole assembly task.
The experimental results show that the SIL method achieved higher rewards compared to the sliding method, as illustrated in the accompanying Figure 11. In terms of success rates, the sliding method performed poorly, while the SIL method showed significant improvement.
Performance comparison of training methods: SIL vs. Sliding.
Experimental setup for robotic peg-in-hole assembly in the real world, including the robotic arm, gripper, camera, and satellite components.
4.5. Transfer to real scene
In this section, we validate the performance of the proposed sim-to-real algorithm in a real-world scenario. We constructed a real satellite component peg-in-hole assembly setup, as shown in the accompanying Figure 12. The objective is to evaluate how well the DRL insertion strategy, which was pretrained in a simulated environment, can adapt and perform in an actual assembly task. The real-world experimental setup consists of several key components, including a UR5 robotic arm, a Robotiq 2F-85 gripper, an Intel RealSense D415 camera, a Force Torque Sensor FT 300-S, and a custom-built satellite assembly platform. The UR5 robotic arm provides flexibility and precision in performing the insertion tasks, while the Robotiq 2F-85 gripper handles the satellite components with accuracy. The Intel RealSense D415 camera offers high-resolution visual feedback essential for precise component alignment. The Force Torque Sensor FT 300-S measures the force and torque during the assembly process, supplying critical feedback for the DRL algorithm. The satellite assembly platform is specifically designed to hold and align the components for the insertion task. By integrating these components, we recreated a realistic and challenging environment for testing the DRL-based insertion strategy. The software architecture of this robotic shaft-hole assembly system uses ROS to aggregate sensor data, which is transmitted via TCP/IP communication to the environment observation module and impedance controller for real-time control of the assembly process. The impedance controller generates control signals based on observed information, guiding the action module to perform assembly tasks.
Real image and corresponding segmented image.
This article proposes a sim-to-real transfer method that utilizes image segmentation to convert visual observations into a canonical space. Figure 13 illustrates real-world images alongside their corresponding segmented results. The U-Net model was trained using a dataset comprising 5000 synthetic RGB image and segmentation mask pairs collected in a simulation environment, along with 50 real-world images used for fine-tuning. The dataset was split into training and testing sets in an 8:2 ratio. Training was conducted on an NVIDIA GeForce RTX 2080ti GPU (11 GB memory) with a batch size of 4 over 300 epochs, using a learning rate of 1e-2, a momentum of 0.9, and the Adam optimizer. The Dice loss function with a deep supervision strategy was utilized to enhance network performance. To improve generalization, data augmentation techniques, including random rotation, horizontal and vertical flipping, and scaling, were applied to both images and labels, ensuring robustness to variations in scale, orientation, and other transformations. The reflective surface of the satellite platform in the real-world images added an extra layer of complexity to the task, presenting a significant challenge for the image segmentation process.
We analyzed the success rate of using segmented images for transfer learning and compared it with transfer learning through domain randomization (DR-DQN) and training the RL algorithm solely on raw images (Vanilla DQN) in the peg-in-hole assembly task. To provide a more comprehensive evaluation, two additional baselines were included: soft actor critic (SAC) [Reference Haarnoja, Zhou, Abbeel and Levine8] and CycleGAN-based domain randomization [Reference Shi, Yuan, Tsitos, Cong, Hadjar, Chen and Zhang26]. SAC was selected as a representative of pure RL approaches due to its stability and efficiency in continuous control tasks. CycleGAN was employed to simulate domain randomization by translating simulated images into the real-world domain, serving as a representative of domain-adaptation-based RL. These additions allow for a more thorough comparison between our proposed method and both conventional RL and domain adaptation techniques. The evaluation of the strategy in the real-world scenario was categorized into three outcomes: complete insertion, indicating the robot fully completed the insertion task; touching the hole, where the peg moved in the correct direction but did not achieve full insertion; and failure, where the robot either moved away from the target or performed unintended actions.
The comparative experimental results are presented in Table I. Out of a total of 20 evaluations, the proposed sim-to-real method achieved a higher success rate. In contrast, the method based solely on raw images failed to succeed in the real-world scenario. The domain randomization-based method successfully completed the assembly task in only 8 out of 20 experiments. This outcome is expected because the real-world visual domain differs significantly from the simulated visual domain. Real-world images contain various textures, shadows, and lighting conditions that are not present in the simulation. Over 20 independent trials, the CycleGAN-based method achieved 13 successful task completions, yielding a success rate comparable to that of our method. However, despite this competitive performance, the GAN-based approach suffers from inherent limitations. Specifically, the generator often exhibits instability during training, with slow convergence and significant sensitivity to hyperparameter tuning. Moreover, the training process incurs high computational costs, making it less practical for real-world deployment scenarios requiring frequent adaptation or limited computing resources. Notably, shadows appear when the robot arm approaches the workbench, creating additional challenges. Therefore, even when the real robot and objects are positioned identically to their simulated counterparts, the real images provide representations that differ from those in the simulation.
Without bridging the gap between simulated and real images, the RL strategy cannot be successfully transferred to the physical environment. We recorded the completion time for 15 successful executions of the assembly task. The average assembly time was 9.6 s. Given the nature of small batch, customized tasks such as shaft–hole assembly for satellite components, this execution time is considered reasonable and meets real-time requirements.
Comparison of experimental results (20 evaluations).
5. Conclusion
In the present work, we have developed a RL-based insertion policy learning method that effectively combines image feedback and exerted force to achieve efficient alignment of satellite components by robots. To achieve sim-to-real transfer, simulation and actual image observations are transformed into canonical space through an image segmentation model, U-Net. By integrating a sim-to-real learning framework for insertion tasks, utilizing segmented images from simulation cameras, and incorporating impedance control, we have demonstrated significant improvements in handling contact-rich tasks.
Despite the promising results, this study has several limitations that warrant further investigation. One notable shortcoming is the reliance on precise modeling for effective sim-to-real transfer. Achieving an accurate model of the tangible interaction between components remains challenging, and discrepancies between virtual and physical environment conditions can affect the capability of the trained agent. Therefore, methods based on simulation training, including ours, are not directly applicable to tasks where the simulation model of the peg and hole cannot be obtained. Additionally, the limitation of our approach is the decrease of texture details when converting original images to segmented images. Our approach is appropriate for robot manipulation tasks where texture information is not important to the task.
Future studies should concentrate on improving the robustness and adaptability of the proposed system. One potential direction is to explore more advanced domain adaptation techniques to further minimize the difference between virtual and physical environments. Moreover, enhancing the scalability of the system to handle a wider variety of assembly tasks and environments could significantly broaden its industrial applicability. Finally, incorporating more sophisticated compliance control strategies and real-time feedback mechanisms could enhance the system’s performance in dynamic and unpredictable conditions.
Author contributions
Ning Zhang conceived and designed the study. Ning Zhang and Yongjia Zhao conducted data gathering. Minghao Yang performed statistical analyses. Ning Zhang and Shuling Dai wrote the article.
Financial support
This work is funded by the National Key Research and Development Program of China (No. 2018AAA0102902).
Competing interests
The authors declare no conflicts of interest exist.
Ethical approval
Not applicable.
