1. Introduction
The rapid development of industry requires customized and adaptable manufacturing processes to quickly respond to customer demands [Reference Jia, She, Cheng, Bao, Chai and Xi1]. The robotic systems should be capable of handling various manufacturing tasks in stochastic, dynamic and changing environments.
In assembly or sorting warehouses, robots fetch components, put them into boxes or on conveyer belts according to sorting rules and deliver boxes to workstations. The new question is whether robots can adapt to new environment and take such multi-step tasks when the components and boxes are totally new or even when the sorting rules are changed. In this context, traditional control methods are unsatisfied to model complex unpredictable processes analytically [Reference Ma, Du, Zhou, Zhang, Liu and Xu2, Reference Rónai and Szabó3, Reference Laursen, Ellekilde and Schultz4]. Therefore, robots should not repetitively execute preprogrammed actions as usual, but learn how to plan and execute actions according to new situations and task requirements.
Deep reinforcement learning (DRL) offers the domain-independent solution, using function approximation and trial-and-error mechanism instead of accurate modeling to learn an effective controller [Reference Nicola, Tagliapietra, Tosello, Navarin, Ghidoni and Menegatti5, Reference Haarnoja, Pong, Zhou, Dalal, Abbeel and Levine6]. There are several common issues to solve, such as defining states and actions, specifying rewards, and designing policy or value function networks. How could robots effectively perform complex multi-step sorting tasks without significant efforts of designing perception, reward and learning systems?
To solve this problem, we investigate an end-to-end approach that learns a critic corresponding to a deterministic policy to do multi-step sorting tasks. The critic takes images as inputs and predicts actions at each individual image pixel. The strengths of proposed system are highlighted on two folds. First, we divide actions into different groups, assuming that some actions are preconditions of others in multi-step tasks. To predict different groups of actions, critic networks are designed based on images at different time steps, with an emphasis on state representation. Second, taking projected heightmap images as inputs without extra high-level image analysis and understanding, the critic model can produce a pixel-wise Q value map for each type of action. The action parameters can also be easily determined by projecting from the corresponding pixels. The main contribution is that this work is a new trial to apply pixel-wise Q value-based critic networks to solve multi-step sorting tasks. We take several experiments to validate the effectiveness of the designed system. The source code of training and testing object sorting policies in simulation environments and real worlds are available at https://github.com/JiatongBao/DRLSorting.
The rest of this paper is organized as follows. Section 2 describes the related work about vision-based robotic manipulation. Section 3 formulates the sorting tasks by the reinforcement learning framework. Section 4 illustrates the method details. The experimental results are provided in Section 5. Finally, Section 6 gives conclusions and future work.
2. Related work
The vision-based robotic manipulation has made important progress with the development of DRL and computer vision. Taking the image representation of the environment as input, the DRL technology is used to control the robot to continuously learn how to complete a manipulation task from scratch through executing a bunch of primitive actions such as pushing, grasping and placing. In this section, we briefly review the related work in these fields.
Robot grasping [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7, Reference Kim, Li and Lee8, Reference Yang, Ni, Gao, Zhang and Tao9, Reference Añazco, Lopez, Park, Oh, Ryu, Al-antari and Kim10, Reference Kim, Park, Kim, Park and Suh11, Reference Chen and Lu12] is a particular research area of object manipulation. In the Visual Pushing and Grasping(VPG) [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7] method, a joint pushing and grasping policy is learned through model-free deep Q learning. The key idea is that pushing can help rearrange cluttered objects for better grasping. Two fully convolutional networks (FCNs) are trained to map from the visual observations to the pixel-wise Q values of pushing and grasping, respectively. Because the grasping posture in VPG is top-down and will not accommodate to picking up a wide variety of objects
$(e.g., \textrm{plates})$
, Tang et al. [Reference Tang, Corsaro, Konidaris, Nikolaidis and Tellex13] optimize the method based on the VPG by learning a 6-degree-of-freedom grasping policy with the help of a planar push, in order to enable grasping the plate from the side. Joshi et al. [Reference Joshi, Kumra and Sahin14] use Grasp-Q-Network to detect robotic grasp based on a double deep Q-learning framework. The Grasp-Q-Network incorporates visual input from multiple camera and motor states to learn a stable grasping strategy. Chen et al. [Reference Chen, Ju and Yang15] propose to apply reinforcement learning algorithms to learning pushing and infer grasping by image morphological processing.
Learning multi-step tasks such as stacking [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16, Reference Yang, Ji, Wu, Lai, Wei, Liu and Setchi17, Reference Liu, Lu, Cong, Yu, Zou and Du18], assembling [Reference Schoettler, Nair, Luo, Bahl, Ojea, Solowjow and Levine19, Reference Li, Jiang, Zhang, Wei and Song20, Reference Luo, Solowjow, Wen, Ojea, Agogino, Tamar and Abbeel21, Reference Beltran-Hernandez, Petit, Ramirez-Alpizar and Harada22, Reference Wang, Zhu, Zhang, Zhou, Dou, Sun, Yao, Xu and Zhang23] and sorting [Reference Nicola, Tagliapietra, Tosello, Navarin, Ghidoni and Menegatti5, Reference Ceola, Tosello, Tagliapietra, Nicola and Ghidoni24] are always more challenging because more primitive actions and action constraints are involved. In ref. [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16], another primitive action of placing is considered. The Q function is approximated using three parallel FCNs, for actions of pushing, grasping and placing, respectively. The multi-step tasks of stacking and making rows are investigated in the experiments. The main concern is the challenge of learning multi-step tasks with sparse rewards, because a solution is improbable through random exploration. Therefore, the Schedule for Positive Task(SPOT) [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16] framework is developed to explore actions within safety zones, identify unsafe regions even without exploring and prioritize its experience to only learn what is useful. Gerrit et al. [Reference Schoettler, Nair, Luo, Bahl, Ojea, Solowjow and Levine19] aim to enable the robot to complete a variety of industrial insertion tasks with visual inputs and different natural reward specifications. For vision-based learning, it uses the L1 distance between the current image and goal image as reward signal. It also learns a policy network using the residual reinforcement learning algorithm, whereas a human-engineered controller for predicting actions should be used. A general formulation of the object sorting problem is proposed in ref. [Reference Nicola, Tagliapietra, Tosello, Navarin, Ghidoni and Menegatti5]. The environment is discretized into a 3D grid of cells. The state of each cell is represented by the semantic properties of the object within it. The state space defined by the states of all cells is fully observed.
It is found that, in the tasks of stacking, making rows or assembling, the pixel-wise Q value-based methods only focus on the postures of objects without consideration of the object attributes like color, shape, etc. Therefore, in the design of network architecture, [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7] and [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16] only take input as the image representation of the environment at the current time step, which is sufficient for the networks to predict actions. However, these networks could fail to predict right actions in the tasks within more action constraints. For example, in a task of sorting by color, a colored object should be grasped and then placed into a box with the same color. However, it is impossible to figure out where to place without information of the grasped object, which cannot always been observed by cameras. The work [Reference Nicola, Tagliapietra, Tosello, Navarin, Ghidoni and Menegatti5] does not directly take images as inputs and needs a precise semantic understanding of the objects and containers. Moreover, the target configuration should be manually defined and incorporated in the representation of the state. In this work, we investigate more challenging object sorting tasks using FCNs to approximate pixel-wise action value functions. The action constraints include both object postures and other object attributes.
3. Problem formulation
We investigate the vision-based multi-step object sorting tasks through DRL [Reference Mnih, Kavukcuoglu and etal25]. A sequence of pushing, grasping and placing operations could be performed in an object sorting task. The task is formulated as a Markov Decision Process
$\lt S,A,P,R,\gamma \gt$
, with state space
$S$
, action space
$A$
, state transition probability matrix
$P\;:\;S\times S\rightarrow \mathbb{R}$
, reward function
$R\;:\;S\times A\rightarrow \mathbb{R}$
, and discount factor
$\gamma \in [0,1]$
. At each time step
$t$
, the robot receives some representations of the environment’s state
$s_t \in S$
, and selects an action
$a_t \in A$
according to a policy
$\pi\;:\;S \rightarrow A$
. One time step later, in part as a consequence of its action, the robot receives a numerical reward
$R_{t+1}$
, and finds itself in a new state
$ s_{t+1}$
.
Formally, a policy is a mapping from states to probabilities of selecting each possible action. The goal is to find an optimal policy
$\pi ^{*}$
in order to maximize the action value function over all policies,
${\pi ^ * }(s) = \arg \mathop{\max }\limits _\pi{Q^\pi }(s,a)$
, where
${Q^\pi }(s,a) ={E_\pi }[{G_t}|{s_t} = s,{a_t} = a]$
is the action value function and defined as the expected return starting from state
$s$
, taking action
$a$
, and then following policy
$\pi$
. The return is defined by
$G_{t} = \sum \limits _{k = 0}^{T - 1}{{\gamma ^k}{R_{t + k + 1}}}$
, that is,
$ \gamma$
-discounted sum on all future rewards from time
$t$
to
$T$
, across planning horizon
$T$
. In the object sorting tasks,
$T$
is the termination time of a task. The object sorting tasks are episodic tasks where each task ends in a terminal state followed by a reset to a new starting episode.
In this work, an off-policy Q-learning method [Reference Hellaby26] is used to train a greedy deterministic policy
$\pi(s_t)$
that chooses actions by maximizing the Q function
$Q^{\pi}(s_t, a_t)$
.
$Q^{\pi}(s_t, a_t)$
measures the expected return of taking action
$a_t$
in state
$s_t$
and then following policy
$\pi$
. It performs as a critic and estimates the value of performing a given action in a given state. In DRL, the Q function is implemented as a deep neural network and trained by iteratively minimizing the error between the temporal difference (TD) and the reward:
\begin{equation} {Q^\pi}({s_t},{a_t}) - {\unicode[Times]{x03B3}}{Q^\pi}({s_{t + 1}},\pi({s_{t + 1}})) \leftrightarrow{R_{t + 1}} \end{equation}
where
$\pi ({s_{t + 1}}) = \arg \mathop{\max }\limits _a{Q^\pi }(s_{t+1},a)$
. From the supervised learning perspective,
$R_{t + 1} + \gamma{Q^\pi }({s_{t + 1}},\pi ({s_{t + 1}}))$
can be regarded as the label for the sample
$(s_t, a_t)$
fed into the Q network. Then, it becomes a regression problem, where the labeled samples are collected through the evolving robot-environment interactions while the model is updated accordingly.
4. Approach
4.1. State descriptions
In the multi-step object sorting tasks, the robot observes each state
$s_t$
by using an RGB-D camera. As in VPG [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7], the camera captures the scene’s colored point cloud which is projected on to the tabletop surface perpendicular to the gravity direction. Accordingly, the heightmap image with RGB and height-from-bottom channels is obtained. The edges of the heightmap are assumed to be the boundaries of the robot’s workspace. In the experiments, the workspace is defined as a square with an area of
$0.448\times 0.448 \,{\rm m}^2$
and a pixel resolution of 224
$\times$
224. Each pixel represents roughly an area of
$4 \,{\rm mm}^2$
.
However, the heightmap at time
$t$
is insufficient for predicting the placing action
$a_t$
in the sorting tasks, because the object held by the robot’s gripper cannot always be observed by the camera. Therefore, in this work, two heightmaps are captured before and after executing the grasping action and merged into one heightmap with 8 channels for representing the state
$s_t$
.
4.2. Primitive actions
Three types of primitive actions including
$\{push, grasp, place\}$
are defined. Each type of action has two parameters, that is, position
$p$
and orientation
$\theta$
. In our scenarios, all actions employ a top-down manipulation posture where the roll and pitch angles of the gripper are fixed. Therefore,
$\theta$
actually denotes the yaw angle.
Specifically,
$push(p,\theta )$
denotes a push that starts with the tip of a closed two-finger gripper at the position
$p$
and moves a fixed distance in the direction
$\theta$
.
$grasp(p,\theta )$
denotes a top-down antipodal grasp where
$p$
and
$\theta$
are the target position and orientation of the tool center point(TCP), respectively. In both actions,
$\theta$
is defined as
$\theta \in \{\frac{2 \pi i}{16}|i\in [0,15]\}$
.
$place(p,\theta )$
denotes a top-down placement in the orientation
$\theta$
where the gripper will be open once it reaches the target position
$p$
. The placing action will be fixed in one orientation in our case. In addition, the 3D position
$p$
and 1D orientation
$\theta$
are projected from a corresponding pixel of the heightmap image.
4.3. Learning the Q functions
In object sorting tasks, a successful grasp is the precondition of a placement attempt. Therefore, all possible pushing and grasping actions are grouped as
$A_{pg}$
while the placing actions are grouped as
$A_{place}$
.
As shown in Fig. 1, for determining the policy
$\pi (s_t) \in A_{pg}$
, we approximate the Q function using two feed-forward fully convolutional networks (FCNs) to extend vanilla deep Q networks (DQN) as in VPG [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7]. Another FCN is learned as the Q function for determining the policy
$\pi (s_t) \in A_{place}$
. The three FCNs are named Pushing Network
$\varPhi _{push}$
, Grasping Network
$\varPhi _{grasp}$
and Placing Network
$\varPhi _{place}$
for each type of primitive action respectively. In addition, the whole model really follows the critic framework of DRL, although the Q function is implemented as multiple branches for the three types of actions. However, in order to learn the synergies between different types of actions, these Q networks should share a deep network or branch. As in the works [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7, Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16], this constraint could be released according to the learned tasks. For the sorting tasks investigated in this work, the critic model is simplified in that three Q networks are learned individually.
The network architecture for learning object sorting tasks.
All FCNs share almost the same network architecture. Each FCN consists of a 121-layer DenseNet [Reference Huang, Liu and Weinberger27] pre-trained on ImageNet [Reference Deng, Dong, Socher, Li, Li and Fei-Fei28], followed by channel-wise concatenation and 2 additional
$1 \times 1$
convolutional layers interleaved with nonlinear activation functions
$\left (\right .$
ReLU
$\left .\right )$
[Reference Nair and Hinton29] and spatial batch normalization [Reference Ioffe and Szegedy30], then bilinearly upsampled. Both
$\varPhi _{push}$
and
$\varPhi _{grasp}$
take as input the 4-channel heightmap image representation of the state
$s_t$
. Each individual network outputs a dense pixel-wise map of Q values with the same image size and resolution as the input, where each individual Q value at a pixel predicts the action value of executing a corresponding primitive action at the projected 3D position
$p$
and 1D orientation
$\theta$
from that pixel. To predict primitive pushing or grasping actions parameterized with different orientations
$\theta \in \{\frac{2 \pi i}{16}|i\in \left [0,15\right ]\}$
, the input heightmap is rotated 16 times; thus, the total outputs are 16 pixel-wise maps of Q values. Differently,
$\varPhi _{place}$
takes as the input the 8-channel heightmap as introduced in Section 4.1. In our case, when an object is successfully grasped, the robot will place the object to the target position in a straight line with no consideration of different orientations. Therefore, there are no rotations of the input heightmap for
$\varPhi _{place}$
and its total output is only 1 pixel-wise map of Q values.
In the evolving interactions between robot and environment, once the reward
$R_{t+1}$
is obtained as a result of taking a primitive action
$a_t$
in the state
$s_t$
, the corresponding network is continuously trained until approximating the optimal action value function by stochastic gradient descent with momentum. The Huber loss function is used to calculate the network loss. Meanwhile, the network is continuously trained with prioritized experience replay [Reference Schaul, Quan, Antonoglou and Silver31] using stochastic rank-based prioritization. In addition, when the robot starts to sort one object, it should first decide to push objects for better grasping or directly grasp an object. The
$\epsilon$
-greedy strategy is applied. With
$1-\epsilon$
probability, the action type
$a_{type}\in \{push,grasp\}$
is determined by comparing the confidences of the optimal push and the optimal grasp. Otherwise,
$a_{type}$
will be assigned with
$push$
or
$grasp$
with a prior probability
$p_{push}$
and
$1-p_{push}$
, respectively. When an object is successfully grasped, the action type
$a_{type}$
should be
$place$
without other choices in our case. For each type of actions, the
$\epsilon$
-greedy strategy is also used to either take actions greedily based on the estimated Q function with
$1-\epsilon$
probability or explore other actions randomly otherwise. For pushing and grasping actions, the common sense constraints [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16] that areas with heights should be the ones to be explored are incorporated in the learning framework to accelerate the learning efficiency. In our experiments,
$\epsilon$
is initialized at 0.5 and then annealed over training to 0.01.
$p_{push}$
is constant at 0.2 while the discount factor
$\gamma$
is constant at 0.65.
4.4. Reward function
We assign reward
$R_{t+1}$
with a baseline value
$B \in \{B_{push},B_{grasp},B_{place}\}$
for each type of primitive action if the robot successfully executes the action
$a_t$
in the state
$s_t$
and the state changes to
$s_{t+1}$
. Each baseline value could be defined according to the difficulty and importance of the specific type of action. In this work, we assign
$B_{push}=0.1$
for pushes that make changes of object postures,
$B_{grasp}=1$
for a successful grasp if an object is grabbed and raised from the tabletop, and
$B_{place}=1.5$
for a successful placement if the grasped object is placed onto the desired area. Based on these baseline values, the reward function is defined as
\begin{equation} R_{t+1} = {\unicode[Times]{x1D7D9}}(s_t,a_t,s_{t+1}) B \end{equation}
where
${\unicode[Times]{x1D7D9}}(s_t,a_t,s_{t+1})$
is an indicator function which equals to 1 if
$a_t$
leads to a successful manipulation and 0 otherwise, and
$B$
is determined according to the action type of
$a_t$
. Considering a special case that the robot would grasp the objects which have been correctly sorted, it will receive a negative reward
$-1$
to punish those actions, physically reset the environment and restart a new training episode.
5. Experiments
In the experiments, we focus on the multi-step tasks of sorting by color. There are a bunch of blocks with different colors on the tabletop. Each block should be placed onto the target area with the same color. The difficulty of this kind of tasks is that the robot should not only notice the grasped block and the current scene but also satisfy the postcondition of a placement attempt, that is, the target area must have the same color as the grasped block.
We take several experiments to investigate (1) how well the proposed method performs on multi-step object sorting tasks (i.e., sort blocks by color), (2) how the robot deals with some distracting objects in the testing scenes, (3) how the robot performs in the testing scenes in totally different configurations of objects and target areas from the training scenes, and (4) how the robot performs in the real-world environments by directly transferring the learned model from simulation to the real world.
5.1. Setup
Environment.
We construct a simulated environment shown in Fig. 2(a) in a real-time physics simulator called CoppeliaSim [Reference Rohmer, Singh and Freese32], where the agent is a UR5 robot arm with an RG2 gripper. Bullet Physics 2.83 is used to simulate the dynamics, and CoppeliaSim’s internal inverse kinematics module is used for robot motion planning. On the corners of the robot workspace, 4 square plates each of which has an area of
$0.15\times 0.15\;\textrm{m}^2$
are placed for holding the objects to be sorted.
Simulation environment and training scenes.
Training.
In the training process, we design three types of sorting-by-color tasks as shown in Fig. 2(b). In the first type of task, there are 4 cuboid blocks with different colors to be sorted. There are also 4 cuboid blocks in the second type of task but with random colors. In the third type of task, there are 6 cuboid blocks with random colors to be sorted. In order to simplify the task, all three tasks share an object color space
$C=\left \{Red,Blue,Green,Yellow\right \}$
. In addition, 4 plates for holding the blocks are fixed with different colors.
Three models named
$Model1$
,
$Model2$
,
$Model3$
are trained for the above three types of sorting-by-color tasks, respectively. All training processes work in a self-supervision manner. The blocks are randomly dropped at the beginning of each episode. The robot then performs task learning by trial and error automatically and continuously. The environment is reset at the termination of an episode. We use PyTorch to implement and train the models on a PC with Intel Core CPU i9-10900KF, NVIDIA GTX 3090 GPU and 32G memory running Ubuntu 16.04.
Testing.
We first design three groups of testing scenes having 4, 6 and 8 cuboid blocks, respectively. The object color space is almost the same as
$C$
used in the training scenes unless incorporating some extra colors. As shown in Fig. 3, each group contains 4 testing scenes where the blocks are scattered, stacked, tightly arranged or with extra colors. Each plate is randomly assigned with a specific color sampled from
$C$
without replacement.
The first three groups of testing scenes.
As shown in Fig. 4, we then design the second three groups of testing scenes the same as the above but the color space of objects and plates is totally new to the robot.
The second three groups of testing scenes.
5.2. Evaluation metrics
We evaluate performance of models in training and testing processes with the following metrics.
-
(1) Action efficiency: this metric is used to evaluate the efficiency of performing sorting operations and defined as
$\frac{{2\times{\rm{\#sorted \,objects}}}}{{\#{\rm{actual\, actions}}}}$
, where the ideal action number of sorting an object is defined as 2 (i.e., grasping an object and then placing it correctly). -
(2) Action success rate: it measures the ratio of the successful action count to the total number of attempts for a specific action.
-
(3) Task completion rate: it calculates the percentage of the objects that have been sorted among all objects at the completion of an episode.
5.3. Results of training
Three models have been trained on the three types of scene setups, respectively. Figure 5 shows the training results. The action efficiency and the success rates of grasping and placing are counted over all actions in the training process with a sliding window size of 500. The task completion rate is calculated and averaged for every 50 episodes. As shown in Fig. 5(a),
$Model1$
has the fastest convergence speed and the highest action efficiency, since there are only 4 objects in the scene and their colors are fixed and different from each other.
$Model2$
and
$Model3$
are trained on the scenes where the object colors are randomly selected from the color space
$C$
and only the object numbers are slightly different. Therefore, their convergence speeds, as well as action efficiencies, are relatively close. The three curves also reveal that it will require more training time in environments with more complex object configurations. When training in the environment with 6 randomly colored objects, the training process almost converges after 30,000 numbers of interactions. It takes about 16 h running on our PC shown in Section 5.1. The training processes are not problematic in different environment setups, and convergence states can be achieved through a number of robot-environment interactions.
Results of training.
The most important factor regarding action efficiency is whether the actions of grasping and placing are executed successfully or not. It can be seen from Fig. 5(b) that the grasp success rates of the three models are almost the same while the convergence speed is much faster. Because there are only a small number of regular blocks in our training scenes, learning how to grasp a cuboid block with common sense constraints seems to be easy for the robot. This also indicates that the grasp actions do not affect action efficiency extensively. Figure 5(c) shows that the success rate of placing is positively correlated to the action efficiency. It is verified that the action efficiency is mainly affected by the placing actions in our setups.
In the process of continuously interacting with the environment, the completion rates of the three models have been increasing by episodes as shown in Fig. 5(d). When the models converge, there are no distinguishing difference among all models in terms of task completion rate. All task completion rates of the models are greater than 95%.
5.4. Results of testing
For each testing scene, we execute 30 episodes. All evaluation metrics are calculated on each episode and then averaged.
We first test the performance of three trained models on the first group of testing scenes as shown in Fig. 3(a). The testing scenes are relatively simple. There are only 4 blocks to be sorted. Evaluation results are summarized in Table I. It is found that the task completion rates of the three models are competitive.
$Model3$
performs the best in terms of action efficiency or success rate of placing, because the testing scenes are relatively simple compared to its training setups. In addition, the average grasp success rates of all three models are very close.
Performance on the first group of testing scenes (%).
Then, the performance of three models is evaluated on the second group of testing scenes as shown in Fig. 3(b). The testing scenes with 6 randomly colored objects are more complex than the first group scenes. It is revealed from Table II that each individual model shows significant difference of performance from others. Since
$Model3$
is trained with 6 randomly colored blocks, it has highest average task completion rate of 97.2% with average action efficiency of 78%. Although
$Model2$
is trained with 4 randomly colored blocks, it still can obtain high task completion rate of 92.3% but with a relatively lower action efficiency of 62.2%.
$Model1$
trained with 4 fixed colored blocks has the worst performance in every metric, because more blocks make the testing scenes with higher complexity and the success rate of placing decreases too much. The average grasp success rates are stable, since a grasp action only cares about the position and orientation of one object.
Performance on the second group of testing scenes (%).
To check the models in more complex scenes, the tasks of sorting 8 objects are considered, as shown in Fig. 3(c). All three models have not been trained with 8 objects. Table III shows that
$Model3$
performs much better than others and obtains average action efficiency of 67.1% and average task completion rate of 91.0%. Again, the grasp success rates are not affected much by the scene configurations. The decreased success rate of placing makes low action efficiency.
Performance on the third group of testing scenes (%).
Tables IV, V and VI show the evaluation results of the testing scenes in Fig. 4(a), (b) and (c), respectively. These scenes are totally new to the models. Considering the results on the three groups of testing scenes,
$Model1$
has lowest action efficiency of 14.7% and lowest task completion rate of 43.3%, which reveals that it is hard for
$Model1$
to handle completely unseen scenes.
$Model2$
performs better than
$Model1$
and obtains the acceptable task completion rate greater than 68% with the action efficiency greater than 36%.
$Model3$
performs best, and its task completion rate always stays above 84%. When dealing with no more than 6 objects, the action efficiency can still be greater than 55%. It is validated that the proposed method generalizes a bit well to similar tasks with novel scenes. However, as the complexities of testing scenes increase, the action efficiencies decrease dramatically, which means that the models need to take more time to complete the tasks. Therefore, it is still challenging for the current trained models to deal with totally new scenes with more objects.
Performance on the fourth group of testing scenes (Mean %).
Performance on the fifth group of testing scenes (Mean %).
Performance on the sixth group of testing scenes (Mean %).
5.5. Real-world experiments
In this part, the performance of
$Model3$
is evaluated on a real robot. As shown in Fig. 6(a), the robot setup includes a Universal Robot UR3 with a Robotiq 2-finger gripper and an Intel RealSense D415 camera. Considering the limited working area of UR3 robot, we set the size of workspace as
$0.28\times 0.28 \,{\rm m}^2$
. On the corners of the robot workspace, 4 square plates each of which has an area of
$0.09\times 0.09 \,{\rm m}^2$
are placed for holding the objects to be sorted. Therefore, the experimental environment in our real scene is a bit different from the simulation. We also design two testing scenes shown in Fig. 6(b). They are shown as two colored heightmap images generated from our system. It should be noticed that the dark black spots on the colored heightmap images are some noises where the depth values captured by the RGB-D camera are invalid and mainly object shadow areas that cannot be observed by the camera. The two testing scenes contain 4 and 6 blocks to be sorted, respectively. After
$Model3$
is trained in the simulation environment, we directly load the model for execution on the real robot, which means that the learned model is transferred from simulation to real world without any extra training in real environment. We run the model on each testing scene for 20 episodes. The evaluation results are shown in Table VII.
Performance on the real-world testing scenes (%).
Real-world robot setup and testing scenes.
It is shown that it can obtain satisfactory success rate of grasping when directly transferring the model from simulation to real world. When dealing with 6 blocks, it can still achieve a success rate of more than 70%. As validated by the works [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7, Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16], the grasp success rate can be improved through training in real environments. On the premise of successful grasping, it is surprised that the success rate of placing is very high with no fine tuning in real-world scenes. When sorting 6 objects, the model has placing success rate of more than 90%. The average action efficiency is around 80%. It should be noticed that, in real-world experiments, when the model plans to grasp an object that has been correctly sorted, the manipulation will not be executed and the episode will not be reset. This is a little different from the setup in the simulations. Thus, we could obtain satisfactory task completion rate of 100%.
5.6. Ablation studies
Freeze of DenseNet. By now, the experimental results are based on the learned model where the 121-layer DenseNet (DenseNet121) is used as the trunk network trained together with the policy. We take comparative experiments on training the network with and without freeze of the weights of DenseNet. In the comparative experiments, we first focus on the task where there are 6 cuboid blocks with random colors to be sorted. The results are shown in Fig. 7.
Results of comparative experiments on training the networks with and without freeze of DenseNet.
It is shown that the model performance decreases dramatically in terms of action efficiency and task completion rate when the trunk network is frozen. Since the whole network becomes a shallow network, the success rate of placing action is much lower and cannot be improved when increasing numbers of robot-environment interactions. It indicates that the shallow neural network is incapable of modeling the placing action in the object sorting tasks, because it should automatically learn and take advantage of task relevant vision features and determine a placing action from a high-dimensional action space within satisfied action constraints (the blocks should be placed onto the plates with the same colors in our setups). However, as shown in Fig. 7(b), the shallow network seems to be powerful enough for learning how to grasp a small number of blocks with regular shapes in the experimental setup, even though the deep network performs better.
Then, the model for sorting 10 blocks with random colors and shapes is studied to reveal its potential capability. The experimental results are also shown in Fig. 7. In order to deal with more complex situations, the robot should perform more attempts of trial and errors. Again, the shallow network performs worse on predicting correct placing actions than the deep network. The deep network for grasping prediction learns more efficiently than the shallow network. The positive aspect of efficient learning of grasp is that the robot has more opportunities to explore and exploit the action space for learning placing actions within a specific time span.
Discount factor
$\gamma$
be zero. We also investigate the discount factor
$\gamma$
in Equation. The experiments are studied on the task of sorting 6 randomly colored cubic blocks with
$\gamma =0.65$
and
$\gamma =0$
comparatively. In the case of
$\gamma =0$
, the networks become FCNs supervised with regression (from trial and error) to infer pixel-wise affordance values according to immediate rewards. The experimental results are shown in Fig. 8. In this sorting task, the success rates of grasping under different discount factor settings evolve closely in the training process. However, the success rate of placing under
$\gamma =0$
is much worse than the one under
$\gamma =0.65$
. The action efficiency follows the same conclusion because it is related to the success rate of placing. Figure 8(d) indicates that the model under
$\gamma =0$
completes with less episodes than the model under
$\gamma =0.65$
after the same numbers of interactions, although the task completion rates averaged on every 50 episodes do not vary significantly. The Q-learning algorithm is expected to quickly update and accurately estimate the value functions which satisfy recursive relationships. As we know, Q learning is an one-step TD method.
$Q^\pi (s_t,a_t)$
can be estimated by
$r_t+Q^\pi (s_{t+1},a_{t+1})$
. If we set
$\gamma =0$
, the Q function would not be updated efficiently.
Results of comparative experiments on training the networks with
$\gamma =0.65$
and
$\gamma =0$
for the task of sorting 6 randomly colored cubic blocks.
5.7. Comparison with related works
This work is mainly inspired by VPG [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7] and SPOT [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16]. Ref. [Reference Nicola, Tagliapietra, Tosello, Navarin, Ghidoni and Menegatti5] is another similar work that focuses on learning the sorting tasks. These works are compared in terms of model characteristics, learned actions and task characteristics, as shown in Table VIII.
Comparison with the related works.
VPG [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7], SPOT [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16] and our work involve multiple types of actions like pushing, grasping and placing. All three works employ end-to-end models with the inputs of RGB-D images and the outputs of action values. In ref. [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7], a bunch of blocks with different colors and shapes are randomly scattered on the tabletop. The work is to study the efficient grasp of objects without replacement, which can benefit from pushing actions. As the real-world example, this model can be used for table clearing with no need of information of objects being grasped. Therefore, the task goal is simple and there are no other task requirements. The work [Reference Hundt, Killeen, Greene, Wu, Kwon, Paxton and Hager16] extends the work of ref. [Reference Zeng, Song, Welker, Lee, Rodriguez and Funkhouser7] and performs object grasping with replacement. The example tasks such as object stacking and making a row of objects are investigated. Although the objects should be manipulated to achieve the requirements of spatial arrangement, there is still no need to know what kind of object being grasped. However, in assembly or sorting warehouse situations, knowing what kind of object being grasped is a fundamental prerequisite. The challenge is that the grasped object cannot always be observed by the cameras. Therefore, we propose a new dual-input placing network that captures the information of the observable environment directly and gets the information of unobservable grasped object indirectly. The learned placing action not only needs to satisfy the requirement of spatial arrangement but also meets the object attribute constraint. The work [Reference Nicola, Tagliapietra, Tosello, Navarin, Ghidoni and Menegatti5] focuses on learning policy in semantic level using only one type of high-level action, that is, the
$move$
action. The
$move$
action should be executed successfully, while it does not consider whether the object can be grasped or not. The environment state is defined as a combination of semantic representations of object clutters (i.e., a 3D grid of cells of fixed size), clusters (i.e., object containers) and target object configurations. The model learns to plan a sequence of actions, in order to bring the environment from the starting state to the desired target in semantic level. It does not learn the sorting task in an end-to-end manner, since the images are not directly taken as inputs.
6. Conclusion
In this work, the method of learning multi-step object sorting tasks from scratch through DRL is presented. A complex sorting task can be decomposed into a sequence of three types of actions including pushing, grasping and placing. The action value functions of three types of actions are learned individually by end-to-end deep models. The preconditions, including whether an object is grasped and what kind of object it is, are considered by the visual inputs from different time steps. The dual-input deep neural network is proposed to learn the placing actions. The postconditions, such as whether the object is placed onto the target area with extra attribute constraint (e.g., color consensus), are considered by the reward function.
To the best of our knowledge, this work is the first to explore learning multi-step object sorting tasks by using pixel-wise Q value-based critic networks, where multiple preconditions and postconditions for a successful sorting should be satisfied. This work shares the merits of end-to-end models with no need of much efforts on processing and analyzing images in semantic level for different tasks. The learned three action models have the satisfactory capabilities in dealing with complex sorting tasks and have good applicability when deploying them into real-world robot systems. The main limitations include the lack of learning the essence of the sorting tasks and the trained models which do not generalize well to totally new scenes with more objects. Future work is to investigate these problems.
Acknowledgment
This research work is supported by the National Natural Science Foundation of China (Grant No. 61806175, 62073322).
Conflicts of interest
No competing financial interests exist.
Authors’ contributions
Jiatong Bao and Guoqing Zhang designed and implemented the task learning model, verified its effectiveness and wrote the paper. Yi Peng and Zhiyu Shao set up the simulated and real experimental environments. Aiguo Song guided the progress and reviewed the paper.
