1. Introduction
Nowadays, the application scenarios of robots are becoming increasingly dynamic, often with strict time constraints, making the motion planning task more challenging. In fact, while in highly structured environment the robot motion might be offline planned, this approach is deemed to fail when coping with unstructured and dynamic scenarios where the time factor is demanding.
In these contexts, the collision detection represents a crucial factor both for motion planning and grasping tasks. Thus, it is necessary to equip the robot with proper sensor technology to perceive the environment and detect collisions in short time in order to have the robot safely replanning its trajectory accordingly. Indeed, as per the literature, the time required to detect a collision could be a bottleneck for the motion planners, depending on the complexity of the task [Reference Pan and Manocha1].
For example, the authors in [Reference Golluccio, Di Lillo, Di Vito, Marino and Antonelli2] solve an objects relocation in cluttered environment problem, and the planner spends a non-negligible amount of time to find a free path, if it exists, using the classical geometrical approaches for the collision detection. Furthermore, as stated in [Reference Di Vito, Bergeron, Meger, Dudek and Antonelli3] and [Reference Kim and Park4], a generic motion planner could be not capable to adopt a replan strategy in short time if the environment changes during the motion of the robot.
Modern learning-based approaches have consistently demonstrated their superiority in terms of computational time over classical methods, which often heavily rely on model descriptions, whether partially or entirely [Reference Schmidt, Vahrenkamp, Wächter and Asfour5]. In fact, data-driven algorithms based on depth images are showing promising results in the fields of robotic grasping and state estimation through classification. In particular, grasping objects in cluttered scenes poses a significant challenge as it requires simultaneously considering the appropriate grasp strategy while avoiding collisions with the surrounding environment. It is very usual to decide a priori a list of possible grasping configurations considering the geometric and physical models of the objects [Reference Huebner, Welke, Przybylski, Vahrenkamp, Asfour, Kragic and Dillmann6, Reference Bohg, Morales, Asfour and Kragic7]; for instance, the authors in [Reference Lenz, Lee and Saxena8] demonstrate that, using neural networks on RGB-D images, it is possible to find the optimal grasp for an object on the Cornell Grasping Dataset. Similarly, in [Reference D’Avella, Sundaram, Friedl, Tripicchio and Roa9], the considered pipeline based on multimodal grasp planner takes as input the RGB and depth images and provides in output the best grasp depending on the arrangement of the objects into the scene. In the study conducted in [Reference Berscheid, Rühr and Kröger10], the authors address the challenge of training a robotic grasping model using depth images and gripper force feedback and propose an algorithm that effectively reduces the amount of data required for training the model. In [Reference Sundermeyer, Mousavian, Triebel and Fox11], an end-to-end network that generates a distribution of 6-DoF parallel-jaw grasps from a depth in an efficient way is addressed; they achieved
$90\%$
of accuracy on objects never seen before, using about
$17$
million of simulated grasp. Differently, in [Reference Mirjalili, Krawez, Silenzi, Blei and Burgard12], the authors proposed a grasping method using foundation models to help the robot understand objects better, such as finding the best grasp points and avoiding certain parts.
On this topic, it is also critical to generate grasps that are robust and that require a reasonable amount of computation time. In fact, in [Reference Kumra, Joshi and Sahin13], they propose the generative residual convolutional neural network (GR-ConvNet), a model able to predict grasp configurations in real time.
Lastly, the authors of [Reference Chen, Niu, Yang, Qian, Li, Wang, Yan and Huang14] present an attention-based approach using two-dimensional encoding to enhance local patterns and to reallocate the distribution of grasping-specific features. Going deep, it uses RGB-D images as input and outputs image-size feature maps, determining the grasp configurations.
A literature review focusing on grasp detection technology, considering significant aspects of analytic and data-driven methods, is reported in [Reference Dong and Zhang15].
Traditionally, image-based solutions have dominated the field, also relying on convolutional neural networks (CNNs) [Reference Krizhevsky, Sutskever and Hinton16]. Recently, 3D point clouds and 3D voxel data representations are also becoming increasingly adopted. For instance, in
$2017$
, a challenge problem in computer vision using 3D data was addressed by [Reference Qi, Su, Mo and Guibas17], where PointNet, a point cloud-based architecture capable of classifying and segmenting
$40$
classes of objects in the ModelNet40 dataset [Reference Wu, Song, Khosla, Yu, Zhang, Tang and Xiao18] was proposed. It is important to mention that this approach requires significant computational power and memory resources. In [Reference Zhou and Tuzel19], an end-to-end trainable architecture for point cloud for the 3D detection called VoxelNet is proposed. It is capable to operate directly on sparse points, outperforming the state-of-the-art LiDAR-based 3D detection methods; whereas, in [Reference Maturana and Scherer20], VoxNet is introduced. The network employs volumetric representation to effectively process 3D data, enhancing robustness in object recognition.
A widely adopted geometric-based collision detection method in motion planning is the Flexible Collision Library (FCL) [Reference Pan, Chitta and Manocha21]. This library combines various techniques to efficiently and accurately compute collision checks. Fast and accurate collision checking methods between objects can be achieved by utilizing object mesh information in the scene [Reference Gilbert, Johnson and Keerthi22]. In the majority of approaches, polyhedral models are commonly used, and many techniques rely on the use of bounding volume hierarchies (BVHs). The complexity of BVH approaches is influenced by the complexity of the polyhedral models. Alternatively, other methods based on the nearest-neighbor technique are employed; however, these methods may face challenges in high-dimensional space, making them unfeasible in certain scenarios [Reference Kleinbort, Salzman and Halperin23]. Therefore, approximation-based modeling is not applicable in high-precision situations [Reference Jiménez, Thomas and Torras24].
Computer vision researchers have considered learning-based techniques that work well with 3D object geometries. In [Reference Das and Yip25], the authors demonstrate that tools for collision detection based on robotics motion planning may be accelerated by performing collision checks considering Machine Learning model. Their model, named Fastron, is capable to represent the configuration space of robot manipulators, which can be used as a proxy collision detector, by replacing the standard geometric collision approaches. Results show that, in a simulated environment, their method outperforms FCL, but it requires a number of samples strictly related to the dimensionality of the space of interest.
Differently from the above approaches, the authors in [Reference Muñoz, Lehner, Moreno, Albu-Schäffer and Roa26] propose CollisionGP, a Gaussian process-based algorithm for modeling a robot’s configuration space and query collision checks. In [Reference Tamizi, Honari, Nozdryn-Plotnicki and Najjaran27], the authors propose Path Planning and Collision Checking Network (PPCNet), an end-to-end framework that reduce planning time significantly.
The authors in [Reference Joho, Schwinn and Safronov28] allow for quickly computing distances for any point in the robot’s workspace. They proposed an algorithm that combines deep learning-based approach for distance computations with geometric collision checkers.
Researchers from NVIDIA and Berkeley introduced a NN called SceneCollisionNet, which leverages raw point clouds of objects and scenes [Reference Danielczuk, Mousavian, Eppner and Fox29]. The network takes the pose of the object as an additional input and generates an output indicating the likelihood of object collision. This is achieved by utilizing a voxel representation within the network architecture. An accuracy of
$93\%$
is obtained considering
$2$
million of collision queries
$10$
times faster than the classic FCL library. Furthermore, in [Reference Murali, Mousavian, Eppner, Fishman and Fox30], the authors render synthetic partial point clouds to train CabiNet, a collision model to predict collisions for poses of SE(3) objects in the scene.
In this work, the focus is on the collision checking between the robot gripper and the surrounding environment for tasks such as replanning and object grasping in clutter. The problem is illustrated in Figure 1, where, in an example scenario involving a cup and bottle and a grasping task, the gripper mesh is projected near the cup, leading to both collision (red) and non-collision (green) configurations.
3D Rendering of an example scenario requiring the robot planning for grasping a cup: the successful (green gripper mesh) and unsuccessful (red gripper mesh), i.e., colliding with surrounding objects/environment, attempts are represented, respectively.
More in detail, in the case under investigation, mesh-based approaches may not be satisfying due to the time required for mesh reconstruction and collision checking. However, it is also possible to perform collision detection using neural network models working with depth images and point clouds. The idea is to assign a desired pose to the gripper and project it onto the camera-captured real scene to check collisions occurrences in real-time.
A collision checker based on deep learning is therefore proposed and an in-depth analysis of the latter is performed in terms of accuracy and computational time using different NN architectures, i.e., fully convolutional neural networks on depth images, PointNet-based methods on point clouds, and an hybrid approach resulting by merging features from both depth images and point clouds. The training of NN models is performed using two datasets of
$60$
k samples each, containing depth images and point clouds, respectively. Additionally, a comparison is made considering the computational time between the best NN model resulting from the previous analysis and geometric approaches, i.e, methods based on distance and KD-Tree and OcTree-based algorithms.
The results demonstrate the superior time performance of the learning-based approach, with the best approach being a customized Point Cloud-based architecture while offering acceptable accuracy. As case study, an industrial application is considered using an experimental setup with a KUKA Agilus robot manipulator, shown in Figure 2, available at the laboratory of Technology
$\&$
Innovation Center (TIC) of KUKA Deutschland GmbH in Augsburg, Germany.
The proposed methods are further validated using a different dataset, i.e., ARMBench [Reference Mitash, Wang, Lu, Terhuja, Garaas, Polido and Nambi31], confirming the performance obtained with the KUKA dataset, thus proving the generality of the reported findings.
As an example to demonstrate the applicability of the approach in a practical industrial scenario, experiments involving the same setup adopted to build the KUKA dataset were performed, consisting of
$10$
collision checking evaluations across different objects configurations [Reference Golluccio32].
Video of the experiments is available as multimedia material attached to this paper. Alternatively, the video can be accessed at the following link: https://youtu.be/0gtcNsYazLA.
The experimental setup composed by the KUKA Agilus robot manipulator with a two fingers gripper and a bin with several objects. The camera is top-mounted in order to have a complete view of the bin. Such setup is used to build the KUKA dataset.
2. Proposed approach and synthetic data generation
The proposed architecture for the collision detection is shown in Figure 3.
Learning-based collision detection: proposed architecture.
In detail, the depth image of the scene from the camera is exploited to build the data structure, which consists of depth images or point clouds and a digital representation of the gripper.
Then, data are sent to a pre-trained neural network whose output is a binary value
$\gamma$
(
$0$
in case of non-collision,
$1$
otherwise), which can be used inside a possible planner or grasping algorithm to evaluate the collision occurrence.
Assumption 1. The proposed approach considers the gripper pose without taking into account the robot configuration. This is suitable for industrial bin-picking applications, as the one considered in the paper, in which the gripper approaches the objects from the top and mainly oriented along the vertical line toward to the bin, and the remaining part of the robot structure does not collide with the environment by design.
In particular, such an assumption holds true by design for SCARA or gantry robots, for example, and for anthropomorphic manipulator as well by starting from proper high-elbow configurations (see Figure 2). In fact, SCARA robots have limited DoFs, typically 4 DoFs, and move primarily in a 2D plane with an added vertical motion for picking. Then, given their constrained movement, the arm does not collide with the environment by design. Gantry robots, operating along fixed Cartesian coordinates, are designed for predictable motion within a defined workspace. Therefore, the assumption holds well because only the gripper moves downward and the rest of the structure is clear from obstacles. For anthropomorphic manipulators, instead, as mentioned above, the proposed approach can be applied only if a high-elbow configuration is enforced, ensuring that the robot arm stays clear of obstacles.
Concerning the data, two different datasets are built: (
$i$
)
$\mathcal{D}_{\text{depth}}$
for the case of depth images and (
$ii$
)
$\mathcal{D}_{\text{pcl}}$
for the case of point clouds. In fact, the learning-based models are capable to extract features from both the depth map and point cloud, learning geometric information about the environment.
With regard to the gripper for which collisions with the environment are required to be checked, its desired configuration is considered
\begin{equation} {{\pmb \eta }}_{\text{g,d}} = \left[\begin{array}{c}\pmb{p}_{\text{g,d}} \\ {{\pmb{\phi} }}_{\text{g,d}} \end{array}\right]\in \boldsymbol {\mathbb{R}}^6, \end{equation}
where
$\pmb{p}_{\text{g,d}}$
and
${{\pmb \phi }}_{\text{g,d}}$
are the desired gripper position and orientation, respectively, with the subscript “g” meaning gripper, and the subscript “d” meaning desired. As outlined in the following sections, this configuration is utilized in combination with the gripper mesh to load its digital depth image or point cloud representation into the environment’s representation.
2.1. Depth images data structure
The procedure for generating
$\mathcal{D}_{\text{depth}}$
is reported in Figure 4, and it works as follows: (
$i$
) the camera acquires the full bin in the environment and provides as output the depth image
${\boldsymbol{I}}_s \in \boldsymbol {\mathbb{R}}^{W \times H}$
with
$W = 640$
and
$H = 480$
; (
$ii$
) in parallel, starting from the mesh of the gripper denoted as
$M_g$
and available offline from the 3D-CAD model, i.e., not acquired online through camera sensor, a virtual camera (with the same intrinsic and extrinsic parameters of the real camera) generates the depth images
${\boldsymbol{I}}_g \in \boldsymbol {\mathbb{R}}^{W \times H}$
relative to the desired configuration of the gripper
${{\pmb\eta }}_{\text{{g,d}}} \in \boldsymbol {\mathbb{R}}^{n}$
with
$n = 6$
. Then, the depth images
${\boldsymbol{I}}_s$
and
${\boldsymbol{I}}_g \in \boldsymbol {\mathbb{R}}^{W \times H}$
are combined into a single image
${\boldsymbol{I}}_c \in \boldsymbol {\mathbb{R}}^{W \times H}$
by the
$\Pi$
block in the picture as follows:
\begin{equation} {\boldsymbol{I}}_c = \Pi ({\boldsymbol{I}}_s,{\boldsymbol{I}}_g)\; . \end{equation}
The function
$\Pi$
considers the minimum depth value for each pair of pixels between the scene
${\boldsymbol{I}}_{s}$
and the gripper
${\boldsymbol{I}}_{g}$
. Thus, the objects pixels that are closer to the camera have a minor depth value and result predominant compared to the depth value for the pixels belonging to the far away objects and covered by the gripper.
Dataset
$\mathcal{D}_{\text{depth}}$
generation: the camera acquires the scene and the gripper is projected into the scene; the output is represented by three depth images, i.e., gripper, scene (without gripper), and complete scene (with gripper).
Finally, in order to reduce the computational load, a crop filter is applied by neglecting parts of the images with no gripper points, i.e., ensuring that the entire gripper is contained without loss of information about its close surrounding space. With reference to Figure 4, this operation is performed by the the block
$\Delta _c$
, which generates cropped images
${\boldsymbol{I}}_{s}^\ast$
,
${\boldsymbol{I}}_{g}^\ast$
, and
${\boldsymbol{I}}_{c}^\ast \in \boldsymbol {\mathbb{R}}^{W^{\ast } \times H^{\ast }}$
, having, in this particular case,
$W^{\ast }=156$
and
$H^{\ast }=156$
, respectively.
The effect of the crop filter is reported in Figure 5.
Application of the crop filter on the complete depth image
${\boldsymbol{I}}_{c}$
; it identifies the center of the gripper and creates a bounding box around it, reducing the images from (640,480) to (156,156).
Summarizing, the dataset
$\mathcal{D}_{\text{depth}}$
is
\begin{equation} \mathcal{D}_{\text{depth}} = \{ [{\boldsymbol{I}}_{c}^\ast \,\, {\boldsymbol{I}}_{s}^\ast \,\, {\boldsymbol{I}}_{g}^\ast ]_1, [{\boldsymbol{I}}_{c}^\ast \,\, {\boldsymbol{I}}_{s}^\ast \,\, {\boldsymbol{I}}_{g}^\ast ]_2,\ldots, [{\boldsymbol{I}}_{c}^\ast \,\, {\boldsymbol{I}}_{s}^\ast \,\, {\boldsymbol{I}}_{g}^\ast ]_{N_s} \}\;, \end{equation}
where
$N_s$
is the number of samples.
It is worth remarking that, in this work, only collisions between the gripper and rest of the scene is considered, the remaining structure of the robot is not considered.
2.2. Point clouds data structure
The dataset
$\mathcal{D}_{\text{pcl}}$
is generated in a different way, according to Figure 6. As well-known, a point cloud is represented by a set of 3D points
$\{P_i|i = 1, \ldots,N\}$
, where each
$P_i \in \boldsymbol {\mathbb{R}}^3$
has coordinates
$[x\ y\ z]$
and makes possible to add some extra feature channels (e.g., color and other details). As in the previous case, the real camera acquires the scene of the cluttered environment (objects within a bin), providing the depth image
${\boldsymbol{I}}_{s}$
. Then, the point cloud generator block
$G_{\text{PC}}$
extracts two point clouds that are
${\boldsymbol{P}}{\boldsymbol{C}}_s \in \boldsymbol {\mathbb{R}}^{N \times 3}$
and
${\boldsymbol{P}}{\boldsymbol{C}}_g \in \boldsymbol {\mathbb{R}}^{N \times 3}$
, with
$N$
number of points, for the scene and the gripper, respectively; the block
$U_{\text{PC}}$
combines them to generate the complete point cloud
${\boldsymbol{P}}{\boldsymbol{C}}_c \in \boldsymbol {\mathbb{R}}^{N \times 4}$
. Due to the different natures of the point clouds (the first one comes from the camera, whereas the second one from the mesh model), a downsampling and padding are applied. The complete point cloud has dimensions of
$N\times 4$
due to the inclusion of an extra fourth column, with values set to either
$0$
or
$1$
to indicate association to the background scene or to the gripper, respectively.
More in detail, the point cloud extraction executed by the block
$G_{\text{PC}}$
is obtained through a conversion of the depth image. In fact, each pixel
$(u, v)$
in the depth image with depth value
$d(u, v)$
is mapped to a 3D point
$(x, y, z)$
using the camera’s intrinsic parameters in eq. (4). Given the intrinsic matrix
${\pmb{K}}$
\begin{equation} \pmb{K} = \left [\begin{array}{c@{\quad}c@{\quad}c} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{array}\right ], \end{equation}
where
$f_x$
,
$f_y$
are the focal lengths in pixels, and
$c_x$
,
$c_y$
are the coordinates of the principal point, respectively, the 3D coordinates can be computed as
\begin{equation} \begin{cases} z = d(u, v)\\ x = (u - c_x) \dfrac {z}{f_x} \\[7pt] y = (v - c_y) \dfrac {z}{f_y} \\ \end{cases} . \end{equation}
The latter transformation in eq. (5). maps the 2D pixel coordinates and depth values into a 3D point obtaining, therefore, the entire point cloud. An example of the acquired-processed depth image and the relative point cloud is reported in Figure 7.
Dataset
$\mathcal{D}_{\text{pcl}}$
generation: the physical camera captures the scene, producing a depth image; then, the PointCloud Generator block
$G_{\text{PC}}$
generates point clouds for both the scene and the gripper; subsequently, the two point clouds are combined using the concatenation block
$U_{\text{PC}}$
to produce the complete point cloud.
On the left, the complete (scene and gripper) depth image
${\boldsymbol{I}}_c$
is reported, whereas in the right part the complete point cloud
${\boldsymbol{P}}{\boldsymbol{C}}_c$
is shown.
Finally, the dataset
$\mathcal{D}_{\text{pcl}}$
resulting from the current point cloud dataset generation is
\begin{equation} \mathcal{D}_{\text{pcl}} = \boldsymbol{PC}_{c_1}, \boldsymbol{PC}_{c_2}, \ldots, \boldsymbol{PC}_{c_{N_s}}\;, \end{equation}
the labels of which are generated using Open3D [Reference Zhou, Park and Koltun33], an open-source library that supports the rapid development of software for handling 3D data, providing a wide selection of data structures and algorithms in C+ + and Python.
2.3. Dataset complexity
Regarding the complexity of the scenarios, the idea is to take into consideration different cases. A basis made of 250 highly diverse cases (bins either empty or filled with objects in a completely random manner) is considered. By randomly sampling these scenarios, the gripper was placed in the scene with random positions and orientations, resulting in a total of
$60k$
configurations, evenly split between
$50\%$
collision and
$50\%$
non-collision cases. This was done to avoid bias and ensure maximum generalization. In particular, the collision label is set, as detailed in Section 3.1 computing the minimum distance between the gripper and the bin (with all the objects inside) point clouds, respectively. If the minimum distance is less than an arbitrary desired threshold (
$1$
cm in our case), then the scenario is labeled as a collision case, non-collision otherwise.
Samples from the dataset are shown in Figure 8. Cases A-B are the easiest, whereas C-D are medium in terms of complexity. The harder cases for the collision checker are E-F since they represent scenarios (objects and bin) never used in the training dataset.
Dataset samples of different complexity: (easy) (a), (b) show a bin with no items and wooden objects, respectively; (medium) (c), (d) contain industrial objects; (difficult) (e), (f) take into consideration plastic fishes and different shape of the bin not seen at the training phase.
3. Collision checker
In the following, the approaches compared for solving the collision checking problem are detailed. First, geometric approach is reported since it is used as baseline. Then, the learning-based methods are described remarking their customizations.
3.1. Geometric-based collision checking
As baseline for collision checking, an algorithm based on libraries Open3DFootnote 1 and TrimeshFootnote 2 is taken into consideration. The latter is a mesh-based library capable of handling triangular meshes for collision detection based on the known general purpose FCL library [Reference Pan, Chitta and Manocha21].
More in detail, in this work, the procedure consists of building an accurate mesh
$M_s$
from the point cloud of the scene
${\boldsymbol{P}}{\boldsymbol{C}}_s$
through the library Open3D using the ball pivoting algorithm. It utilizes the continuity property of spherical balls with a given radius [Reference Peng, Jiang, Liao, Niemeyer, Pollefeys and Geiger34]. The mesh of the gripper
$M_g$
is then projected into the scene with a desired pose
${{\pmb{\eta }}}_{\text{g,d}}$
and, finally, the collision checker available from the library Trimesh is executed on the generated objects to compute the collision condition
$\gamma$
, i.e., collision/non-collision state, and the minimum distance between the nearest two points belonging to the scene and to the gripper, respectively. The entire flow of the algorithm is reported in Algorithm 1. It is worth mentioning that this procedure is meant to be used only as baseline, since it cannot be used in real time due to the time related to the mesh reconstruction as detailed in Section 4.
Mesh-based Collision Checker.
3.2. CNNs-based collision checking
This work considers three different architectures based on CNNs: two fully convolutional neural networks (FCNNs), one utilizing a single depth image and the other utilizing two depth images, respectively, and the well-known ResNet18 network customized for this application. These learning-based models differ from the basic versions as they are adapted to handle various data representations, i.e., depth images versus RGB images, incorporating additional layers to accommodate these differences. These first two architectures are reported in Figure 9, while the ResNet18-based architecture is shown in Figure 10.
3.2.1 FCNNs -
$1$
depth image (FCNN1D) and
$2$
depth images (FCNN2D)
Both the FCNN models consist of four blocks, each made of three convolutional layers (CONV), followed by max pooling (MP) and dropout (D) layers. The last layer is a convolutional layer with a kernel of size
$1$
(CONV with K = (
$1$
,
$1$
)), and it is used for compressing the output as a binary classifier. Furthermore,
$g(\!\cdot\!)$
represents the activation function. In this case, there are two activation functions
$g_1 (\!\cdot\!)$
and
$g_2 (\!\cdot\!)$
.
The difference between the two models is represented by the input. In particular, as shown in Figure 9, the FCNN1D takes as input
${\boldsymbol{I}}_{c}^\ast$
, an image
$(156{\times} 156)$
with
$1$
channel that contains the gripper projected into the scene. Instead, the model FCNN2D considers two depth images
${\boldsymbol{I}}_{s}^\ast$
and
${\boldsymbol{I}}_{g}^\ast$
, representing the scene and gripper which, combined together, result in a depth image with input size of
$(156{,}156)$
and
$2$
channels, i.e., the image is composed by two layers and, therefore, threaded by the neural network as a multi-channel depth image.
Fully convolutional neural network (FCNN) architecture: FCNN1D takes one single-channel depth image (156, 156, 1) as input (left violet input part); FCNN2D takes two depth images (156, 156, 2), i.e., multi-channel depth image (right yellow input part); both of them return collision/non-collision state as output.
3.2.2 ResNet18
It is a convolutional neural network made of 18 layers (see Figure 10). In the transfer learning context, there is the possibility to use a pre-trained version of the network, which is trained on more than
$1$
million of images present in the ImageNet database [Reference Deng, Dong, Socher, Li, Li and Fei-Fei35]. It is capable of classifying more than
$1000$
different objects present in the dataset. The original version of the network considers images
$(224{,\,}224)$
, but a customized version is implemented in this work. In the detail, only the deep architecture is used (not pre-trained) and adapted to work with the images of the dataset
$\mathcal{D}_{\text{depth}}$
. A well-known problem with very deep neural networks is the vanishing gradient when it is back-propagated to earlier layers because the repeated multiplication may make the gradient very small. Based on this, ResNet uses the concept of residual blocks, which incorporate shortcut skip connections in order to jump over some layers.
ResNet18 architecture: it takes two depth images
${\boldsymbol{I}}_s$
,
${\boldsymbol{I}}_g$
$(156,156,2)$
as input, i.e., a multi-channel depth image, and classifies the output in collision/non-collision.
3.3. PointNet-based collision checking
When working with point clouds, due to the irregularity of structure, it is common to pass through a 3D voxel grids transformation [Reference Zhou and Tuzel19]. Unfortunately, this procedure is really hard and expensive from the computational point of view. For this reason, this problem has been faced in literature.
3.3.1. (Standard) PointNet
The authors presented in [Reference Qi, Su, Mo and Guibas17] a deep learning framework, called PointNet, that receives in input unordered points. In its original version, the proposed architecture is capable of classifying and segmenting
$40$
different classes of objects, through reorganizations of the points received in input. In this work, the segmentation from the network is not considered, since the point cloud
$\boldsymbol{PC}_{c}$
is built considering the information about the belonging of the points.
On the latter, initially a transformation through the T-Net block is applied (e.g., rotation about z-axis), which does not affect them in terms of shape or change of relevant features but perform an input transform and feature transform, respectively. This first T-Net aligns the input point cloud before any feature extraction takes place. Once aligned, the features are upscaled to 64 dimensions through a shared MLP, marking the start of feature extraction. A second transformation with a second T-Net, i.e., feature transform, is then applied to the 64-dimensional features to derive the local features. In particular, these two T-Nets are neural networks designed to learn proper transformation matrices that perform normalization for a given input. During the forward pass, the input point cloud and then the features vector are multiplied by these learned transformation matrices, allowing PointNet to adaptively learn from various point cloud representations in an end-to-end manner. After the input and feature transform, another transformation to a different space with dimensionality
$1024$
is applied in the second MLP. To this regard, an important aspect is that each point has
$1024$
features and the network can learn the shape of the elements present in the data. The segmentation block contains additional information about the points, i.e., which mesh (gripper or bin) they belong to, and it is fused with the rest of features before the max pooling operations. With such architecture, even if the order of the input changes, the result is the same. Finally, there is a fully connected layer to map the features in one of the
$k$
categories to classify. In this particular case, as already mentioned, the classes are two, namely collision and non-collision, as shown in Figure 3. The overall architecture is reported in Figure 11, where every block is part of the current model.
PointNet and Fast-PointNet architectures with and without orange blocks, respectively: they take a point cloud
${\boldsymbol{P}}{\boldsymbol{C}}_c$
as input and classify the output in collision/non-collision.
3.3.2. Fast-PointNet
A custom version of PointNet, named Fast-PointNet, is proposed as well. In fact, since the camera used to acquire the data is fixed in the space, the T-Net block can be neglected. From the computational point of view, if the point cloud is composed of a large number of points, the multiplication with these matrices could take varying amounts of time. Due to this consideration, this custom version is faster than the previous architecture.
The segmentation block containing additional information about the points is considered as well. The latter block, as proven by tests, with further information about the points helps at improving the accuracy of the NN. The resulting architecture is shown in Figure 11 where, in particular, the orange blocks are neglected while the segmentation block is included in the model.
3.4. Hybrid collision checking
Finally, a combination of FCNNs and Fast-PointNet, entitled MixNet, is proposed. It is a network that fuses the depth image
${\boldsymbol{I}}_{s}^\ast$
of the entire scene and the point cloud of the gripper
$\boldsymbol{PC}_g$
. These two architectures are combined in such a way that the features extracted from the depth image of the scene
${\boldsymbol{I}}_s$
and the point cloud of the gripper
${\boldsymbol{P}}{\boldsymbol{C}}_g$
are fused all together. The architecture is reported in Figure 12.
MixNet Architecture: it takes a depth image
${\boldsymbol{I}}^\ast _s$
$(156,156,1)$
and a point cloud
${\boldsymbol{P}}{\boldsymbol{C}}_g$
as input and classifies the output in collision/non-collision.
3.5. Learning-based collision checker
The entire flow for the learning-based collision checker, both for depth images and point cloud data structures, is reported in Algorithm 2. In particular, based on the approaches proposed above, the input of the algorithm is made of the depth image of the scene
${\boldsymbol{I}}_s$
provided by the camera, the mesh of the gripper
$M_g$
, the desired pose
${{\pmb{\eta }}}_{\text{{g,d}}}$
for the gripper and the architecture to use, choosing among FCNN1D, FCNN2D, ResNet18, PointNet, Fast-PointNet, and MixNet. The output of the algorithm is, as described above, a scalar value
$\gamma$
that is the prediction of the neural checker, which contains the information about the collision. Then, the ApplyTransformation function loads the mesh of the gripper
$M_g$
with a specific pose
${{\pmb {\eta }}}_{\text{g,d}}$
. The DepthGenerator generates the synthetic depth image for the gripper
${\boldsymbol{I}}_g$
. At the same time, the PointCloudGenerator builds the point clouds for the gripper
${\boldsymbol{P}}{\boldsymbol{C}}_g$
and for the scene
${\boldsymbol{P}}{\boldsymbol{C}}_s$
; the latters are combined through the function
$\Pi (\cdot, \cdot )$
as per eq. (2), which projects the gripper into the real scene, generating the complete depth image
${\boldsymbol{I}}_c$
(a filter to reduce the resolution is applied to the resulting point cloud). Differently, the Union function is applied to the point clouds, computing the final point-cloud image
${\boldsymbol{P}}{\boldsymbol{C}}_c$
. At this point, the architecture to use as a collision checker is selected, and the related samples are fed into the corresponding model.
Learning-based Collision Checker.
KUKA dataset evaluation results: applied model with corresponding accuracy, precision, recall, F1-Score, CPU time, and GPU time (
$\mu \pm \sigma$
) are reported, respectively.
4. Evaluation of the proposed models
As mentioned above, the models have been trained using a dataset consisting of
$60$
k depth images and point clouds. It is made of balanced classes (
$50\%$
collision and
$50\%$
non-collision). The chosen loss function is binary cross-entropy, and the training optimization is performed using the stochastic gradient descent (SGD) optimizer with momentum and regularization. Furthermore, the datasets
$\mathcal{D}_{\text{depth}}$
and
$\mathcal{D}_{\text{pcl}}$
are divided into three sub-datasets following the rule
$70\%$
for the training set,
$15\%$
for the validation set and
$15\%$
for the test set. After training the proposed models, the latter are used for collision prediction in scenes containing previously unseen objects, in order to evaluate the generalization capability of the network. In particular, all the learning-based models have been trained e validated on a laptop equipped with CPU 11th Gen Intel Core
$^{\text{TM}}$
i7-11850H 2.50 GHz
$\times$
16 and GPU NVIDIA T1200.
Obtained results are reported in Table I and show that the methods based on the point clouds outperform the others in terms of precision, F1-score and recall, and that Fast-PointNet represents the best choice for most of the main parameters. Regarding computational cost, Fast-PointNet holds a significant advantage over PointNet on both GPU and CPU. It also yields comparable results to other methods, especially in the more compelling scenario of GPU usage.
In general, the resulting performance is remarkable if compared to geometric approaches. For instance, the geometric-based (FCL-based) checker, used as baseline for the dataset generation, takes
$\approx \!\!1.90$
s for reconstructing the mesh from the related high resolution point cloud, making this approach not suitable for fast real-time applications.
4.1. Comparison of Fast-PointNet with geometric point cloud-based approaches
Since Fast-PointNet proves to be overall the most performing approach, a comparison with other classic geometric methods based on point clouds is addressed. The considered algorithms are:
-
• NP
$_{\text{dist}}$
—Norm-based distance computation using NumPy
Footnote
3
[Reference Harris, Millman, van der Walt, Gommers, Virtanen, Cournapeau, Wieser, Taylor, Berg, Smith, Kern, Picus, Hoyer, van Kerkwijk, Brett, Haldane, del Río, Wiebe, Peterson, Gérard-Marchant, Sheppard, Reddy, Weckesser, Abbasi, Gohlke and Oliphant36]: it uses linear algebra functions to provide efficient low level implementations of standard algorithms for distance computation. It computes the distances among all the points for the two point clouds
${\boldsymbol{P}}{\boldsymbol{C}}_{s}$
and
${\boldsymbol{P}}{\boldsymbol{C}}_{g}$
. Then, according to the camera resolution and kinematics characteristics of the gripper, a threshold-based heuristic is considered as decision-making for collision checking; -
• O3D
$_{\text{dist}}$
—Norm-based distance computation using Open3D
Footnote
4
[Reference Zhou, Park and Koltun33]: for each point in the source point cloud
${\boldsymbol{P}}{\boldsymbol{C}}_s$
, it computes the distance to the target point cloud
${\boldsymbol{P}}{\boldsymbol{C}}_g$
, exploiting CUDA. Afterward, for detecting collision, the same approach used in NP
$_{\text{dist}}$
is applied; -
• SP
$_{\text{KDTree}}$
—KDTree using SciPy
Footnote
5
[Reference Muja and Lowe37]: the construction of a KD Tree is made of recursively, partitioning the points in the space, forming a binary tree. Each internal node in the tree functions has a hyperplane, splitting the space into two regions. Then, it considers a query to computes the distances of all for nearest neighbors provided by point cloud target using a multi-threading approach; -
• PCL
$_{\text{OcTree}}$
—OcTree using Point Cloud Library (PCL)
Footnote
6
[Reference de Queiroz, Garcia, Chou and Florencio38]: it is a tree data structure where each internal node has eight children. This structure is mainly used for spatial partitioning of 3D point clouds due to their useful description of 3D space, which facilitates the quick finding of nearby points.
The time analysis results are reported in Figure 13. More in detail, the chart illustrates statistics on the time complexity (in s) required to correctly detect collisions between the robot gripper and its environment using point cloud representation. In particular, each element group represents the distribution of the times for the
$i$
-th method, highlighting its median, maximum, and minimum values, first and third quartile, and potential outliers. Specifically, approaches based on well-known algorithms like OcTree and KDTree, as well as methods based on geometric distances are compared along with with our best model based on point cloud, i.e., Fast-PointNet, remarking the time performance of the latter. In particular, due to the time gap between the Fast-PointNet and the other methods, it has been necessary to use a logaritmic scale on the chart to better display the results.
Box charts relative to time complexity in log scale of: NP
$_{{\rm dist}}$
(NumPy), O3D
$_{{\rm dist}}$
(Open3D), SP
$_{{\rm KDTree}}$
(SciPy), PCL
$_{{\rm OcTree}}$
(Point Cloud Library), and our proposed method Fast-PointNet.
4.2. Evaluation on the Amazon Robotic Manipulation Benchmark (ARMBench) dataset
With the aim of corroborating the findings obtained with the KUKA dataset, results obtained with the ARMBench dataset are presented in this section. It is a large-scale dataset collected in an Amazon warehouse from containers with heterogeneous contents. In particular, it contains depth images with several annotations as, e.g., objects types and positions/orientation. Such depth images are adapted to the proposed framework, allowing for their integration in the same way as the data presented in Section 2.3. In detail, as for the KUKA dataset, a set of
$60k$
scenarios has been generated by randomly projecting the gripper mesh into Amazon depth images, in such a way to have, as for the KUKA dataset,
$50\%$
of collision and
$50\%$
of non-collision cases. Examples of depth images generated using the ARMBench dataset are reported in Figure 14. The latter shows:
-
•
$I_c$
(top-left)—the complete depth image, namely the scene and gripper together; -
•
$I^\ast _c$
(bottom-left)—the complete depth image cropped around the gripper with a resolution of
$156 \times 156$
px; -
•
$I^\ast _s$
(top-right)—the depth image of the scene alone cropped around the gripper with a resolution of
$156 \times 156$
px; -
•
$I^\ast _g$
(bottom-right)—the depth image of the gripper alone cropped with a resolution of
$156 \times 156$
px.
The results are reported in Table II. Notably, they are very close to the performance obtained with the KUKA dataset, proving the generality and effectiveness of the reported findings and remarking the Fast-PointNet outperforming the others in terms of accuracy and time as well, even if, regarding the latter, it is comparable to FCCN1D-2D, as already observed for the KUKA dataset.
ARMBench dataset evaluation results: applied model with corresponding accuracy, precision, recall, F1-Score, CPU time, and GPU time (
$\mu \pm \sigma$
) are reported, respectively.
Depth images based on Amazon ARMbench dataset:
${\boldsymbol{I}}_c$
is the complete depth image, i.e., scene and gripper;
${\boldsymbol{I}}^\ast _c$
is the complete depth image cropped around the gripper with a resolution of
$156 \times 156$
px,
${\boldsymbol{I}}^\ast _s$
is like the previous one without the gripper;
${\boldsymbol{I}}^\ast _g$
is related to the gripper alone cropped with a resolution of
$156 \times 156$
px.
4.3. Industrial application with KUKA Agilus robot
The overall framework was finally exploited for the scenario under consideration on the real hardware. In detail, the test campaign was carried out in Python environment using a combination of PyTorch and robot operating system (ROS). The considered scenario is the same used to build the KUKA dataset and depicted in Figure 2.
Industrial application with KUKA Agilus robot in the Technology & Innovation Center at KUKA Deutschland GmbH in Augsburg (Germany). The learning-based approach is evaluated in
$10$
different real cases reporting the front, top views with the depth image showing a bounding box for the gripper target pose.
The experiments consist of
$10$
different cases
$C_i$
(
$i=1,2,\cdots, 10$
), as reported in Figure 15, where, for each scenario, it is reported the front and top view of the setup and the corresponding depth image of the bin alone with a bounding box showing the target pose of the gripper as projected in the depth image for collision checking. The case
$C_1$
represents a straightforward scenario made of two sub-cases: non-collision and collision, respectively. In the first part an empty bin is considered, making it easy for the algorithms to detect the absence of collisions for the desired assigned configuration of the gripper. Then, a tower of objects is added in correspondence of the same desired position, in order to generate intentionally a collision with the gripper. In case
$C_2$
, the bin is filled with different objects, representing a more complex detection scenario compared to the previous one. The cases
$C_3$
and
$C_4$
involve different bins and objects in the scene, respectively, to test the generalization capacity of the models. Specifically, in the case
$C_3$
, the gray bin has a different orientation and height compared to the one used in previous cases. Furthermore, in
$C_4$
the objects involved are plastic fish, and the collision occurs only with the initial part of the fingers, resulting in a hard collision to detect. In the previous cases, the final desired pose of the gripper is the same, while in the following remaining cases different final configurations are considered. In particular,
$C_5$
,
$C_6$
, and
$C_9$
are such as that the non-collision occur by moving the gripper very close to the objects, while cases
$C_7$
,
$C_8$
, and
$C_{10}$
consider collisions with minor contact between the gripper and the environment.
5. Conclusion and future work
In this work, the problem of checking collisions between the robot gripper and the surrounding environment in short time for tasks, such as planning and object grasping in clutter, was considered. Starting from the mesh of the robot gripper, corresponding depth images or point-clouds were generated and combined with the depth image provided by the camera about the surrounding environment. Deep learning tools were, then, exploited to evaluate the occurrence of collisions. The validation tests showed the effectiveness of the learning-based approaches highlighting the Fast-PointNet method outperforming the other considered methods in terms of accuracy, reaching an accuracy of
$\approx 93\%$
on a dataset made of
$60$
k samples, with an evaluation time which is comparable with the methods based on depth images (order of
$10^{-1}$
ms). A comparison was made with geometric approaches based on point clouds as well, remarking the higher performance of the Fast-PointNet over the other methods in terms of computational time. Furthermore, an additional dataset, i.e., the ARMBench by Amazon, was used to evaluate the learning based models obtaining results close to performances seen for the KUKA dataset and, therefore, showing the robustness and scalability of the proposed learning-based models.
As future work, another learning-based approach, Vision Transformers (ViT), will be considered for comparison. ViT features a different architecture from CNNs, leveraging the self-attention mechanism commonly used in large language models (LLMs). Moreover, the framework will be extended to account for other parts of the robot’s structure beyond the gripper, making the proposed collision-checking approach applicable to additional tasks. However, it is worth noting that the proposed collision detection method has a wide range of applications, not only in robotics but also in other fields, such as automotive, which could be the subject of further investigation.
Acknowledgments
This work was performed at KUKA, Germany. We would like to thank Dr. Jonas Schwinn and Dr. Dominik Joho for their helpful supervision and suggestions and Dr. Kirill Safronov, Mr. Giuseppe Monetti, and Dr. João Oliveira for their advices.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0263574725000517.
Financial support
The authors declare that this work was supported by H2020-ICT project CANOPIES (Grant Agreement N. 101016906), by Project “Ecosistema dell’innovazione—Rome Technopole” financed by EU in NextGenerationEU plan through MUR Decree n. 1051 23.06.2022—CUP H33C22000420001, by Project CONCERTO-A COgNitive arChitecture for sEamless human-Robot inTeractiOn, CUP H53C24001050006, and from Project COM3, CUP H53D23000610006, both funded by EU in NextGenera-tionEU plan through the Italian “Bando Prin 2022 - D.D. 104 del 02-02-2022” by MUR.
Conflicts of interest
The authors declare no conflicts of interest exist.
Ethical approval
Not applicable.
