3.1 Data augmentation and generation

Basically, our synthetic data generation method is fed with three inputs: (i) a picture showing an empty cabinet, (ii) a catalog including all the available components that can be installed in a cabinet, and (iii) a limited number of real pictures that will be manipulated in order to add noisy elements in the generated data, by exploiting a suitable strategy described in the following.

The core idea is to enrich the ground truth (consisting of a limited number of images) with synthetic images, where the area of the empty cabinet is filled with random components picked from the catalog. Notice that, at this stage, we are not interested in generating compliant panels, since our only objective at this stage is to build a suitable object detector that is capable of recognizing both the component and its geometrical position and extension. The size of the catalog and the randomness of the composition allow us to generate a suitable number of images where each component can be included with a suitable frequency, thus making the result dataset robust to object detection and segmentation learning tasks.

This simple approach can be further combined with other image augmentation strategies (Image Processing module in Figure 1) with the aim of yielding a training set that includes a sufficient and diversified number of examples for learning the model. In particular, our framework also includes traditional data augmentation strategies that is, Gaussian Blur and PerspectiveTransform. As regards the former, the idea consists in introducing imperfections into data so as to make component detection more resilient to data changes, it is obtained by averaging contiguous pixel values. The last one allows for applying random four-point perspective transformations to images.

The resulting dataset from this process will include all the necessary features for the training phase: (i) a large number of different pictures, (ii) the position of each component, (iii) the type of each component. Notably, since each component depicted in the synthetic pictures is randomly placed, the detection model will be forced to learn the intrinsic features of each component, instead of considering positional features, that may vary in the different schematics. Within a cabinet, there are other “auxiliary” elements that are simplified in a schematic, mainly separation boxes, metal runners, and cables. For simplicity, we call them noise to randomly add to the generated images in order to make the object detection model able to distinguish and ignore these elements.

Figure 2, show some examples of the data generation process. We can observe the empty cabinet (Figure 2a), and two instances where it is filled with random components (Figure 2b and 2c). Notice that the synthetic data does not necessarily represent a realistic situation. As already mentioned, this is not an issue since our purpose here is to strengthen the object detection and segmentation phase, which is discussed below.

Fig. 2.

Input and output of the component detection approach.

3.2 Component detection

The Model Building module in Figure 1 allows for training the deep architecture used to perform the component detection. For this, we adopted the Mask R-CNN convolutional neural architecture proposed by He et al. (Reference He, Gkioxari, Dollár and Girshick2017). In general, R-CNN (Region based CNN) refers to a family of neural architectures adopting a Multi-shot approach. The underlying idea is a two-step process: first, different bounding boxes across possible regions of interest (RoIs) are extracted; then, such regions are independently evaluated through a CNN architecture in order to map them to any of the proposed classes (see Figure 3).

Fig. 3.

R-CNN working flow.

Mask R-CNN extends a specific architecture named Faster R-CNN (Ren et al. Reference Ren, He, Girshick and Sun2015) that includes two main components: (i) Region Proposal Network (RPN), a deep neural network aimed at extracting RoIs from the picture, and (ii) Fast R-CNN, a neural architecture that performs classification, by scaling a region to a predefined size thus enabling the computation of a set of CNN feature maps. The main advantage of the Faster R-CNN architecture is a suitable trade-off between competitive accuracy in terms of object recognition, and relative speed in the recognition phase. By contrast, other approaches based on Single-Shot architectures such as YOLO (Redmon et al. Reference Redmon, Divvala, Girshick and Farhadi2016) or SSD (Liu et al. Reference Liu, Anguelov, Erhan, Szegedy, Reed, Fu and Berg2016) focus on fast recognition, at the cost of recognition accuracy. This is clearly not acceptable in our scenario, where we aim at checking compliance, and missing a component in the picture would result in a failure in the check. Mask R-CNN further improves Faster R-CNN by introducing a further branch for predicting segmentation masks on each Region of Interest. The recognition of the mask is crucial in our scenario since it allows to precisely identify the geometrical position of the component within the panel. Technically, Mask R-CNN rebuilds the mask by resorting to an alignment component and a mask head, composed of two convolutional layers and capable of generating a mask for each RoI in order to segment the picture in a pixel-to-pixel fashion.

Mask R-CNN relies on a backbone convolutional architecture. In our framework, we used ResNet (Residual Network) (He et al. Reference He, Zhang, Ren and Sun2016), a very deep CNN architecture characterized by residual blocks and skip connections. These two features guarantee both, fast convergence in the training stage and expressiveness/accuracy in the recognition phase. We further strengthened the training phase by exploiting Transfer Learning. In particular, we used a ResNet pre-trained on COCO dataset, ^{Footnote 1} which was also fine-tuned for our specific scenario, by exploiting the generated dataset with the artificially generated labeled components.

Figure 2d shows the output of the recognition phase, on a real picture representing a true panel. We can see that the model successfully recognizes all available components, and additionally devises a contour of their geometric extension. These contours represent one of the inputs to the reasoning component.

4.1 Input specification

In ASP the input specification is made by a set of “facts,” which are assertions that model true sentences. Thus, the labeled schematic of the circuit (we informally refer to it as cad), and the output of the Mask R-CNN net (exemplified in Figure 2a) are converted in a set of ASP facts of the following form:

These facts provide information about the components like their label, id, top-left and bottom-right coordinates, and membership. In particular, the membership is valued with “cad” if the object modeled is part of the schematic of the panel, and “net” if it is recognized by the neural network in the actual picture we are comparing to the schematic. Moreover, we also compute a graph of topological relations among objects, providing information on relative position and distance among objects. The relative position and the distance among components are actually calculated by our ASP program, but for simplicity, we assume here they are given in input as facts of the form:

The between predicate denotes the neighbors for the component ID along the direction DIR having MEM as membership; additionally, the manhattan predicate specifies the manhattan distance between the two components ID1 and ID2, where the terms MEM1 and MEM2 stand for their membership.

4.2 ASP program

We now present ASP program (see Encoding 1) that encodes in a uniform way (w.r.t. the input instance provided as a set of facts) the compliance problem. First, the graph is preprocessed (lines 2-3), by calculating useful information about the relative positions of the objects. Next, according to the “guess-and-check” programming methodology, a disjunctive rule guesses the mapping between “cad” components of the schematic and “net” components predicted by the neural network (see lines 6-7).

Encoding 1

ASP program modeling compliance

The disjunctive rule can be read as follows: “Given a cad component and a net component of the same type, the two can be mapped, or not.” The candidate solutions are filtered out by the constraints in lines 9-13, ensuring that the same element of the cad is not mapped twice, and the same element of the net is not mapped twice.

The optimal mapping is obtained by weak constraints in lines 15-35. In detail, the program first minimizes the cad elements without a mapping (lines 15-16), then (also in order of priority) the weak constraints in lines 18-31 ensure that “If a cad component ID1 is mapped to a net component ID2, ID1 neighbors should be mapped to ID2 neighbors.”

The mapping is further optimized considering the distance (lines 33-35) between cad components and net components. The distance is optimal when the elements are in the same position in “net” and $\text{"cad."}$ Finally, the program identifies components that are absent or in excess w.r.t. the schematic by rules in lines 37-40.

5.1 Experimental setup

We set up the experimentation by considering the extreme scenario where no labeled examples are available. Therefore, our training set used includes only the synthetically generated images (by using the data augmentation techniques described in Section 3), while the real pictures of the EPCs are used to evaluate the predictive performances. The final result of this process is a training set composed of $\sim$ 10,000 colored images synthetically generated with size ( $320 \times 320$ ) and a test set of 32 images depicting real control panels with the same size as the training ones.

The model discussed in Section 3 has been implemented in the form of a python prototype based on TensorFlow ^{Footnote 2} library. The experiments were performed on an NVidia DGX Station equipped with 4 GPU V100 32GB. As described in Section 3, a ResNet instance (including 101 layers) is used as the backbone of the component detection model, the Mask R-CNN, that is trained over 200 epochs with $batch\_size = 2$ , while Adam is adopted as optimizer with learning rate $\mathit{lr}= 10^{-4}$ .

To assess the capability of the proposed approach in detecting the components installed in the ECPs, a number of traditional measures and well-known metrics for the Object Recognition tasks have been used. In this sub-section, we briefly introduce and define such measures. The first measures we consider are the standard Precision and Recall metrics, defined as $p = \frac{TP}{TP + FP}$ and $r = \frac{TP}{TP + FN}$ . Here, TP, FP, FN, and TN denote respectively the number of cases that are: positive and correctly classified, positive and incorrectly classified, negative and incorrectly classified, and negative and correctly classified. Hence, a Precision-Recall Curve can be obtained by computing and plotting the precision against the recall for different threshold values (i.e. the detection probabilities of the model).

In an object detection scenario, precision and recall represent the capability of the prediction model to identify the boxes that contain the target objects. In particular, for a given object the focus is on comparing the true bounding box with the predicted bounding box, and the TP, FP, FN, and TN values depend on the degree of overlap between these two boxes. Given two boxes, the Intersection Over Union (IoU) is defined as the fraction of the overlapping area between the ground truth b and the predicted bounding box $\hat{b}$ :

(1) \begin{equation} \mathit{IoU}(b,\hat{b}) = \frac{b \cap \hat{b}}{b \cup \hat{b}}\end{equation}

Then, given a threshold $\theta$ , an object with a true bounding box b and a predicted bounding box $\hat{b}$ is positive if $\mathit{IoU}(b,\hat{b})>\theta$ , and negative otherwise. For a given $\theta$ , it is possible to devise a precision-recall curve by plotting all p/r values relative to all objects and interpolating the resulting curve (He et al. Reference He, Gkioxari, Dollár and Girshick2020; Ren et al. Reference Ren, He, Girshick and Sun2015).

Since the values of precision and recall are defined on a given $\theta$ threshold, we can define (He et al. Reference He, Gkioxari, Dollár and Girshick2020; Ren et al. Reference Ren, He, Girshick and Sun2015) Average Precision and Recall as the area represented by integrating over all possible thresholds:

(2) \begin{equation} \mathit{AP} = \int_{0}^{1} p(\theta) \, d\theta \quad \mbox{(and resp.)} \quad \mathit{AR} = \int_{0}^{1} r(\theta) \, d\theta \, .\end{equation}

Finally, by averaging $\mathit{AP}$ (resp. $\mathit{AR}$ ) over all class components we can finally obtain the mean average precision ( $\mathit{mAP}$ ) and mean average recall ( $\mathit{mAR}$ ) measures.

5.2 Evaluation results

Here, we discuss the results in terms of the effectiveness of the DL-Based detection model and the scalability of the ASP module. For the first aspect, the detection model exhibits optimal performances for both the quality measures, exhibiting values of $\mathit{mAP}=0.954$ and $\mathit{mAR}=0.935$ . In order to evaluate the operational applicability of our approach in a real scenario, we conducted a further analysis by considering the values of precision and recall on a fixed $\mathit{IoU}$ threshold $\theta=0.5$ . Basically, in this test case, precision and recall, respectively, represent the capability of the model to correctly recognize the components depicted in the picture and the percentage of recognized components w.r.t. the ground truth.

Figure 4a reports the resulting precision-recall curve. The resulting area is $0.947$ , which denotes a good performance of the detection model also considering the operational case. Figure 4b shows a more detailed picture of the model performances by plotting the pr-curve for each instance. As expected, for almost all instances, the yielded curves highlight the good predictive accuracy of the model in recognizing the different types of components, except for one case in which the quality is slightly lower. The above evaluation shows that the component detection module is effective in recognizing the components of a panel: in particular, prediction errors can occur in rare cases with inaccurate image acquisition (e.g. non-frontal framing or inclusion of elements external to the cabinet) as the catalog provides only a limited number of component perspectives. An example of such behavior is depicted in Figure 5, where accuracy is affected by the wrong perspective of the image. Since the ASP program performs the compliance task with optimal accuracy in our benchmark images, the accuracy of the system corresponds with the one of the neural model.

Fig. 4.

Precision-Recall Curves on test set.

Fig. 5.

An example of inaccurate image acquisition. The side perspective of the image does not match the component images in the catalog leading to inaccurate predictions.

One might wonder whether the ASP component is efficient thus in a further experiment the execution time of the ASP-based component was measured. We generated instances of compliance testing in a range of 6–50 labels (types of components), and of 12–75 components and averaged over 500 samples the execution time needed by our ASP engine DLV2 (Alviano et al. Reference Alviano, Calimeri, Dodaro, Fuscà, Leone, Perri, Ricca, Veltri and Zangari2017) to solve the instances. The results reported in Figure 6 show that our system provides answers in a short time, in the order of seconds for instances sized as real-world ones, and performance is acceptable (avg. 1.93 s, max about 18 s) also for instances of 75 components.

Fig. 6.

Performance of the ASP-Based component (Execution time).