2.1 Object detection models

Object detection is one of the core techniques used in computer vision and image processing. There are several fast and powerful real-time object detection systems, such as Fast r-CNN^{(Reference Girshick7)}, Faster r-CNN^{(Reference Ren, He, Girshick and Sun8)}, Xception^{(Reference Chollet9)}, Yolo^{(Reference Redmon, Divvala, Girshick and Farhadi10)}, or EfficientNet^{(Reference Tan and Le11)(Reference Tan, Pang and Le12)}. These methods have been updated and represent considerable improvements with respect to former CNN models. For example, Fast r-CNN^{(Reference Girshick7)} is an evolution of the VGG16 network, a region-based CNN, using the Caffe framework with multitasking, in which training is carried out in a single stage, thus avoiding any disk storage. Faster r-CNN^{(Reference Ren, He, Girshick and Sun8)} builds on Fast r-CNN by introducing a Region Proposal Network (RPN) with the aim of proposing regions of interest at the same time that feature maps are being generated.

Xception^{(Reference Chollet9)} is also an evolution of the VGG16 network that uses inception layers^{(Reference Szegedy, Liu, Jia, Sermanet, Reed and Anguelov13)}. These are neural network layers that independently look for correlations across channels and at spatial pixels. Xception is based on a linear pipeline of depth-wise separable convolution layers, efficiently implemented in TensorFlow and forms the base of the Facebook object detector software.

Yolo^{(Reference Redmon, Divvala, Girshick and Farhadi10)} is a family of algorithms used in many research studies on object detection. Proposed by Redmon et al., Yolo, named after the slogan “you look only once”, frames object detection as a regression problem to spatially separate bounding boxes and associated class probabilities. Yolo processes images by first resizing the input image, then secondly, a single convolutional neural network runs on the image. Finally, the resulting detections are thresholded based on the model’s confidence. Newer versions of Yolo propose the use of larger neural networks than the precious version and/or show faster execution. Yolo predicts bounding boxes using dimensional clusters as anchor boxes^{(Reference Redmon and Farhadi14)}. Yolo offers the advantage of allowing multi-label predictions; this is, an object can be detected as two (or more) different labels at the same time. In this way, a friendly drone and a malicious drone can both be detected and labelled as drones, while at the same time, if their visual appearance is known, they could also be labelled as a threat or non-threat.

EfficientNet^{(Reference Tan and Le11)} is a brand-new state-of-the-art object detection model that, together with its recent evolution, EfficientNet^{(Reference Tan, Pang and Le12)}, has become very popular in a short time thanks to its accuracy and efficiency. One of the key improvements is its novel Bi-directional Feature Pyramid Network (BiFPN), which allows information to flow in different directions: top down and bottom up. Secondly, EfficientNet uses a fast, normalised fusion technique, which adds an additional weight for each input feature, thus identifying the importance of each input feature. Finally, it introduces a scaling method, which jointly resizes the resolution/depth/width of the model to better fit with different resource constraints. EfficientNet-B0 is a version of EfficientNet adapted for small-size objects.

2.2 Object detection models for on-board UAV processing

Object detection methods have also been implemented to detect objects from UAVs with different target applications such as surveillance and disaster management.

For example, real-time object detection has been performed to detect humans by using videos captured from a UAV^{(Reference Bhattarai, Nakamura and Mozumder15)}, addressing constraints such as computation time, viewing scale and altitude. The results are also visualised in a Geographic Information System platform by geo-localising objects to world coordinates.

In other research, a technique was proposed^{(Reference Rudol and Doherty16)} to detect humans at a high frame rate by using an autonomous UAV. A map of points of interest is built by geo-locating the positions of the detected humans. The video sequences, which are streamed from two video sources, thermal and colour, are collected on board the UAV and fused to increase the successful detection rate.

Furthermore, Yang et al.^{(Reference Yang, Yurtsever, Renganathan, Redmill and ÖzgÜner17)} recently proposed a real-time detection and warning system based on artificial intelligence (AI) to monitor social distancing between people during the coronavirus disease 2019 (COVID-19) pandamic⁽¹⁸⁾. A fixed monocular camera is used to detect individuals in a region of interest (ROI), and the distances between them are measured in real time without data recording. The proposed method was tested across real-world datasets to measure its generality and performance.

2.3 Drone detection models

In 2017, the European SafeShore project, in parallel with the IEEE AVSS conference, launched the Drone-vs-Bird Challenge to improve methods for detecting UAVs close to coastal borders, where they can easily be confused with birds. The challenge aims to correctly label drones and birds appearing in a video stream. From the papers presented in the first and second editions, in 2017 and 2019, respectively, it can be concluded that new advances are being driven by using CNNs with more and more layers, larger training datasets and improved implementations. In addition, the exploitation of temporal information is a key issue in differentiating drones from birds. The best paper of 2019^{(Reference Craye and Ardjoune19)} proposed a 110-layer CNN based on a semantic segmentation U-Net network originally designed for medical image processing, adding some dilation layers to improve detection of small objects. Posterior spatial–temporal filtering allowed an F1-score of $0.73$ to be obtained. The set of training and test images used in those works, although challenging, were taken from the ground with the sky as background.

Drone-Net⁽²⁰⁾ is a deep learning model, available as open source, trained with 2,664 real drone images. It contains 24 convolutional layers in total, two of which are detection layers called Yolo layers. The details of the Drone-Net CNN are shown in Fig. B.1 in the Appendix. This model is considered as the state-of-the-art model for drone detection, and its accuracy results will be compared with those achieved by the new models presented herein.

Xiaoping et al.^{(Reference Xiaoping, Songze, Boxing, Yanhong and Feng21)} proposed a dynamic drone detection method based on two consecutive inter-frame differences. They combine a Support Vector Machines (SVM) classifier with the traditional Histogram of Oriented Gradient (HOG) detection algorithm, and add an intermediate step based on a Fisher Linear Discriminant (FLD) to reduce the dimensionality of the HOG features. Using a dataset of 500 images, their results show an accuracy above $90 \%$ , similar to other drone detection algorithms, but with improvements in terms of the detection time that allow the processing of up to 10 images per second. However, since this method is based on the difference between consecutive images, it is not suitable for a moving camera.

Hu et al.^{(Reference Hu, Duan, Mao, Zhou and Zhou22)} proposed an object detection method, called DiagonalNet, by using an improved hourglass CNN as its backbone network and generating confidence diagonal lines as the detection result. A large dataset (10,974 sample images) was created by processing videos and photos with different backgrounds and lighting, augmented by rotating and flipping each image with random angles. The images contain six types of UAVs, including multirotor and helicopter devices, and are labelled manually. The proposed algorithm detects UAV quickly (at 31 images per second) and accurately (with mean average precision above 90%). However, the experiments were all carried out indoors and in close proximity to the target drone.

With the objective of improving swarm cooperative flight and low-altitude security, Jin et al.^{(Reference Jin, Jiang, Qi, Lin and Song23)} proposed a Six-Dimensional (6D) pose estimation algorithm for quadrotors. The proposed algorithm, based on the Xception network and pre-trained with ImageNet, includes a novel Relational Graph Network (RGN) to improve the performance of the drone pose detection. The pose is obtained by recognising eight key points (nose, rotors, etc.) of a quadrotor. After training with 340 images, simulation experiments showed a mean average precision of $0.94$ in position and $0.74$ in velocity, representing an enhancement of $9.7\%$ compared with the baseline network. Given the real-time capabilities of the algorithm (30 frames per second), it was also tested with real flights. However, the mean average precision dropped to $0.65$ – $0.75$ . Moreover, the accuracy of the 6D pose results decreased for small-sized drones.

Carrio et al.^{(Reference Carrio, Tordesillas, Vemprala, Saripalli, Campoy and How24)} used AirSim to train a CNN detection network with 16 layers by automatically labelling depth maps. Depth maps were obtained by stereo-matching of the Red–Green–Blue (RGB) image pairs of the virtual ZED stereo camera on the AirSim drone. The ground-truth labels of the depth maps were generated automatically by colour segmentation of the visual image. After training with 470 images, the detection system was integrated on board a small drone and tested while the drone navigated in the environment. The results showed that the system can simultaneously detect drones of different sizes and shapes with mean average precision of 0.65–0.75 and localise them with a maximum error of 10% of the truth distance when flying in linear motion encounters. The solution is limited to a maximum distance of 8m with relative speeds up to 2.3m/s.

Wyder et al.^{(Reference Wyder, Chen, Lasrado, Pelles, Kwiatkowski, Comas, Kennedy, Mangla, Huang, Hu, Xiong, Aharoni, Chuang and Lipson25)} presented a UAV platform to detect and counter a small UAV in an indoor environment where Global Positioning System (GPS) is not available. An image dataset is used to train a Tiny Yolo object detection algorithm. This algorithm, combined with a simple visual-servoing approach, is also validated in a physical platform. It successfully tracked and followed a target drone, even with an object detection accuracy limited to 77%.

4.1 Tools

The following training tools are used for model creation:

• • AirSim simulator

Figure 1.

AirSim training images.

AirSim^{(Reference Shah, Dey, Lovett and Kapoor27)} is a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles such as cars or drones. AirSim is built as an Unreal Engine⁽²⁸⁾ plugin. Unreal Engine provides ultra-realistic rendering and strong graphical features for AirSim. Many environments are available for use in AirSim. In this work, the suburban neighbourhood is selected to capture images.

• • Darknet framework

Darknet^{(Reference Redmon29,30)} is a framework for training and testing of neural networks, written in C. The C language provides an efficient solution for general object detection in real time. We use Yolo-V3^{(Reference Redmon and Farhadi31)} with Darknet-53, the original 53-layer CNN shown in Fig. A.1, which achieves the highest measured floating point operations per second. In addition to the implementation of the algorithms, the Darknet framework also provides several pre-trained CNN models.

• • Local desktop computer and Google Cloud Platform

The proposed models are trained on a desktop with an NVIDIA GeForce GTX 1060 graphical processing unit (GPU) with 6GB RAM graphic co-processor and Intel i7 processor with 16GB of memory. In addition, the Google Cloud Platform provides a colaboratory service^{(Reference Bisong32)} (“Google Colab”), which allows you to write and execute python in an internet browser. Google Colab allows the execution of codes on Google’s cloud serves, which include powerful hardware including GPUs and tensor processing unit (TPUs). Neural networks were also tested on a Google cloud server with a Tesla T4-16GB GPU.

4.2 Drone image dataset and auto-labelling

Training a CNN requires a large dataset of labelled images. In this work, new drone images for the training dataset were captured by using the AirSim simulator. AirSim provides a public Application Programming Interface (API) for receiving parameters related to the drone and environment. We created a dataset of 2,000 images for training, $1,280\times960$ in size, captured in AirSim by using a drone flying randomly in the environment. Then, the captured images are auto-labelled by mapping the drones position in world coordinates to the image coordinates. Some of the image samples used in training can be seen in Fig. 1.

The procedure for projecting from 3D world coordinates to the image plane includes a few steps, which are explained in detail below:

• • The pinhole camera model

The pinhole camera model defines the geometric relationship between a 3D point in the scene and its corresponding 2D projection onto the image plane. This geometric mapping from 3D to 2D is a perspective projection. In the AirSim simulator, a pinhole camera model is available and mounted onto the drones for capturing images. The pinhole camera models in AirSim do not include the geometric distortion caused by lenses. Figure 2 shows a schematic view of the pinhole camera projection. The following paragraphs explain the different coordinate systems in more detail:

• • Forward projection

Figure 2.

Pinhole camera projection visualisation.

Figure 3.

Projection summary.

The order of the forward projection is shown in Fig. 3. Firstly, the world coordinates of the drone which is found in the image are converted to camera coordinates by using quaternions and a rotation matrix. The rotations are described as a yaw–pitch–roll sequence, and the rotation matrix can be obtained by using the Euler angles which are available in the simulator as shown in Equation (1).

Quaternions are applied to the coordinate rotations and to relate them to the Euler angles^{(Reference Stevens, Lewis and Johnson33)}. The quaternion for a yaw–pitch–roll sequence are presented in Equation (2)

(1) {\fontsize{8}{10}\selectfont\begin{align}R = \left[\begin{matrix}\cos (\theta) \cos (\psi)&\quad \cos (\theta) \sin (\psi) &\quad - \sin (\theta) \\ \\[-7pt]-\cos(\phi) \sin(\psi)+\sin(\phi) \sin(\theta) \cos(\psi)&\quad \cos(\phi) \sin(\psi)+\sin(\phi)\sin(\theta) \sin(\psi)&\quad \sin(\phi) \cos(\theta) \\ \\[-7pt]\sin(\phi) \sin(\psi)+\cos(\phi) \sin(\theta) \cos(\psi)&\quad -\sin(\phi) \cos(\psi)+\cos(\phi)\sin(\theta) \sin(\psi) &\quad \cos(\phi) \cos(\theta) \end{matrix}\right]\end{align}}

(2) \begin{equation} q=\begin{bmatrix} \cos(\frac{\phi}{2}) \cos(\frac{\theta}{2}) \cos(\frac{\psi}{2}) + \sin(\frac{\phi}{2}) \sin(\frac{\theta}{2}) \sin(\frac{\psi}{2}) \\ \\[-7pt] \sin(\frac{\phi}{2}) \cos(\frac{\theta}{2}) \cos(\frac{\psi}{2}) - \cos(\frac{\phi}{2}) \sin(\frac{\theta}{2}) \sin(\frac{\psi}{2}) \\ \\[-7pt] \cos(\frac{\phi}{2}) \sin(\frac{\theta}{2}) \cos(\frac{\psi}{2}) + \sin(\frac{\phi}{2}) \cos(\frac{\theta}{2}) \sin(\frac{\psi}{2}) \\ \\[-7pt] \cos(\frac{\phi}{2}) \cos(\frac{\theta}{2}) \sin(\frac{\psi}{2}) - \sin(\frac{\phi}{2}) \sin(\frac{\theta}{2}) \cos(\frac{\psi}{2}) \end{bmatrix} \end{equation}

The drone captures an image of the other drone, which is visible in the camera at certain angles when the field of view of the camera is less than $60^\circ$ . Secondly, the image coordinates are obtained by using the perspective projection of the camera, which uses the camera matrix received from the simulator. Finally, the pixel values are calculated by moving the origin to the upper-left corner of the screen. The mapping geometry is presented in Fig. 2.

Figure 4.

Darknet-53 CNN used as backbone for model 2.

4.3 Transfer learning

The main purpose of Transfer Learning (TL) is to improve the learning performance by using the experience from successfully pre-trained models^{(Reference Taylor and Stone34)}. TL can be used for different goals and in different situations. For instance, Drone-Net is built by using transfer learning to refine a Darknet model for detecting drones.

4.4 Proposed models

In this section, the new models proposed to improve the current results of Drone-Net are presented.

• • Model 1 uses the same CNN architecture as Drone-Net, shown in Fig. B.1 in the Appendix. From the pre-trained Yolo network, up to 16 convolutional layers are transferred. After transferring these convolutional layers, the network is trained with 2,000 new images obtained from the AirSim simulator and another 1,000 images taken from the Drone-Net training set. The main purpose is to use the pre-trained weights from Drone-Net with the hope that the new model can detect real drones as well as drones from AirSim images.

• • Model 2 is built by using the pre-trained weights from Darknet-53 for the neural network model based on the Yolo-v3^{(Reference Redmon and Farhadi31)} architecture. The neural network model is based on the Yolo-v3^{(Reference Redmon and Farhadi31)} architecture. In this model, the Darknet-53 model is implemented and the default Yolo-v3 network is modified to detect only the drone class. The Yolo-v3 network shown in Fig. 4 contains 107 layers: 75 convolutional layers, 23 shortcut layers, 4 routes, 2 upsamples and 3 Yolo detection layers. The predictions in Fig. 4 show that Yolo-v3 detects objects in three different layers. The model summary can be seen in Fig. C.1 of the Appendix. In this model, up to 74 convolutional layers from Darknet-53 model are transferred, then the training is extended with the 3,000 images: 2,000 images from the AirSim simulator as before, plus another 1,000 images taken from the Drone-Net training set.

• • Model 3 is constructed and optimised by using the EfficientNet-B0 object detection algorithm to detect a drone. EfficientNet-B0 contains 145 layers and 2 detection layers. Up to 132 convolutional layers from Efficient-D0 model are transferred. The EfficientNet-B0 model summary used in training can be seen in Fig. 15 in the Appendix. The training set includes the 3,000 images: 2,000 images from the AirSim simulator and 1,000 images taken from the Drone-Net training set. The models proposed here have backbone networks such as Drone-Net, Darknet-53 and EfficientNet-B0. The details of these backbones are presented in Table 1. In addition, in Table 2, the model details are explained. For instance, model 1 uses up to 16 Drone-Net backbone network convolutional layers, and model 2 has a backbone network from Darknet-53 with up to 74 convolutional layers. Model 3 has more layers in total than the other models thanks to the EfficientNet-B0 model, which has 145 layers in total, and it uses up to 132 convolutional layers from EfficientNet-B0.

Table 1

Backbone models used in training

Table 2

Model details for training (all 6,000 iterations)