2.1 Object detection

Target detection is an important research direction of computer vision, which is used to lock the target position in a single-frame image. Research on ship detection began in 2002, code-named ‘Spartan scout,’ in the form of a project to self-control a pilotless ship. The ship was equipped with a ship detection system that could detect and track ship objects at sea (Rao et al., Reference Rao, Wang, Hu and Mullane2014). Later, Guang et al. (Reference Guang, Qichao and Feng2011) proposed a method of region segmentation to extract the features of different segmentation regions; then, a machine-learning algorithm was used to detect the ships in each region of the image. Kim et al. (Reference Kim, Park and Yu2010) used background estimation to enable ship detection and merged with AIS to implement relevant ship information matching. Liu (Reference Liu2010) detected ships in an inland waterway by separating the sky–water line and incorporating the average ship features in the region of interest.

In recent years, deep learning has been developing rapidly and is widely used in the field of object detection. Object detection algorithms are divided into two categories: object detection models based on classification, including region-based convolutional neural network (R-CNN) (Girshick et al., Reference Girshick, Donahue, Darrell and Malik2014), Fast R-CNN (Girshick, Reference Girshick2015) and Faster R-CNN (Ren et al., Reference Ren, He, Girshick and Sun2017), and detection models based on regression, including the YOLO (Redmon et al., Reference Redmon, Divvala, Girshick and Farhadi2016) and single shot multibox detector (SSD) (Liu et al., Reference Liu, Anguelov, Erhan, Szegedy, Reed, Fu and Berg2016) series.

Object detection methods based on deep learning have been widely used in various fields, including ship detection. Betti et al. (Reference Betti, Michelozzi, Bracci and Masini2020) constructed a YOLOv3 detector based on the Keras application programming interface to detect four types of ships: cargo ships, warships, oil tankers and tugboats. He et al. (Reference He, Yi, Mu and Zhang2019) designed a method based on the combination of a Gabor filter and Faster R-CNN to effectively improve the accuracy of ship object detection in satellite images. Wang et al. (Reference Wang, Wang, Hong, Cheng and Fu2017) proposed a ship detection method based on the SSD algorithm and transfer learning according to different image input sizes; two models, SSD-300 and SSD-500, were designed to test the detection performance. Guo et al. (Reference Guo, Yang, Wang, Song and Gao2020) proposed a rotational Libra CNN method and introduced the concept of a balanced feature pyramid to improve the detection effect for different sizes of ships. At present, deep-learning algorithms are widely used in ship object detection. However, in real circumstances, the types of ships are diverse, their sizes are different and the water traffic environment is complex; in addition, it is necessary to consider recognition under night visual conditions, which poses a huge challenge to water object detection.

2.2 Object tracking

Object tracking is a key technology used to solve the real-time locking of moving objects in video feeds. Classic object-tracking algorithms include mean shift, particle filter and correlation filter. These methods mainly model the current object region and then find similar regions in the next image. Well-known correlation filtering algorithms include circulant structure tracking with kernels (CSK) (Henriques et al., Reference Henriques, Rui, Martins and Batista2012) and kernelised correlation filter (KCF) (Henriques et al., Reference Henriques, Caseiro, Martins and Batista2015), both of which determine the moving direction of an object by finding the relationship between two adjacent frames.

Compared to general motion environments, water traffic environments are more complex, including the partial or complete occlusion of adjacent ships, attitude and illumination changes, sudden scale changes and motion blur (Chen et al., Reference Chen, Yuan, Wu, Zhang and Zheng2014; Li and Zhu, Reference Li and Zhu2014; Li et al., Reference Li, Tian, Zuo, Zhang and Yang2018). Chen et al. (Reference Chen, Li, Tian and Chao2017) used the difference in the grey peak to detect the position of a ship and tracked ships based on the mean-shift algorithm. Dong et al. (Reference Dong, Zheng, Li, Tian and Liu2019) designed an object-tracking algorithm based on an improved KCF, in which a Kalman filter module was added to predict the position of the ship tracking object in the next frame, and the concept of object-tracking critical probability was used to evaluate whether the object tracking was abnormal. However, in maritime video feeds, because of occlusion, ships are often not detected. Cheng et al. (Reference Cheng, Zhilin and University2019) improved the KCF algorithm by adding a scale-transform frame and tracking-effect detection method to solve the object occlusion problem or tracking failures caused by object detection. Chen et al. (Reference Chen, Xu, Yang, Wu, Tang and Zhao2020) proposed an enhanced ship-tracking framework based on KCF and a curve-fitting algorithm and used a data anomaly detection and correction program to correct the positions of occluded ships.

With the development of deep-learning algorithms, such algorithms have been applied to object tracking. This method trains a CNN model to predict the object region in the next frame (Dorai et al., Reference Dorai, Chausse, Gazzah and Amara2017; Pang et al., Reference Pang, Coz, Yu, Luaces and Diez2017). Recently, such algorithms have appeared in maritime applications, where researchers (Shan et al., Reference Shan, Zhou, Liu, Zhang and Huang2020) have used a modified Siamese network combined with multi-region proposal networks to build a tracking pipeline to track maritime ships. ResNet-50 with a feature pyramid network structure was used as the CNN of the Siamese detection subnetwork.

However, research concerning multi-object ships is relatively sparse. Vivone et al. (Reference Vivone, Braca and Horstmann2015) used a prior knowledge-based multi-object tracking method, prior information given by the ship channel, and its related motion model as the basic components of a variable structure interactive multi-model process and used the joint probabilistic data association rules to deal with false and missed detections. Xiao and Gang (Reference Xiao and Gang2011) employed the continuously adaptive mean shift (camshaft) multi-ship tracking algorithm and multi-feature adaptive fusion based on colour, shape and texture to improve the robustness of the model.

3.1 Increased contrast at night

Computer vision began to dabble in maritime video-surveillance and finds many both civilian and military applications (Grimaldi et al., Reference Grimaldi, Bechar, Lelore, Guis and Bouchara2015). Therefore, it is necessary to improve the image quality before further analysis. Yang et al. (Reference Yang, Nie and Liu2019) estimated the initial brightness through Max-RGB and then refined it with the WLS filter. Based on the well-constructed brightness, the enhanced image at a low light level was obtained. In the inland waterway, the problem of night image detection mainly came from the low contrast between the ship target and background.

In this paper, we chose the RGB colour space based on the visible light image. The differences in red (R), green (G) and blue (B) in three dimensions cause the image to present different colour representations; however, it can be difficult to express the deep image characteristics. By constructing the pixel matrices hue (H), saturation (S) and brightness (V), the image can be easily quantified according to its colour, saturation and brightness. For image P, as shown in Figure 2(a), the corresponding S and V grey matrices are calculated as

(3.1)\begin{equation}\left\{ \begin{array}{@{}l@{}} {{S_{ij}} = 1 - \dfrac{{3\,\textrm{min}({{R_{ij}},{G_{ij}},{B_{ij}}} )}}{{{R_{ij}} + {G_{ij}} + {B_{ij}}}}\; \; \; \; }\\ {{V_{ij}} = \dfrac{1}{{\sqrt 3 }}\textrm{(}{R_{ij}} + {G_{ij}} + {B_{ij}}\textrm{)} } \end{array} \right.\end{equation}

where ${R_{ij}}$, ${G_{ij}}$, and ${B_{ij}}$ represent the three colour weights for the pixel coordinates $({i,j} )$ after normalisation.

Figure 2.

Image enhancement process. (a) Original image. (b) Gradient characteristic image. (c) Greyscale image of gradient characteristic image. (d) Saturation image. (e) Brightness image. (f) Grey image after enhancement processing

A generic ship tends to show high saturation and low brightness in night images; accordingly, the pixels in the ship region have higher S and lower V. The surface region in the background, conversely, exhibits lower S and higher V, e.g., ${S_{\textrm{boat}}} > {S_{\textrm{river}}}$ and ${V_{\textrm{boat}}} < {V_{\textrm{river}}}$. At the same time, because of the influence of the illumination on both sides of the water surface, the pixels have high gradient characteristics; therefore, the Sobel operator can be used to extract the gradient image of $P:L$, as shown in Figure 2(b), and then transform it into the grey pixel matrix $L\mathrm{^{\prime}}$, as shown in Figure 2(c). Then, the contrast of the water region pixels in S and V can be enhanced:

(3.2)\begin{equation} \begin{cases} {S_{ij}} = \dfrac{1}{{\textrm{exp}\left( {\frac{{L{^{\prime}_{ij}}}}{{{P_{\textrm{max}}}}}} \right)}}{S_{ij}}\\ {{V_{ij}} = \textrm{min}\left( {{V_{ij}} \times \textrm{exp}\left(\dfrac{{L{^{\prime}_{ij}}}}{{{P_{\textrm{max}}}}}\right),\quad {P_{\textrm{max}}}} \right)}\end{cases} \end{equation}

where ${P_{\textrm{max}}} = 255$. The processed greyscale images of S and V are shown in Figure 2(d) and (e), respectively. It is obvious from the figure that the ship region is more obvious in the saturation image S than in the brightness image V. The ship region can be enhanced by subtracting the two images:

(3.3)\begin{equation}{T_{ij}} = {S_{ij}} - \textrm{min}({S_{ij}}, \ V_{ij} - \varepsilon ).\end{equation}

where ${T_{ij}}$ = grey image after enhancement processing and the parameter $\varepsilon = 40\varepsilon$. The grey image T is shown in Figure 2(f). Now that the ship region is clearly expressed, the contrast enhancement calculation for the object region can be carried out using

(3.4)\begin{equation}{P_{ij}} = \left\{ {\begin{array}{*{20}{l}} {\textrm{min}\left( {{P_{\textrm{max}}} \times {{\left( {\dfrac{{{P_{ij}}}}{{{P_{\textrm{max}}}}}} \right)}^\beta },\ {P_{\textrm{max}}}} \right)}& {{T_{ij}} > 0}\\ {{T_{ij}}}& {{T_{ij}} = 0} \end{array}} \right.\end{equation}

where ${P_{ij}}$ = pixel after contrast enhancement and $\beta$ = controllable parameter, where $\beta = 1.15$. Each pixel in T is compared with 0, and the ship region is segmented from T. Then, the RGB component of the corresponding position in the original image P is exponentially calculated. The processed image P is shown in Figure 2(f). The ship object in the figure shows higher contrast compared with the background region.

3.2 Self-attention mechanism

The self-attention mechanism is a recently proposed improved machine-learning method that enables the model to fully consider the relationship between the features of the samples in training and enhances the feature learning of key regions. This mechanism has been widely used in natural language processing and image learning (Cao et al., Reference Cao, Chen, Guo and Shi2020). Because of the small size of the convolution kernel, CNNs only capture regional features with the same size and cannot effectively reflect the relationships between pixels in different regions. In this paper, we improve the network structure of SSD, based on the self-attention mechanism, so that it can analyse the correlations of all the pixels in the different scale feature maps, making the correlation of the extracted features stronger.

First, we input the feature map x with a structure $c \times w \times h$, use a 1 × 1 convolution kernel to convolute twice, and compress the channel number according to $c = c \div 8$ and obtain matrices F and G with structures $c^{\prime} \times w \times h$, which are then transformed into $c^{\prime} \times n$, where $n = \; w \times h$. The third convolution operation is performed using a $1 \times 1$ convolution kernel without channel number compression, and a matrix H with a $c \times w \times h$ structure is obtained, which is then transformed into $c \times n$. The matrices F and G are used to calculate the attention matrix

(3.5)\begin{equation}{e_{i,j}} = \frac{{\textrm{exp}({s_{ij}})}}{{\mathop \sum \nolimits_{i = 1}^N \textrm{exp}({s_{ij}})}},\; \quad \textrm{where} \; {s_{ij}} = f{({X_j})^T}g({{X_i}} ),\end{equation}

where ${e_{i,j}}$ = element values of the attention matrix E, which has dimensions $n \; \times \; n$; ${s_{ij}}$ = relative weight of the i position to the j position; N = number of elements in the characteristic graph matrix; and $\; f({{x_i}} )= {W_f}{x_i}$, $g({{x_j}} )= {W_g}{x_j}$, where ${W_f}$ and ${W_g}$ represent $1 \times 1$ convolutions.

We continue to multiply the matrices H and E to obtain a matrix with dimensions $c \times n$ as the self-attention map:

(3.6)\begin{equation}{q_{i,j}} = y(h({X_i})e({X_j})),\end{equation}

where ${q_{i,j}}$ = elements in the matrix Q and y = learning parameter, which is used to represent the dependence of the network on the relationship between the long-distance regions. $h({{x_j}} )= {W_h}{x_i}$, where ${W_h}$ is a $1 \times 1$ convolution.

The matrix Q is expanded into a characteristic graph with a $c \times w \times h$ structure as the output result of the self-attention module, as shown in Figure 3. The input structure is a characteristic graph with a $c \times w \times h$ structure. Because the structure of the entire process characteristic graph remains unchanged, the self-attention module can be directly added to the convolution network without changing its basic structure.

Figure 3.

Calculation method for a self-attention characteristic graph

3.3 The detector

The basic network of SSD was improved compared with VGG16. The full connection layer of the original VGG16 was deleted, and the depth of the convolution layer was increased. In addition, four convolution layers were added after the network to extract the network features. In the process of the network moving forward, a series of prediction scores of bounding box sets and categories are generated and then the final detection results are determined via nonmaximum suppression (NMS). As the depth of the network increases, the size of the feature map decreases and the NMS layer collects the object information at each scale; therefore, the default box generated by SSD is multiscale.

As shown in Figure 4, based on the SSD network, the self-attention module is added after the 4th, 7th, 8th, 9th, 10th and 11th layers. The output characteristic diagram of the original layer is taken as the input of the self-attention module, and the corresponding feature map is the output. With the deepening of the network, the scale of the generated feature map decreases, and different scale feature maps produce different default boxes. NMS is used again to fuse the self-attention feature maps of the different receptive fields, and the final detection result is calculated.

Figure 4.

Improved SSD network

3.4 The tracker

This component of the algorithm must continuously track all the ship objects detected by the detector. Multiple-object tracking (MOT) is used to track and extract the trajectories of multiple objects of interest in a video sequence. As opposed to single-object tracking, MOT must identify different objects in each frame. For new objects, it must generate new trajectories, and for objects that have left the field of view of the camera, it must terminate the tracking of their trajectories.

The traditional multi-object tracking algorithm is based on correlation filtering, with its representative algorithm being a KCF-based multi-object tracking algorithm (Henriques et al., Reference Henriques, Caseiro, Martins and Batista2015) that uses multithreading to track multiple single objects. The SORT (Bewley et al., Reference Bewley, Ge, Ott, Ramos and Upcroft2016) algorithm proposed in 2016 regards multi-object tracking as a data association problem. SORT is based on accurate object detection, according to the location information in the detection box, and uses Kalman filtering and the Hungarian algorithm to match the tracking objects of the front and back two frames of the image. However, because SORT ignores the surface characteristics of the object being detected, it is easy to lose an object.

DeepSORT (Wojke et al., Reference Wojke, Bewley and Paulus2017) has been improved, combining object motion and surface feature information and using the Mahalanobis and cosine distances as a measure of the similarity between the motion and depth features within the detection box, while at the same time using the Hungarian algorithm to cascade match the predicted track with the detections in the current frame, giving priority to the objects that have not been lost. Then, use the minimum IOU threshold to filter the low confidence matching to reduce error matching, and set a threshold to remove trackers that loop too many times. The algorithm structure is shown in Figure 5.

Figure 5.

DeepSORT algorithm processes

3.5 Multimodal data fusion

1. (1) Perspective transformation

Perspective transformation is the process of projecting an image onto a new view plane by establishing a projection matrix. The projection process is the product of the coordinate vector and the transformation matrix, as

(3.7)\begin{equation}[{x^{\prime} \; \; y^{\prime} \; \; w^{\prime}} ]= [{u \; \; v \; \; w} ]\left[ {\begin{array}{*{20}{c}} {{a_{11 \; \; }}\; {a_{12 \; \; \; }}{a_{13\; }}}\\ {{a_{21 \; \; }}\; {a_{22 \; \; \; }}{a_{23\; }}}\\ {{a_{31 \; \; }}\; {a_{32 \; \; \; }}{a_{33\; }}} \end{array}} \right],\end{equation}

where u and v = coordinates of the original plane; the parameter $w = 1$; the perspective matrix contains the zoom, rotation and translation operations of the plane; and ${a_{33}} = 1$. The transformed coordinates are represented as ($u\mathrm{^{\prime}},v\mathrm{^{\prime}}$), where $u\mathrm{^{\prime}} = x\mathrm{^{\prime}}/w\mathrm{^{\prime}}$ and $v\mathrm{^{\prime}} = y\mathrm{^{\prime}}/w\mathrm{^{\prime}}$. Entering this into Equation (3.7), we obtain Equation (3.8). According to this formula, the parameters in the perspective matrix can be calculated by simply identifying four sets of key points:

(3.8)\begin{equation}\left\{ {\begin{array}{@{}l} {u^{\prime} = \dfrac{{{a_{11}} \times u + {a_{21}} \times v + {a_{31}} \times 1}}{{{a_{13}} \times u + {a_{23}} \times v + 1 \times 1}}}\\ {v^{\prime} = \dfrac{{{a_{12}} \times u + {a_{22}} \times v + {a_{32}} \times 1}}{{{a_{13}} \times u + {a_{23}} \times v + 1 \times 1}}} \end{array}} \right..\end{equation}

In this paper, the principle of perspective transformation is used to project the image coordinate system onto the longitude and latitude coordinate system. The process of transforming the pixel coordinate to longitude and latitude coordinates is called the forward perspective, while the opposite is called the reverse perspective. If we reverse perspective the plane of the river water, the effect is shown in Figure 6, which demonstrates the reverse perspective effect of a square virtual grid with a side length of 0⋅00045 in the longitude and latitude coordinate system.

1. (1) Ship position estimation

Figure 6.

Perspective effect of a virtual grid in world coordinates on an image

The previous steps transform any pixel coordinate into longitude and latitude coordinates on the river surface in an image. However, the longitude and latitude data in AIS are confirmed by the shipborne GPS, which is often close to the bow or stern of the ship; therefore, there may be a gap of tens of meters between the two sets of coordinates. This paper takes the ship centre as the estimated positioning point and marks the pixel coordinates of the ship centre position corresponding to the river surface in the video.

This is shown in Figure 7, where the point O on the long side of the detection box is the selected water surface projection point. A ray is traced along the angle $\mathrm{\theta }$ to the right boundary and intersects with the boundary box at the point F, where $\mathrm{\theta }$ is obtained by the DeepSORT tracker, such that the line segment $OF$ can approximately represent the waterline of the ship. The position of B in world coordinates is

(3.9)\begin{equation}P_B^\mathrm{^{\prime}} = \frac{1}{2} \cdot ({\textrm{Transform}({{P_O}} )+ \textrm{Transform}({{P_F}} )} ), \; \end{equation}

where P is used to represent the pixel coordinates of each point and $\textrm{Transform}\; ()$ represent the forward perspectives. From point B along the vertical travel direction, a ray is traced into the detection box to the ship positioning point S. The length of this line segment in world coordinates, $l_{BS}^{\prime}$, is determined by the length/width ratio of the ship, such that

(3.10)\begin{equation}l_{BS}^{\prime} = \frac{1}{2} \cdot \delta \cdot \textrm{Distance}({\textrm{Transform}({{P_O}} ),\textrm{Transform}({{P_F}} )} ),\end{equation}

where $\delta$ represents the length/width ratio, which for bulk carriers, tankers and river ships is approximately $1:6$ and for yachts, cruise ships and tugboats is approximately $1:3$; $\textrm{Distance}\; ()$ is used to calculate the Euclidean distance between two points. On this basis, the position S in the world coordinate system $P_S^\mathrm{^{\prime}}$ can be obtained as

(3.11)\begin{equation}P_S^\mathrm{^{\prime}} = P_B^\mathrm{^{\prime}} + ({l_{BS}^{\prime} \cdot \textrm{sin}(\beta ),\; \; l_{BS}^{\prime} \cdot \textrm{cos}(\beta )} ),\; \end{equation}

where $\beta$ = azimuth of point S with respect to point B in world coordinates. Then, we can obtain the pixel coordinates of the point $S$:

(3.12)\begin{equation}{P_S} = \textrm{Transform}^{\prime}({P_S^\mathrm{^{\prime}}} ),\end{equation}

where $\textrm{Transform}^{\prime}()$ = reverse perspective.

1. (1) AIS signal prediction and data fusion

Figure 7.

Estimation method for the ship GPS positioning points

Next, we demarcate the object region within the longitude and latitude coordinate system. We receive the dynamic AIS information in the object region and extract the Maritime Mobile Service Identity (MMSI) number, longitude and latitude, direction and speed information. We track the latest signal points of the ship according to the MMSI number, where ${D_{mmsi,t}} = \{{\textrm{lon},\textrm{lat},\textrm{sog},\textrm{cog}} \}$. The AIS signal is updated only every few seconds and is out of sync with the video signal; therefore, it is necessary to track the ship and continuously predict its position at every moment ${A_{mmsi,t^{\prime}}} = \{{\textrm{lon}^{\prime},\textrm{lat}^{\prime}} \}$ according to ${D_{mmsi,t}}$, such that

(3.13)\begin{align}\textrm{lon}^{\prime} & = \textrm{lon} + ({t^{\prime} - t} )\cdot \textrm{sog} \cdot \textrm{sin}({\textrm{cog} \cdot \pi /180} ),\end{align}

(3.14)\begin{align}\textrm{lat}^{\prime} & = \textrm{lat} + ({t^{\prime} - t} )\cdot \textrm{sog} \cdot \textrm{cos}({\textrm{cog} \cdot \pi /180} ),\end{align}

Therefore, the position of AIS objects at each moment are expressed as ${P_A}$, the position of video ship objects after the positive perspective transformation are expressed at the same moment as $P_S^\mathrm{^{\prime}}$, two points constitute the vector ${R_1} = \overrightarrow {P_S^\mathrm{^{\prime}}{P_K}}$. Their forward directions are represented by the vector ${R_2} = P_S^\mathrm{^{\prime}} + \overrightarrow {P_O^\mathrm{^{\prime}}P_F^\mathrm{^{\prime}}}$, and we calculate the distance between the two as

(3.15)\begin{equation}d = \textrm{Normalisation}({P_S^\mathrm{^{\prime}},\; {P_A}_2} )- \left|{\frac{{{R_1} \cdot {R_2}}}{{{R_1}{R_2}}}} \right|+ 1,\end{equation}

where $\textrm{Normalisation}()$ represents a normalised function, the first half of the formula represents the Euclidean distance between two points, and the second half represents the cosine similarity between the vectors. The range of values of distance is $[{0,2} ]$.

Using the near-matching mechanism, information pairs with smaller distances can be matched first.