3.1. Adaptive dilated convolution feature extraction model

Let the input features be $X^{(i)} \in \mathbb{R}^{H \times W \times D}$ , where $i=1, \ldots, N$ , N is the total number of scene elements in the dataset; H and W are the height and width of the image, respectively; D is the number of channels of the image. Firstly, we need to extract the features from the input elements $X^{(i)}$ . The feature extraction model is as follows:

(1) \begin{equation} \begin{array}{c} \displaystyle\left \{\alpha ^{(i)}, \beta _{1}^{(i)}, \beta _{2}^{(i)}, \ldots, \beta _{J}^{(i)}\right \}=P_{1}\left (X^{(i)} ; \xi _{1}\right ) \\[5pt] \displaystyle\alpha ^{(i)} \in \mathbb{R}^{D_{\alpha }} \\[5pt] \displaystyle\beta _{j}^{(i)} \in \mathbb{R}^{D_{r}, j}=1, \ldots, J \end{array} \end{equation}

where $\alpha ^{(i)}$ is the feature vector of $D_{\alpha }$ dimension, which encodes the semantic information of the overall scene. $\left \{\beta _{j}^{(i)}\right \}_{j=1, \ldots, J}$ is the feature vector of $D_{\beta }$ dimension, which encodes the local semantic information. J is the number of feature regions in the scene, and $\xi _{1}$ is the parameter of feature encoding model $P_{1}$ .

In the process of underwater archaeology, underwater heritage exists buried by mud and sand, and some features are missing, which brings great challenges to the extraction of features. Extracting the overall scene features $\alpha ^{(i)}$ and local features $\left \{\beta _{j}^{(i)}\right \}_{j=1, \ldots, J}$ is a prerequisite for generating accurate scene parsing. Convolutional neural network (CNN) can extract high-level semantic features better. However, in the standard convolution, the field of receptive is fixed, and features with different sizes of objects cannot be extracted simultaneously. In contrast, dilated convolution adjusts the size and shape of the convolution block by using the learned dilated coefficients, and has a flexible size and shape of the receptive field. At the same time, multi-scale feature information is extracted without loss of spatial resolution. To extract both overall and local features, we propose an adaptive dilated convolutional network. In the paper, the proposed method removes the two pooling layers in the convolutional neural network to obtain more detailed local features, to obtain higher-resolution feature maps. This paper adaptively adjusts different dilated coefficients according to task requirements. For small target classes requiring detailed feature learning, a smaller dilated coefficient (less than 1) is used to learn more detailed local features $\left \{\beta _{j}^{(i)}\right \}_{j=1, \ldots, J}$ . For large target classes that require global features, use a larger dilated coefficient (greater than 1) to learn the broader overall feature $\alpha ^{(i)}$ .

Figure 2.

Overview of adaptive dilated convolution.

Figure 2 provides the overview of the proposed adaptive dilated convolution. Assume that the input and output features of the dilated convolution are $X^{(i)} \in \mathbb{R}^{H \times W \times D}$ and $F^{(i)} \in \mathbb{R}^{H \times W \times D}$ , respectively. The dilated coefficients are learned from an additional regression layer that takes the feature map X as input and outputs an dilated coefficient map $\mathrm{R} \in \mathbb{R}^{H \times W \times 8}$ . The size and shape of receptive field are adjusted by the vector coefficients in R. Since the position of the dilation coefficient vectors is adaptive, the dilated coefficient maps R has the same size as the output feature maps F. Thus, all the convolution patches have their respective dilated coefficient vectors to obtain the receptive field of the target size and shape.

We use formula (2) to initialize the dilated regression layer:

(2) \begin{equation} \left \{\begin{array}{l} \displaystyle\eta _{0}(a)=\varepsilon \\[5pt] \displaystyle\mu _{0}=1 \\[5pt] \displaystyle\varepsilon \sim \mathcal{N}\left (0, \sigma ^{2}\right ),\,\, \sigma \ll 1 \end{array}\right. \end{equation}

where $\eta _{0}$ and $\mu _{0}$ denote the initialized deviations of the convolution kernel and the regression layer, and $a$ denotes the position in the convolution kernel. The convolution kernel is set to a value close to 0 and the bias is 1, so the generated dilated coefficients are almost approach to 1. The training starts with the standard convolution, and the adaptive dilated coefficients are progressively learned during the training process to obtain the appropriate values.

Reconstructing convolutional patches extracts a set of patches from the feature map, each of which has a certain shape and size. The number, size, and shape of the convolution patches are related to the size of the input feature map and the receptive field size of the network, which are adaptively reshaped based on the dilated coefficient R. These patches are then reconstructed into a new feature map by interpolating between the patches to fill in the missing pixels. (The missing pixels in the patches refer to the areas in the reconstructed feature map that may have gaps or pixel values that were not directly observed or sampled. These missing pixels occur because the patches are extracted from the original feature map, and during the reconstruction process, interpolation is used to fill in the gaps between the patches). This approach allows the network to consider larger scenes when performing convolutional operations and can increase the perceptual field of the feature map. In traditional convolutional neural networks, reducing the size and resolution of the feature maps by using operations such as pooling layers and step convolution can lead to loss of information and reduced resolution. To overcome this problem, a bilinear sampling approach is used. The feature vectors are then sampled from the convolution patch in a bilinear interpolation to perform element-by-element multiplication with the kernel. We apply the adaptive dilated convolution to the last layer of the CNN. For each location (the red dot as an example), the associated convolutional patch is reshaped using the off-dilated coefficient vector learned from the regression layer to obtain a perceptual field with flexible size and shape. When the dilated coefficient is equal to 1, the adaptive convolution is the standard convolution; when the dilated coefficient is less than 1, the convolution patch shrinks and the perceptual field shrinks; when the dilated coefficient is greater than 1, the convolution patch expands so that the perceptual field is enlarged. Figure 3 illustrates receptive fields with flexible sizes learned from dilated convolution. We perform experiments using different sizes of convolution kernels, and the experimental results show that 1×1, 3×3, 5×5, and 7×7 convolution kernels perform the best convolution, and other sizes lose some of their ability to convolve feature information at different scales. Here, the convolution kernels are operated for different perceptual fields rather than changing the size of the kernel by applying an expansion factor. These convolution kernels are of fixed size and are used to capture feature information at different scales under different receptive fields. For each receptive field, they capture local details, medium-scale structures, and larger-scale contextual information, respectively.

Figure 3.

Illustration of receptive fields with flexible sizes in adaptive dilated convolutions.

Let $R^t$ be the 8-dimensional vector of the adaptive dilated coefficient at position $t$ , $R^{t}=\left [r_{x 1}^{t}, r_{x 2}^{t}, r_{x 3}^{t}, r_{x 4}^{t}, r_{y 1}^{t}, r_{y 2}^{t}, r_{y 3}^{t}, r_{y 4}^{t}\right ]^{T}$ . Suppose that for $F^t$ , the associated convolution patch in $X$ is $W^t$ , and its center $\left (s^{t}, v^{t}\right )$ is any convex quadrilateral, the convolution kernel is $K\in \mathbb{R}^{D \times (2k+1) \times (2k+1)}$ , where $k$ is a constant. The shape of $W^t$ is determined by $R^t$ by changing the position of its four corners, each of which is controlled by two components of $R^t$ , and the values of all components in $R^t$ are determined by the position of the corners. Feature vectors $(2k+1)\times (2k+1)$ were selected from the convolution patch $W^t$ for element multiplication. The coordinates of these feature vectors $\left (x_{i j}, y_{i j}\right )$ were determined by the new positions of corner points, and it can be expressed as:

(3) \begin{equation} \left \{\begin{aligned} x_{i j}= \,& s^{t}+\varphi i j \frac{r_{x 1}+r_{x 2}-r_{x 3}-r_{x 4}}{4 k}+\varphi i \frac{r_{x 1}+r_{x 2}+r_{x 3}+r_{x 4}}{4} \\ & +\varphi j \frac{r_{x 1}-r_{x 2}+r_{x 3}-r_{x 4}}{4}+\varphi k \frac{r_{x 1}-r_{x 2}-r_{x 3}+r_{x 4}}{4} \\ y_{i j}= \,& v^{t}+\varphi i j \frac{r_{y 1}+r_{y 2}-r_{y 3}-r_{y 4}}{4 k}+\varphi i \frac{r_{y 1}+r_{y 2}+r_{y 3}+r_{y 4}}{4} \\ & +\varphi j \frac{r_{y 1}-r_{y 2}+r_{y 3}-r_{y 4}}{4}+\varphi k \frac{r_{y 1}-r_{y 2}-r_{y 3}+r_{y 4}}{4} \end{aligned}\right. \end{equation}

where $s^{t}$ , $v^{t}$ , $\varphi$ are integers, ${i}, {j} \in [-{k}, {k}]$ .

Since the dilated coefficient is a real value, the eigenvector can be obtained by bilinear interpolation. Suppose the convolution patch after bilinear interpolation is $O_t$ , and it can be expressed as:

(4) \begin{equation} O_{i j}^{t}=\sum _{n, m} W_{n m}^{t} \max \left (0,1-\left |x_{i j}^{\prime \prime }-m\right |\right ) \max \left (0,1-\left |y_{i j}^{\prime \prime }-n\right |\right ) \end{equation}

where $W_{n m}^{t}=W^{t}(n, m)$ , $ \mathrm{n}, \mathrm{m} \in [-\mathrm{\varphi k}, \mathrm{\varphi k}]$ . So the forward propagation of convolution is:

(5) \begin{equation} F^{t}=\sum _{i, j} K_{i j} O_{i j}^{t}+\mu \end{equation}

where ${K_{ij}}={K}({i}, {j})$ denotes the multiplication process of all output channel elements and $\mu$ is the bias.

Accordingly, in the case of backpropagation, the gradient change is:

(6) \begin{eqnarray} g\left (O_{i j}^{t}\right )&=&\left (K_{i j}\right )^{T} g\left (F^{t}\right ) \nonumber\\ g\left (K_{i j}\right )&=&g\left (F^{t}\right )\left (O_{i j}^{t}\right )^{T} \\ g(\mu )&=&g\left (F^{t}\right )\nonumber\end{eqnarray}

We obtain $\frac{\partial O_{i j}^{t}}{\partial W_{i j}^{t}}$ for the bilinear interpolation equation and then obtain $\frac{\partial o_{i j}^{t}}{\partial x_{i j}}$ and $\frac{\partial O_{i j}^{t}}{\partial y_{i j}}$ for the partial derivatives of the corresponding coordinates. Since the coordinates $x_{i j}$ and $y_{i j}$ depend on the vector of perspective coefficients $R^t$ , we can use the following partial derivatives of the perspective coefficients to obtain the gradient of $R^t$ .

(7) \begin{equation} \begin{array}{l}{\left [\begin{array}{l} \displaystyle\partial x_{i j}/ \partial r_{x 1}^{t} \\ \displaystyle\partial x_{i j}/ \partial r_{x 2}^{t} \\ \displaystyle\partial x_{i j}/ \partial r_{x 3}^{t} \\ \displaystyle\partial x_{i j}/ \partial r_{x 4}^{t} \end{array}\right ]=\left [\begin{array}{l} \displaystyle i j \varphi +i k \varphi +j k \varphi +k^{2} \varphi \\ \displaystyle i j \varphi +i k \varphi -j k \varphi -k^{2} \varphi \\ \displaystyle -i j \varphi +i k \varphi +j k \varphi -k^{2} \varphi \\ \displaystyle -i j \varphi +i k \varphi -j k \varphi +k^{2} \varphi \end{array}\right ]/ 4 k,} \\[30pt] {\left [\begin{array}{l}\displaystyle \partial y_{i j}/ \partial r_{y 1}^{t} \\ \displaystyle\partial y_{i j}/ \partial r_{y 2}^{t} \\ \displaystyle\partial y_{i j}/ \partial r_{y 3}^{t} \\ \displaystyle\partial y_{i j}/ \partial r_{y 4}^{t} \end{array}\right ]=\left [\begin{array}{c} \displaystyle i j \varphi +i k \varphi +j k \varphi +k^{2} \varphi \\ \displaystyle -i j \varphi -i k \varphi +j k \varphi +k^{2} \varphi \\\displaystyle -i j \varphi +i k \varphi +j k \varphi -k^{2} \varphi \\ \displaystyle i j \varphi -i k \varphi +j k \varphi -k^{2} \varphi \end{array}\right ]/ 4 k} \end{array} \end{equation}

Finally, using these partial derivatives, the gradients of the perspective coefficient mapping $R$ and the input feature mapping $X$ are obtained by the chain rule.

(8) \begin{eqnarray} g\left (r_{x_{-}}^{t}\right )&=&\sum _{i, j}\left (\frac{\partial x_{i j}}{\partial r_{x_{-}}^{t}} \frac{\partial V_{i j}^{t}}{\partial x_{i j}}\right )^{T} g\left (O_{i j}^{t}\right ), \nonumber\\ g\left (r_{y_{-}}^{t}\right )&=&\sum _{i, j}\left (\frac{\partial y_{i j}}{\partial r_{y_{-}}^{t}} \frac{\partial V_{i j}^{t}}{\partial y_{i j}}\right )^{T} g\left (O_{i j}^{t}\right ), \\ g\left (W_{n m}^{t}\right )&=&\sum _{i, j} \frac{\partial O_{i j}^{t}}{\partial W_{n m}^{t}} g\left (O_{i j}^{t}\right ), \nonumber\end{eqnarray}

where $r_{x_{-}}^{t}$ denotes any component of $\left \{r_{x 1}^{t}, r_{x 2}^{t}, r_{x 3}^{t}, r_{x 4}^{t}\right \}$ and $r_{y_{-}}^{t}$ denotes any component of $\left \{ r_{y_{-}}^{1}, r_{y_{2}}^{t}, r_{y_{3}}^{t}, y_{x_{4}}^{t}\right \}$ .

3.2. Context feature coding model

There are interconnections between scene features, and each scene feature is related to the overall scene, and these associations help feature classification and generate more accurate scene parsing results. By contextual feature encoding model $P_{2}$ , the contextual association features between scene as a whole and localities are learned. As shown in equation (6):

(9) \begin{equation} \begin{aligned} \displaystyle\left \{\gamma _{1}^{(i)}, \gamma _{2}^{(i)}, \ldots, \gamma _{J}^{(i)}\right \} &=P_{2}\left (\alpha ^{(i)}, \beta _{1}^{(i)}, \beta _{2}^{(i)}, \ldots, \beta _{J}^{(i)} ; \xi _{2}\right ) \\[5pt] \gamma _{i}^{(i)} \in \mathbb{R}^{D_{\gamma }}&, j =1, \ldots, J \end{aligned} \end{equation}

where $\xi _{2}$ is the parameter of the contextual feature encoding model. $\left \{\gamma _{j}^{(i)}\right \}_{j=1, \ldots J}$ is the $D_{\gamma }$ -dimensional vector of contextual features. Each contextual feature $\gamma _{i}^{(i)}$ encodes the semantic relationship between the jth feature and the whole scene as well as the relationship between the jth feature and other features.

The structure of graph is irregular and has infinite dimensionality, which can describe the semantic contextual relationship between feature points well, so graph convolutional network is used to learn the contextual relationship of scene features. The matrix E is used to represent the nodes of the scene feature graph, A is the adjacency matrix to represent the feature semantic contextual relationship, and the matrices E and A are the inputs of the graph convolutional network. The propagation between each layer in the graph convolutional neural network can be expressed as:

(10) \begin{equation} F^{k+1}=f\left (\tilde{D}^{-\frac{1}{2}} \tilde{E} \tilde{D}^{-\frac{1}{2}} F^{k} \rho ^{k}\right ) \end{equation}

where $\tilde{A}=A+I$ , where I is the unit matrix, $\tilde{D}$ is the degree matrix of $\tilde{E}$ , $F^{k+1}$ is the feature of k + 1 layers, the feature representation of the input layer X is E, $f(\cdot )$ is the nonlinear activation function, and $\rho ^{k}$ is the parameter of learning. The final output of the graph convolutional network is the updated features of node $X^{(i)}$ in the graph, which can be aggregated into a scene feature vector for feature semantic relationship inference.

3.3. Enhancement classification network

Let the input of scene classification model $P_{3}$ be scene features $\alpha ^{(i)}$ , and the output scene classification probability prediction result can be expressed as:

(11) \begin{equation} \begin{array}{c} \displaystyle\left \{p_{1}^{(i)}, p_{2}^{(i)}, \ldots, p_{M}^{(i)}\right \}=P_{3}\left (\alpha ^{(i)} ; \xi _{3}\right ) \\[6pt] \displaystyle p_{m}^{(i)} \in [0,1],\,\, m=1, \ldots, M \end{array} \end{equation}

where $\xi _{3}$ is the parameter of the scene classification model, M is the total number of scene classes in the dataset, and $p_m^{(i)}$ is the initial probability that the image $X^{(i)}$ predicted by the scene classification model belongs to the mth class of scenes. The network includes a convolution layer with a filter size of $1\times 1$ , and a softmax regression layer, which can generate scene classification results $\left \{p_{m}^{(i)}\right \} m=1, \ldots, H \times W$ based on the input scene global feature $\alpha ^{(i)}$ .

The underwater heritage is located in an environment with turbid water, and the acquired object features are blurred, which can easily confuse the objects in the process of classification. The reinforced classification network is mainly used to discriminate the categories with similar probability values, from which the correct object category is judged. The input of the enhanced classification network is the initial probability value of $p_m^{(i)}$ , the image features F and the original image X cascaded together. where the image features F are used as a reference during the enhancement classification to improve the enhancement classification accuracy. Since the initial probability $p_m^{(i)}$ , the image features F and the original image X may have a wide range of values, these features are normalized separately. The general diagram of enhancement classification network is shown in Fig. 4.

Figure 4.

The general diagram of enhancement classification network.

Based on this input, a series of convolutional layers are then used for enhanced classification. First, a 3 $\times$ 3 convolution is used, which extracts information about the contextual relationships between multiple pixels. We introduce a multi-classifier here and then use a 1 $\times$ 1 convolution to predict the enhanced classification $\alpha _{c}^{(i)} \in \mathbb{R}^{H \times W \times D}$ . Each of these elements $\alpha _{c}^{(i)}$ represents the probability value that the ith pixel in the image belongs to the cth object class. This probability value is normalized by the multinomial logistic regression (i.e., softmax) function as follows:

(12) \begin{equation} p_{c}^{(i)}=\frac{\exp \left (\alpha _{c}^{(i)}\right )}{\sum _{j=1}^{C} \exp \left (\alpha _{j}^{(i)}\right )} \end{equation}

where $p_{c}^{(i)}$ is the normalized probability. The set of all class probabilities $\left \{p_{c}^{(i)}\right \}_{C=1, \ldots, C}$ is the probability distribution $P(Y \mid X, c)$ sought by the scene semantic parsing task.

In the model training phase, after generating scene classification results, the scene classification loss is calculated. The loss of the strengthened classifier is the multiclassification cross-entropy loss function:

(13) \begin{equation} L_{1}=-\frac{1}{N} \sum _{i=1}^{N} \sum _{c=1}^{C} y_{i, c} \times \log p_{c}^{(i)} \end{equation}

where $y_{i, c} \in \{0,1\}$ is the scene classification label. $y_{(i,c)}=1$ means that the image $X^{(i)}$ belongs to the cth category scene, and $y_{(i,c)}=0$ means vice versa. Since the similarity of probability scoring of multiple categories can easily lead to misclassification, regularization is used to avoid this situation. The difference-based regularization method makes each category probability scoring as large as possible. Here, second-order moments are used as the difference-based regularization:

(14) \begin{equation} Q=1-\frac{1}{N} \sum _{i=1}^{N} \sum _{c=1}^{C}\left (p_{c}^{(i)}\right )^{2} \end{equation}

where $Q \in [0,1-1/ C]$ . The larger the difference between the probability scoring $\left \{p_{c}^{(i)}\right \}_{C=1, \ldots, C}$ , the smaller the value of Q. Thus, the loss function can be reduced to increase the difference between the probabilities.

3.4. Scene parsing model

To generate the scene semantic parsing results, this chapter adopts the scene parsing model $P_{4}$ to predict the category of each region based on the local features and contextual relationship features, so as to obtain the final scene parsing results. Since there are two input features, local features and contextual features, the scene parsing model $P_{4}$ first generates two sets of classification probabilities based on these two features and then integrates the two sets of probabilities. To obtain more detailed predictions, the scene resolution model $P_{4}$ predicts a set of classification probabilities based on the pixel-level local features $\left \{\beta _{j^{\prime }}^{(i)}\right \}_{t^{\prime }=1, \ldots, J^{\prime }}$ , where $J^{\prime }$ is the number of pixels in the image $X^{(i)}$ . And another set of classification probabilities are predicted based on the hyperpixel-level contextual relationship features $\left \{\gamma _{j}^{(i)}\right \}_{j=1, \ldots, J}$ , where J is the number of hyperpixels. After converting the probability results predicted based on the hyperpixel-level contextual relationship features to pixel-level results, they are then integrated with the results predicted based on the pixel-level features. The integration is done by taking the higher of the two probability scores for each pixel for each category as the final classification probability.

The scenario parsing model $P_{4}$ can be modeled as follows:

(15) \begin{equation} \begin{array}{c} \displaystyle\left \{q_{j^{\prime }, 1^{\prime }}^{(i)}, q_{j^{\prime }, 2}^{(i)}, \ldots, q_{j^{\prime }, L^{(i)}}^{(i)}=P_{4}\left (\beta _{j^{\prime }}^{(i)}, \gamma _{j}^{(i)} ; \xi _{4}\right )\right. \\[5pt] \displaystyle q_{j^{\prime }, k}^{(i)} \in [0,1],\,\, k=1, \ldots, K \end{array} \end{equation}

where $\xi _4$ is the parameter of the scene resolution model, K is the total number of object classes in the dataset, and $q_{j^{\prime }, k}$ is the probability that the j’th pixel in the image $X^{(i)}$ predicted by the scene resolution model belongs to the kth class of objects. In the model training phase, the scene resolution loss is used to constrain the learning of the model. The scene parsing loss uses a pixel-level cross-entropy loss function:

(16) \begin{equation} L_{2}=-\frac{1}{N} \sum _{i=1}^{N} \frac{1}{J^{\prime }} \sum _{j^{\prime }=1}^{J^{\prime }} \sum _{k=1}^{K} y_{i, j^{\prime }, k} \times \log q_{t^{\prime }, k}^{(i)} \end{equation}

where $y_{i, j^{\prime }, k}$ is the label of whether the j’th pixel belongs to the kth class of objects. Ultimately, a joint optimization approach can be used to combine the scene classification loss function and the scene resolution loss function, while optimizing all the models proposed in this chapter:

(17) \begin{equation} \begin{aligned} L &=L_{1}+\lambda L_{2} \\ &=-\frac{1}{N} \sum _{i=1}^{N}\left [\sum _{c=1}^{C} y_{i, c} \times \log p_{c}^{(i)}\right .\\ &\quad\left .+\lambda \frac{1}{J^{\prime }} \sum _{t^{\prime }=1}^{J^{\prime }} \sum _{k=1}^{K} y_{i, j^{\prime }, k} \times \log q_{j^{\prime }, k}^{(i)}\right ] \end{aligned} \end{equation}

where $\lambda$ is a scaling factor that controls the ratio of scene classification loss to scene parsing loss.

4.1. Training dataset and evaluation index

In order to evaluate the scene resolution performance of the proposed methods in this paper, the test results are compared with the current state-of-the-art methods. All methods are tested on the Underwater Shipwreck Scenes (USS) dataset, which contains 1163 images of eight categories: wreck, statue, porcelain, sediment, reef, water, plant, and fish. The data of this dataset are a homemade dataset generated by collecting underwater archaeology images online and then by manual annotation. Pixel accuracy (PAcc) and mean intersection-over-union accuracy (mIoU) are used as the evaluation criteria of the proposed algorithm in this paper. Pixel accuracy counts the percentage of correctly classified pixels in the whole dataset. The mean intersection-over-union (mIoU) is calculated separately for each object class, and then, the intersection-over-union values of all object classes are averaged as the final result.

4.2. Experiment details

In this experiment, training and testing are performed on a small server with GTX2080 GPU and 64G RAM. In order to demonstrate the objectivity of the proposed method in the comparison experiments, the Pytorch deep learning tool is used for implementation, and the ResNet convolutional neural network is used for deep network construction. The experimental training parameters are: initial learning rate = 0.0001, epoch = 300, momentum = 0.9, weight decay = 0.0001, and random gradient descent method is used for training. The learning rate is dynamically adjusted using the “poly” strategy.

4.3. Ablation experiment

We validate the effectiveness of adaptive dilated convolution (ADC) and enhancement classification network (ECN) in the proposed algorithm in this paper by ablation experiments. The experiments are conducted on the USS dataset. We use ResNet-50 as the baseline for global enhancement network (GENet) ablation experiments. The experimental results are shown in Table I and include the recognition accuracy of each category in addition to PAcc and mIoU. The visual results are shown in Fig. 5. From Table I, it can be seen that PAcc and mIoU in the baseline network, reached 79.6% and 63.8%. After adding ADC to the baseline, PAcc and mIoU are improved, which are 82.3% and 65.1%, respectively. Some categories such as “statue,” “fish,” and “plant” far exceeded the baseline. This indicates that ADC helps to refine the details of small objects. After adding ECN to the baseline, both PAcc and mIoU were higher than the baseline, 83.1%, and 65.5%, respectively. The recognition accuracy of individual categories is higher than the baseline, especially “wreck,” “reef,” “sediment,” and “water” which were much higher than the baseline. When ADC and ECN were added to the baseline at the same time, the data were significantly improved, and the PAcc and mIoU were as high as 86.9% and 68%, which increased by 7.3% and 4.2% compared to the baseline. The ADC and ECN numbers are much better than adding either one of them. This is because the adaptive convolutional network can change the receptive field according to the size of the object, to extract more features. Strengthening the classification network increases the distance between similar categories and makes classification results more accurate. The combination of the two not only improves the evaluation indexes but also enables the analysis of more complex scenarios. The experimental results demonstrated the effectiveness of ADC and ECN.

Table I.

Experimental data of ADC and ECN ablation.

The black bolded font in the table indicates the excellence metrics for each algorithm.

Figure 5.

Visual effect of the ablation experiment.

From the figure, it can be seen that the baseline network is less effective in recognizing details and small objects (e.g., fish). After adding the ADC network, the overall recognition effect is significantly improved. In the second graph of the second column, it can be seen that the number of recognized fish has significantly increased contour clarity, indicating a better recognition rate for small objects. Adding the EDC network, the effect graph can see the different object boundary contours are clear and the recognition is better. By adding both ADC and EDC networks, the overall visual effect is significantly improved. The visual effect graph illustrates that both ADC and EDC networks proposed in this paper can effectively improve the algorithm’s performance.

4.4. Comparison of different dilated convolutional networks

Dilated convolutional networks were proposed long ago [Reference Yu and Koltun37], and many current studies on dilated convolution have been improved upon. In order to verify the advancedness of the proposed adaptive dilated convolution, the proposed dilated network is replaced by the current state-of-the-art expanded convolutional network for comparison experiments. The experimental results are shown in Table II. The visual effect graph is shown in Fig. 6.

Table II.

Comparison results of different dilated convolutions.

The black bolded font in the table indicates the excellence metrics for each algorithm.

Figure 6.

Visual effect of different dilated convolution comparison.

As can be seen in Table II, the adaptive dilated convolution proposed in this paper performs better in terms of performance, with the highest PAcc and mIoU. For each category, the recognition accuracy performs best overall. However, other methods also have the highest recognition accuracy for a single category. The highest accuracy of porcelain recognition by DCLS is 82.3%, which is higher than 0.3% in the dilated convolutional network proposed in this paper. The maximum pixel accuracy of AFDC for wreck is 88.1%, which is higher than the proposed dilated convolutional network by 0.4%. The highest recognition accuracy of ADC for water is 86.0%, which is 0.1% higher than the proposed algorithm. All the above methods obtain more context information by increasing receptive field, but the overall recognition accuracy in complex scenes with both large and small objects is slightly lower than that of the network proposed in this paper. The proposed adaptive dilated convolution can change the size and shape of the receptive field arbitrarily according to the size and shape of different target categories, thereby achieving better performance in PAcc and mIoU on the USS dataset.

Figure 7.

General scene parsing visual effect.

As can be seen in Fig. 6, the visual effects of different dilated convolutions for different classes of object recognition are basically the same. However, the adaptive dilated convolution proposed in this paper outperforms the other dilated convolutions in terms of the overall object recognition contour and the recognition accuracy of small objects. From the visual results, it can be seen that DCLS performs poorly on low-resolution images because using a learnable spatial sampling rate can result in small kernel sizes that do not capture sufficient contextual information, thus affecting segmentation performance. Although AFDCNet can adapt to different object sizes, its performance is not as good as other networks when dealing with dense small objects. As can be seen from the third image in the first row, when the target part in the scene is occluded or deformed, the prediction performance of ADC is significantly lower than other methods.

4.5. Experimental results and analysis under different conditions

4.5.1. The parsing results under the general condition

To evaluate the performance of the proposed algorithm in regular scenes, it is compared with the existing more advanced methods. The visual results of its detection are shown in Fig. 7, class pixel accuracy and PAcc as shown in Table III, and the class intersection-over-union and mIoU as shown in Table IV.

Table III.

Class pixel accuracy and PAcc under regular conditions.

The black bolded font in the table indicates the excellence metrics for each algorithm.

Table IV.

Class intersection-over-union and mIoU under conventional conditions.

The black bolded font in the table indicates the excellence metrics for each algorithm.

From the experimental results data in Table III and Table IV, we can see that the PAcc and mIoU of the proposed algorithm are higher than other algorithms, 86.9% and 68%, respectively. The pixel accuracy and mIoU of all eight categories are improved compared with DFANet, which has increased the perception field, but it is difficult to identify different object categories accurately because of the inconsistent size and large difference of object categories in underwater shipwreck scenes. Especially for small objects porcelain and fish, the recognition rate is the worst among all methods, 80.5% and 73.6%. Although DFANet increases the receptive field, the size of object classes in underwater shipwreck scenes is inconsistent and varies greatly. Therefore, it is difficult to recognize different object categories accurately. DecoupleSegNets uses a feature pyramid-based approach to process different scales of input images, but it is still sensitive to the input image size. The accuracy of porcelain recognition is higher than the proposed algorithm by 0.7%, while all other categories are lower than the algorithm in this paper. The attention mechanism and non-local neural network structure of ANNNet make it have certain black box nature and poor interpretation for some categories. SegFormer has a certain limitation on the size of the input image, which needs to be adjusted within a certain range. And the adaptive dilated convolution proposed in this paper can automatically adjust the size of the perceptual field according to the object category. It can fully extract multi-scale features of different target classes. The category pixel accuracy and mIoU of the proposed method are mostly higher than the other three algorithms. The algorithm proposed in this paper has the highest recognition accuracy for statue, reef, sediment, and fish under general conditions and is significantly higher than the other compared algorithms. The above comparative experimental results show that adequate feature learning, construction of contextual semantic relations, and effectively enhanced classification can significantly improve the performance of the proposed method on the USS dataset.

From the visual effect figure, it can be found that for dense and relatively small objects (fish), it is easy to recognize multiple as one. The boundary recognition result is blurred when different objects intersect. There is a phenomenon of miscalculation in the figure, for example, “reef” is recognized as “wreck.” Overall, the overall recognition degree of the proposed method for different object categories and the boundary contours are relatively good, which proves that the performance of the proposed algorithm on the USS dataset is better than other algorithms.

Figure 8.

Semi-burial situation of cultural relics scene parsing visual effect.

4.5.2. The parsing results under the semi-burial situation of cultural relics

To verify the effectiveness of the proposed algorithm under different situations. The visual effects are compared with other methods in the case of semi-buried cultural relics, and the visual effects are shown in Fig. 8. Class pixel accuracy and PAcc with are shown in Table V, and class intersection-over-union and mIoU are shown in Table VI.

From the experimental results data in Table V and Table VI, it can be seen that the PAcc and mIoU of the algorithm proposed in this paper are higher than the other algorithms in the case of semi-buried cultural relics, 85.4% and 65.4%, respectively. From the data in the table, it can be seen that most of the target categories of the algorithm proposed in this paper are higher than other algorithms. The pixel accuracy and mIoU of the proposed algorithm are lower than those of the DFANet algorithm by 1.2% and 0.6%, respectively. The pixel accuracy and mIoU for fish are lower than the algorithm ANNNet by 0.6% and 0.3%, respectively. Pixel accuracy and mIoU for the plant are lower than the algorithm DecoupleSegNets by 1% and 0.4%, respectively. However, the pixel accuracy and mIoU of semi-buried artifacts (wreck, statue, porcelain) are significantly higher than the other algorithms. In the case of semi-buried artifacts, adaptive dilated convolution can use the dilated coefficient of the convolution kernel to expand the perceptual field, which in turn improves the feature extraction ability of the target class. At the same time, adaptive dilated convolution can also utilize the size and shape of the convolution kernel for feature enhancement and compensation to improve the expression ability and recognition performance of the target category. The comparison results of experimental data show that adjusting the perceptual field size according to the object size can fully extract object features, fuse contextual information features, and improve the segmentation accuracy of object categories.

It can be seen from the visual effect figure that the recognition of the semi-buried cultural relics algorithm proposed in this paper has the highest completeness in the recognition of cultural relics. As can be seen in the experimental effect plots of other methods, there exist buried cultural relics with plants or sediment being recognized as part of the cultural relics, resulting in blurred edges of the object class. For some smaller object categories, there also exists the phenomenon that they cannot be recognized. The comprehensive experimental effect figure can be found that the proposed method in the paper has a high degree of completeness in artifact recognition and a relatively good edge recognition effect, which further proves the effectiveness of the proposed algorithm.

Table V.

Class pixel accuracy and PAcc under semi-buried conditions for cultural relics.

The black bolded font in the table indicates the excellence metrics for each algorithm.

4.5.3. The parsing results under the turbid water condition

Due to the presence of a large amount of sediment in the site where the cultural relics are located, the water quality of the environment in which they are located is turbid. In order to further verify the effectiveness of the proposed algorithm, the visual effect diagram is shown in Fig. 9 for comparison with other algorithms in the case of turbid water, the category detection accuracy and detection pixel accuracy are shown in Table VII, the category cross-accuracy and mIoU are shown in Table VIII.

Table VI.

Class intersection-over-union and mIoU under semi-buried conditions for cultural relics.

The black bolded font in the table indicates the excellence metrics for each algorithm.

Figure 9.

Visual effect of scene parsing under turbidity condition.

Table VII.

Class pixel accuracy and pixel accuracy under turbid water conditions.

The black bolded font in the table indicates the excellence metrics for each algorithm.

Table VIII.

Class intersection-over-union and mIoU under turbid water conditions.

The black bolded font in the table indicates the excellence metrics for each algorithm.

From the experimental result data in Table VII and Table VIII, we can see that the PAcc and mIoU of the proposed algorithm in this paper are higher than other algorithms, which are 84.9% and 65.5%, respectively. Meanwhile, the category pixel accuracy and mIoU of eight categories are improved, which are higher than other algorithms. Compared with DFANet, the proposed algorithm improves pixel accuracy by 3.4% and mIoU by 1.3%. The pixel accuracy of the algorithm presented in this paper is 3% and 1.2% higher than that of DecoupleSegNets. The pixel accuracy and mIoU of ANNNet are 2.9% and 1% lower than that of the proposed algorithm. Compared with SegFormer, the pixel accuracy of the algorithm proposed in this paper is improved by 2% and that of mIoU by 0.8%. When there are ambiguous and difficult-to-judge samples, a single classifier may not be able to accurately classify these samples because they may belong to multiple classes or not to any one class. However, stepwise classification using multiple classifiers can better handle these ambiguous samples. In a multi-classifier model, each classifier is responsible for classifying a set of categories. First, the initial classifier classifies all samples, and misclassified or ambiguous samples, they are sent to the next classifier for further classification. This process can be repeated several times until all samples are assigned to the last classifier for final classification. The experimental results show that the proposed algorithm can effectively improve the segmentation performance of object categories under the turbid water condition, which verifies the effectiveness of the proposed network.

As can be seen from the visual effect diagram, all the category boundaries are blurred, which is caused by the blurring of the extracted features due to the turbidity of the water, which makes it difficult to discriminate in the classification process. And it can be seen from the figure that there is the phenomenon of object category misclassification. Due to the blurred water quality, the features are difficult to extract, and some objects are identified as similar backgrounds. For example, “porcelain” is identified as “sediment,” and “statue” is identified as “reef.” From the last line of the effect, we can see that our proposed method has a good visual effect, which proves the effectiveness of the proposed method in turbid water conditions.

4.6. Comparative experiments on public datasets

To verify the generalization of the algorithm proposed in this paper, we conducted comparison experiments with current state-of-the-art algorithms on a public dataset containing the ADE20K dataset, the Cityscapes dataset, and the SUIM dataset. ADE20K covers various annotations of scenes, objects, and object parts, and contains various objects in natural space environments. Each image has an average of 19.5 instances and 10.5 object classes with 150 semantic categories. The Cityscapes dataset contains 20,210 training images, 2000 validation images, and 3351 test images. 30 different categories of city street data from 50 different cities for different periods and seasons are included in the Cityscapes dataset. The SUIM dataset is the first publicly available large-scale underwater semantic segmentation dataset with 2975 training images, 1525 test images, and 500 validation images. The dataset has 1525 images and contains 8 categories: Background (waterbody) (BW), Human divers (HD), Aquatic plants and sea-grass (PF), Wrecks or ruins (WR), Robots (RO), Reefs and invertebrates (RI), Fish and vertebrates (FV), and Sea-floor and rocks (SR).

4.6.1. ADE20K and Cityscapes comparison test

Since the experimental results on the ADE20K and Cityscapes datasets are similar, we put the two experimental results together for analysis. Due to the large number of scene categories in these two datasets only PAcc and mIoU are reported in this paper, and the experimental data are shown in Tables IX and X, and the visual effect graphs of scene resolution are shown in Figs. 10 and 11. The experimental data in Table IX shows that the proposed algorithm achieves the highest PAcc and mIoU on both ADE20K dataset and the Cityscapes dataset. DFAT is not fine enough for object edge detail processing, resulting in lower data and a less accurate visual effect of scene parsing results. The attention mechanism in ANNNet is based on a homogeneous image grid, which is less effective in resolving irregular shapes in the scene. SegFormer performs differently on different size input images. The overall effect of the proposed algorithm and the current state-of-the-art algorithm on visual scene resolution is the same, but the proposed algorithm performs better in detail. The proposed algorithm can make a consistent prediction for large objects and an accurate prediction for small objects by expanding adaptive convolution.

Table IX.

PAcc and mIoU of the ADE20K dataset.

The black bolded font in the table indicates the excellence metrics for each algorithm.

4.6.2. SUIM dataset comparison test

To further verify the advancedness and performance of the algorithm proposed in this paper for underwater images, we conducted comparison experiments between current advanced algorithms and the algorithm proposed in this paper on the SUIM dataset. In addition, we added the SUIMNet method to this set of comparison experiments. The experimental data are shown in Table XI, and the visual effect graph of scene resolution is shown in Fig. 12.

Figure 10.

ADE20K dataset scene parsing effect.

Table X.

PAcc and mIoU of the Cityscapes dataset.

The black bolded font in the table indicates the excellence metrics for each algorithm.

The data in the table show that the proposed algorithm in this paper has the highest PAcc and mIoU among the comparison methods, which are 93.1% and 81.1%, respectively. In terms of category pixel accuracy, the algorithm proposed in this paper is slightly lower than other algorithms for the whole part of the category. In terms of category pixel accuracy, SUIMNet has the highest accuracy for BW and FV pixels, which is 0.2% and 0.8% higher than the proposed algorithm. decoupleSegNets has the best recognition for HD with 90.2%, which is 0.3% higher than the proposed algorithm. ANNNet has the highest recognition accuracy for WR with 86.3%, which is 0.5% higher than the proposed algorithm. The visual resolution of this paper is clearer than that of this paper, as can be seen from the visual effect graph. This is because the SUIM dataset is larger and can provide more samples to train the model, thus better capturing the statistical patterns of the data. In addition, the SUIM dataset already contains many high-quality underwater images with clearer category features than the USS dataset. The higher-clarity images are easier to be classified during the training process because the object boundaries and details are clearer.

Table XI.

PAcc and mIoU of the SUIM dataset.

The black bolded font in the table indicates the excellence metrics for each algorithm.

Figure 11.

Cityscapes dataset scene parsing effect.

Figure 12.

SUIM dataset scene parsing effect.