2.1 Subjects

Twelve male subjects were recruited to participate in this study. Their mean age was 38.2 years (SD = 4.1). All subjects had extensive experience in simulated helicopter flying, and the average simulated flight time was 3,160h (SD = 1126.4). Their binocular vision was normal, and they agreed to record their eye movement data during the experiment. Subjects were told they could stop the experiment at any time. The experiment was reviewed and approved by the Nanjing University of Aeronautics & Astronautics institutional review board. The experiments were carried out according to the Declaration of Helsinki. All participants provided written informed consent.

2.2 Equipment

The layout of the helicopter cockpit simulation platform was shown in Fig. 1. Flight control and display devices in the helicopter cockpit simulation platform included a cyclic control stick, a collective pitch stick, pedals, a head-up display and two digital instrument displays. The screen sizes of the visual display and the digital instrument display were 40 inches and 14 inches respectively, the resolutions were 1,920 × 1,080 and 1,366 × 768, and the refresh rates were both 60Hz. In the experiment, the helicopter model we selected from the helicopter cockpit simulation platform was Kamov Ka-52.

Figure 1.

Overall schematic diagram of helicopter cockpit simulation platform.

Eye-tracking used the SMI ETG 2W glasses-type eye-tracking devices. We used eye-tracking devices that allowed subjects to freely turn their heads and tracked a greater range than desktop fixed eye-tracking devices. Eye-tracking was accurate to 0.5° with a range of 80° horizontally and 60° vertically. Begaze SMI software (version 3.5) was used to analyse eye movement data.

2.3 Experimental design

Participants were required to fly the helicopter on the cockpit simulation platform to complete the flight task of the designated route, as shown in Fig. 2(a). Start at waypoint A, climb to 2,000 feet to start cruising and then descend to 1,000 feet at waypoint B. Then climb to 3,000 feet to start cruising and descend to 1,000 feet at waypoint C. Finally, climb to 2,500 feet to start cruising, descend before reaching waypoint A and land at waypoint A. It took about 15min to complete an airline flight. The English warning information in red font appeared in the lower right corner of the screen on the right side of the digital instrument display, with the content of “missile approaching”, as shown in Fig. 2(b), and the duration was 15s. The warning information were divided into the presence or absence of flickering (bright 1s, dark 0.5s) in terms of vision, and the presence or absence of audible warnings (deep-deep-deep-deep, ring 0.35s, off 0.35s) in terms of hearing. The displays of warning information were divided into four encoding forms, as shown in Table 1. During the flight, each coded warning information appeared four times randomly. When participants noticed the warning information, they were required to report the occurrence of the warning information to the note-takers.

Table 1.

Details of the four warning coding types

Figure 2.

Schematic diagram of the flight route and warning location.

2.4 Procedure

Participants were carefully introduced to the research content and experimental environment. The participants then wore and calibrated eye-tracking devices and started flight tasks in sequence. Eye movement data were recorded by an eye-tracking device. Recording was started when participants started the flight task and stopped when all tasks were completed. Due to the limited number of personnel with sufficient flight experience, in order to collect sufficient experimental data, each participant repeated four flight tasks, and each flight had a rest time of 10min. But it should be noted that the type and appearance time of the warning information in each flight task were random. A total of 48 sets of experiments were conducted and 752 sets of valid eye movement data based on warning information were collected. Participants were paid after completing all flight tasks.

2.5 Data analysis

After sorting and processing the collected eye movement data, 25 eye movement features were obtained. According to the meaning of eye movement features, eye movement features were divided into eye features and visual features, as shown in Table 2. To compare the effect of warning information on various eye movement feature data, an analysis of variance was used to determine the level of importance of eye movement features. The ANOVA method ranked the importance of features based on the F value that is the ratio of the mean square between and the mean square within for each feature. The calculation formula of F was shown in Equation (1).

(1) $$F = {{\left[ {{n_1}\left( {{{\bar x}_1} - \bar x} \right) + {n_2}\left( {{{\bar x}_2} - \bar x} \right) + \cdots + {{\left. {{n_k}\left( {{{\bar x}_k} - \bar x} \right)} \right]} \mathord{\left/ {\vphantom {{\left. {{n_k}\left( {{{\bar x}_k} - \bar x} \right)} \right]} {\left( {k - 1} \right)}}} \right.} {\left( {k - 1} \right)}}} \right.} \over {\left[ {\sum {{{\left( {{{\bar x}_{1i}} - \bar x} \right)}^2} + \sum {{{\left( {{{\bar x}_{2i}} - \bar x} \right)}^2} + \cdots + {{\left. {\sum {{{\left( {{{\bar x}_{ki}} - \bar x} \right)}^2}} } \right]} \mathord{\left/ {\vphantom {{\left. {\sum {{{\left( {{{\bar x}_{ki}} - \bar x} \right)}^2}} } \right]} {\left( {n - k} \right)}}} \right.} {\left( {n - k} \right)}}} } } \right.}}$$

Table 2.

The serial number, abbreviation and meaning of each eye movement feature

The eye movement features were sorted according to their importance, and each eye movement feature was classified by Euclidean distance. The above ANOVA and classification were performed using IBM SPSS Statistics 22 software.

About correlation calculation and heat map generation, we used the Python software. By inputting eye movement data and running relevant programs, the relevant thermodynamic diagrams of each feature of eye movement data were obtained.

For a traditional deep learning model, the more layers of the network, the stronger the corresponding nonlinear expression ability and the more features learned by the model. However, with the increase of the number of network layers, it was difficult for the nonlinear expression of the traditional multi-layer network structure to represent the identity mapping, so the model may suffer from network degradation problem. Noise interference was common in eye movement data, which affected the accuracy of the model to identify the pilot perception of warning information. To address the above problems, this paper proposed a method of apply DRSN model to eye movement data. The DRSN model was able to overcome the difficulty that traditional learning models cannot achieve identity mapping on nonlinear transformations when training data samples in deep networks. At the same time, the interference of noise data samples and redundant data samples on feature threshold extraction was suppressed.

When DRSN was based on back-propagation for model training, its loss was not only back-propagated layer-by-layer through convolutional layers, etc., but also back-propagated more conveniently through the identity mapping of residual terms. Then, soft threshold was used to denoise the data, and a better model was obtained [Reference Zhao, Zhong, Fu, Tang and Pecht33].

It was assumed that the required solution mapping was H(x ^l ), and this problem was transformed into the residual mapping function F(x ^l )=H(x ^l )-x ^l for solving the network. Compared with the ReLU function, the soft threshold was more flexible to set the eigenvalue interval. In the residual shrinkage network, the threshold was automatically adjusted according to the situation of the sample itself through the attention mechanism. A Part of the DRSN model was shown in Fig. 3, where the size of the input x ^l was C × N, and after passing through the hidden layer 1, the ReLU function was used to obtain x ^l+1 as the input of the hidden layer 2. In the hidden layer 2, a small sub-network was constructed to learn a set of thresholds γ between 0 and 1, and then the soft thresholding of the features was performed and the residual term F(x ^l ) was added to obtain the output x ^l+2. The output of each layer was as follows:

(2) \begin{align} & {x^{l + 1}} = {\mathop{\rm Re}\nolimits} LU\left( {{w^{l + 1}}{x^l} + {b^{l + 1}}} \right)\nonumber \\ & x' = {w^{l + 2}}{\mathop{\rm Re}\nolimits} LU\left( {{w^{l + 1}}{x^l} + {b^{l + 1}}} \right) + {b^{l + 2}} \end{align}

In Equation (2), w and b were the weight vector and the bias vector, respectively.

Figure 3.

Local frame diagram of DRSN model.

Equation (3) was the soft thresholding result obtained by comparing $x'$ each dimension with the corresponding threshold γ.

(3) \begin{align} & x' = {\begin{cases} {x' - \gamma } & \quad\quad\quad {x' \gt \gamma } \\ 0 &\quad { - \gamma \le x' \le \gamma } \\ {x' + \gamma } & \quad\quad {x' \lt - \gamma } \end{cases}} \nonumber \\ & {x^{l + 2}} = x' + F\left( {{x^l}} \right) \end{align}

At the same time, to compare the superiority of DRSN algorithm model, according to the reading and induction of literatures, three machine learning algorithm models widely used in eye movement data were selected for comparison, namely SVM, RF and BPNN. For the four machine learning algorithm models, when selecting the eye movement data features, they were classified according to the importance of the eye movement features, and the eye movement features of different importance levels were selected as the data input of the model. Finally, the best eye movement feature suitable for the model was selected. The specific architecture and parameters of the four algorithm models were shown in Tables 3 and 4.

Table 3.

The architecture of the DRSN model

Table 4.

Specific parameters of SVM, RF and BPNN models