Nomenclature

AI

Artificial intelligence

AUC

Area under the curve

CAE

Convolutional auto-encoder

CBLSTM

Convolutional bi-directional long short-term memory

CNN

Convolutional neural network

DR

Discrimination and reconstruction

FN

False negative

FP

False positive

FSE

Fujitsu systems europe

GAN

Generative adversarial network

DL

Deep learning

DR

Discrimination and reconstruction

LSTM

Long short-term memory

MAD

Multivariate anomaly detection

MHM

Machine health monitoring

RNN

Recurrent neural networks

RUL

Remaining useful life

STD

Standard deviation

SVM

Supporting vector machine

TN

True negative

TP

True positive

Symbols

C _t

centroid of statistical features for training data

D _v

distance to centroid for validation data

μ

mean value

σ

standard deviation (STD)

υ

boundary factor for one-class support vector machines

S

skewness value

τ

detection threshold

2.0 The Airbus helicopter sensor fault dataset

One of the main challenges in aerospace industry is to test the validity of flight test data from heavily instrumented aircraft due to possible faulty sensors. Ensuring that healthy sensors are used for aircraft health monitoring is of vital importance because faulty sensors can lead to incorrect or misleading detection and diagnosis of aircraft faults, which in turn affects the safe flight of aircraft. Because of the sheer volume of measured signals that need to be validated, manual validation is no longer possible. It is crucial to automate the validation process. For this purpose, Airbus collected and released a set of helicopter vibration measurement data from different flight tests – publicly available on https://doi.org/10.3929/ethz-b-000415151.

Damage tolerant structures and redundant systems are relevant design features for aircraft safety. Multiple vibration sensors installed on a helicopter may arguably be seen as an increased redundancy and tolerance of sensor damage. In all operating conditions of the helicopter collected from different flights, multiple accelerometers were placed at different positions of the helicopter, in different directions (longitudinal, vertical, lateral) to measure the vibration signals with a constant sampling rate of 1,024Hz and sampling length of 1 minute. The training data is composed of 1,677 accelerometer data sequences from healthy sensors. The testing (or validation) data has 594 sequences consisting of streams from either healthy or faulty sensors. Measurement locations and directions in the testing data may or may not be identical to those of the training data. All signals in the dataset were normalised so that absolute values do not have physical meaning.

With this dataset, Airbus hosted a worldwide AI challenge in 2019 to classify the testing sequences into healthy and faulty sequences. Firstly, only the training data were released for model training to ensure that all the models were trained without a priori knowledge about the testing data. After the submission of trained models, Airbus released the testing data.

3.0 Review of results by deep learning methods

Among the 140 competing teams, FSE won the first prize with an F1-score of 0.93 using a DL model based on multivariate anomaly detection with the generative adversarial network (MAD-GAN) (https://www.fujitsu.com/emeia/about/resources/news/press-releases/2019/emeai-20191211-fujitsu-wins-first-prize-for-predictive.html). There is no publication about the details of the winning method. In the original paper of MAD-GAN [Reference Li, Chen, Jin, Shi, Goh and Ng4], an unsupervised multivariate anomaly detection method was proposed based on GANs. The authors used the long short-term memory based recurrent neural networks (LSTM-RNN) as the base models for both the generator and discriminator in the GAN framework. Their MAD-GAN framework treated the entire variable set concurrently and each data stream independently to capture the latent interactions amongst the variables. They used a novel anomaly score (DR-score) to detect anomalies through the discrimination and reconstruction (DR) phases. The test results showed that the proposed MAD-GAN is an effective method in detecting anomalies caused by cyber attacks on some complex real-world digital systems. Apparently, FSE’s winning result proved the efficacy of MAD-GAN in detecting anomalies caused by faulty sensor measurements.

Garcia et al. [Reference Garcia, Michau, Ducoffe, Gupta and Fink5] stressed the fact that it is uncommon to find application cases of unsupervised DL (e.g. AE and CNN)-based anomaly detection. They compared six image encoding strategies such as Gramian angular field, Markov transition field, recurrence plot, grey scale encoding, spectrogram and scalogram to transform the raw time series data into images for a CAE network. They defined a more robust encoding method by modifying each of these six existing algorithms. Training the DL model only on healthy condition data, they extracted the 99th percentile in the distribution of the residuals of all sub-series to define the detection threshold τ. They then monitored the maximum residual over the sub-series (for the detection of local anomalies) and measured it against the threshold τ beyond which an anomaly is considered detected. Using the Airbus dataset, they conducted a comprehensive study comparing the modified and the existing encoding methods and showed an improved performance by using the encoded images against using the raw time series. All the modified versions were observed to perform better than their un-modified counterparts, in which the scalogram indicated the best performance with an F1-score of 0.91 and area under the curve (AUC) score of 0.92.

There are several highly cited studies in the literature that demonstrated improved MHM results delivered by DL, which may potentially be applied to the Airbus dataset. General consensus is that it is difficult to feed the raw sensory data like the Airbus dataset directly into classification and regression models. This is why many previous works focused on feature extraction/fusion methods that requires expensive human labour and high-quality expert knowledge. Based on the development of DL methods in previous years, which redefined representation learning from raw data, Zhao et al. [Reference Zhao, Yan, Wang and Mao7] designed a deep neural network structure of convolutional bi-directional long short-term memory networks (CBLSTM) to address the raw sensory data. The CBLSTM firstly used CNN to extract local features that are robust and informative from the sequential input. The bi-directional LSTM was used to encode temporal information, where the LSTM captures long-term dependencies and models the sequential data, and the bi-directional structure allows the past and future contexts to be captured. Then, fully connected layers and the linear regression layer were stacked on top of bi-directional LSTMs to predict the target value. They applied this method to a tool wear test dataset, and found that the CBLSTM was able to predict the actual tool wear with raw sensory data input. The experimental results showed that their model outperformed several state-of-the-art baseline methods.

Li et al. [Reference Li, Zhang and Ding8] proposed a novel intelligent remaining useful life (RUL) prediction method based on DL. The time-frequency domain information was explored for prognostics, and multi-scale feature extraction was implemented using the CNN. A popular rolling bearing dataset prepared from the PRONOSTIA test platform was used to show the effectiveness of the proposed method. By comparing other approaches, they demonstrated the superiority of the proposed method with highly accurate RUL prediction. Using transfer learning to enable and accelerate the training of deep neural network, Shao et al. [Reference Shao, McAleer, Yan and Baldi9] developed a new DL framework to achieve highly accurate machine fault diagnosis. They demonstrated that the proposed method was faster to train (e.g. less epochs) and more accurate (e.g. quantified by both positive and negative predictions) by comparing with other existing methods. They first converted original sensor data to images using a wavelet transform to obtain time-frequency distribution images. They used a pretrained network to extract lower-level features. They then fine-tuned the higher levels of the neural network architecture with labelled time-frequency images. They used three datasets of induction motors, gearboxes and bearings with sizes of 6,000, 9,000 and 5,000 time-sequences, respectively, to verify the effectiveness and generalisation of the proposed technique. They achieved a test accuracy of nearly 100% on each dataset, and a significant improvement from the previous benchmark of 94.8–99.64% in accuracy with the gearbox dataset.

4.0 Results by non-deep learning-based techniques

We will use the results from the above two studies with the Airbus dataset where F1-scores were 0.93 and 0.91 by FSE and Garcia et al., respectively, as the benchmark for us to compare with simple statistical analyses, a distance-to-centroid method and a one-class SVM classifier.

4.1 Simple statistical analysis method

Using basic statistical analyses, we propose a new simple statistical technique in the paper to classify the sensor health data. We firstly calculate the two most commonly used statistics, i.e. the mean μ and the standard deviation (STD) σ, for every sequence (1,677 in total) in the training data. We then visualise the distributions of the mean and STD values by a 2D histogram (histogram2 in Matlab) as shown in Fig. 1. As we can see that the distributions are widely spread on both side of the mean values and mostly on the right side of the STD values. With a given rate of outliers, e.g. 5% (or 2.5% on either side) as opposed to the two-sigma principle for a Gaussian distribution, we can draw the boundary lines as indicated by the four red dashed lines in Fig. 1. This rate can be prescribed based on the practical requirements of false positive (or false alarm) rate or false negative (or missed detection) rate. For example, in aerospace industry false negative can be fatal thus it should be minimised by using a relatively large rate of outliers, such as the 5% chosen here.

Figure 1.

2D histogram of mean and STD values in the training dataset (red dashed lines are bounds set by 95th percentiles).

In classification with the testing (or validation) data, we obtain the μ and σ values for each of the 594 sequences and compare them with the boundary values obtained from the training data. If either the μ or σ value from a sequence in the testing data goes beyond the respective boundary values, we classify this as positive, i.e. the sequence was from a faulty sensor. This represents the essence of our simple statistics-based classification method. The classification result is presented in a confusion matrix shown in Fig. 2. In the testing data, the ground truth is that there are 297 negative sequences (measured by healthy sensors) and 297 positive sequences (measured by faulty sensors). We can see that the true positive (TP) rate, or the recall score, is very high at 99.66%, and a good F1-score of 0.961, which is in fact better than the results delivered by the DL-based methods at considerably reduced computational cost and complexity. The F1-score here is defined as:

\begin{equation}{\rm{F}}1{\rm{\;}} = {\rm{\;}}{{2 \times {\rm{TP}}} \over {2 \times {\rm{TP\;}} + {\rm{\;FP}} + {\rm{\;FN}}}}\end{equation}

where TP, FP and FN are true positive, false positive and false negative, respectively.

Figure 2.

Confusion matrix using a simple statistical analysis method with a recall score of 99.66% (296/297) and a F1-score of 96.1%.

When other outlier rates are chosen, the results are summarised in Table 1. We can see the results at the outlier rate of 3% have improved the classification performance with an F1-score of 0.9732. If we would know a priori the ground truth in the testing data, we could further optimise the F1-score by searching for the outlier rate. We found that the best F1-score of 0.98 can be achieved by setting the outlier rate to 3.6% while using the μ and σ values as the features. We can replace the STD with the skewness S (a 3rd order statistic), where the corresponding results can be seen in Fig. 3 and Table 2. Obviously, replacing STD with skewness produces lower F1 scores. The mean plus skewness combination may not be able to detect the possible faulty-sensor sequence of all-zero values, which has a STD of zero and a skewness of infinity (or NAN – not a number).

Table 1.

Classification with mean and STD as features

Table 2.

Classification with mean and skewness as features

Figure 3.

2D histogram of mean and skewness values in the training dataset.

The corresponding results with the three-feature combination of μ, σ and S are listed in Table 3. We can see that an extra feature of skewness only improves the performance marginally from the μ and σ combination at the outlier rates of 1 and 3%, and produces weaker performance at the 5% outlier rates. Further with the three-feature combination, we have found the highest recall score (100%) at outlier rate of 3.85%, and the highest F1-score (98.01%) at 3.57% outlier rate, as shown in Fig. 4. It is worth noting that the detection criterion is through an ‘OR’ logical operator, where as long as one of the features goes out of bounds, we will have a positive (or faulty sequence) detection. Perhaps the ‘OR’ operation among the three features made the difference for the superb performance by the simple statistical analysis method, which will be explored further in Section 4.2. In addition, we added the fourth feature of kurtosis (the 4th order statistic) and found that the added feature does not help improve the performance. Despite the fact that it would be difficult to choose the right outlier rate before knowing the ground truth a priori, we can demonstrate that the simple statistical analysis method with any reasonable outlier rate is at least comparable to the two DL-based methods discussed previously.

Table 3.

Classification with mean, STD and skewness as features

Figure 4.

Confusion matrices using a simple statistical analysis method with the highest recall score of 100% (left) and the highest F1-score of 98.01% (right).

4.2 Absolute distance to centroid method

To verify that the ‘OR’ operation among the three features was the key to achieve the remarkable performance by the simple statistical analysis method, we can fuse the three features into a distance-to-centroid metric for classification with the selected outlier rates, which is another novelty of the paper. The outlier is measured by the absolute distance to the centroid of the three statistical features. Using the same statistical features such as mean, STD and skewness for the training data, we firstly calculate the centroid (same as the median for this Airbus dataset), denoted as C _t = [C _μ , C _σ , C _S ]. For the validation data, we calculate the features (mean μ, STD σ and skewness S) for each of the 594 sequences, then we obtain the absolute distances D _v by the sum of the three absolute differences

\begin{equation}{D_v} = \left| {\mu - {C_\mu }} \right| + \left| {\sigma - {C_\sigma }} \right| + \left| {S - {C_S}} \right|\;\end{equation}

With two selected rates of outliers at 3 and 5%, we obtained the confusion matrices shown in Fig. 5. We can see that the absolute distance to centroid method is not performing as well as the simple statistical method with the ‘OR’ operation and the benchmark DL methods by the F1-score, despite that the former has the highest precision score of 100%. The low recall score (79% and 82%), meaning too many missed detections of faulty sensors, tend to prevent this method from being the ideal choice for this application which is to identify faulty sensors and for other aerospace applications. The results presented in Fig. 5 confirms the importance of the ‘OR’ operation among the three features to achieving the better performance by the simple statistical analysis method.

Figure 5.

Confusion matrices using the absolute distance to centroid method with the highest precision score of 100% but lower F1-scores than those in Section 4.1 and the benchmark scores by the DL methods.

4.3 One-class SVM classification with simple statistics

SVMs were developed to classify datapoints separated by complex (i.e., non-linear) multidimensional boundaries [Reference Cortes and Vapnik10]. Subsequently, SVMs were extended to address single-class classification problems, where the points belonging to the single class are separated by a hyperplane from the points regarded as ‘outliers’. For this reason, one-class SVMs are considered as robust unsupervised outlier detection algorithms, and they have been widely used in machine fault monitoring and anomaly detection problems over the past two decades [Reference Fernández-Francos, Marténez-Rego, Fontenla-Romero and Alonso-Betanzos11, Reference Widodo and Yang12]. One advantage of using one-class SVMs for machine condition monitoring is that they only require healthy data to fit a hyperplane that is then used to identify anomalous behaviour (i.e. points outside the hyperplane). They also have the benefit of providing a strong generalisation ability using relatively small-sized datasets [Reference Bi, Bennett, Embrechts, Breneman and Song13], which is a crucial aspect for machine condition monitoring applications.

Therefore, we tested the ability of a one-class SVM to identify the anomalous or faulty sequences in the Airbus dataset. The one class SVM was set up with radial-basis-functions kernel and a boundary factor of υ = 0.1 (usually the default value). Instead of giving the full sequences to the SVM, we used the same pre-computed statistical features (mean, STD, skewness and kurtosis) as inputs. When using mean and STD only, a F1-score of 97% was achieved (Fig. 6), which compares well with the F1-score of 96% in Fig. 2. We also conducted a leave-one-out analysis to see which features were the most important out of the four tested, the accuracy metrics for each case are presented in Fig. 7.

Figure 6.

Classification result using a one-class SVM with mean and STD as features.

Figure 7.

Leave-one-out analysis showing the effect of excluding one of four different statistical features from the one-class SVM classification.

It seems that the most important feature is the mean, as when it is left out the accuracy drops significantly, while when leaving out the kurtosis, all four accuracy-metrics improve to almost 95%. This is in agreement with the previous results with simple statistical analysis method.

We further show in Fig. 8 the performance of different combinations of statistical features, including mean-STD-skewness as well as the same combination including peak-to-peak (maximum minus minimum of each sequence). We can see that the mean-STD-skewness combination performs slightly worse than mean-STD only, while including peak-to-peak produces the best result with a recall score of 0.98 and F1-score of 0.98.

Figure 8.

Classification results of one-class SVM with different combinations of statistical features.

4.4 Analysis of the ground truth data

To analyse the ground truth data, we propose a simple way of visualising the training and validation data. Figure 9 displays scatter plots of two statistical features, μ and σ, along with zoomed-in versions of two marked areas in Figs 10 and 11. The marked areas contain 26 and 33 positive data sequences acquired by faulty sensors, which are challenging to distinguish from negative points in the training data (represented by green circles). In particular, the area in Fig. 11 presents a difficult case, requiring a lower threshold of σ < 0.06 for better classification. Without this threshold, the best classification result achievable through simple statistical analysis is an F1-score of 93.5%. However, with the lower σ threshold at 0.06, the F1-score can be increased to 98.34% by adjusting other bounds or thresholds that have a lesser impact on the F1-score (see Fig. 12). In addition, nine out of the ten false positive (FP) points shown in Fig. 12 can be located within the area of σ < 0.06 in Fig. 11.

Figure 9.

Scatter plot of the two statistical features.

Figure 10.

Scatter plot of the two statistical features – zoomed version of Fig. 9 (dashed box with 26 positives is the first difficult area in to classify).

Figure 11.

Scatter plot of the two statistical features – zoomed version of Fig. 9 (dashed oval with 33 positives is the second difficult area to classify).

Figure 12.

Classify with mean of [−0.2, 0.7], or STD of [0.06, 1.5] or skewness [−0.6, 0.7] where the lower threshold of STD at 0.06 is the most influential bound.

It is suspected that the distance-to-centroid method misclassified most of these two difficult areas, resulting in a F1-score of only 90%. These areas account for approximately −10% in true positive (TP) cases and +10% in false positive (FP) cases if misclassified, i.e. (26+33)/594 = 0.0993, which may explain why the distance-to-centroid method reaches a ceiling F1-score of around 90%, as shown in Fig. 5. This suspicion is supported by Fig. 13, where 54 false negative (FN) points are marked by red asterisks. It is clear from Fig. 13 that the dashed circle area in Fig. 11 is mostly misclassified (33 out of 43).

Figure 13.

Classify with distance to centroid method at 5% outlier rate, misclassified (FN) points in red ^* symbols and the dashed circle area is mostly (33 FN out of 43) misclassified.

5.0 Concluding remarks

In traditional machine learning, unsupervised classification can be a challenging problem. Deep learning-based methods have demonstrated significant potential to address this challenge. With the Airbus dataset, deep learning methods, such as the GAN and CAE, can deliver good performance in classifying sensor data into ‘good’ or ‘bad’ when the deep learning models are trained on ‘good’ data only. In this paper, we are not disputing the effectiveness of deep learning methods, rather we want to remind people that traditional statistical analysis and machine learning methods can often perform as well as, and sometimes better than, the newer and more sophisticated deep learning methods. In the example of the Airbus dataset, most of our simple statistical analysis methods and the one-class SVM with simple statistical features can outperform the deep learning counterparts. The absolute distance to centroid method performed less effectively than the DL learning benchmarks. However, one might argue that we could have seen the testing data prior to forming our framework, which would make the comparison unfair. Our counter argument would be to begin by using proven simpler methods as benchmarks before starting the journey of applying more complex and computationally expensive deep learning methods. For example, we have demonstrated that we could use the mean and standard deviation and 5% outlier in the training data, refer back to Fig. 1, to set up the boundaries/thresholds for anomaly detection without the need of any fine tuning by the testing data.

In conclusion, for MHM problems, it is not necessarily true that DL methods are always superior to traditional methods, and it is a good practice to start solving a MHM problem with simpler methods. Similar views and arguments can be found in a case study by Wang et al. [Reference Wang, Taylor and Rees14] in another DL application to MHM. Based on the results present here, we therefore recommend that where possible all future published studies of applying DL methods to MHM include appropriately selected traditional reference methods.