1. Introduction
Integrating deep learning techniques and artificial intelligence (AI) has become a driver for revolutionary improvements in the constantly changing healthcare scene. One promising avenue within this intersection is the use of deep learning for intelligent healthcare emotion analysis, specifically focusing on the identification of pain through facial expressions and feature analysis. Pain assessment is a crucial aspect of patient care, influencing diagnosis, treatment, and overall well-being. Traditional pain assessment methods often rely on self-reporting, which can be challenging in cases where patients are non-verbal, paediatric, or cognitively impaired. The fusion of deep learning, facial expression analysis, and feature extraction holds the potential to revolutionise pain assessment, providing a more objective and nuanced understanding of patients’ emotional states.
Pain is a complex and subjective experience, and its accurate assessment is pivotal in ensuring timely and effective medical interventions. Traditional pain assessment tools, while valuable, have limitations, particularly in cases where patients face communication barriers. Non-verbal cues, especially facial expressions, have long been recognised as crucial indicators of pain [Reference Bisogni, Castiglione, Hossain, Narducci and andUmer1]. However, interpreting these cues manually poses challenges in terms of accuracy and objectivity. The advent of deep learning offers a promising avenue to overcome these challenges, as it can enable automated and more precise emotion analysis based on facial expressions [Reference Rodriguez, Cucurull, Gonzàlez, Gonfaus, Nasrollahi, Moeslund and Roca2]. Facial expressions are potent communicators of emotional states, and the link between specific facial expressions and pain has been extensively studied. Pain-related facial expressions, commonly referred to as pain faces, include components such as brow lowering, eye tightening, and grimacing [Reference Pandit, Schmitt, Cummins and andSchuller3]. These expressions not only convey the presence of pain but also offer insights into its intensity and nature. Leveraging deep learning to analyse these intricate facial expressions provides an opportunity to quantify pain objectively and in real time, facilitating more accurate and timely interventions.
Deep learning, a subset of machine learning, has demonstrated unparalleled capabilities in processing and analysing complex datasets. Convolutional neural networks (CNNs) and recurrent neural networks are well-liked deep learning architectures that have achieved outstanding results in sequence analysis and an equally impressive record in image analysis [Reference Egede, Song, Olugbade, Wang, Amanda, Meng and Bianchi-Berthouze4]. Applying these deep learning techniques to facial expression analysis allows for the extraction of nuanced features, enabling a more sophisticated understanding of emotional states such as pain. Beyond the recognition of facial expressions, feature analysis plays a crucial role in identifying patterns associated with pain. Features can encompass a range of attributes, including facial muscle movements, colour variations, and spatial configurations. Extracting these features using deep learning algorithms enhances the granularity of pain assessment, contributing to a more comprehensive and accurate model [Reference Kashif, Hussain, Munir, Siddiqui, Abbasi, Aakif and Song5]. Feature analysis is particularly beneficial in distinguishing between different pain intensities and types.
While the potential benefits of a deep learning approach for intelligent healthcare emotion analysis are substantial, challenges persist. Variability in individual expressions, cultural influences, and the dynamic nature of pain itself pose challenges in creating a universally applicable model [Reference Altameem and Altameem6]. The good news is that these problems also provide chances to study and improve algorithms, which can make the suggested method more resilient and applicable in other contexts. With a focus on pain diagnosis using facial expressions and feature analysis, this research seeks to investigate and build a deep learning-based framework enabling intelligent healthcare emotion analysis.
The successful implementation of a deep learning approach for intelligent healthcare emotion analysis has the potential to revolutionise pain assessment in diverse healthcare settings. Beyond its applications in pain management, the developed model could find utility in various medical contexts, including mental health, neurology, and paediatrics. Moreover, the research contributes to the broader field of AI in healthcare, paving the way for more sophisticated and objective assessments of patients’ emotional states. A patient-centered approach is integral to modern healthcare, emphasising the importance of understanding and addressing patients’ experiences and needs. The proposed deep learning framework aligns seamlessly with the principles of patient-centred care by offering a more personalised and precise method of pain assessment. This, in turn, can lead to tailored interventions and improved patient outcomes [Reference Bargshady7].
Throughout the research process, ethical considerations will be paramount. Patient consent, privacy, and the responsible use of sensitive data will be strictly adhered to. Transparency in the algorithmic decision-making process and considerations of potential biases in the dataset will be addressed to ensure the ethical integrity of the research [Reference Leo, Carcagnì, Mazzeo, Spagnolo, Cazzato and andDistante8]. The subsequent sections of this thesis will delve into a comprehensive review of existing literature, detailing the theoretical frameworks and methodologies employed in previous studies related to deep learning, facial expression analysis, and pain assessment. What follows is an explanation of the study’s methodology, which will detail the research strategy, data gathering procedures, and analytical approaches. Results, analysis, and debates will be presented in subsequent chapters, with implications for healthcare practice, technology development, and future research addressing any remaining questions.
The integration of deep learning techniques into healthcare emotion analysis, specifically targeting pain identification through facial expressions and feature analysis, represents a significant leap forward in patient care. This research seeks to contribute to the evolving landscape of intelligent healthcare, offering a more nuanced and objective approach to pain assessment with the potential to enhance the overall quality of healthcare delivery.
2. Literature survey
Human emotions are impromptu cognitive states of feelings, and the study of facial recognition is demanding. The use of machine learning and neural networks enables the identification of emotions, while features are collected from pictures through the application of CNN. The CNN model achieves an accuracy of 80% for the classification of four emotions and 72% for the classification of five emotions. The accuracy of CNN Pattern 2 is 79% for the number four and 72% for the number five [Reference Sowmya, Alex, Kanavalli, Supreeth, Shruthi and Rohith9]. With a preliminary accuracy of 79.8% for seven basic emotions, one study that used a tagged dataset with more than 32,000 photos demonstrated CNN’s promise in Facial Emotion Recognition (FER) without requiring much optimisation. Although more improvement in model training and data processing may improve FER accuracy and dependability in practical applications, these results highlight deep learning’s capacity to identify a variety of emotional expressions [Reference Babajee, Suddul, Armoogum and Foogooa41].
Stacking ensemble learning methods have been increasingly utilised in machine learning to analyse student populations, offering valuable insights for identifying the most effective method to predict symptoms of social anxiety. Yang et al. initially employed the Jenks Natural Breaks technique to classify features. They subsequently utilised the Apriori algorithm to examine correlations between features based on the classification findings. This allowed them to identify behaviour aspects that are connected with students’ mental health state. The researchers finally developed a Particle Difference Neural Network model to forecast the mental health state of pupils. The suggested model exhibited superior performance compared to conventional machine learning and deep learning models. It demonstrated rapid convergence and achieved more accurate predictions of students’ mental health conditions [Reference Li, Xu, Chen, Huang, He and Xie10]. Furthermore, Suri introduced a fusion model that combines Random Forest (RF), logistic regression (LR), and AdaBoost. This fusion model was utilised to create a psychiatric crisis early warning system, as described in their work [Reference Cheng, Zhang, Zhang, Meng, Hong, Li and Wang11]. In addition, Zhang et al. developed an ensemble that stacks a learning model that utilises numerous algorithms to forecast student success in blended learning. The model comprises polynomial Naive Bayes, AdaBoost, and Gradient Boosting as the main learners, with LR serving as the secondary learner. This arrangement forms a two-layer fusion framework [Reference Ayache and Alti12].
A different research [Reference Huang, Xia, Li, Wang and Feng13] proposes a framework for the identification of facial expressions. The Active Shape Model is utilised to conduct feature selection on facial images. These characteristics are utilised for training several classifiers, including a quadratic classifier (Discriminant Analysis (DA)), a multilayer perceptron, and others. The quadratic classifier was determined to have the most precise categorisation outcomes. Multiple studies argue that using certain regions of the face, where pain is visibly present and expressed, would be more beneficial than using the complete facial picture as input data. By following this approach, the model will undergo training to focus on the areas that are most significant for the expression of pain, while excluding any irrelevant information or disturbances. Huang et al. [Reference Xin, Lin, Yang and Zheng14] first identified the facial areas of the left eye, right eye, nose, and mouth. They then used a multi-stream CNN to extract features, which included four sub-CNNs, one for each area. An important aspect of this system was that the extracted characteristics were given learnt weights to ensure that each region’s contribution to pain expression was appropriately accounted for. In a study conducted in [Reference Cui, Huang, Ni and Feng15], researchers integrated a 9-layer CNN with an attention mechanism (AM). This AM assigned varying weights to different parts of the face based on their expressiveness. The result was the generation of attention face maps, which led to predictions that were up to 19% more accurate. The authors introduced a multi-scale regional attention network (MSRAN) in ref. [Reference Li, Zhu and Zhao16], which employs several cropping areas from each video frame. Moreover, it integrates a self-attention and a connection attention module to highlight pain-related areas and investigate their correlation. In addition to generating saliency maps, Li et al. [Reference Schroff, Kalenichenko and Philbin17] utilised the concept of [Reference Peng, Huang and Zhang18] to integrate contrastive and multi-task training using an autoencoder. Furthermore, akin to the detection of fundamental facial expressions, a significant obstacle in estimating pain intensity lies in the fact that some individual attributes, such as facial shapes, can lead to substantial variations in the same emotional state. Consequently, it is sometimes difficult to differentiate between two neighbouring degrees of pain expression due to the substantial diversity within each intensity. Peng et al. [Reference Xin, Li, Yang, Lin and Zheng19] examined face shape data and created a sophisticated multi-task network that analysed the connection between pain detection and shape. This approach significantly improved the accuracy of pain assessment. In a similar manner, Xin et al. [Reference Dai, Broekens and Truong20] introduced a new multi-task framework that integrates a CNN feature learning module with an autoencoder attention component. This framework not only estimates the subject identity but also takes into account the fact that different people exhibit distinct pain symptoms. The studies demonstrated exceptional performance on publicly accessible datasets.
Many researchers argue that most studies train models, whether with deep or shallow architectures, on dataset-specific characteristics rather than genuine pain-related variables. In addition, the majority of research examining the facial manifestation of pain employ validation techniques on the same dataset, while giving less consideration to performance across different datasets. These techniques are not feasible in real-world scenarios. Dai et al. [Reference Andrade21] conducted experiments to solve these challenges by combining pain and emotion-detection datasets. Their goal was to construct a real-time pain assessment system that had improved generalisation capabilities. Their research emphasises the significance of evaluating different datasets, conducting tests in real time, and the necessity of a pain dataset that is both balanced and environmentally valid [Reference Zhu and Ramanan22]. Table I summarizes the literature review below.
Summary of literature review.
2.1. Research gap
The proposed model addresses several research gaps identified in the existing literature. One critical research gap that the model successfully tackles is the need for an integrated approach to pain intensity prediction using facial expressions. Existing methods often focus on either statistical or deep learning-based techniques independently, lacking a comprehensive strategy for combining the strengths of both. How adaptable is the unified framework to diverse healthcare scenarios, accounting for variations in data types, processing requirements, and security constraints across different contexts? How effectively does the model tackle noise reduction during image preparation and handle dynamic and real-time scenarios, where noise levels may exhibit unpredictable variations? In scenerios such as Cross-cultural effectiveness and potential biases across different demographic settings. The identified research gaps are addressed in the proposed system. However, exploring the computational efficiency and resource requirements of the proposed Smart Automated System (SASys) is also crucial to ascertain its applicability in resource-constrained healthcare environments.
3. Proposed method
A growing number of people are turning to cloud-based mobile apps that can instantly analyse films and transmit data. Like any other programme, this one has a front end and a back end. Separate portions of the back end and front end run on different devices; the former uses the cloud, and the latter uses a mobile device. The use of cloud computing has greatly improved the accessibility of data processing and computing resources. Devices that have enough RAM and processing power should be able to run even the most complex apps with no problem when employing our proposed approaches. In this section, an SASys is introduced with the objective of predicting the severity of pain on a human facial region. With any luck, this endeavour will deliver high-quality modules for an intelligent healthcare framework by bringing computation and cache closer to the user. In order to put the proposed method into action, this challenge has been divided into four distinct pieces. You can see the proposed block diagram below (Figure 1).
Proposed block diagram.
3.1. Image preparation
Important to the study at hand is the collection of pain-related facial expressions from video sources. When pain-based facial expressions are added to the data collection process, a variety of noises are produced. Due to the substantial effect these noises have on the picture quality and the information accessible inside it, inappropriate experimenting takes place. In order to extract the face area and determine emotions connected to pain, a colour picture J is taken into consideration in the image preparation operation. For accurate face detection across all possible expressions, use a tree-structured component model [Reference Safavian and Landgrebe23]. There are 68 landmark points calculated for the frontal face using our method, as opposed to 39 for the profile face. Normalisation is applied to the obtained facial region following face detection. This process utilised a bilinear image interpolation approach to standardise the retrieved face region G, using a fixed picture size of O × O. The proposed SASys’s face detection process is shown in Figure 2.
The task of picture preprocessing for the proposed Smart Automated System.
In this step, we use the normalised facial area G to compute the features. In this case, we have employed (i) approaches based on deep learning and (ii) statistical feature representation to extract useful and unique features, with the latter being followed by machine learning-based classification techniques. Each of these approaches has been tested independently. According to the proposed method, the best course of action is to combine the outcomes of all available strategies. Both performance and the proposed plan are strengthened by performance integration.
Because colour picture J provides a more comprehensive collection of information than greyscale photos, it is an essential component of the image preparation process for improving the precision of facial expression analysis as it pertains to pain. Colour photos are superior because they pick up on finer details, such as changes in skin tone and the little fluctuations in facial colouration that accompany pain emotions. With these further details, face landmarks and characteristics may be detected more accurately, which in turn improves the effectiveness of the analytical processes that follow. One further thing that makes the analysis more robust is utilising a tree-structured component model for reliable face identification. This makes sure that the collected facial expressions are accurately aligned and normalised.
3.2. Methods based on statistics and then machine learning
What we call “feature extraction” is actually a series of steps that start with an input image and end with a collection of traits that make the input picture stand out. This face preparation was successful, as seen in Figure 2. There is a regular distribution of tones or an inconsistent distribution of tones in the extracted face regions. By taking these textural aspects into account during feature computation, we were able to extract more discriminant and disjunctive features, improve image quality owing to various noise artefacts, and reduce issues in intra- and inter-class similarity or dissimilarity. In discriminatory feature representation techniques, structural, statistical, and transformed-based methods are the mainstays for evaluating textures of facial areas. When facial muscles are relaxed, emotions can be seen. The feature extraction module interprets computer vision perceptions and extracts permissible non-rigid information in order to express action units. The structure of the framework and numerous feature descriptions make it easy to modify the flexible face details. Attributes can be either geometric or dependent on how something looks. Using reference points and coordinates, the geometric-based feature extraction method retrieves a plethora of geometric measurements. In order to extract features, the appearance-based technique additionally makes use of pixel intensity values.
Examining the texture pattern statistically is more practical and convenient than using methods based on structures or transformations. This analytic approach assesses more data from the intensity values of the image’s pixels. Gathering and displaying image-based features based on look is made easier with its support. Because G, the preprocessed facial image, could contain patterns that are neither regular nor non-regular [Reference Saxena27]. This is why statistical methods are superior at analysing irregular patterns as well as regular ones. In this case, feature computation has considered both global and local-to-global feature representation methodologies. The greyscaled version of the facial image
$G\ of\ O\,\times O\times 3$
is created during the statistical-based feature calculation. Next, its global feature representation, denoted as gɦ, is obtained by computing the Local Binary Pattern (LocBP) and Histogram of Oriented Gradients (HoOG) features from the entire G. In order to get more specific information, every G is split into two equal parts, one vertically
$\left(\frac{\boldsymbol{\mathsf{g}}_{im}}{\boldsymbol{\mathsf{g}}_{\mathfrak{wm}}}\right)$
or four equal parts,
$(g_{\mathfrak{mh}})$
. In this context,
$g_{\mathfrak{h}}$
denotes a globally computed feature, g_im denotes a horizontally local feature,
$\boldsymbol{\mathsf{g}}_{\mathfrak{wm}}$
denotes a vertically local feature, and fLG denotes a locally to globally calculated feature. A feature vector called
$\boldsymbol{\mathsf{g}}_{im}$
is created by merging features from both the horizontal and vertical halves of G. The feature vector
$\boldsymbol{\mathsf{g}}_{\mathfrak{wm}}$
is also produced by merging features from the two vertical halves of G. A feature vector
$(g_{\mathfrak{mh}})$
is created by merging features taken from the four vertical regions (first horizontally and then vertically) of G.
Feature vectors, shown in Figure 3, represent the processes of feature extraction
$\boldsymbol{\mathsf{g}}_{\mathfrak{h}}$
,
$g_{{i}^{m}}$
, and
$g_{{\mathfrak{w}^{m}}}$
.
Methodological strategy for statistically representing facial features in the GH region.
In order to proceed to machine learning classification, one must first retrieve characteristics from several methods (Figure 3). One goal of the classification assignment is to identify the types of pain intensity as seen on the patients’ faces. Due to the fact that the feature vectors that were recovered include a variety of discriminant values, the classifier is able to use these features to create predictions in the classification job [Reference Szegedy, Liu, Jia, Sermanet, Reed, Anguelov and Rabinovich28]. Some classifiers that have been used for classification are decision tree (DT) [Reference Safavian and Landgrebe23], support vector machine (SVM) [Reference Al Amrani, Lazaar and El Kadiri24], K-nearest neighbour (KNN) [Reference Manocha and Girolami25], and LR [26]. A classifier’s input, learning strategy, and target function (model) are its three main components. For the same training and testing datasets, these classifiers produce different results because their underlying mathematics are different. There are two possible categories for facial pain detection in this research: second-class (PN) and pain (P) issues and third-class (PN, PM, and PH) issues. Data classification techniques with several levels of granularity have therefore been employed. If we look at the F1-score and the proportion of correct recognitions (accuracy), we can observe how these parameters affect the classification performance of the suggested system.
Different types of characteristics provide different sized and composed feature vectors when using statistical and appearance-based approaches. As an example, the statistical-based feature vectors (
$g_{\mathfrak{h}}$
) are usually 1 × 81 pixels in size and are generated using LocBP and HoOG features extracted from the whole picture. In order to get more specific details, these vectors are then divided into local components, which provide local feature vectors that are generally 1 × 81 in size and can be horizontally (
$g_{im}$
) or vertically (
$g_{wm}$
) orientated. These vectors, when joined, provide a complete picture of the texture patterns in the face image.
Pixel intensity data and geometric measurements are used to construct feature vectors for the appearance-based algorithms. The features that make up these vectors are both detailed and discriminative since they incorporate both global feature representation and local-to-global features. While their exact dimensions are implementation-dependent, these vectors often have a common structure that integrates both local and global characteristics. These specific and discriminative feature vectors have a major impact on how well machine learning classification works for predicting pain intensity. Classifiers that use a mix of global and local characteristics are better able to pick up on both broad and subtle facial emotions, which in turn allows for more reliable predictions.
3.3. A novel deep learning-based approach
Lately, multi-task deep learning algorithms have demonstrated potential in extracting distinctive characteristics from photos and videos collected in various regions of the world using CNN approaches. Due to their exceptional performance in challenging image processing applications, CNN approaches have gained significant popularity. CNNs employ diverse kernels to execute mixed convolution operations on input pictures. After performing convolutional operations, neural networks that utilise weight sharing to provide different network training parameters implement the pooling approach [Reference Tian, Kanade and Cohn29]. CNN models are now being utilised to address and resolve several computer vision problems. These include but are not limited to scene interpretation, object recognition, object detection, face recognition, texture classification, and numerous more [Reference Umer, Rout, Pero and Nappi30]. Traditional statistical and structural computer vision methods are not as effective as recent edge caching of generic CNN feature representation [Reference Szegedy, Liu, Jia, Sermanet, Reed, Anguelov and Rabinovich28] in the task of recognising facial emotions in individuals. These general CNN characteristics are more effective for dealing with occlusion and articulation in face pictures collected in an unconstrained setting. A suggested approach in ref. [Reference Szegedy, Liu, Jia, Sermanet, Reed, Anguelov and Rabinovich28] utilises deep learning features and data augmentation to accurately identify facial emotions.
For this situation, we employ multi-task convolutional neural network (MTCNN) structures that depend on deep learning to acquire features, enabling the classification of pictures into two or three distinct pain categories based on facial expression analysis [Reference Zhang, Tian, Kong, Zhong and Fu32]. Expression changes can be caused by several factors such as age, head-pose, identification bias, and illumination variance. Additionally, the facial expression identification system is plagued by overfitting, bias, and a lack of appropriate training data. The offered techniques [Reference Ioffe and Szegedy33] aim to address the computational complexity of the CNN model and provide solutions to these challenges. MTCNN architectures possess a hybrid design that enables them to extract both shape and texture information from images. The feature descriptors of the proposed approach derive advantages from both the geometric and visual aspects of the shape and texture data. Graphics processing units (GPUs) can address the primary challenge of the CNN model, which is its substantial computing demand. GPUs have the ability to handle various computational tasks, such as managing complex data patterns, employing techniques for pretraining and transfer learning, applying methods for fine-tuning on existing CNN models, and dealing with abstraction and multiple levels of representation. The MTCNN architecture was trained using a vast database, and the network weights were fine-tuned according to the number of classes in the issue [Reference Liu, Li, Shan and Chen34].
This study presents the MTCNN architecture that employs a deep learning methodology to analyse face attributes in order to identify discriminatory patterns and classify them accordingly. The proposed CNN architecture consists of six to eight layers of profound image perturbation. Beginning with convolution, each layer in the architecture performs operations prior to applying the rectified linear unit to the convoluted feature maps. The revised feature maps were utilised to perform max pooling, which extracted discriminant features. The key characteristics were normalised and subsequently passed to the batch normalisation layer. After the completion of each iteration of the CNN, two interconnected layers are appended to refine the collected features from the upper levels. To optimise the performance of the recommended CNN, we integrate various novel approaches to tackle issues related to data imbalance and overfitting. Typical techniques in this area include transfer learning, fine-tuning, and normalisation. Utilising optimisers, batch normalisation, label smoothing, and normalisation can improve the performance of the MTCNN architecture.
The fundamental component of a MTCNN is the convolution operation, which serves as the primary operator in this network. The primary function of this layer is to conduct convolution operations, which are used to extract information from pictures. By employing a kernel or filter with dimensions u × u in the convolution process, feature maps are generated to improve the input image. The convolution operation entails the act of sliding the filter over the input and subsequently applying non-linearity. The matrix product is overlaid at each position on the feature map. Ultimately, a technique for modifying parameters is employed to enhance the efficiency of the proposed SASys. A down-sampled feature map may be generated by using max pooling, which involves selecting the highest value inside each patch of the feature map. It is customary to incorporate it after a convolutional layer. The max-pooling layer has four benefits: (i) enhanced translation quality, (ii) reduced processing expenses, (iii) faster matching, and (iv) improved accuracy. A MTCNN utilises a fully connected layer to produce a vector that is directly proportional to the number of class labels. This fully connected layer combines all the feature mappings into a unified vector representing the class scores. Dense layers employ weight values to establish connections between all input and output levels. Batch normalisation serves as a regulariser, allowing the model to benefit from accelerated learning rates [Reference Bergstra, Breuleux, Bastien, Lamblin, Pascanu, Desjardins and Bengio35]. Feature fusion from convolutional and pooling layers is happening, and those features are scheduled for multi-tasking. It minimises the required training stages and required time while enhancing accuracy for tasks involving photo classification. Batch normalisation enhances the gradient flow throughout the network by reducing the sensitivity of gradients to variations in parameter size or initial values. Based on the information presented, we propose a MTCNN architecture that incorporates the previously mentioned layers addressed in this study. The suggested MTCNN architecture may be observed in Figure 4.
The suggested system’s CNN architecture.
In these circumstances, the image will be consistently sent to the input layer. In order to obtain the intricate image (feature map) for each filter, the convolution process is carried out using separate kernels (filter banks). Due to the growing number of filters and the exacerbation of convolution challenges with bigger enhanced pictures, the network will be required to do extensive calculations. Parameters in this context operate in a manner akin to finely adjusted weights in sets of filters. Increasing the number of network parameters in max-pooling layers intensifies the computational complexity [Reference Lucey, Cohn, Prkachin, Solomon and Matthews36]. In the max-pooling layer, a 2 × 2 filter size is consistently used on each feature map received from the previous layer, wherever it is feasible. The averaging, minimising, or maximum values are subsequently used to calculate a two-down-sample step, initially in a horizontal and then in a vertical direction, using four-digit numbers. Following it is a layer that is completely interconnected. Once the characteristics of each layer are converted into a one-dimensional vector in the fully connected layer, the process is considered finished [37]. Like other fully linked layers, the dense layer also performs a linear operation [Reference Liu, Li, Shan and Chen34]. The probability scores are the final result of a linear operation that is triggered using a function such as Softmax to establish a connection between the input and output.
Crucial elements that affect the performance of the proposed MTCNN architecture are transfer learning and fine-tuning. To derive more complex feature representations, one can construct a more selective collection of features for the suggested SASys by modifying the parameters of the underlying CNN model. By performing model retraining following fine-tuning, specific layers and their associated parameters can be immobilised, resulting in a reduction of computational burdens. Transfer learning enables the use of a pre-trained model on one problem to be applied to other related problems. A “pre-trained model” refers to a model that has undergone training before to its use. We propose a CNN architecture that employs a transfer learning approach to achieve deep learning. The new model incorporates either the pre-trained model’s weights or a set of layers to tackle similar difficulties or employs both simultaneously. Transfer learning has two advantages: less generalisation error and reduced training duration.
3.4. Methods of fusion
To improve the suggested SASys’s performance over pain classification, we use multiple score-level fusion techniques to merge the scores of the SVM classifier and the scores of the deep learning-based approach. The SVM classifier uses features generated from the statistical approach. These methods encompass weighted sum, sum, and product. Using the statistical approach with the LocBP feature, the deep learning-based method, and the statistical strategy with the HoOG feature, I will denote the post-classification scores as
$\varepsilon _{1}, \varepsilon _{2}\ and\ \varepsilon _{3}$
, respectively. To summarise the rule-based approaches, consider the following: (i) The formula for sum is
$\varepsilon =\varepsilon _{1}+\varepsilon _{2}+\varepsilon _{3}$
. (ii) The formula for product is
$\varepsilon =\varepsilon _{1}\times \varepsilon _{2}\times \varepsilon _{3}$
. (iii) The formula for weighted sum is
$\varepsilon =x_{1}\times \varepsilon _{1}+x_{2}\times \varepsilon _{2}+{x_{3}}\times \varepsilon _{3}$
, where ε is the fused score and
$x_{1},x_{2}\ and\ x_{3}$
that correlate to the corresponding weights
$x=x_{1}+x_{2}+x_{3}$
.
Several benefits and features of the SVM classifier impacted the choice to combine results from deep learning approaches with scores from the SVM utilising fusion techniques such as weighted sum, sum, and product. SVMs’ capacity to locate the best hyperplane for classification jobs and their efficacy in dealing with high-dimensional data have made them famous. Because of this, they can accurately detect the nuanced changes in facial expressions that correspond to varying degrees of discomfort.
When dealing with overlapping classes or ambiguous facial expressions, SVMs can be very helpful since they give a clear margin of separation between classes. We can improve the accuracy and robustness of pain categorisation by merging SVM ratings with scores from deep learning-based approaches. We used a number of approaches to rectify the problem of an unbalanced dataset. We started by increasing the sample size in classes that were under-represented by using data augmentation approaches. Second, to make sure the classifier gets a fair representation of all classes, we used balanced sampling approaches during training. Finally, we improved the system’s performance in managing unbalanced data by adjusting the weights in the fusion approaches to prioritise the minority classes.
4. Experiments
Windows 10, an Intel Core i5 3.20 GHz processor, 16 GB of RAM, and Python were the tools utilised to construct the SASys in this study. For various tasks, various Python libraries have been used. The suggested CNN architecture and other deep learning-oriented approaches have made use of the Keras and Theano packages.
Two picture datasets were used to test the proposed SASys in this work. The primary dataset we utilise is the UNBC-Master set of expressions related to shoulder discomfort. Each of the three physiotherapy clinics in this database has recorded the cases of 129 patients – 63 males and 66 females – who have complained of shoulder pain. On the grounds of the Master’s University, the action unfolds. While submitting data, participants probably dealt with a wide variety of shoulder pain issues, including tendinitis, bursitis, rotator cuff injuries, arthritis, bone spurs, dislocations, subluxations, impingement syndromes, and capsulitis, among others. Text labels on the images can read anything from “no pain” to “high-intensity pain” in this instance. According to the results, the photos can be categorised as either nonaggressive (PN) (
$v_{1}$
), covertly aggressive (PM) (
$v_{2}$
), or openly aggressive (PH) (
$v_{3}$
). You may see examples of these three types of descriptions in Table II. This database is used to create the images seen in Figure 5. We also use the 2D Face set database
$E_{2}$
with the pain-expression set [46]. Ten males and thirteen females are depicted in these 599 photos. This database typically has a two-class problem. A total of 298 images display the “Pain - Nil” (
$X_{1}$
) expression, while an additional 298 display the “Pain” (
$X_{2}$
) expression. You may see some examples of images from this database in Figure 6 (Table II).
Information about the UNBC photo library as it relates to the proposed system.
Information retrieved from the UNBC database and certain samples.
Various E2 database extracts utilised.
4.1. Label verification
Annotations made by experts were cross-validated with patient self-reports of pain levels in certain instances. The labels should accurately represent the patients’ genuine pain sensations, and this system helps with that. The designations for pain severity were provided by qualified medical specialists who carefully observed each patient’s facial expressions. To ensure the labels are accurate, these professionals have vast experience in pain evaluation. However, there are still obstacles to overcome before we can get clinically relevant ground truth data. To further investigate other ways of enhancing label dependability, further study is required.
5. Results and discussion
In this study, the face region is extracted during picture preprocessing using the tree-structured component model, with the goal of normalising the input picture into a G shape with dimensions
$200\times 200\times 3$
. The three levels of pain are “Pain - Nil,” “Pain - Moderate,” and “Pain - High.” Isolating the pain level is the objective of this endeavour. Sequential classification approaches based on deep learning and statistics and machine learning are both utilised in the proposed system. Combining the results of many operations improves the efficiency of the suggested system. Hence, using Scheme1 (P1), the face region is used to produce feature vectors G
$G g_{\mathfrak{h}}^{MCQ}\in \mathbb{S}^{1\times 256}$
and
$G g_{\mathfrak{h}}^{HOG}\in \mathbb{S}^{1\times 81}$
. The feature vectors obtained from the face regions are fed into a multitude of classifiers. Each dataset is split in half: half is used for training, and half is used for testing. Afterwards, following a 10-fold cross-validation operation, the suggestion system’s average performance on testing data was made public.
After comparing the proposed system to the alternatives – which included LR, KNN, DT, RF, and SVM, multi-task CNN with autoencoder (multi-task CNN + AE), MSRAN, 9-layer CNN with AM (9L-CNN + AM) – Table III reveals that the former performed better. Therefore, our next experiments will make use of the SVM classifier. From each facial region G, feature vectors
$g_{\mathfrak{h}}^{HOG}\in \mathbb{S}^{1\times 81},\boldsymbol{\mathsf{g}}_{\mathfrak{T}}\mathfrak{M}^{HOG}\in \mathbb{S}^{1\times 162}, \boldsymbol{\mathsf{g}}_{w}\mathfrak{M}^{HOG}\in \mathbb{S}^{1\times 162},\ and\ \boldsymbol{\mathsf{g}}_{m}\mathfrak{h}^{HOG}\in \mathbb{S}^{1\times 324}$
are obtained using HoOG features, using Scheme1 (
$Q_{1}$
), Scheme2 (
$Q_{2}$
), Scheme3 (
$Q_{3}$
), and Scheme4 (
$Q_{4}$
). With the help of Monte Carlo Quantification (MCQ) features, the G facial region yields the feature vectors
$g_{\mathfrak{h}}^{MCQ}\in \mathbb{S}^{1\times 256}$
,
$g_{\mathfrak{T}}^{MCQ}\in \mathbb{S}^{1\times 512}, g_{w}^{MCQ}\in \mathbb{S}^{1\times 512},\ and\ g_{m}^{MCQ}\in \mathbb{S}^{1\times 1024}$
.This kind of feature extraction aims to improve the suggested system’s performance by extracting an increasing number of local features.
Image database E2 of the proposed system was described.
The results for HoOG and MCQ characteristics show that Scheme 4 (
$Q_{4}$
) performs better.
5.1. Classification and learning from features
First, there is picture preprocessing; second, there is feature learning with classification; and last, there is feature extraction with classification procedure. The TSPM approach, which was previously described, and this method of picture preparation are identical. A subsequent step involves resizing the normalised face region G to 96 × 96 pixels. The following stage is to input the reduced face pictures from the training set into the proposed CNN architecture (Figure 4). The architecture calls for an image with dimensions of 96 × 96 × 3. In addition to feature computation, the CNN architecture is trained to accomplish pain classification, as seen in Figure 4. To have a better understanding of how these designs function, we’ve begun the process of testing out several configurations for training the CNN architecture. Another option is to use the remaining testing samples to assess how well the trained CNN model performed (Table IV).
The outcomes of utilising multiple ML classifiers following the feature extraction process from Scheme 1 (P1).
Batches and epochs are two elements that impact parameter learning, which is a crucial aspect of CNN designs. Both of these factors influence the architecture’s learning capacities when the network is being trained. There was a happy medium between 700 epochs and 16 batches, which enhanced the suggested SASys’s performance. The suggested system’s performance has been enhanced through the application of transfer learning and fine-tuning. Table V displays the results of the proposed system’s operation, broken down by class:
$E_{2}$
, UNBC 3, and UNBC 2. These results have been reported using the following metrics: accuracy, F1-score, training time, and testing time.
Using a deep learning-based technique, the proposed system’s efficiency.
In Table V, we can see that the accuracy rates for the various classes at UNBC range from 82.47% for the second-class problem to 80.34% for the third-class problem and 75.97% for the second-class
$E_{2}$
problem. When contrasted with the results obtained from the statistical feature technique, the deep learning-based method yields better results in the proposed system. It follows that deep learning facial traits are more accurately portrayed in pain analysis than statistical features. Combining the best parts of both approaches, this study makes the proposed system more efficient. The combined performance will be discussed later on.
5.2. Fusion
Statistical methods (HoOG and MCQ) and deep learning-based features are employed to derive the classification scores in this case. While doing the fusion, many different post-classification approaches were used. These kinds of scores are produced by CNN models and machine learning classifiers. We use the sum-rule, product-rule, and weighted sum-rule approaches for post-fusion fusion after we classify the scores. In terms of the product-rule fusion method, the table data clearly shows that the proposed system performs better than the Ssum-rule, product-rule, and weighted sum-rule approaches. This is because the suggested approach combines classification scores from LocBP, HoOG, and CNN feature representation schemes. In terms of accuracy, the proposed system achieves 82.06% for third-class UNBC and 75.67% for second-class E2 database as shown in Table VI and Table VII. The results of these tests are considered in the comparisons that follow.
Pain in the shoulders as measured using the UNBC-Master database and compared to competing image-based Smart Automated Systems.
We evaluate the proposed image-based Smart Automated System against its rivals using the
$E_{2}$
database.
5.3. Parameter fine-tuning
Improving the suggested system’s efficiency requires fine-tuning its settings. By modifying the learning rate, batch size, and number of epochs, we adjusted the CNN architecture for fine-tuning. The network was trained using pre-learnt weights that were obtained by transfer learning. Our datasets were used for fine-tuning. This method lessens the likelihood of overfitting and aids in attaining greater generalisation. We used a machine that has 16 GB of RAM and an Intel Core i5 CPU. Depending on the size of the dataset, training the CNN architecture might take several minutes, with each epoch taking around 38.67 s. It was practical for use in real-time applications since the inference time was just about 0.04 s per picture.
The system’s architecture and results show that it can perform pain categorisation jobs well, considering these considerations. The present validation gives a strong basis and shows that the system is already working well in its tested situations; nevertheless, further validation of its generalisability might be achieved with other datasets.
6. Discussion
By combining statistical methodologies with deep learning techniques, our methodology improves predictive accuracy and increases the method’s clinical application for pain intensity prediction. Colour photos, precise feature extraction, and fusion methods all work together to provide a strong system that can handle many real-time applications and healthcare settings. The computational complexity of deep learning methods is a possible downside; however; it may be reduced by implementation optimisation and the use of cloud computing resources. It may also be difficult to get the large datasets needed for the cross-cultural validation of the system.
This document details the proposed system’s general architecture, including the procedures and components that make it tick. It details on how the Smart Automated System uses mobile and cloud components, deep learning models, and machine learning classifiers for pain identification. The study goes into depth on several feature extraction technologies, including deep learning techniques (such MTCNN) and statistical methods (like HoOG and MCQ). Both the method and the features (geometric, texture-based, etc.) that are retrieved are detailed in it.
While the study doesn’t go into technical detail about particular hyperparameters or training methods, it does outline the overall techniques, including the usage of CNNs and different classifiers. The training process includes assessing the system’s performance on various datasets, and it also makes reference to utilising libraries and tools like as Keras and Theano. Results and performance measures are included in the article to illustrate the comparison of various models and methodologies. Without going into detail about each hyperparameter and training stage, this data gives an indirect idea of how efficient and successful the suggested strategies are. The study does a good job of outlining the proposed system’s architecture, methodology, and performance, although it could be lacking in technical details on hyperparameters and training techniques. Based on the broad approaches presented, readers may grasp the strategy and evaluate its repeatability with this information.
UNBC database accuracy percentage against other models.
$E_{2}$
database accuracy percentage against other models.
The experimental validation was restricted to just two datasets, as mentioned in the statement. To be sure our results would hold up under different conditions, we used a 10-fold cross-validation in our trials. To account for the ten folds, a total of ten trials were conducted in each experiment. This pertains to the testing scope of the proposed system. It implies that evaluating the system on a smaller set of datasets could give some idea of how well it works but that larger sets of more varied real-world clinical data are required to prove its efficacy and flexibility. Nonetheless, the offered information demonstrates that the suggested approach solves the problem by utilising a strong methodology and advanced methodologies. Additional datasets may not be required right away for the reasons given below:
-
• The suggested system employs a multi-pronged strategy, drawing on deep learning and statistical methodologies to identify and categorise pain-related facial characteristics. The findings are better than the existing approaches as mentioned in Figs.7 and 8, which proves that this integration improves its performance and resilience.
-
• Cutting-Edge Methods: An array of fusion algorithms including MTCNN are among the cutting-edge approaches utilised by the system. It appears that the system is constructed to handle different input circumstances since these approaches are meant to successfully handle complicated picture data and variations.
-
• Extensive Testing and Outcomes: Notable datasets including the UNBC-Master and the E2 Face set databases were used to test the system, and it achieved high accuracy and F1-scores. These tests show that the system works well in a variety of settings and situations.
Upholding patient privacy rights requires securing data storage, obtaining user consent, and limiting access to this data. Furthermore, unequal performance among groups may result from possible biases in training data, such as inadequate representation of distinct demographics. This can affect patient outcomes and equity in healthcare settings by producing forecasts for under-represented groups that are less accurate or even detrimental. The creation and implementation of the system would gain depth and legitimacy if these ethical issues were addressed in the study, along with possible methods for reducing privacy threats and guaranteeing varied, inclusive datasets.
The suggested system has been tested against several models and has demonstrated equal or superior performance in each. This lends credence to its present efficacy as it signifies that it has been validated against recognised criteria. Tables VI and VII represent our proposed model versus other recent state-of-the art models. Our model shows improved accuracy over other models in the UNBC dataset as well as the E2 database.
7. Conclusion
In conclusion, the suggested method is a novel approach to intelligent healthcare emotion analysis that identifies pain using facial expressions. Cloud-based mobile apps with separate front-end and back-end components operating on multiple devices improve data processing and computing accessibility. The SASys combines statistical and deep learning feature extraction approaches to improve system robustness and performance. Face detection and normalisation during picture preprocessing are essential for feature extraction. Comparing statistical feature representation using HoOG and LocBP with machine learning classifiers to enhance deep learning utilising multi-task CNNs, the MTCNN architecture performs well in face feature discrimination for pain categorisation. Combining categorisation scores from statistics and deep learning methodologies employing weighted sum, sum, and product rule methods improves SASys performance. The suggested technique outperforms machine learning classifiers and competes with state-of-the-art image-based SASys in healthcare on multiple datasets.
7.1. Future work
The paper offers a thorough overview of the proposed system for pain classification, showcasing the innovative use of deep learning and machine learning techniques. It effectively details the preprocessing steps, feature extraction methods, and classification approaches and provides a comparative analysis of various classifiers and fusion techniques. To further enhance the study’s impact and facilitate reproducibility, it would be beneficial to include additional technical details. Providing specific information on the CNN architecture, such as the number and types of layers, activation functions, optimisation algorithms, and configurations for transfer learning and fine-tuning, along with comprehensive training procedures including batch size, learning rate schedules, and criteria for stopping training, would significantly enrich the paper. These additions would enable other researchers to replicate and build upon this valuable work.
While the experimental validation has so far been confined to just two datasets under controlled conditions, further extensive testing on diverse, real-world clinical data is essential to demonstrate robust performance and generalisability.
Author contribution
In this collaborative study on intelligent healthcare emotion analysis, specifically targeting pain recognition through facial expressions, each of the four authors contributed significantly to the overall research and development process. Below is a detailed breakdown of their respective contributions:
Author 1: Dr. U. Samson Ebenezar
Role: Lead Researcher and System Design
The primary responsibility of the first author was to frame the research objectives and design the overall system architecture. This involved conducting an extensive review of the latest trends and challenges in healthcare emotion analysis, with a specific focus on pain detection using facial expressions. He also played a pivotal role in identifying the research gap and proposing a cloud-based mobile application approach for the system, which integrated separate front-end and back-end components for optimised data processing. Additionally, they were responsible for defining the core framework of the SASys and coordinating the implementation of the MTCNN as the deep learning model of choice.
Author 2: Dr. S P. Manikandan
Role: Literature Review and Algorithm Development
Dr. S P. Manikandan contributed to the analysis of the current literature on healthcare sentiment analysis, particularly in the context of facial expression-based pain detection. They meticulously reviewed past research papers, identifying limitations and potential improvements. They focused on integrating statistical methods, such as HoOG and LocBP, to improve the feature extraction process. In addition to that, he played an instrumental role in comparing traditional machine learning classifiers with the proposed deep learning architecture, conducting experiments, and analysing the performance of both approaches. Their contribution was critical in establishing the effectiveness of deep learning-based techniques over conventional methods.
Author 3: Dr. P. Gururama Senthilvel
Role: Data Processing and Model Optimisation
The third author Dr. P. Gururama Senthilvel took the lead in data preprocessing and feature extraction techniques, including the critical task of detecting faces and normalising them for optimal feature extraction. This process was essential for enhancing the accuracy of the system. He was responsible for fine-tuning the deep learning models, specifically the MTCNN, to ensure its efficiency in multi-task learning. They also handled the integration of the statistical and deep learning-based classification systems, resulting in a robust and efficient hybrid model. The contribution of this author was key in ensuring the system’s performance was both resilient and accurate, ultimately enhancing the overall results of the study.
Author 4: Dr. C. Sivasankar
Role: Evaluation, Results Analysis, and Paper Writing
Dr. C. Sivasankar was primarily responsible for conducting the experimental evaluation of the proposed method, including the implementation of various performance metrics. They designed and presented the results through detailed graphs, comparing the performance of the newly proposed algorithm with existing models. They also worked closely with the other authors in analysing the results, identifying areas of improvement, and validating the effectiveness of the suggested method. He contributed significantly to the preparation of the journal paper, formulating the conclusion and proposing future enhancements. Their work in result interpretation and comparison with state-of-the-art SASys was essential in demonstrating the superiority of the proposed approach.
Collaborative Efforts
Each author worked closely together to refine the methodology, iterating on ideas and making collective decisions about algorithmic improvements and enhancements. Through regular discussions and feedback sessions, the research paper evolved into a comprehensive study showcasing the innovative deep learning-based system for pain detection using facial expressions. All four authors jointly contributed to ensuring that the system outperformed traditional classifiers, delivering an effective, accurate, and resilient healthcare emotion analysis tool.
This structured breakdown highlights the unique contributions of each author to the success of this research paper, underlining the collaborative nature of the study.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interests
The authors declare no conflicts of interest exist.
Ethical standards
Not applicable
