3.2.1. Feature extraction

The proposed hybrid classification model relies on two key features for activity classification: one extracted from images and the other from human motion data, revealing the movement of each joint in terms of x, y, and z direction values.

1. 1. Joints Motion Data The motion data for joints was generated using the open-source framework MediaPipe, extracting 33 joint information per frame. However, not all joints offer useful information for activity recognition [Reference Chen, Wang, Xue and He57]. Rigorous experiments were conducted on the datasets to identify the 15 informative joints that consistently contribute to all activities, as depicted in Figure 3.

2. 2. Generate image as feature

   • • It’s impossible to understand the performed activities from a single image clearly.

   • • Utilizing image frames with a temporal skip of four image frame, RGB motion history images are generated. However, they are sensitive to illumination and cluttered backgrounds.

   • • Next, skeleton joint information is obtained from each image frame and appended to a new black image, creating a skeleton motion history image.

   • • Generate one skeleton motion history image for every 40 frames and input it into the ResNet50-ViT for further processing.

   Figure 3.

   Informative joints selection for analysis.

   
   Figure 4.

   Skeleton motion history images representation for each activity.

   
   Figure 5.

   Geometric and kinematic features representation.

   

   Some sample images related to LNMIIT-SMD are shown in Figure 4.

3. 3. Geometric and Kinematic feature Computation The cornerstone for assessing diverse features and constructing feature vectors is the 3D coordinates of human skeleton joints. These vectors are meticulously derived through manual calculation, leveraging MediaPipe features generated for each frame. The distinctiveness of these features effortlessly differentiates each activity. The dataset specifically focuses on 15 selected joints, outlined in Figure 3, for precise activity classification.

   • • Calculation of Angles Between Skeleton Joints: To create a visual representation of the skeleton, we connect the 3D coordinates of essential body joints with lines. In our dataset, we concentrate on six key joints – left and right elbows, left and right knees, and left and right hips – to calculate angle features. Please refer to Figure 5 for a detailed explanation of the angle calculation method. The angle is determined by averaging the differences between the hip-and-knee and ankle-and-knee values. In the context where joint1 represents the joint hip, joint2 corresponds to the joint knee, and joint3 denotes the joint ankle, the angle is computed using the following equation:

     (1) \begin{equation} \Theta = \frac {joint1joint2joint3}{joint1joint2 \cdot joint2joint3} \end{equation}
     (2) \begin{equation} joint1joint2joint3 = joint1_{1} \cdot joint1_{2} + joint2_{1} \cdot joint2_{2} + joint3_{1} \cdot joint3_{2} \end{equation}
     (3) \begin{equation} joint1_{1} = x1 - x2,\quad joint2_{1} = y1 - y2,\quad joint3_{1} = z1 - z2 \end{equation}
     (4) \begin{equation} joint1_{2} = x3 - x2,\quad joint2_{2} = y3 - y2,\quad joint3_{2} = z3 - z2 \end{equation}
     (5) \begin{equation} joint1joint2 = \sqrt {joint1_{1}^2 + joint2_{1}^2 + joint3_{1}^2} \end{equation}
     (6) \begin{equation} joint2joint3 = \sqrt {joint1_{2}^2 + joint2_{2}^2 + joint3_{2}^2} \end{equation}
     (7) \begin{equation} Angle = \frac {\cos ^{-1}(\Theta) \cdot 180}{\pi } \end{equation}

     The variables $joint1_{2}$ , $joint2_{2}$ , and $joint3_{2}$ represent the $x$ , $y$ , and $z$ components of the respective vectors from $joint2$ to $joint1$ and $joint3$ .

   • • Velocity Computation: The computation of the differences in the coordinates of the human skeleton within time t and t + 1 yields the velocity of the X, Y, and Z axes. Apart from evaluating velocity, the three-dimensional acceleration is ascertained by contrasting the variations among the succeeding frames. The mean acceleration and mean velocity are two of our most important features that can be calculated using velocity and acceleration values.

   • • Distance Computation: The feature vector includes the distances between hand-to-hand, hand-to-head, and hand-to-spine base positions. In essence, for each data point in the dataset, extra information is included, specifying the lengths or spatial gaps between hands, hands and head, and hands and spine base. This expansion enhances the dataset with crucial details about the relative positions of these essential body parts, providing a more comprehensive perspective for analysis or modeling purposes.

   • • Height Computation: The height is determined by measuring the distance between the uppermost and lowermost joints of the body. Since height can vary significantly among individuals, considering this factor is essential for standardizing or normalizing other feature vectors. Essentially, by adjusting for height differences, we ensure that the analysis and comparison of additional features are not disproportionately affected by individual height variations. This normalization process improves the accuracy and reliability of the overall feature analysis across diverse body types.

   • • Windowing and Segmentation: Employing all the data for training at once is not an effective strategy. Instead, using a windowing or segmentation strategy is optimal. Segmentation is typically utilized in the preprocessing phase to make data analysis more manageable, with windowing being the most common approach. The dataset was collected using a sampling rate of 30 Hz (30 samples per second). Data is split into allotted frames (Windows) of 2.57 s (80 characteristics set) with a 50% intersection with the previous window, as illustrated in Figure 6.

Figure 6.

Sliding window approach for action recognition.

3.3.1. illuminationRevive architecture

Recognizing activities under varying illumination conditions presents a significant challenge. The illuminationRevive architecture has been developed to tackle this issue, focusing on restoring underexposed images. This architecture comprises an autoencoder network, which serves as an image-to-image translation network to enhance image illumination. Additionally, a post-image processing step is included to further improve the visual quality of the resulting image.

3.3.2. Image-image translation network

This network comprises downsampling and upsampling blocks, with the downsampling path followed by the upsampling path, incorporating skip connections to retain crucial spatial information as shown in Figure 7. In the encoding phase, input images undergo processing through a series of convolutional layers. It commences with the encoder_conv1 layer, which extracts basic features using 32 filters with a 3 x 3 kernel size and ReLU activation function. Subsequent max-pooling layers (encoder_maxpool1 and encoder_maxpool2) downsample the feature maps to retain important information while reducing spatial dimensions. The process advances with deeper convolutional layers (encoder_conv2 and encoder_conv3), each extracting higher-level features with increasing complexity using 64 and 128 filters, respectively. In the decoding phase, the feature maps undergo reconstruction through corresponding convolutional layers (decoder_conv1, decoder_conv2, and decoder_conv3) with identical filter configurations as their encoder counterparts. Upsampling layers (decoder_upsample1, decoder_upsample2, and decoder_upsample3) are employed to increase spatial dimensions, and concatenation layers (concat1, concat2, and concat3) merge the upsampled features with those from the encoder layers to preserve spatial information. While the network produces improved illumination output, it may result in the loss of some pixel information. The loss function incorporates exposure loss and illumination smoothness loss, along with the Adam optimizer set to a learning rate of 0.0001.

Figure 7.

Image-to-image translation network.

3.3.3. Loss function

Exposure loss and illumination smoothness loss have been utilized as loss functions to train the autoencoder. Illumination smoothness loss quantifies the smoothness of illumination across an image, calculated as the sum of squared gradients of pixel intensities in the image. Exposure loss measures the discrepancy between the desired exposure level ( $ E_{\text{target}}$ ) and the actual exposure level ( $ E_{\text{input}}$ ) of an image. It is calculated as the absolute difference between the target and input exposure values. Let’s denote the exposure loss as $L_{\text{exposure}}$ and the illumination smoothness loss as $L_{\text{smooth}}$ . The combined loss function $L_{\text{total}}$ would then be:

(8) \begin{equation} L_{\text{total}} = L_{\text{exposure}} + \lambda \cdot L_{\text{smooth}} \end{equation}

where $\lambda$ is a hyperparameter that controls the trade-off between the exposure loss and the smoothness regularization. After several experiments, the value of $\lambda$ is set to 1.75. The variation of Peak signal to noise ratio (PSNR) and structural index similarity (SSIM) with variation in $\lambda$ is shown in Figure 8. Mathematically, the exposure loss can be defined as the difference between the desired exposure level $E_{\text{target}}$ and the actual exposure level $E_{\text{input}}$ of the image:

(9) \begin{equation} L_{\text{exposure}} = \lvert E_{\text{target}} - E_{\text{input}} \rvert \end{equation}

Figure 8.

SSIM and PSNR variation w.r.t lambda( $\lambda$ ).

Meanwhile, the illumination smoothness loss can be calculated as described earlier:

(10) \begin{equation} L_{\text{smooth}} = \sum _{i=1}^{H} \sum _{j=1}^{W} \left ( \nabla I_{\text{input}}(i, j) \right)^2 \end{equation}

3.3.4. Dataset used for image-image translation network

The network was trained using the LOL dataset(LOW-Light dataset) [Reference Guo and Hu58], which consists of 500 image pairs. Each pair includes both a low-light version and its corresponding normal-light counterpart. The dataset is divided into 350 training pairs and 150 testing pairs. Low-light images exhibit noise typical of low-light photography, posing a challenge for image processing algorithms.

3.3.5. Post image processing process

Additionally, the dataset primarily consists of indoor scenes, presenting a variety of indoor lighting conditions. To address this issue, a demosaicing process is applied to estimate missing color information at each pixel, thereby reconstructing a complete, full-color image. Subsequently, white balance is employed to correct undesirable color casts, ensuring accurate color representation. Following this, a color correction process is implemented to eliminate unwanted color casts caused by recording lighting conditions and to compensate for environmental effects such as reflections or the color of surrounding objects. To further enhance the quality of the images, a non-local means denoising technique is utilized, effectively reducing noise while preserving essential image characteristics. Finally, gamma correction is applied to fine-tune overall brightness and contrast, thereby enhancing the visual appeal of the final results. This comprehensive approach addresses the complexities associated with activity recognition in challenging illumination scenarios. The overall process is shown in Figure 9.

Algorithm 1

Detect Image Illumination

Figure 9.

Restoration of underexposed image.

Table III.

Accuracy of different approaches for human activity recognition.

Table IV.

Configuration parameters for ViT.

Figure 10.

Proposed hybrid model for activity detection/Classification architecture.

Table V.

Analysis of different architectures leading to the proposed classification model.

4.1.1. Existing datasets

To validate the generalizability of the classification model, it has been evaluated on existing datasets, namely NTU-RGBD [Reference Shahroudy, Liu, Ng and Wang65], MSR Daily Activity [Reference Hossain, Deb, Sakib and Sarker66], and I3D Post. These datasets were carefully selected based on literature papers and our target area. Only activities similar to LNMIIT-SMD were chosen from the MSR Daily Activity [Reference Hossain, Deb, Sakib and Sarker66] and I3D Post human action dataset [Reference Pareek and Thakkar9] for testing. Here is a brief description of these datasets.

1. 1. NTU RGB + D Dataset: NTU-RGBD dataset consists of 60 activity classes. The NTU-RGBD dataset is a large-scale RGBD dataset for HAR and pose estimation tasks. It includes RGB (color) and depth films shot by numerous sensors in various indoor environments. The RGBD videos in the dataset were collected from three different camera views: frontal, top, and side. Multiple videos are recorded from these numerous perspectives for each activity, offering diverse perspectives and capturing different parts of the activity. Fifteen activities that we are targeting have been selected out of 60 classes. The classes are drinking water, eating, sitting down, standing up, shaking head, making a call, falling, headache, chest pain, back pain, neck pain, sneezing, walking with a walker, handshaking, and walking.

2. 2. MSR Daily Activity Dataset [Reference Hossain, Deb, Sakib and Sarker66]: The MSR daily Action Dataset encompasses a diverse range of human activities, offering researchers a comprehensive dataset for action recognition studies. With 16 distinct activities, including drinking, eating, reading books, using laptops, and more, it provides a rich variety of actions performed in real-world settings. Each activity is meticulously annotated with action labels and skeletal joint positions, enabling the development and evaluation of algorithms for action recognition using RGB and depth data.

3. 3. I3D Post Dataset [Reference Pareek and Thakkar9]: The I3D Post Dataset encompasses a diverse array of human actions, offering researchers a comprehensive dataset for action recognition studies. With activities ranging from walking, running, and jumping to more intricate actions like hand-waving, handshaking, and facial expressions, it offers a rich variety of actions performed in real-world scenarios.

Table VI.

Test results of various methodologies on the LNMIIT-SMD dataset.

Figure 11.

Effect of illuminationRevive architecture: (a) Output without illuminationRevive architecture (b) Output with illuminationRevive architecture.

Figure 12.

Graphical representation of test parameters for different illumination correction methods.

4.3.1. Parametric analysis

The proposed model and other existing classification models have been used to validate the performance of the suggested model. The development and finalization of the hybrid model have been described in the Table V. NTU-RGBD and LNMIIT-SMD datasets have been used to train the classification models. Some of the sample images of NTU RGB + D Dataset and its corresponding SMHIs are shown in Figure 15. To assess the performance and generalizability of the model, it has been tested on two additional existing datasets: the MSR daily activity dataset and the I3D post dataset. The results have been shown in Table IX.

1. 1. The training and validation sets were divided in a 60:20 ratio, with the remaining 20% used for testing.

2. 2. A series of experiments has been performed to select the batch size. Models have been tested with different batch sizes, as shown in Figure 16. Finally, a batch size of 128 has been finalized for all.

3. 3. Other hyperparameters are shown in Table VII.

The comparative table in Table VIII illustrates the performance of various DL models across different datasets.

1. 1. Among the models, the proposed Hybrid DL Architecture demonstrates superior performance across all datasets. With a training accuracy of 97.60% and a testing accuracy of 97.12% on the NTU-RGBD dataset, and a training accuracy of 99.45% and a testing accuracy of 98.75% on the LNMIIT-SMD dataset, it outperforms other models significantly.

2. 2. Notably, for the MSR Daily Activity dataset, the proposed hybrid DL architecture achieves a testing accuracy of 97.12%, surpassing other models such as CNN, LSTM, Convolutional-LSTM Hybrid Architecture, ViT-LSTM Architecture, GRU-INC Architecture, and LKA-GCN Architecture, which achieved testing accuracies ranging from 90.75% to 96.52%.

3. 3. Similarly, on the I3D Post dataset, the proposed model achieves a testing accuracy of 96.90%, outperforming the other models, which achieved testing accuracies ranging from 89.75% to 94.12%. In comparison to the MSR dataset, the proposed model shows an improvement of approximately 6.37% in testing accuracy over the nearest competitor, demonstrating its effectiveness in activity recognition tasks.

4. 4. On the I3D Post dataset, the proposed model outperforms the nearest competitor by approximately 2.78% in testing accuracy, highlighting its superiority. These results suggest that the proposed hybrid DL architecture is more effective in accurately recognizing activities compared to existing models across multiple datasets.

Comparatively, other models such as CNN, LSTM, Convolutional-LSTM Hybrid Architecture, ViT-LSTM Architecture, GRU-INC Architecture, and LKA-GCN Architecture also exhibit competitive performance but fall short in achieving the high accuracies attained by the proposed Hybrid DL Architecture. These results underscore the efficacy of the proposed model in classification tasks across different datasets.

Table VII.

Hyperparameters used for model training.

Table VIII.

Comparative analysis of different classification models on LNMIIT-SMD and NTU-RGBD dataset.

Figure 16.

Batch size selection w.r.t accuracy for different classification models.

The Table XI presents performance metrics for various DL models and a proposed model across different activities on the LNMIIT-SMD dataset. Among the activities, “Eating” is handled with high precision and recall by all models, with the proposed architecture achieving a precision of 0.97, recall of 0.96, and an F1-score of 0.96, showcasing its effectiveness. Similarly, for “Exercise”, all models perform well, but the proposed architecture stands out with a precision of 0.99, recall of 0.98, and an F1-score of 0.98, demonstrating its superior capability in recognizing this activity.

Figure 17.

Accuracy comparison for different deep learning models on the NTU RGB + D dataset.

‘Handshake’ is excellently detected by all models, with perfect or near-perfect scores across precision, recall, and F1-score, highlighting the robustness of the proposed architecture. For “Headache”, “Situps”, “Vomiting”, and “Walking”, the proposed model consistently demonstrates competitive performance, often ranking among the top performers with high precision, recall, and F1-scores. Overall, the proposed architecture exhibits strong performance across various activities, suggesting its suitability for activity recognition tasks on the LNMIIT-SMD dataset when compared to existing DL models. The suggested model outperforms other models across activities in the NTU-RGBD dataset, with percentage variations ranging from -3% to + 3% as shown in Figure 17. Notably, LSTM and ConvLSTM models show minor variations of -1% to 0%. In contrast, GRU-INC and LKA-GCN models exhibit variations of -1% to + 1%, while ViT-LSTM demonstrates variations of -2% to + 2%. Overall, the proposed model consistently maintains a slight edge over other models, highlighting its superior performance for activity recognition. The proposed model demonstrates significant performance improvements across various activities compared to other models on the MSR daily activity dataset and I3D post multiview human action dataset. For instance, in activities like drinking water, eating, and standing up, the proposed model achieves a higher accuracy percentage compared to CNN, LSTM, ConvLSTM, ViT-LSTM, GRU-INC, and LKA-GCN models as shown in Figure 18. Specifically, the proposed model outperforms others by 1.5%, 2%, and 1%, respectively, in these activities. Moreover, in tasks such as sitting down, walking, and handshaking, the proposed model showcases remarkable accuracy percentages, surpassing other models by margins ranging from 0.5% to 1.5%. Figure 19 illustrates the variation of accuracy and loss across different classification models, while Figure 21 presents the variations in precision, recall, and F1-score.

4.3.2. Statistical analysis

Table X presents the results of tukey test analysis comparing the proposed classification model with other models. The “meandiff” column shows the mean difference between the groups, while $p-adj$ represents the adjusted p-value for each comparison. A lower $p-adj$ value indicates a more statistically significant difference.

Statistically, the proposed model shows significant differences compared to some models but not all. For instance, compared to the CNN model, the proposed model has a mean difference of 2.8571 with a p-adj value of 0.1065, suggesting a potentially significant difference, although not extremely significant. Similarly, compared to GRU-INC, LKA-GCN, and LSTM models, the proposed model exhibits a mean difference of 3.0, with p-adj values of 0.0781, indicating potentially significant differences. However, compared to Convolutional-LSTM and VisionTransformer models, the proposed model’s mean differences are 2.4286 and -2.2857, respectively, with p-adj values of 0.244 and 0.3097, suggesting less significant differences.

Figure 18.

Accuracy comparison for different deep learning models on MSR daily activity and I3D post dataset.

Table IX.

Model testing accuracy on MSR daily activity dataset and I3D post human action dataset.

Table X.

Tukey test for comparative analysis among different classification models.

Overall, the ANOVA analysis indicates that the proposed model is statistically better than other classification models.

Figure 20 illustrates the statistical variation of accuracies for different classification models.

Figure 19.

Variations in accuracy and loss across existing and proposed classification models. (a) Accuracy, loss variation w.r.t to epochs for CNN architecture. (b) Accuracy, loss variation w.r.t to epochs for LSTM architecture. (c) Accuracy, loss variation w.r.t to epochs for conLSTM architecture. (d) Accuracy, loss variation w.r.t to epochs for vision transformer-LSTM architecture. (e) Accuracy, loss variation w.r.t to epochs for proposed architecture.

Figure 20.

ANOVA analysis for different classification models.

Table XI.

Performance metrics for existing deep learning models and proposed model.

Figure 21.

Variations in precision, recall, and F1-score across different deep learning models. (a) Variations in precision, recall, and F1-score for CNN architecture. (b) Variations in precision, recall, and F1-score for LSTM architecture. (c) Variations in precision, recall, and F1-score for convLSTM architecture. (d) Variations in precision, recall, and F1-score for vision transformer-LSTM architecture. (e) Variations in precision, recall, and F1-score for vision proposed architecture.

Figure 22.

Confusion matrices for existing classification architectures and proposed classification model for LNNMIIT-SMD (a) Confusion metrics for CNN architecture. (b)Confusion metrics for LSTM architecture. (c) Confusion metrics for convLSTM architecture. (d) Confusion metrics for vision transformer-LSTM architecture. (e) Confusion metrics for proposed architecture.

4.3.3. Matric analysis

For matric analysis, confusion matric has been generated for all classification models, which is displayed in Figure 22.

• • The confusion matrix, like the CNN model, highlights its classification performance, achieving 95.00% accuracy in “eating” and 100% accuracy in “handshake.” “Exercise” is mostly classified accurately at 95.19%, while “vomiting” shows 96.51% accuracy. Minor misclassifications occur in “headache” and “situps”.

• • The LSTM model struggles with a mere 94.17% accuracy in classifying “eating” instances, though it achieves a perfect 100% accuracy for “handshake.” Despite mostly accurate identification of “exercise” instances at 96.30%, minor misclassifications are observed, and “vomiting” instances show a relatively weak accuracy of 95.35%.

• • The ConvLSTM model demonstrates commendable accuracy, notably achieving 95.00% in identifying “eating” instances and flawless accuracy for “handshake.” Additionally, “exercise” instances are predominantly accurately identified at 96.30%, with minimal misclassifications, while “vomiting” instances exhibit a predominantly accurate classification of 98.06%.

• • The ViT-LSTM model shows strong performance, with 95.56% accuracy in classifying “eating” instances and perfect accuracy for “handshake.” “Exercise” is mostly accurate at 95.53%, with minor misclassifications, while “headache” is at 96.20%. “Situps” and “vomiting” exhibit high accuracies at 98.83% and 98.02% respectively.

• • The GRU-INC model achieves 95.71% accuracy in classifying “eating” instances but struggles with misclassifications in “exercise” and “headache,” contributing to a 2.86% misclassification rate. Impressively, it accurately identifies “handshake” instances at 100%, despite their absence in the dataset. “Vomiting” instances are mostly accurately classified at 92.86%, with some misclassifications. Overall, the model’s confusion matrix reflects its mixed performance across activities.

• • The LKA-GCN model achieved 95.71% accuracy in classifying “eating” instances, with perfect classification for “handshake.” While “exercise” instances are predominantly identified accurately at 92.86%, minor misclassifications occur, along with a high accuracy of 97.14% for “headache” instances.

• • The proposed model shines with exceptional accuracy, notably achieving 97.08% for “eating” and perfect 100% for “handshake.” “Exercise” instances are predominantly accurately classified at 98.15%, while “headache” and “situps” exhibit high accuracies of 97.51% and 98.17%, respectively. Despite minor misclassifications in activities like “vomiting” and “walking,” the overall accuracy remains impressively high, with percentages summing up to 100% in each row, showcasing the model’s reliability across activities.

The proposed model outperforms others, demonstrating fewer misclassification errors in comparison to alternative models.