3.2.1 Text cleaning

Text cleaning is an important practice in the world of NLP and text analysis. Text cleaning refers to the process of changing unstructured text into a more standardized and usable format through the removal or modification of unnecessary entities. By unnecessary entities, we typically mean special characters, HTML tags, punctuation, or other symbols that do not provide any meaningful content. For example, typical raw text from an IMDb review may contain special characters such as exclamation marks (!), emojis, or HTML tags, which should be cleaned to improve text quality and consistency. Text cleaning aims to standardize the text by removing noise and also irrelevant data so that the improved data can help a model’s performance and reduce computation time spent on the model. For example, the review “I LOVED this movie!! It was amazing!!1” would appear more as “I loved this movie it was amazing.” Once the unnecessary punctuation, HTML tags, and emoji are removed, and all text is made lowercase. This process enhances the data quality and allows ML models to focus on the content without interference from irrelevant considerations. There are multiple ways of cleaning up text (e.g., regular expression, NLP libraries, creating rules, or human intervention), but challenges can arise depending on the task, e.g., if context or sentiment can be determined by characters like an emoji. It is critical to evaluate the cleaning process based on the task needed and that only meaningful information is retained.

3.2.2 Language localization

Language localization (standardization) refers to the general process of taking multiple variants (such as British English and American English) of languages and converting them into standardized forms to create consistency in textual data. In this process, the size of overall vocabulary is decreased through standardizing words that have the same meaning but different spelling (e.g., “favourite” to “favorite”), and it makes the growth and efficacy of NLP models more effective. It enables NLP models to learn correctly by knowing different words are the same type of token rather than separate tokens. Generally, standardization is usually carried out using mapping dictionaries or language tools and is imperative for ensuring that there is linguistic consistency across sets of data.

3.2.3 Stopword removal

Stop word removal is taking away common words, or stop words, like “the,” “is,” and “was” that appear frequently in text but do not add substantially meaningful information. Removing stop words reduces noise and guarantees important stemming words are kept, which helps determining sentiment or meaning. Stop words are helpful at clarifying what the model is trying to do by consolidating their vocabulary and focusing on the primary structure typically removed. There are instances, such as “not,” where the stop word is kept if it alters the meaning of the sentence.

3.2.4 Lemmatization

Lemmatization (lemma) is the process of reducing words to their dictionary or base form using vocabulary and grammar rules. It helps group related words (e.g., “running” and “ran”) under a single form (“run”) while preserving meaning, improving model accuracy, and reducing redundancy.

3.2.5 Negation handling

Negation handling links negation words (like “not” and “don’t”) to the words they modify (e.g., “not good”) to preserve negative sentiment. This prevents misinterpretation, especially when stop words are removed, and ensures models accurately capture the intended meaning in tasks like sentiment analysis.

3.2.6 Term frequency–inverse document frequency (TF-IDF)

A statistical method called TF-IDF is used to evaluate a word’s significance in a single document in relation to how frequently it occurs in a corpus, which is a collection of documents (Dai et al., Reference Dai, Li, Luo, Zhao, Hong, Zhu and Liu2024). It combines two essential elements: IDF, which evaluates a word’s individuality or rarity across the corpus, and TF, which measures how often a term appears in a specific document. The TF-IDF value is calculated using the following mathematical expression:

(1) \begin{align} TF_{IDF\left (t,d,D\right )}=TF(t,d)\times IDF(t,D) \end{align}

Where

(2) \begin{align} TF\left (t,d\right )=\frac {f(t,d)}{\sum _{t^{\prime }\epsilon d} f(t^{\prime },d)} \end{align}

(3) \begin{align} IDF\left (t,D\right )=log\frac {|D|}{|\{d\epsilon D\;:\;t\epsilon d\}|} \end{align}

Here, $|\textrm {D}|$ denotes the overall quantity of documents, $|\{d\epsilon D\;:\;t\epsilon d\}|$ denotes the number of documents that include term t, and f(t,d) indicates how often the phrase “t” appears in document “d.” TF-IDF highlights terms that are more pertinent to a given text by giving phrases that occur often in a text but typically throughout the corpus greater scores.

3.2.7 Word embedding using word2Vec

A word embedding method based on neural networks is called Word2Vec, which uses a large text corpus for training to create vector representations of words (Johnson et al., Reference Johnson, Murty and Navakanth2024). It produces dense vectors that effectively record the semantic connections between words. The two main techniques used by Word2Vec are skip-gram and CBOW. The CBOW model aims to predict a target word based on its surrounding context words, whereas the skip-gram model forecasts context. The following expressions define the objective functions for each of these models:

(4) \begin{align} J_{CBOW}=-logp(w_{t}|w_{t-m},\ldots ,w_{t-1},w_{t+1},\ldots ,w_{t+m})\\[-30pt] \nonumber \end{align}

(5) \begin{align} J_{Skip-gram}=-logp(w_{t-m},\ldots ,w_{t-1},w_{t+1},\ldots ,w_{t+m|}w_{t}) \\[-13pt] \nonumber \end{align}

Here, $w_{t}$ denotes the target word, while $w_{t-m},\ldots ,w_{t-1},w_{t+1},\ldots ,w_{t+m}$ signify the surrounding context-related terms. The subsequent word embeddings are dense vectors that effectively capture semantic linkages, with nearby vectors in the vector space representing words with similar meanings.

3.3.1 Stacked bidirectional LSTM

The typical design of the LSTM cell (Wang et al., Reference Wang, Qiao, Yan, Tang, Li and Ning2022), as depicted in Figure 2, is structured to effectively mitigate the vanishing gradient problem while enabling the efficient flow of important details throughout the LSTM network. Traditional Recurrent Neural Networks (RNNs) often struggle to learn long-term relationships between feature vectors as a result of the vanishing gradient issue during training [30]. To address this, the LSTM architecture introduces three key gates: the input gate $(i_{t})$ , the forget gate $(f_{t})$ , and the output gate $(o_{t})$ , which collectively controls the flow of information through the activation vector for memory cells.

Forget gate: The purpose of this process is to decide how much data from the prior cell state $c_{t-1}$ ought to be kept or eliminated, according to the prior hidden state $h_{t-1}$ and the present input $x_{t}$ . The forget gate produces an output ranging from 0 to 1, indicating the degree of information retention, and is mathematically represented by Equation (6).

(6) \begin{align} f_{t}=Sigmoid\ (w_{xf}x_{t}+w_{hf}h_{t-1}+b_{f}) \end{align}

Here, $b_{f}$ represents the bias vector, $w_{xf}$ signifies the weight matrix connecting the current input $x_{t}$ to the forget gate $f_{t}$ , and $w_{xf}$ represents the matrix of weights linking the prior hidden state $h_{t-1}$ to the forget gate $f_{t}$ .

Figure 2.

Structure of LSTM.

Input gate: This component establishes the amount of current input $x_{t\ }$ that ought to become part of the cell state $c_{t}$ . The behavior of the input gate is defined by Equation (7).

(7) \begin{align} O_{t}=Sigmoid(w_{xo}x_{t}+w_{ho}h_{t-1}+b_{0}) \end{align}

Here, $b_{f}$ denotes the bias vector, the input $x_{t}$ is connected to the input gate $i_{t\ }$ by a weight matrix called $w_{xi}$ and $w_{hi}$ is the weight matrix linking the previous hidden state $h_{t-1}$ to the input gate $i_{t\ }$ .

Output Gate: This gate regulates how much information from the current cell state $c_{t}$ is passed to the LSTM network through the output $h_{t}$ . Each LSTM gate functions as a fully connected network, receiving a vector as input and producing a real-valued output. The output gate is defined by Equation (8).

(8) \begin{align} O_{t}=Sigmoid(w_{xo}x_{t}+w_{ho}h_{t-1}+b_{0}) \end{align}

Here, $b_{f}$ signifies the bias vector, $w_{xo}$ $x_{t}$ to the output gate $O_{t}$ , and $w_{ho}$ is the weight matrix linking the previous hidden state $h_{t-1}$ to the output gate $O_{t}$ .

The LSTM cell produces two final outputs: the cell state $c_{t}$ and the hidden layer output $h_{t}$ , which are represented by Equations (9) and (10), respectively.

(9) \begin{align} C_{t}=f_{\ t}\ \odot \ C_{t-1}+i_{t}\odot {\widetilde {C}}_{t} \end{align}

The equation (11) for ${\widetilde {C}}_{t}$ , which represents the intermediate cell input state as

(10) \begin{align} \tilde {C}_t = \tanh (W_{xc}x_t + W_{hc}h_{t-1} + b_c) \end{align}

Figure 3.

Structure of Bi-LSTM.

Here, $b_{f}$ denotes the bias vector, $w_{xc}$ represents the matrix of weights that relates the input $x_{t}$ to the candidate cell state, and ${\widetilde {C}}_{t}$ and $w_{hc}$ represent the matrix of weights that links the preceding hidden state $h_{t-1}$ to the candidate cell state ${\widetilde {C}}_{t}$ .

LSTM networks are limited in their ability to get context from upcoming tokens on webpages. To address this, a Bidirectional LSTM (Bi-LSTM) layer is employed, comprising two independent LSTM layers that reverse the direction of input processing and contribute to a shared output (Graves et al., Reference Graves, Jaitly and Mohamed2013). Because of this framework, the model may take use of long-term dependencies from both past and future text contexts on web pages.

Two distinct LSTM layers, which analyze the input sequence in opposite directions—from the future to the past and from the past to the future—constitute a bi-LSTM network, as shown in Figure 3. Accurately interpreting sentiment in text requires the model to gather contextual information from both previous and subsequent words, which is made possible by bidirectional processing. Using the present input $x_{t}$ and the previous hidden state ${\vec {h}}_{t-1}$ the forward LSTM computes the hidden state ${\vec {h}}_{t}$ as it processes the sequence from left to right. The word embeddings are the same for both the forward and backward LSTM parameters, despite their separation. At each time step t, the forward and backward hidden states are concatenated to create the final hidden state in the Bi-LSTM as defined in Equation (12).

(11) \begin{align} h_{t}=\big [{\vec {h}}_{t},{\hat {h}}_{t}\big] \end{align}

For SA, a deep hierarchical model can significantly enhance performance (Wang et al., Reference Wang, Gan, Sun, Zhang, Lu and Kang2019). However, stacking BiLSTM layers proves to be particularly effective within shallow learning frameworks. Both local contextual cues and complex latent symmetrical patterns in text are captured by the suggested model using a layered BiLSTM architecture. In this setup, each lower BiLSTM output of one layer is sent to the subsequent layer as input, allowing the model to progressively extract richer and more abstract features relevant to sentiment understanding. The Figure 4 below demonstrates the architectural framework of the Stacked Bidirectional LSTM model.

Figure 4.

Structure of stacked Bi-LSTM.

Figure 4. Long description

A diagram of the structure of stacked Bi-LSTM. The diagram shows the flow of data through multiple layers of a Bi-LSTM network. The input layer consists of a sequence of inputs labeled as X subscript t-1, X subscript t, and X subscript t+1. These inputs are fed into the first bidirectional LSTM layer, which contains multiple LSTM units processing the data in both forward and backward directions. The output from the first layer is then passed to the third bidirectional LSTM layer, which also processes the data in both forward and backward directions using LSTM units. Finally, the output layer receives the processed data and generates a sequence of outputs labeled as Y subscript t-1, Y subscript t, Y subscript t+1, and Y subscript t+2. The diagram illustrates the bidirectional flow of information through the LSTM layers, highlighting the interaction between forward and backward processing units.

3.3.2 Multi-head attention mechanism (MHA)

The MHA mechanism improves SA by allowing the method to concentrate on several key words and phrases at once, capturing diverse contextual cues and prioritizing key sentiment-driving components for improved accuracy and performance. Figure 5 shows the MHA mechanism’s structure.

In SA, traditional attention mechanisms fall short by capturing emotional cues from only a single representational level, limiting the model’s understanding of nuanced expressions. To address this, the MHA mechanism is employed, which applies the input feature matrix and undergoes several linear transformations. This enables the model to pick up sentiment-relevant patterns from various subspaces, enabling a more in-depth and diverse interpretation of the text’s emotional content.

Fundamentally, MHA is composed of several self-attention units, each designed to extract distinct sentiment cues. Three core matrices are used in each unit: the global feature vector matrix T, the query (Q), the key (K), and the value (V) are the results of a multi-channel CNN that captures significant sentiment-related textual features. According to Equation (12):

(12) \begin{align} Q=K=V=T \end{align}

At the basis of the self-attention mechanism found in SA is the Scald Dot-product Attention (SDA). This method assists the model to assess how important each word is with respect to all the other words in a sentence, which is important to understand the emotional tone of the sentence and its contextual sentiment. The SDA function achieves this by calculating the measure of similarity using the dot product between the query (Q) and the key (K). The dot product indicates how much focus one word should have on another word. In order to avoid the outcome being too large and prevent the model from returning the result exploding, the number is divided by the square root of the dimensionality of the key to scale it down ( $\sqrt {dk}$ ). The scaled values are passed through a Softmax function, which transforms them into a set of attention weights. The value matrix (V) is then weighted by these attention weights, leading to a weighted representation highlighting the most sentiment-related portions of the text. This process is formally defined in Equation (13):

(13) \begin{align} SDA\left (Q,K,V\right )=SoftMax\left (\frac {{QK}^{T}}{\sqrt {d_{k}}}\right )V \end{align}

In SA, the MHA approach enhances the model sensitivity to different emotional cues by applying different linear transformations to Q, K, and V by three different sets of learnable parameters called $W_{i}^{Q},W_{i}^{K},W_{i}^{V}$ for each attention head. The results of the linear transformations to Q, K, and V are then put into the SDA module. These transformed matrices are then fed into the Scaled Dot-Product Attention (SDA) module. Each head gains the ability to concentrate on certain sentiment-relevant elements of the input text, such as emotion-bearing words, negations, or context-specific polarity. The output from each head, denoted as ${head}_{i}$ , captures a unique perspective on the sentiment dynamics within the sequence, as shown in Equation (14):

(14) \begin{align} {head}_{i}=SDA \big({QW}_{i}^{Q},{KW}_{i}^{K},{VW}_{i}^{V}\big) \end{align}

The outputs from each attention head, ${head}_{1}\ to {head}_{h}$ , are concatenated to form a single matrix. This aggregated representation is then multiplied by a learnable weight matrix W to perform a final linear transformation. The result is the complete output of the MHA mechanism, as defined in Equation (17), where h denotes the number of attention heads involved.

(15) \begin{align} Head=MultiHead (Q,K,V) =Concat({head}_{1},\ldots ,{head}_{h})W \end{align}

Figure 5.

Structure of the multi-head attention mechanism.

3.3.3 Residual-normalized GRU (ResNorm-GRU)

Following the attention layers, the ResNorm-GRU layer is integrated to refine the sequential representation. A GRU (Graves et al., Reference Graves, Jaitly and Mohamed2013) is a specialized architecture within the RNN family, developed to handle sequential data more efficiently. The accompanying figure demonstrates the internal structure of a GRU. Notably, GRUs differ from LSTM networks in several key ways: they use fewer parameters, which simplifies the training process, and they employ two distinct gates—the update gate $Z_{t}$ and the reset gate $r_{t}$ . GRU-based models face key challenges with long and complex sequences, including vanishing or exploding gradients that hinder long-range dependency learning, unstable gate activations causing inconsistent training, and performance degradation when stacking multiple layers. These issues limit the effectiveness and robustness of standard GRU architectures in sentiment analysis of lengthy, nuanced reviews. To address these limitations, a residual-normalized GRU (ResNorm-GRU) is introduced.

(i) Layer Normalization (LN):

Applied to the update gate, reset gate, and candidate hidden state. LN normalizes pre-activations, making the model less sensitive to weight initialization and ensuring stable gradient flow.

(16) \begin{align} Update gate\;:\; z_{t}=\sigma (LN(W_{z}x_{t})+LN(U_{z}h_{t-1})+b_{z}) \end{align}

(17) \begin{align} Reset gate\;:\; r_{t}=\ \sigma (LN(W_{r}x_{t})+LN(U_{r}h_{t-1})+b_{r}) \end{align}

(18) \begin{align} \text{Candidate hidden state: } \widetilde {h}_{t} = \tanh ( \!\operatorname {LN}(W_{h}x_{t}) + \operatorname {LN}(U_{h}(r_{t}\odot h_{t-1})) + b_{h} ) \end{align}

(ii) Residual Enhancement:

After computing the GRU hidden state, a shortcut connection adds the input $x_{t}$ directly to the GRU output.

(19) \begin{align} \text{GRU Update} \, : \ h_t^{gru} = (1-z_t)\odot h_{t-1} + z_t\odot \widetilde {h}_t \end{align}

(20) \begin{align} \text{Residual output} \, : \ h_t = h_t^{gru} + P(x_t) \end{align}

where $P\left (x_{t}\right )$ is an identity mapping if dimensions match, or a linear projection if they differ.

After feature extraction, the next step involves using Conditional Random Fields (CRFs) to refine the predictions by modeling the dependencies between output labels and improving overall classification accuracy.

3.3.4 Attentive conditional random field (Attn-CRF)

Traditional CRF are widely used in sequence labeling tasks because they model dependencies between output labels. However, CRFs treat all input features equally, which limits performance when the input contains sentiment-driving words mixed with irrelevant terms.

To address this, the Attn-CRF is proposed. It integrates attention weighting into the CRF framework, utilizing CRF’s proficiency in modelling label dependencies while enabling the model to concentrate on more significant tokens.

(i) Attention Integration:

Attention weights $({\alpha }_{t})$ are computed from hidden states $h_{t}$ .

(21) \begin{align} \alpha _t = \frac {\exp (\! \operatorname {score}(h_t) )} {\sum _{k=1}^{T}\exp(\! \operatorname {score}(h_k))} \end{align}

Where the score $ (h_{t} )$ is computed using dot product or feedforward attention.

(ii) Weighted Features for CRF:

The attention weights refine the hidden states:

(22) \begin{align} \hat {h}_t = \alpha _t \cdot h_t \end{align}

The weighted features are fed into the CRF layer.

(iii) CRF Sequence Modeling:

The CRF computes the conditional probability of a label sequence $y=(y_{1},y_{2},\ldots ..,y_{T})$ given weighted features h:

(23) \begin{align} P(y \mid \hat {h}) = \frac {\exp \left (\sum _{t}\psi (y_{t-1},y_t,\hat {h}_t)\right )} {\sum _{y'}\exp \left (\sum _{t}\psi (y'_{t-1},y'_t,\hat {h}_t)\right )} \end{align}

where $\psi$ is the potential function that considers both unary scores and pairwise scores

(iv). Decoding:

The Viterbi algorithm with attention-weighted features is used to determine the most likely label sequence.

(24) \begin{align} Y^{*} = \underset {y}{\arg \max }\; P(y \mid \hat {h}) \end{align}

The following algorithm 1 represents the Attn-CRF.

Algorithm 1.

Attentive conditional random field

Algorithm 1 Long description

An equation representing an algorithm for attentive conditional random field. The input is X, which is a sequence of elements X1, X2, up to XT. The output is Y star, which is a sequence of elements Y1, Y2, up to YT. The algorithm starts by encoding the input X into H. Then, for each time step t from 1 to T, it calculates an attention score based on the encoded value ht. Next, it applies a softmax function to the attention scores to get at. It then updates ht to get h hat t, which is a product of at and ht. Finally, for each time step t from 1 to T, it calculates unary scores for each label y in the label set, considering the previous label y prev. It then computes the potential as the sum of the unary score and the transition score. The Viterbi decoding is applied to the potential to get the final output Y star.