3.1 Redefining SEPSIS

Problematic sentences in English news exhibit distinct characteristics that render them impure. Upon thorough examination of these properties, we have categorized these issues as SEPSIS properties. The term SEPSIS was initially introduced by Das and Chatterji (Reference Das and Chatterji2023), where the authors defined it as SpEculation, oPinion, biaS, and dIStortion. However, we found the definition of SEPSIS to be unclear in our understanding. Therefore, we have refined the definition, referencing each term from existing literature and incorporating proper dictionary definitions within the context of news reporting. In this study, we adopt the acronym SEPSIS to represent the four key attributes of these sentences: SpEculation, oPinion, biaS, and twISt. The subsequent section elaborates on these properties in the context of English news sentences.

• • Speculation—the activity of guessing possible answers to a question without having enough information to be certain or the act of speculating in order to make a profit.Footnote ^g In the domain of news journalism, it involves making assumptions, often expressed in future tense. The inclusion of speculative language in news articles has the potential to undermine the fundamental journalistic value of objectivity.

• Example: This political party may win the election.

  Here, the author speculates that the political party may win, where it totally depends on the people who vote. It generally refers to a future incidence.

• • Opinion—consists of two key components: a target g and a sentiment s on the target, that is (g, s), where g can be any entity or aspect of the entity about which an opinion has been expressed, and s is a positive, negative, or neutral sentiment, is given by Liu et al. Reference Liu2022. In news reporting, a neutral opinion is desirable. Here we are considering negative and positive sentiment.

  Example: This political party has an advantage in this election.

  Here, the writer has the opinion that this political party has an advantage. Here, the writer put some baseless opinions.

• • Bias—According to D’Alessio and Allen (Reference D’Alessio and Allen2000), media bias can be categorized into three distinct forms: (1) gatekeeping, (2) coverage, and (3) statement bias. Gatekeeping bias involves the selection of stories out of the potential stories; coverage bias reflects the amount of attention given to specific perspectives in the media; statement bias, in contrast, indicates how an author’s personal viewpoint is integrated into the text and it focuses on the favorability of coverage towards one party or the other.

  Here, our definition of bias is almost similar to the Statement Bias. However, we are using it at the sentence level with the inclusion of all types of news analysis. While Statement Bias only focuses on politics, we consider Bias in all the other domains namely, Sports, COVID- $19$ , and Entertainment. Along with this, we also add the following things Bias refers to the partiality or inclination of the author to convey a specific perspective or viewpoint by using the voice or statements of other individuals. It occurs when the author, intentionally or unintentionally, presents information in a way that favors one viewpoint over others or modifies facts to promote a specific agenda.

• Example: It would be very good if this political party won the election.

  This sentence shows his biasness towards a party and wants to spread the biasness to the engaged readers.

• • Twist—to change the information so that it gives the message you want it to give, especially in a way that is dishonest.Footnote ^h In news reporting, twist refers to presenting information in a convoluted or complex manner that makes it difficult for the engaged readers to understand the intended meaning easily. This can involve intricate language, vague expressions, and a lack of clarity for the engaged readers. Such twisting of news can impact the readability and transparency of the information being conveyed, potentially causing confusion or misinterpretation among readers.

• Example: Don’t just win, win with a significant number of votes, as per the party leaders’ perspective.

  In this instance, the sentence exhibits complexity. Here, the focus shifts from emphasizing the viewpoints of certain party leaders to highlighting the victory in the election.

This definition is not exhaustive, and its characteristics are mutually inclusive in nature. We have considered these four properties in problematic sentences based on the observation. The sentence, phrase, and intent having SEPSIS properties are called a Septic sentence, phrase, and intent, respectively. One Septic sentence may have more than one SEPSIS characteristic.

We have considered Pure as the opposite of Septic. Following are the cases when we consider a sentence as Pure.

• • A sentence is considered Pure if it does not have SEPSIS properties. Consider the example

  There were six other Indian firms competing.

  This sentence states a number that is assumed to be verified by the news agency. Hence, it doesn’t have any SEPSIS property. Thus, this is considered a Pure sentence.

• • A sentence is considered Pure if it is a remark made by a person or a group. Now we consider two examples

  Example 1: Lockett also said, “I think medical treatment will be done correctly once we go to the hospital.”

  Example 2: Mohan challenged, “Our team will win.”

  Example-1 sentence is considered as Pure as it is said straightforwardly by a person, whereas Example 2 is regarded as a Septic sentence as the phrases outside quotations have SEPSIS property. Thus, for any sentence, the statement inside the quotation is Pure. Depending on the remaining part, the sentence is considered Pure or Septic.

• • A sentence is considered Pure if its interpretation relies on contextual information. Consider the example

  We regret and apologize for this mistake.

  Here, we don’t know the actual mistake. It is referring to some previous incident. Thus, we consider this type of sentence as Pure.

Table 1 presents diverse examples illustrating the classification of sentences into “Septic” and “Pure” categories based on the SEPSIS framework. In example 1, the sentence is classified as Pure because the quoted speech is assumed to be accurate, and the unquoted part is considered pure. Example 2 is classified as “Septic” due to the added sentiment of being “full of praise.” Example 3 is labeled as Septic because the fact is presented by an anonymous entity. Example 4 is Pure as it represents a straightforward cause–effect relationship. In contrast, Example 5 is classified as Septic because the cause–effect relationship is based on an assumption or probability. Examples 6 and 7 are both labeled as Septic as they involve either part (cause or effect) being probabilistic. Example 8 is Pure since the word “Yet” is a continuation of the previous sentence and does not introduce a negative effect. Examples 9 to 12 are classified as “Septic” due to various reasons such as questionable psychiatric effects, uncertainty in decision-making, emphasis on positive government initiatives, and lack of clarity on the extent of deliberation. Finally, Example 13 is Pure as there is no Septic phrase in the statement. This section provides a brief overview of each example’s classification based on the Septic or Pure criteria.

Table 1.

Examples of sentences classified as Septic or Pure

Table 1 Long description

A table with two columns labeled Example and Class, and 13 rows. The Example column contains sentences illustrating the classification of sentences into Septic and Pure categories based on the SEPSIS framework. The Class column contains the classification of each sentence as either Septic or Pure. Row 1: Prabin Kumar has played a big role in our aim towards a New India, New UP, the PM said. Pure. Row 2: Rahul was full of praise for his bowlers, saying: The last five games we struggled with our death bowling, but that was a positive today. The boys showed courage and pulled things back. You expect them (SRH) to score 230-plus after that start. Septic. Row 3: According to a member of this community, the Prime Minister had a meeting in the United States in 2015 and England in 2017. Septic. Row 4: India was declared champion on the basis of the number of boundaries as the game did not end in a super over. Pure. Row 5: In the nail-biting super over, the mentor finally breathes a sigh of relief, witnessing the disciple Akash hit a crucial six. Septic. Row 6: Probably due to distress, Lee could not perform his best. Septic. Row 7: But we could not meet their hope. Septic. Row 8: Yet by courtesy of Centurion Chulbul Pandey, Kenya built a big score. Pure. Row 9: Coronavirus has shown its devastating impact on all the patient’s psyche with yet another positive patient in Junagadh taking his own life out of depression. Septic. Row 10: President Donald Trump, under pressure over his handling of the coronavirus outbreak, announced the federal government would ship 150 million rapid tests to the U.S. for opening schools and ensuring safety at centers for senior citizens. Septic. Row 11: Uttar Pradesh will achieve the one crore COVID-19 tests mark on Monday with the help of optimistic Govt. initiatives. Septic. Row 12: The FIH, after much deliberation, decided to bring hockey back on the pitch after the COVID-19 era. Septic. Row 13: Though a few malls in the city lifted their shutters on Tuesday, some others will take a day or two to clean and sanitize the premises before welcoming visitors. Pure.

3.2 Septic sentence dataset preparation

In this section, we provide a comprehensive overview of the dataset used in our study. The dataset contains 7290 news sentences from different genres and publications. News from publications all around the world, not only in India, was used to create a substantial variation in the dataset. BBC, CNN, The Times of India and The Hindu are among them. However, the majority of the articles are from Indian publishers. Political, sports, criminal, business, education, celebrity, and COVID- $19$ are the major domains of the dataset.

The dataset initially contains about $1900$ and $500$ sentences. These are annotated as Pure and Septic, respectively. It is evident from the figure that there are more Pure sentences than Septic sentences. Thus, the dataset is imbalanced in nature. We have worked on improving the dataset by normalizing it. This is achieved by adding more Septic sentences to the dataset. We normalized the dataset more by oversampling the 500 Septic sentences with the Generative Pretrained Transformer (GPT v3.5) (Floridi and Chiriatti Reference Floridi and Chiriatti2020; Dale Reference Dale2021), which makes comparable sentences of a given sentence by replacing the words in the sentence with their synonyms or similar words. Using this model, we have increased the count of Septic sentences from $500$ to $1900$ . This way, our current dataset contains $1900$ Pure and $1900$ Septic sentences, that is a total of $3800$ sentences.

While GPT-3.5 was utilized to generate additional Septic sentences to balance the dataset, we acknowledge several limitations. The generated sentences may not always introduce significant variability or could lead to overfitting if they lack sufficient contextual diversity. To mitigate these issues, we reviewed a sample of generated sentences for quality and coherence, employed bias detection methods, and combined GPT-3.5-generated data with other augmentation techniques. We also optimized the data generation process to manage computational costs effectively. Despite these measures, we recognize that GPT-3.5’s inherent limitations might impact the quality of augmented data, and we are continuously exploring ways to improve this aspect.

We have created a baseline system using these $3800$ sentences. Then, $5000$ COVID- $19$ news sentences are annotated by the system. We have scrutinized these annotations and made the required changes. We found $1745$ Septic sentences in the 5000 system annotated sentences. We took $1745$ Pure sentences from the rest and added these $3490$ sentences to our prior dataset, totaling $7290$ balanced dataset. The distribution of the dataset over the different domains namely politics, sports, criminal justice, business, education, entertainment, and COVID- $19$ . We have chosen $10$ % of the total sentences from our dataset for testing.

We performed stratified splitting to maintain the distribution of categories across both training and test sets. Specifically, for each category, 10% of the total sentences were set aside for testing, while the remaining 90% were used for training the models. This approach ensures that the test data accurately reflects the overall distribution of news categories and provides a robust evaluation of model performance across different news types. The distribution of training and testing data is illustrated in Figure 1.

For this study, our annotators were selected based on their extensive background in linguistic analysis and domain-specific knowledge. They underwent rigorous training, which included guidelines on annotation standards, handling ambiguities, and maintaining consistency across annotations. The training sessions covered specific methodologies for identifying and labeling sentences as Pure or Septic, ensuring that all annotators adhered to a uniform criterion. Additionally, we conducted regular calibration meetings to review annotations and resolve discrepancies. This comprehensive approach was designed to ensure high-quality and reliable annotations for our dataset.

Septic sentence annotation varies from person to person. Thus, inter-annotator agreement is necessary for the validation of the dataset. We analyze the degree of agreement between them using Fleiss’ kappa (Reference Fleiss1971). Fleiss Kappa’s score is calculated to check the agreement between the three annotators regarding sentence classification. Three annotators have annotated the same set of 100 sentences. The Fleiss’ Kappa score among these three raters is $0.72$ , which indicates a substantial agreement (Takala et al. Reference Takala, Malo, Sinha and Ahlgren2014).

3.3 Details of models

In our study on classifying Septic news articles, we employed various models with distinct architectures, namely CNN, LSTM, BiLSTM, and then used the hybrid neural network framework of CNN-BiLSTM and CNN-BiLSTM with an attention mechanism. The accuracy and performance of classifiers can be significantly influenced by the presence of irrelevant and redundant features. To address this, we outline the details of relevant linguistic features extracted from English news content for Septic Sentence analysis.

• • Lexical features: This category includes word-level features, such as the number of words, frequency of large words (length more than 10), unique words, presence of named entities (person, location, organization, and miscellaneous), the existence of quoted phrases (“quotation marks”), presence of exclamation mark(!), presence of Question mark(?), and presence of English pronominal anaphora.

• • Syntactic features: Syntactic features involve the frequency of function words and phrases. The Term Frequency-Inverse Document Frequency metric is employed as a statistical weighting measure to assess the importance of terms in documents within the dataset. Term importance increases with the frequency of word occurrence.

• • Domain-specific features: In this section, we integrate specific features relevant to the news domain, such as the presence of words from a predefined list of clue phrases. A segment of the created small list of clue phrases is presented in Table 2 based on the dataset analysis.

Table 2.

List of clue phrases indicating Pure and Septic sentences

Table 2 Long description

A table with two columns and two rows labeled Pure and Septic. The Pure row lists the clue phrases: Says, Informed, Disclosed, Speaks, Speak, Tell me, Told, Announced. The Septic row lists the clue phrases: Check it out, Is checked, Question asked, The question has been, The question arises, Promise, Promised, The question has started to arise, Not confident, It seems, Have taken up the challenge, Throw the challenge, Take up the challenge, Challenge, It seems, Will prove, Policy taken, Claim, Claimed, The news goes sour, The impression of terror, No sign of panic, Warns again, Yet there was relief, Not confident, Assured, Fear, Have feared, Do not be afraid, Apprehensive, The charge to give, Became stronger, Complain, Complained, The complaint was made, Strong message, Standing in the stands, Stand up, Under the cannon, Getting down on the field, Recognized, Recognition, Acceptance, The complaint arose, Complaints arise, Instigated, What a hint, Got it.

Figure 1.

Distribution of news sentences across categories in our dataset. (Blue bars: 90% training data, Red bars: 10% test data).

The linguistic features are appended with Keras embedding layer for each model. Now we discuss each model with their specific configurations.

The Simple CNN model represents a streamlined convolutional architecture. Initially, we experiment with the default parameters for CNN. It starts with a Keras embedding layer for word representations with the addition of the mentioned word linguistic features, followed by a single convolutional layer, followed by global max-pooling for feature extraction, and Dense Layers for binary classification. In addition to being computationally effective, this approach identifies significant local patterns in the data. The efficacy of the Simple CNN Model is maintained while computation efficiency is increased. It has a Dropout Layer with a rate of 0.4 after the Embedding Layer, which has an output dimension of 16 and an input length of 100. Utilizing the rectified linear unit activation function, the Conv1D Layer featured 50 filters and a kernel size of 10. The architecture was finished using a Dense Layer, a GlobalMaxPool1D Layer, and a sigmoid activation function.

To identify sequential dependencies in the input data, the LSTM model makes use of Long Short-Term Memory networks, a kind of recurrent neural network (RNN). For jobs where the order of the input sequence is important, LSTMs are a good fit. An embedding layer, an LSTM layer for sequential learning, and a global max-pooling layer for feature extraction make up the model architecture. The Global Max-Pooling Layer, LSTM Layer, Dense Layer, and Embedding Layer make up the LSTM Model, which focuses on sequential dependencies. This architecture does a great job of comprehending the sentences in sequential news articles.

To identify sequential dependencies in the input data, the LSTM model makes use of LSTM networks, a kind of RNN. For jobs where the order of the input sequence is important, LSTMs are a good fit. An embedding layer, an LSTM layer for sequential learning, and a global max-pooling layer for feature extraction make up the model architecture. The Global Max-Pooling Layer, LSTM Layer, Dense Layer, and Embedding Layer make up the LSTM Model, which focuses on sequential dependencies. This architecture does a great job of comprehending the sentences in sequential news articles.

CNNs and bidirectional long short-term memory networks (BiLSTMs) are combined to create the CNN-BiLSTM hybrid neural network. Long-range dependencies are captured by the BiLSTM layer, whereas local patterns are extracted by the first convolutional layer. This architecture works well for jobs requiring both local and global context information because it is especially good at learning hierarchical features. This model consists of a Dense Layer for classification, Global Max-Pooling Layer, Layer, Convolutional Layer, and Embedding Layer. This architecture, which can be seen in Figure 5, is made to recognize long-term dependencies as well as local patterns in the input sequences.

The CNN-BiLSTM architecture incorporates attention mechanisms through the CNN-BiLSTM with attention model. By allowing the model to concentrate on particular segments of the input sequence, attention mechanisms give the model a way to assess the relative relevance of various components. This model with attention enhancement is good at managing lengthy sequences and finding important information in them. For thorough sentence categorization, it has an Embedding Layer, Bidirectional LSTM Layer, Convolutional Layer, attention mechanism, Global Max-Pooling Layer, and Dense Layers.

In the CNN-BiLSTM with attention model, the attention mechanism is employed to enhance the model’s ability to focus on important parts of the input sequence. Here’s how it is integrated:

• • Attention layer: Applied to the output of the BiLSTM. It computes attention scores to weigh the importance of different time steps in the sequence.

• • Attention scores calculation: Attention scores are derived by comparing each time step in the LSTM output against every other time step.

• • Attended values: The attention scores are used to aggregate and emphasize the most relevant parts of the sequence, which are then passed through a global max pooling layer.

• • Reshaping: The attended values are reshaped to match the input requirements of the dense layer.

This approach allows the model to focus on more informative parts of the sequence, improving overall performance by leveraging context more effectively.

3.4 Evaluation and comparative analysis

We performed tests using a standard setup in order to assess the models. Every model was trained using a batch size of 128 over a period of 12 epochs. The binary crossentropy loss function and the Adam optimizer were applied. An 80:20 ratio was used to divide the dataset into training and testing sets. In order to reduce overfitting, early quitting with a patience of two was also used throughout training.

We performed Bayesian hyperparameter tuning for the best possible model performance, which is essential for figuring out the model’s predictive ability. The learning rate (1e-4 to 1e-1), LSTM units (10 to 100), Conv1D filters (10 to 100), kernel size (3 to 10), Dense layer units (32 to 128), and dropout rate (0.1 to 0.9) were the parameters we specified for our vast hyperparameter search space. Throughout the optimization process, a thorough investigation of various configurations was guaranteed by this methodical specification.

Since Bayesian hyperparameter optimization provides superior accuracy than other methods, we used it to fine-tune the configuration of the standalone CNN model as well as the hybrid CNN-BiLSTM model. This method was used since it has been shown to improve model accuracy. The optimized hyperparameters for the CNN-BiLSTM model were found to be “dense_units”: 106.0, “dropout_rate”: 0.5294, “filters”: 72.0, “kernel_size”: 8.0, “lstm_units”: 14.0, while for the CNN model, the best parameters were “dense_units”: 63.0, “dropout_rate”: 0.5147, “filters”: 83.0, “kernel_size”: 9.0.

Performance measures, namely recall, accuracy, precision, and F1 score, were used to measure how well the models worked. Figure 4 illustrates the confusion matrices and corresponding outputs of the proposed models across different experimental setups. We have provided further insights into the performance of our proposed models in Figure 2 with 12 epochs and without early stopping and Figure 3 with 12 epochs with early stopping of patient 2. It provides a dynamic perspective on model learning by showcasing the Training and Validation Accuracy over Epochs. A critical metric for comprehending the convergence of the model is also provided by the Training Loss over Epochs. The discriminative power of our models is demonstrated by the receiver operating characteristic curve, which graphically depicts the trade-off between true positive rate and false positive rate. The model performance is examined concerning these findings, emphasizing the effect of various configurations on classification accuracy. Finally, the diverse set of models and configurations investigated in this study offers insights into effective approaches for Septic content classification in news articles.

We conducted a thorough evaluation of the models using several objective metrics: Accuracy, Mean Absolute Error (MAE), Correlation, F1-Score, and Unweighted Average Recall (UAR). Accuracy measures the proportion of correct classifications, MAE quantifies the average error magnitude, and Correlation indicates the strength of the relationship between predicted and actual values. The F1-Score balances precision and recall, while UAR averages recall across classes to handle imbalanced data effectively. These metrics collectively offer a comprehensive assessment of each model’s performance, ensuring a robust and objective comparison. The detailed evaluation results are summarized in Table 3. These evaluations provide a comprehensive view of each model’s performance, addressing both predictive accuracy and the ability to handle imbalanced data. The diverse set of models and configurations investigated in this study offers valuable insights into effective approaches for Septic content classification in news articles.

Table 3.

Performance metrics of different models for Septic sentence identification

Table 3 Long description

The table presents performance metrics for five different models used for Septic sentence identification. It includes six columns: Model, Accuracy, MAE, Correlation, F1-Score, Accuracy, and UAR. The models listed are CNN, LSTM, BiLSTM, CNN + BiLSTM, and BiLSTM + CNN with Attention. Each row provides specific values for these metrics. Row 1: Model, CNN; Accuracy, 0.8731; MAE, 0.1754; Correlation, 0.7932; F1-Score, 0.8731; Accuracy, 0.8731; UAR, 0.8736. Row 2: Model, LSTM; Accuracy, 0.8607; MAE, 0.2205; Correlation, 0.7588; F1-Score, 0.8598; Accuracy, 0.8607; UAR, 0.8575. Row 3: Model, BiLSTM; Accuracy, 0.8738; MAE, 0.1853; Correlation, 0.7927; F1-Score, 0.8736; Accuracy, 0.8738; UAR, 0.8725. Row 4: Model, CNN + BiLSTM; Accuracy, 0.8827; MAE, 0.1587; Correlation, 0.7933; F1-Score, 0.8827; Accuracy, 0.8827; UAR, 0.8824. Row 5: Model, BiLSTM + CNN with Attention; Accuracy, 0.8669; MAE, 0.1531; Correlation, 0.7666; F1-Score, 0.8667; Accuracy, 0.8669; UAR, 0.8657.

Figure 2.

Comparison charts of our proposed models without early stopping.

Figure 3.

Comparison charts of our proposed models with early stopping with patience of 2.

Figure 3. Long description

Panel A: A line graph titled Training Accuracy Comparison shows accuracy on the vertical axis and epochs on the horizontal axis. The graph compares the performance of five models: CNN, LSTM, BiLSTM, CNN + BiLSTM, and CNN + BiLSTM + Attention. All models show an increase in accuracy over epochs, with CNN + BiLSTM + Attention achieving the highest accuracy. Panel B: A line graph titled Validation Accuracy Comparison shows accuracy on the vertical axis and epochs on the horizontal axis. This graph also compares the same five models. The CNN + BiLSTM + Attention model starts with the lowest accuracy but improves significantly by the third epoch. Panel C: A line graph titled Training Loss Comparison shows loss on the vertical axis and epochs on the horizontal axis. The graph compares the same five models, with all models showing a decrease in loss over epochs. The CNN + BiLSTM + Attention model has the lowest loss by the fifth epoch. Panel D: A line graph titled ROC Curve Comparison shows the true positive rate on the vertical axis and the false positive rate on the horizontal axis. The graph compares the ROC curves of the five models, with all models achieving high area under the curve (AUC) values around 0.94.

Figure 4.

Confusion matrices of our proposed models.

Figure 5.

CNN-BiLSTM hybrid neural network architecture.

Figure 5. Long description

A diagram of a CNN-BiLSTM hybrid neural network architecture. The diagram illustrates the flow of data through various layers of the network. The process begins with the Input layer, which feeds into the Embedding layer. This is followed by a Conv1D layer, then a BiLSTM layer. The output from the BiLSTM layer is passed through a GlobalMaxPooling1D layer. The pooled output is then processed by a Dense layer, followed by a Dropout layer, and finally, the Output layer.

The CNN + BiLSTM model demonstrates the highest performance, achieving the best accuracy (0.8827), F1-Score (0.8827), and UAR (0.8824), with a relatively low MAE (0.1587) and high correlation (0.7933). The BiLSTM model shows strong performance but with slightly lower metrics compared to CNN + BiLSTM. The CNN model also performs well but does not reach the level of the CNN + BiLSTM model. The LSTM model has the lowest accuracy and F1-Score, indicating it is less effective compared to the others. The BiLSTM + CNN with attention model offers balanced performance but does not outperform the CNN + BiLSTM model. This comprehensive evaluation provides a thorough comparison of the models’ effectiveness and their ability to manage imbalanced data.

We also performed extensive evaluation using K-fold cross-validation with K = 5 and early stopping to mitigate overfitting. The results showed the following mean accuracies: CNN model achieved 0.7603, LSTM model reached 0.7971, BiLSTM model had 0.7886, BiLSTM + CNN model obtained 0.8596, and BiLSTM + CNN + attention mechanism resulted in 0.8436. This approach ensured a robust and reliable assessment of each model’s performance across different subsets of the data.

4.1 Corpus annotation

Annotators manually annotate the Septic phrases in Septic sentences. In the previously mentioned $7290$ sentences, total $3645$ Septic sentences are present. Three annotators have annotated these $3645$ Septic sentences using “ $\lt \lt \ldots \ldots \gt \gt$ ” tags for Septic phrases. The above example is annotated as “The KBD leaders have $\lt \lt$ adopted new tactics $\gt \gt$ to meet, $\lt \lt$ beating $\gt \gt$ the COVID protocols in Eastern India.” The Septic parts of the sentence are “adopted new tactics” and “beating.” To learn the statistical model, we convert the Septic phrases to IOB (Inside–outside–beginning) format.

4.2 Septic phrase identification model

Experiments have been conducted on the CRF framework to determine the best features for Septic phrase Identification in English. The following are the features used for experiments in the Septic phrase Identification task.

• • Contextual Feature: Contextual features are the words that come before and after a specific word. This feature consists of the current word (CURR WORD), up to the previous three words (PREV $3$ WORDS), and up to the next three words (NEXT $3$ WORDS).

• • Named Entity (NE): The NE tag of the current word (CURR NER), the NE tag up to two previous words (PREV $2$ NER), and the NE tag up to two next words (NEXT $2$ NER) are also used as a feature. A CRF-based NER taggerFootnote ⁱ has been used to get the named entity.

• • Parts of Speech (POS): The POS tag of the current word (CURR NER), up to the previous two words (PREV $2$ POS TAG), and up to the next two words (NEXT $2$ POS TAG) have been employed as another feature. In this case, we have used the Stanford Log-linear Part-Of-Speech TaggerFootnote ^j that was built using fine-grained POS tags and the Penn Treebank tag set.

We have achieved $90.2$ % accuracy by considering the optimum features. The optimum feature combination includes the current word, the previous two words, the next two words, the current POS tag, the previous two POS tags, the next two POS tags, the current Named Entity Recognition (NER) tag, the previous two NER tags, and the next two NER tags.

Due to data limitations, it’s advisable to avoid using neural networks for sequence labeling tasks. CRF tends to perform better for these types of tasks under such constraints. The superior performance of the CRF model for Septic phrase identification can be attributed to its ability to leverage a set of features and contextual information. Specifically, we used an optimal feature combination that includes the current word, the previous two words, the next two words, the current POS tag, the previous two POS tags, the next two POS tags, the current Named Entity Recognition (NER) tag, the previous two NER tags, and the next two NER tags. This comprehensive feature set allows the CRF model to capture intricate dependencies and relationships within the text, leading to more accurate and robust identification of Septic sentences.

5.1 Dataset preparation

In our intent identification study, we employ our Septic phrase-annotated corpus. The same group of annotators once again annotates the sentences by using $\lt \lt ..\gt \gt$ for the Septic phrase, as mentioned previously, and $[[\ldots \ldots ]]$ for the intent part. The mentioned sentence is annotated as—“The KBD leaders have $\lt \lt$ adopted new tactics $\gt \gt$ to $[[$ meet $]]$ , $\lt \lt$ beating $\gt \gt$ the $[[$ COVID protocols $]]$ in Eastern India.” In this sentence, beating is the Septic phrase for the intent adopted new tactics and adopted new tactics is the Septic phrase for the intent COVID protocols. This way, we annotated 800 Septic Sentences to learn a statistical model.

Our intent and Septic phrase annotated dataset is represented in IOB format to use in a statistical model. We also add some features to the dataset. For this task, we have pre-processed the data as follows.

1. 1. First, we tokenize each sentence.

2. 2. A Part-Of-Speech Tagger (POS Tagger) and a Named Entity Recognizer (NER) have been applied to assign tags to each token Manning et al. (Reference Manning, Surdeanu, Bauer, Finkel, Bethard and McClosky2014).

3. 3. For each token, we apply the IOB (Inside–outside–beginning) format to mark it as part of intent or a normal phrase. We have used B-Intent, I-Intent, and O to refer to the Beginning Intent word, Internal Intent word, and other words, respectively. The mentioned example English sentence with these tags is shown in Table 4.

5.2 Intent phrase identification model

In this paper, to train the intent identification model we utilize the CRF++ 0.58,Footnote ^k which employs the CRF algorithm. In this case, we’re utilizing the context word, Named Entity tag, POS tag, and the Septic phrase as features to identify the Intent phrases in news sentences.

Table 4.

Intent annotation in a Septic sentence. ORG: organization, GPE: geo-political entity

Table 4 Long description

A table with six columns and twelve rows. The columns are labeled TOKEN, POSTAG, NER, SEPTIC, and INTENT. The table presents data for each token in a sentence, including part-of-speech tags, named entity recognition, and intent annotations. Each row lists a token, its part-of-speech tag, named entity recognition tag, septic annotation, and intent annotation. The tokens include words like The, KBD, leaders, have, adopted, new, tactics, to, meet, beating, the, COVID, protocols, in, Eastern, India, and a period. The annotations include tags like DT, NN, NNS, VBP, VBN, JJ, NNS, TO, VB, VBG, DT, NNP, NNS, IN, NNP, NNP, and punctuation marks. The NER column includes tags like B-ORG, B-GPE, and I-GPE. The SEPTIC column includes tags like B-Septic and I-Septic. The INTENT column includes tags like B-Intent and I-Intent.

Out of 800 intent annotated Septic sentences, $750$ sentences are used for training, and the rest $50$ sentences are used for testing. We have shown the feature details and corresponding accuracy values in Table 5. The highest accuracy we have achieved for the intent phrases identification task is $91.50$ %.

Table 5.

Accuracy of intent identification task based on the CRF tagger using different feature sets

Table 5 Long description

A table with four rows and three columns. The columns are labeled Feature Id, Feature details, and Accuracy (in percent). The rows are as follows: Row 1: Feature Id, f0; Feature details, CURR WORD + PREV WORD + NEXT WORD + CURR POS TAG + CURR NER; Accuracy (in percent), 90.80. Row 2: Feature Id, f1; Feature details, CURR WORD + PREV 2 WORDS + NEXT 2 WORDS + CURR POS + CURR NER + CURR SEPTIC; Accuracy (in percent), 90.90. Row 3: Feature Id, f2; Feature details, CURR WORD + PREV 2 WORDS + NEXT 2 WORDS + CURR POS TAG + PREV 2 POS TAG + NEXT 2 POS TAG + CURR NER + PREV 2 NER + NEXT 2 NER + CURR SEPTIC + PREV 2 SEPTIC + NEXT 2 SEPTIC; Accuracy (in percent), 91.40. Row 4: Feature Id, f3; Feature details, CURR WORD + PREV 2 WORD + NEXT 2 WORD + CURR POS TAG + PREV 2 POS TAG + NEXT 2 POS TAG + CURR NER + PREV 2 NER + NEXT 2 NER + CURR SEPTIC + PREV 2 SEPTIC + NEXT 2 SEPTIC; Accuracy (in percent), 91.50.

The highest accuracy was achieved by experimenting with different combinations of features, including the current word, the previous two words, the next two words, the current POS tags, the previous two POS tags, the next two POS tags, the current Named Entity Recognition (NER) tag, the previous two NER tags, the next two NER tags, the current SEPTIC label, the previous two SEPTIC labels, and the next two SEPTIC labels.

5.3 Grouping of intents

We have devised three intent classes based on the manual analysis of the intent phrases in Septic sentences. These intent classes are mentioned below.

1. 1. Class type 1: The topics related to Politics, COVID protocol, and Government initiatives are covered in this class. The intents like felicitating doctors and COVID worriers, raising awareness, and vaccinating the citizen, essential workers prioritized getting the vaccine, as a backup, the Indian Railways have turned trains into isolation wards, no mask no oil, wearing mask, and hand sanitizer even when you are at home, avoiding gathering, the guideline of the WHO, no mask during a political campaign, and campaign while Covid is considered in this class.

2. 2. Class type 2: This class includes the topics of the spread of the COVID- $19$ outbreak, testing, research, and vaccination. The intents for the COVID- $19$ outbreak comprise the low testing rate, scarcity of beds, death tolls, overflowing hospitals, fighting for oxygen, and COVID- $19$ hotspots. Under Testing, Research, & Vaccination, the intents noted are RT-PCR kits shortage, increased number of tests, risk of transmitting SARS-CoV- $2$ , reducing COVID- $19$ symptoms, Self-taken swabs for SARS-CoV- $2$ , and most extensive vaccination campaign.

3. 3. Class type 3: The impact of COVID- $19$ on human life is considered in this class. According to our observations, a few intents under this class are food crisis during a pandemic, loss of jobs, financial crisis, supply chain crisis, travel ban, risk of falling into extreme poverty, loss of human life, lack of social protection, and costly living.

For this investigation, we have used $500$ Septic sentences on COVID- $19$ . There are $550$ intents (phrases) in these $500$ sentences. Each of the intents is manually placed in one of these three classes. Here, these non-exhaustive classes are created by observing this dataset only. The distribution of these classes in the $550$ COVID- $19$ Septic intents in $500$ Septic sentences are shown in Figure 6. It can be observed that the Class type $3$ is made Septic with the highest frequency.

Figure 6.

The distribution of intent classes on COVID- $19$ data.

6.1 Dataset used

For the Fake news application, we have chosen an existing datasetFootnote ^l of Patwa et al. (Reference Patwa, Sharma, Pykl, Guptha, Kumari, Akhtar, Ekbal, Das and Chakraborty2021). The dataset consists of $10,700$ Fake and real news stories on COVID- $19$ . These stories are gathered from verified websites namely Politifact,Footnote ^m newsChecker,Footnote ⁿ Boomlive,Footnote ^o and others, as well as tools like Google fact-check-explorerFootnote ^p and the IFCN chatbot.Footnote ^q Verified Twitter accounts are used to get real news from Twitter (Patwa et al. Reference Patwa, Sharma, Pykl, Guptha, Kumari, Akhtar, Ekbal, Das and Chakraborty2021).

Since our Septic sentence classifier works on sentence labels, the Fake news stories are first broken down into sentences. Then, our classifier classifies the sentences as Septic (1) or Pure (0). Here, we have experimented with different matrices while the results of the Septic sentence classifier are combined. Each and every sentence is supplemented with an extra feature generated from the Septic sentence classifier. Since the Septic classifier works on individual sentences, the news sentences are first broken down on sentence level, and then, the classifier is run upon those individual sentences. Finally, for each news, the results from the CNN-BiLSTM model are combined according to six different matrices, namely Mean, Max, Min, Median, Mode, and Existence of sepsis in the news sentence.

The results from the classifier for each sentence are combined by calculating the mean. For example, if a Fake news story contains $2$ Septic sentences and $4$ Pure sentences then its SEPSIS feature value is $(1+1+0+0+0+0)/6 = 0.33$ . This real value is considered the SEPSIS feature. The flowchart of how the Septic sentence is used as a feature of Fake news detection is shown in Figure 7.

Figure 7.

Procedure of Ablation Study.

6.2 Methodology and experimental results

We first apply baseline machine learning algorithms for the experiment with bag-of-word and term frequency-inverse document frequency (tf-idf) features to classify the Fake and real news. Then, we use the SEPSIS and Pure property as a feature, along with the bag-of-word and tf-idf features. We split the dataset into training and test datasets in the ratio of 80:20. The dataset is class-wise balanced, with real news accounting for $52.34$ percent of the samples and Fake news accounting for $47.66$ percent. We have done the following pre-processing steps.

• • Removing Twitter handles, garbage characters, URLs, and numerical values from the sentences

• • Removed all the stopwords using nltk corpus

• • Stemming and Lemmatization

• • Vectorization of the sentences to encode them into numbers

• • Post-padding to make sentences of the same length

We employed the identical models as detailed in the referenced paper. The machine learning algorithms utilized include DT, Logistic Regression, Support Vector Machine with a linear kernel, Gradient Boost, Random Forest, and Passive-Aggressive Classifier. Table 6 illustrates the performance comparison of these algorithms with and without the inclusion of the SEPSIS feature.

Table 6.

Accuracy (Acc), weighted average precision(P), weighted average recall (P), and weighted average f1 score (F1) of ML models for fake news identification

Table 6 Long description

The table compares the performance of various machine learning models for fake news identification, with and without the SEPSIS feature. It has two main sections: Without SEPSIS feature and With SEPSIS feature, each containing columns for Accuracy (Acc), weighted average precision (P), weighted average recall (R), and weighted average f1 score (F1). The models listed are Decision tree, Logistic regression, Support vector machine, Gradient boost, Passive aggressive classifier, and Random Forest. Each model’s performance metrics are provided for both scenarios. For example, the Decision tree model has an accuracy of 85.34 without SEPSIS and 85.84 with SEPSIS. The table provides a detailed comparison of how the inclusion of the SEPSIS feature affects the performance of each model.

Including the SEPSIS feature for every classifier increases the accuracy percentage. Thus from the comparison table, it can be observed that the SEPSIS property of the sentences is essential in identifying Fake news. It gives the highest change in accuracy for the Random Forest classifier.