1. Introduction
Code-switching in written natural language text occurs when the author chooses to switch from one language to at least one other. During the last two decades, linguists, sociolinguists, and psycholinguists have put forward several definitions for code-switching. Swann and Sinka (Reference Swann, Sinka, Graddol, Leith, Swann, Rhys and Gillen2007) believed that the scholar’s discipline informed the choice of definitions. Most individuals understand code-switching to occur when mixing between two (or among several) languages. According to Milroy and Muysken (Reference Milroy and Muysken1995), this term is “the alternative use by bilinguals of two or more languages in the same conversation.” Myers-Scotton (Reference Myers-Scotton2006) defined code-switching similarly as “the use of two or more languages in the same dialog.” When a bilingual person switches between two languages, this might be due to several reasons or motivations. Some of these are pointed out by Grosjean (Reference Grosjean1982). For example, code-switching might occur when some bilinguals cannot find an appropriate translation for what they want to say or there is no suitable expression or word in the language being used. Also, some situations, attitudes, emotions, and messages generate code-switching.
Code-switching in online communication is prevalent. Hale (Reference Hale2014) reported that more than 10% of Twitter users wrote in multiple languages; this has also been spotted on other social media platforms (Johnson Reference Johnson, Jones and Uribe-Jongbloed2013; Androutsopoulos Reference Androutsopoulos2013; Jurgens, Dimitrov, and Ruths Reference Jurgens, Dimitrov and Ruths2014; Nguyen et al. Reference Nguyen, Dogruöz, Rosé and de Jong2016). Gupta et al. (Reference Gupta, Bali, Banchs, Choudhury and Rosso2014) stated that users in social media tend to switch from one script, for example Arabic, to another, such as Roman. The occurrence of code-switching in online communication presents a challenge for natural language processing (NLP) tools, since many of them have been prepared for texts written in only one language.
This phenomenon is of particular interest when processing the Arabic language because of its frequent occurrence and because of the many dialects of Arabic in use. When processing Arabic text, it is important to identify where and when code-switching occurs, because more appropriate language resources can be applied to the task and thus make a significant improvement in processing performance.
However, relatively few studies of code-switching for Arabic texts exist, not only on its frequency but also on the development of software to identify occurrences automatically. A contributory factor is that dialect identification for Arabic has been found to be more difficult than for other languages. This paper will seek to address this gap in the research.
The automatic detection of code-switching has been achieved using various approaches such as n-grams (e.g., Oco et al. Reference Oco, Wong, Ilao and Roxas2013; Bacatan et al. Reference Bacatan, Castillo, Majan, Palermo and Sagum2014), and dictionary-based methods. Today, detection rates of up to 99% can be reached, even for small input. The work in this paper uses the Tawa toolkit (Teahan Reference Teahan2018), which uses the prediction by partial matching (PPM) compression scheme; it also uses the Waikato Environment for Knowledge Analysis (Weka) data analytic tool as a second method for the automatic detection of code-switching in Arabic text. It provides a comparison between the traditional machine learning classifier algorithm which is the sequential minimal optimization (SMO) and the PPM compression-based approach. This paper explains in more detail about creating two types of Arabic corpora, code-switching and non-code-switching, for use in training and testing. Next, it outlines the experiments performed on Arabic Facebook text to evaluate the PPM classifier produced by the Tawa toolkit and the SMO classifier provided by Weka. Finally, it provides a conclusion for this study.
2. Related work
Since the mid-1900s, linguists have studied the code-switching phenomenon. In contrast, the NLP community has started to address it only recently. Solorio et al. (Reference Solorio, Blair, Maharjan, Bethard, Diab, Ghoneim, Hawwari, AlGhamdi, Hirschberg, Chang and Fung2014) pointed out that code-switching has posed new research questions, and they expected an increase in NLP research addressing code-switching in the coming years.
Oco and Roxas (Reference Oco and Roxas2012) developed pattern-matching refinements (PMRs) to the automatic detection of code-switching using a dictionary-based approach. They achieved a high accuracy of 94.51%, a marked improvement in accuracy rates over the other dictionary-based approaches, which were in the range of 75.22–76.26%. The disadvantages of a dictionary-based approach are that dictionaries for some languages may not be available and that some words are not in the dictionaries (Oco et al. Reference Oco, Wong, Ilao and Roxas2013).
Lignos and Marcus (Reference Lignos and Marcus2013) produced a system that outlined the problems of both social media and code-switching in language and detection status. They collected two corpora from Twitter, containing about 6.8 million Spanish tweets and 2.8 million English tweets, to model the two languages. They then annotated using crowdsourcing for tens of thousands of Spanish tweets, around 11% of which included code-switching. This system achieved a 0.936
$F$
-measure in detecting code-switching tweets and 96.9% word-level accuracy.
Summary of our corpora produced for this research
Dialect detection in Arabic is crucial for almost all NLP tasks and has recently gained strong interest among Arabic NLP researchers. One of the earliest works in this area was by Elfardy and Diab (Reference Elfardy and Diab2012) and addressed the automatic detection of code-switching in Arabic online text by identifying token-level dialectal words. They mentioned that identifying code-switching in written text is a very challenging task, since an accompanying speech signal does not exist. They produced a system called Automatic Identification of Dialectal Arabic (AIDA) that comprised a Modern Standard Arabic (MSA) morphological analyzer, dictionaries, sound-change rules and a set of language models to perform token-level dialect identification. It achieved a token-level F-score of 84.9%.
Elfardy, Al-Badrashiny, and Diab (Reference Elfardy, Al-Badrashiny and Diab2014) used a Naïve-Bayes classifier provided by the Weka toolkit (Hall et al. Reference Hall, Frank, Holmes, Pfahringer, Reutemann and Witten2009) to detect code-switching between the Egyptian Arabic Dialect (EAD) and MSA. They used the code-switching portion from the Arabic Online Commentary Dataset built by Zaidan and Callison-Burch (Reference Zaidan and Callison-Burch2011). It obtained an accuracy of 51.9%.
Several studies, such as Malmasi, Refaee, and Dras (Reference Malmasi, Refaee and Dras2015) and Malmasi and Zampieri (Reference Malmasi and Zampieri2016, Reference Malmasi and Zampieri2017), have addressed Arabic dialect identification. Ali (Reference Ali2018) used a character-level convolutional neural network approach to classify Arabic dialects; this achieved an F1-score of 57.6%. This result was obtained using a recurrent layer before the convolution layer. Alshutayri et al. (Reference Alshutayri, Atwell, Alosaimy, Dickins, Ingleby and Watson2016) examined several classifiers provided by Weka to classify Arabic dialects. They pointed out that the SMO algorithm achieved the best accuracy. (SMO has also been used in this paper as a traditional machine learning classifier for comparison).
Alkhazi and Teahan (Reference Alkhazi and Teahan2017) used a PPM character-based compression scheme to segment Classical and Modern Standard Arabic. It achieved an accuracy of 95.5% and an average F-measure of 0.954 (recall 0.955 and precision 0.958).
3. New Arabic corpora
Most NLP research for the Arabic language focuses on MSA; research in Arabic dialects is sparse. To evaluate a compression-based approach and traditional machine learning classifiers for the automatic detection of code-switching, it was necessary to build a code-switching corpus containing samples of Arabic code-switching and non-code-switching corpora. We therefore created the Bangor Arabic–English Code-switching (BAEC) corpus for use as a testing set. The following non-code-switching corpora have also been created for this research to use as training sets:
• Saudi dialect corpus (SDC);
• Egyptian dialect corpus (EDC).
Table 1 lists the number of words, number of characters, and overall size in kilobytes for each of these corpora.
3.1 Methodology
Different methods were used to create our new corpora. The Facebook ScraperFootnote a (FS) system helped us to extract data from Arabic Facebook pages. For SDC, we could not collect enough text from Facebook alone, since most Saudi Facebook users tend to use non-colloquial Arabic. Hence, we moved to Twitter, the third most popular social network platform in Saudi Arabia (Global Media Insight Website 2019) (see Figure 1). Manual cut-and-paste techniques were used to extract data from Twitter and websites. For non-code-switching corpora, extensive cleaning removed emojis, punctuation marks, URLs, and non-Arabic words. For the Saudi and Egyptian dialect corpora, we also removed Quranic Arabic and Hadith (Prophet Mohammad’s speech, words, and actions), since they are classified as Classical Arabic.
Most popular social network platforms in Saudi Arabia 2018 (in millions; Global Media Insight Website 2019).
3.1.1 Sampling method
A judgmental, non-probability-type sampling method was chosen to collect our data. The procedure for collecting a sample is based on personal judgment, so the researcher uses his or her own experience and knowledge to select a sample (Doyle Reference Doyle2011). When we built our corpora, therefore, we looked at the user’s location (if available) as well as reading each post and tweet to verify its class (whether written in a Saudi dialect or an Egyptian dialect). This required substantial time and effort, taking approximately 3 months’ work.
3.1.2 Verifying the quality of the tagging
Annotated Arabic dialects are more challenging than MSA because they do not have clear spelling standards and conventions. They were not commonly written until recently, whereas MSA has official orthographic standards and conventions. Today, Arabic dialects are often used in social media. Two Saudi university researchers with extensive knowledge in Egyptian dialect verified the quality of tagging. If they disagreed on a particular word, whether in MSA or an Arabic dialect, they looked at the Arabic online dictionary (www.almaany.com) and came to an agreement. This dictionary contains all MSA words with their meanings.
3.2 Code-switching corpus
The term “code-switching corpus” refers to a body of text consisting of two or more languages under study (Yu et al. Reference Yu, He, Chien and Tseng2013). Because Arabic and English are the primary languages in this research, we built an Arabic–English code-switching corpus. To our knowledge, there are no available Arabic code-switching corpora derived from Facebook, so it was necessary to build a new corpus for our research.
The main objective of this research is to detect code-switching in Arabic text from Facebook. We therefore collected our corpus from different Arabic Facebook pages containing code-switching and used it to build the BAEC corpus, which focuses on switching between Arabic and English. It consists of 45,251 words and is 436 KB in size (see Table 1). It was collected from different Facebook pages using the FS system. It includes code-switching between: MSA and English; the Saudi dialect and English; and the Egyptian dialect and English. Manually annotated, it has been produced in XML. A sample taken from the BAEC corpus is shown in Figure 2.
A sample from the BAEC corpus.
We used the following rules when we annotated the BAEC corpus: (1) if the sentence was written in Arabic letters and had no Saudi or Egyptian dialect word, we annotated it as
$< \mathrm{MSA} >$
; (2) if a phrase was written in Arabic letters and contained any Egyptian dialect word, we annotated it as
$< \mathrm{Egypt} >$
(this rule was also applied to the Saudi dialect); (3) if the word or the phrase was written in English letters, we tagged it as
$< \mathrm{English} >$
. We found that tagging the BAEC corpus was much more complex than we first thought, as it has a lot of emojis, URLs, English numbers, Arabic numbers, English hashtags, Arabic hashtags, and non-Arabic words.
In annotating the BAEC corpus, we found some unforeseen issues. For example, some Arabic Facebook users write some English words using Arabic letters, such as منشن “mention”, اونلاين “online”, كورس “course”, شير “share”, and so on. These words have been annotated as translated English
$< \mathrm{TEng} >$
. Also, some users mix Arabic with English words to produce one mixed word, based on a normal word such as “class”, where they add Arabic letters ال and then write الكلاس “Alclass”. Another habit is to use Arabic grammar in an English word, for example, making “class” plural using Arabic grammar rules for pluralization and then writing it as كلاسات “claassaat”. We annotated these kinds of words as
$< \mathrm{MAE}>$
, which means a Mixed Arabic and English word.
In fact, the detection of TEng and MAE words provided one of the biggest challenges for NLP tools, since the issue was unforeseen and we had insufficient training data to provide an effective means to identify these phenomena.
3.3 Non-code-switching corpora
The term “non-code-switching corpora” refers to bodies of text consisting of one language (Yu et al. Reference Yu, He, Chien and Tseng2013). Because Arabic is the primary language in this research, we built two Arabic corpora, the SDC and the EDC (see Table 1), to be used as non-code-switching corpora for training language models.
3.3.1 Saudi dialect corpus (SDC)
There are many dialectal varieties in Saudi Arabia, such as the Najdi dialect (Arabic: اللهجة النجدية) spoken in the central region of Saudi Arabia by approximately 4 million speakers, the Hejazi dialect (Arabic: اللهجة الحجاز ية) spoken throughout the Hejaz region (the west region) of the country by around 14 million speakers, and the Gulf dialect (Arabic: اللهجة الخليجية) spoken in the east region of Saudi Arabia, near the Gulf region, by around 7 million speakers (Eberhard, Simons, and Fennig Reference Eberhard, Simons and Fennig2017). The Gulf dialect spreads to Bahrain, Qatar, UAE, and Iraq (Eberhard et al. Reference Eberhard, Simons and Fennig2017). In fact, MSA is used throughout all Saudi regions, mixed with each dialect. A map of the dialect usage in Saudi Arabia is shown in Figure 3.
A map of Saudi Arabia showing the locations of the dialects (based on Al-Moghrabi Reference Al-Moghrabi2015).
A 210,396-word corpus called the SDC was built for training the Saudi model, containing the mixed dialects of Saudi Arabia. It was collected from social media platforms, such as Facebook and Twitter, and is 2018 KB in size (see Table 1).
3.3.2 Egyptian dialect corpus (EDC)
The Egyptian dialect is one of the most widely spoken Arabic dialects. It is used by around 64 million speakers (Eberhard et al. Reference Eberhard, Simons and Fennig2017). In fact, Egyptian TV and cinema spread their dialect to all Arab countries. It is considered the most widely understood dialect in the Arab world. For historical reasons, some frequently used words are shared between the Egyptian dialect and the Hejazi dialect (one of the Saudi dialects used in the west of Saudi), which makes distinguishing between these two dialects a challenging task.
The EDC that we constructed consists of 218,149 words and is 2024 KB in size (see Table 1). It was also collected from the social media platform Facebook.
3.4 Analyzing the non-code-switching corpus
To analyze the new corpora described above, we investigated the top 10 most frequent words from each corpus. This information allowed us to identify how often words are used in different corpora and the similarities and differences between them.
Table 2 illustrates the top 10 most frequent words from the EDC and the SDC. There are some similarities between the Saudi dialect and the Egyptian dialect. For example, the word من “of” is the second most frequent word in both dialects. The word في “in” is the most frequent word in the Saudi dialect; in contrast, it is the third most frequent word in the Egyptian dialect.
The top 10 most frequent words from the EDC and from the SDC
4. Tawa
Tawa is a compression-based toolkit based on the API designed by Cleary and Teahan (Reference Cleary and Teahan1997); it is an updated version of the text mining toolkit (TMT) (Mahoui et al. Reference Mahoui, Teahan, Thirumalaiswamy Sekhar and Chilukuri2008). Tawa adopts the PPM text compression algorithm developed by Cleary and Witten in Reference Cleary and Witten1984. It is a character-based model that predicts an upcoming symbol using the previous symbols with a fixed context. Every possible upcoming symbol is assigned a probability based on the frequency of previous occurrences. If a symbol has not been seen before in a particular context, the method will “escape” to another lower-order context in order to predict the symbol. This is called the escape method and is used to combine the predictions of all character contexts (Cleary and Witten Reference Cleary and Witten1984).
The main aim of the Tawa toolkit is to facilitate the design and implementation of applications that need textual models, such as word/language segmentation, text classification, and a wide range of text mining applications, by protecting users from modeling details and estimating process details. It consists of nine main applications, such as classify, codelength, train, markup, segment, and so on (Teahan Reference Teahan2018). This study concentrates on two applications provided by the Tawa toolkit: building models and language segmentation.
For language segmentation using Tawa, we use multiple models trained on representative text, using the toolkit’s train tool, from each of the languages under research. We then use the markup tool, which utilizes the Viterbi algorithm to find the segmentation with the best compression with all possible segmentation search paths extended at the same time, discarding the poorly performing alternatives (Teahan Reference Teahan2018).
5. Weka
Weka is a data mining tool that contains different machine learning algorithms for classification, regression, clustering, and so on. It has a graphical user interface that makes it easy to use (Hall et al. Reference Hall, Frank, Holmes, Pfahringer, Reutemann and Witten2009). Weka implements several classifiers, such as Naïve-Bayes, J48, ZeroR, SMO, and so on. As mentioned earlier, Alshutayri et al. (Reference Alshutayri, Atwell, Alosaimy, Dickins, Ingleby and Watson2016) pointed out that the SMO algorithm achieved the best accuracy rate when they classified Arabic dialects, so we used the SMO classifier in our experiments.
The support vector machine (SVM) is a supervised machine learning algorithm used for regression analysis and classification. It has been applied to different NLP problems, such as part-of-speech tagging, information extraction, and so on. On unseen data, the SVM classifier has a better generalization capability than other classifiers (Li, Bontcheva, and Cunningham Reference Li, Bontcheva and Cunningham2009). One disadvantage of SVM is that it is slow, especially when applied to a very large classification problem. The enhanced algorithm for SVM, SMO, solves SVM problems by dividing a big quadratic programming (QP) problem into a chain of smaller QP problems; this leads to improved results and computation time (Platt Reference Platt1998).
6. Experiments and results
Three experiments were performed as part of the evaluation of the compression-based approach (provided by Tawa) and of traditional machine learning classifiers such as the SMO classifier (provided by Weka) to detect code-switching in Arabic Facebook text. These were to: (1) detect code-switching among the Egyptian dialect and English; (2) detect code-switching among the Egyptian dialect, the Saudi dialect, and English; and (3) detect code-switching among the Egyptian dialect, the Saudi dialect, MSA, and English.
Choosing suitable training corpora is the first step in building language models. As indicated in Section 3.3, the SDC and EDC were created for this purpose. The corpus selected for training MSA was built by Alkahtani, a PhD student at Bangor University (Alkahtani Reference Alkahtani2015). The Brown corpus was selected for training English.
The BAEC corpus (described in Section 3.2) was used in these experiments as a testing corpus. First, a cleaning process removed all MAE and TEng words (described in Section 3.2) because we did not have enough training data. Numbers, emojis, and punctuation marks were also removed, leaving only pure Arabic and English to process, as we thought this would reduce the errors and enhance the accuracy.
Confusion matrix for two classes
A confusion matrix has been used to evaluate the performance of the automatic detection of code-switching. A confusion matrix is a table that summarizes the classification and segmentation performance. The often-used case is a two-class confusion matrix that is used to present the positive and negative classes for some binary classification problems. In this case, the four cells of the matrix are true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), as shown in Table 3 (Sammut and Webb Reference Sammut and Webb2017). TP is the number of correct predictions that are positive, FN is the number of incorrect predictions that are negative, FP is the number of incorrect of predictions that are positive, and TN is the number of correct predictions that are negative.
From these four outcomes, four measures of classification performance can be defined as
\begin{equation}Accuracy = \frac{TP+TN}{TP+TN+FP+FN}\end{equation}
\begin{equation}Recall = \frac{TP}{TP+FN}\end{equation}
\begin{equation}Precision = \frac{TP}{TP+FP}\end{equation}
\begin{equation}F\textrm{-}measure = \frac{2 Precision Recall}{Precision + Recall}\end{equation}
6.1 First experiment
The first experiment was conducted to evaluate the PPM compression algorithm and focused on the detection of code-switching between the Egyptian dialect and English. The testing text was manually extracted from the BAEC corpus for cases that contained only the Egyptian dialect and English, around 17,761 words and 79,975 characters.
Automatic detection of code-switching between the Egyptian dialect and English, using the PPM compression algorithm, obtained an accuracy of 99.8%, an average recall of 0.996, an average precision of 0.999, and an average
$F-$
measure of 0.998. Table 4 shows the result of this experiment. Some sample output from the PPM classifier is shown in Figure 4. The bolded values are the highest scores in each column.
The results of the first experiment
Sample output from the first experiment’s output using PPM.
However, we have noticed that most occurrences of English text that were incorrectly predicted as an Egyptian dialect were abbreviations, such as CV, PDF, and BBC. This is because the Brown corpus, built from American English written texts, does not include abbreviations. The use of a new English corpus for abbreviations should therefore further reduce the number of errors.
We repeated this experiment using Weka so we could discover the best result. We used the SMO algorithm with the StringToWordVector filter and the WordTokenizer filter to detect code-switching between the Egyptian dialect and English. The WordTokenizer filter divides the text into words. We achieved an accuracy of 55%, an average recall of 0.55, an average precision of 0.671, and an average F-measure of 0.44. Table 4 shows the result of this experiment.
After that, we examined the SMO classifier with the CharacterNGramTokenize filter which divides text into n-grams. We tried four different types of n-grams, UniGram, BiGram, TriGram, and FourGram. Table 4 shows that UniGram achieved an accuracy of 97.5% which is a higher accuracy rate than the other n-grams models.
The results of the second experiment
An example of confusion between the Saudi and Egyptian dialects from the second experiment’s output using PPM.
6.2 Second experiment
The second experiment was conducted to evaluate the automatic detection of code-switching among the Egyptian dialect, the Saudi dialect, and English. As in the first experiment, the testing text was extracted from the BAEC corpus for cases that contained only the Egyptian dialect, the Saudi dialect, and English, around 18,957 words and 85,099 characters.
Using PPM in this experiment produced an accuracy of 97.8%, an average recall of 0.899, an average precision of 0.977, and an average F-measure of 0.932. Table 5 shows the result of this experiment.
Figure 5 illustrates some Saudi dialect predicted as Egyptian dialect, such as (highlighted in yellow in Figure 5), and some Saudi dialect correctly predicted as Saudi dialect, such as (highlighted in green in Figure 5). Also, the word “opt”, an abbreviation in architecture, was predicted as a Saudi dialect word. In this testing text, several English abbreviations were predicted as Saudi dialect or Egyptian dialect words, such as HR, IBDL, PMP, CTRL C, and so on. The results of the first and second experiments show a clear need to build a new English training corpus that contains possible abbreviations.
Repeating the second experiment, using the SMO classifier with the StringToWordVector filter and the WordTokenizer filter, produced an accuracy of 48.2%, an average recall of 0.482, an average precision of 0.571, and an average
$F-$
measure of 0.378. Table 5 shows the result of this experiment. We also examined the SMO classifier with the CharacterNGramTokenize filter and tried different types of n-grams, as shown in Table 5. Table 5 shows that UniGram achieved an accuracy of 80.7% which is a higher accuracy rate than the other n-grams models.
6.3 Third experiment
The third experiment was considered a more complex task for the PPM classifer since it had four different classes: the Egyptian dialect, the Saudi dialect, MSA, and English. The testing file, of around 5002 words and 23,668 characters, was also manually extracted from the BAEC corpus containing the Egyptian dialect, the Saudi dialect, MSA, and English.
Using PPM for the third experiment obtained an accuracy of 53.261%, an average recall of 0.539, an average precision of 0.562, and an average F-measure of 0.551. Table 6 shows the result of this experiment.
The results of the third experiment
Figure 6 shows a sample of the confusion between the Egyptian dialect and MSA in the third experiment. All these sentences are in the Egyptian dialect but were predicted as MSA. We speculate that the reason for this disappointing result was that the MSA corpus used to train the MSA model did not represent the MSA found in Facebook, since it was built from news websites. To prove this, we examined the overall compression code lengths of the sample marked-up text for the different model configurations, as shown in Table 7.
Sample of some Egyptian dialect sentences predicted as MSA from the third experiment’s output using PPM.
Minimum code lengths for different models
The Egyptian, Saudi, and English models have the lowest minimum code lengths with 258722.891 bits, so these are the more appropriate models for this testing file. Adding a fourth model, MSA, to these models resulted in an increase in the minimum code length.
Repeating the third experiment using the SMO classifier with the StringToWordVector filter and the WordTokenizer filter obtained an accuracy of 26.3%, an average recall of 0.263, an average precision of 0.351, and an average F-measure of 0.239. Table 6 shows the result of this experiment. We then examined the SMO classifier with the CharacterNGramTokenize filter and tried different types of n-grams as shown in Table 6. Table 6 shows that UniGram achieved an accuracy of 60.2% which is a higher accuracy rate than the other n-grams models.
7. Conclusion
This paper discussed our creation of several new Arabic corpora, the production of a code-switching corpus, BAEC, that contains samples of Arabic code-switching and the production of two non-code-switching corpora, SDC and EDC.
We compared the traditional machine learning classifier SMO and the PPM compression-based approach to the automatic detection of code-switching in Arabic text. Our experiments showed, first, that PPM achieves a higher accuracy rate than SMO when the training corpus correctly represents the language or dialect under study. When this condition is satisfied, therefore, the compression-based approach will be more effective for automatically detecting code-switching in written Arabic text. Second, when Arabic and English are classified using SMO, the CharacterNGramTokenize filter is a more appropriate filter to use than the WordTokenizer filter because the difference between these two languages is best modeled using characters. Third, the CharacterNGramTokenize filter is also more appropriate for comparison between SMO and PPM, since PPM is a character-based model.
The first experiment focused on the detection of code-switching among the Egyptian dialect and English. PPM obtained an accuracy of 99.8% on testing data from the BAEC corpus, 2.3% higher than the SMO classifier’s accuracy. The second experiment investigated the automatic detection of code-switching among the Egyptian dialect, the Saudi dialect, and English. PPM achieved an accuracy of 97.8%, 17.1% higher than the SMO classifier. Finally, the third experiment detected code-switching among the Egyptian dialect, the Saudi dialect, MSA, and English. The SMO classifier obtained an accuracy of 60.2%, 6.9% higher than PPM.
Clearly, the MSA corpus used to train the MSA PPM model in the third experiment did not represent MSA text in Facebook, since it was built from news websites. As part of future work, a possible solution to overcome this issue is to build a new MSA Facebook corpus, trained on MSA text specially taken from Facebook. In addition, to distinguish between MSA and Arabic dialects is very difficult because most Arabic users, especially Saudis and Egyptians, mix MSA with their dialects. Finally, the use of a new English corpus containing all the possible abbreviations should improve the results further.
Acknowledgements
We are grateful for Saudi government sponsorship of the first author, Taghreed Tarmom.
