2.1 Division of the Arabic dialects

Geographical location is the main factor producing many Arabic dialects. Most studies categorise Arabic dialects generally into eastern and western dialects (Barkat, Ohala, and Pellegrino Reference Barkat, Ohala and Pellegrino1999; Ghazali, Hamdi, and Barkat Reference Ghazali, Hamdi and Barkat2002; Watson Reference Watson2002; Kaye and Rosenhouse Reference Kaye, Rosenhouse and Hetzron2005). The eastern dialects include those spoken in the Arabian Peninsula (Saudi Arabia, Oman, Yemen, the United Arab Emirates, Qatar, Kuwait, Bahrain), Mesopotamia (Iraqi dialects), Levantine (Syrian, Lebanese, Jordanian, and Palestinian dialects), and Egypt (Egyptian dialects). The western dialects are those spoken in countries located west of Egypt, including Libya, Tunisia, Algeria, Morocco and Mauritania.

The annual reference on the languages of the world (Ethnologue) in its 22nd edition defines 36 Arabic dialects. These dialects are spoken in the Middle East and North Africa in the area spreading from Morocco and Mauritania in the west to Oman in the east, as well as countries from north-central and northeast Africa, such as Sudan and Chad. Ethnologue considers Tajiki, Uzbeki, Cypriot and Maltese, spoken in central Asia and island countries in the Mediterranean sea, also among Arabic dialects (Eberhard et al. Reference Eberhard, Simons and Fennig2019).

The Glottolog, a bibliographic database of the world’s languages and language families, classifies colloquial Arabic into 5 groups, composed of 38 dialects (Hammarström, Forkel, and Haspelmath Reference Hammarström, Forkel and Haspelmath2018). The five groups are Arabian Peninsula Arabic, Eastern Arabic, Egyptic Arabic, Levantine Arabic, and North African Arabic. The two Arabic dialects that are mentioned in the Glottolog, but not in the Ethnologue, are Nubi and Khorasan Arabic.

Although geographical location plays an effective role in obtaining various Arabic dialects, several additional factors impact the appearance of Arabic dialects, even within the same geographical region. These factors include lifestyle, religion, education and socio-economic status (Watson Reference Watson2002; Holes Reference Holes2004).

According to Ethnologue (Reference Eberhard, Simons and Fennig2019), there are approximately 89 million Egyptian Arabic speakers, roughly 22.4 million of whom speak Sa’idi, 2.2 million speak Eastern Egyptian Bedawi, and 64.4 million speak Modern Egyptian. Modern Egyptian includes north and south central Delta Arabic and Cairene Arabic, which is widely understood in the Arab world and used in media (Eberhard et al. Reference Eberhard, Simons and Fennig2019). This makes Modern Egyptian the most popular dialect in the Arab world. Table 1 summarises the number of speakers of the 36 Arabic dialects.

Table 1.

Number of speakers of the 36 Arabic dialects according to Ethnologue (Reference Eberhard, Simons and Fennig2019)

2.2 Challenges of processing dialectal Arabic

There are several difficulties facing the NLP research community when processing dialectal Arabic. These challenges arise in part due to the inherent properties of the Arabic language itself, while the remaining challenges occur as a result of the differences between Arabic dialects on one side and MSA on the other side, as well as the differences between Arabic dialects themselves.

Complex morphology. Dialectal Arabic inherits the complex morphology of MSA due to the rich inflectional morphology (i.e., affixes) that models aspect, case, gender, mood, number, person, state and voice. Therefore, the word takes many forms. Moreover, there are a large number of clitics, such as conjunctions and future particles, that can be attached to the word. The attached clitics significantly increase the number of possible forms of the word (Habash et al. Reference Habash, Eryani, Khalifa, Rambow, Abdulrahim, Erdmann, Faraj, Zaghouani, Bouamor, Zalmout, Hassan, Al-Shargi, Alkhereyf, Abdulkareem, Eskander, Salameh and Saddiki2018). Although MSA is morphologically richer than dialectal Arabic in some dimensions, such as gender, case and mood (e.g., Arabic dialects mostly lack the dual form found in MSA), Arabic dialects generally have a more complex cliticisation system than MSA (e.g., circumfix negation in Arabic dialects) (Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2014). In addition, dialectal Arabic has clitics and affixes that are not part of MSA morphology (Habash et al. Reference Habash, Eskander and Hawwari2012b).

Cross-dialectal differences. Linguistic differences do not only appear between MSA and dialectal Arabic, but also between Arabic dialects. The linguistic diversity of Arabic dialects emerges at all levels including morphology, syntax, lexicon and phonology. Moreover, distinct degrees of linguistic distance occur based on geographical distance, education, social class and other potential factors (Diab and Habash Reference Diab and Habash2007). Consequently, the complex relationship between Arabic dialects creates a scenario in which NLP applications are impeded, since a pre-made tool that is suited to one Arabic variety will not perform well for another (Habash et al. Reference Habash, Eskander and Hawwari2012b; Shoufan and Alameri Reference Shoufan and Alameri2015).

Limited resources. MSA has received much attention in Arabic NLP; a large number of resources have been built so far and many proposed semi-automatic approaches have been used to generate more annotated data (Habash Reference Habash2010). However, these data cannot be utilised when addressing dialectal Arabic texts due to the linguistic differences between MSA and Arabic dialects. Therefore, many works on Arabic NLP that handle the web and social media texts have created their own annotated resources to fill in the gaps. More effort should be made as the current available dialectal Arabic datasets are limited in terms of size, scope and scale (Diab and Habash Reference Diab and Habash2007; Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2014; Habash et al. Reference Habash, Eryani, Khalifa, Rambow, Abdulrahim, Erdmann, Faraj, Zaghouani, Bouamor, Zalmout, Hassan, Al-Shargi, Alkhereyf, Abdulkareem, Eskander, Salameh and Saddiki2018).

Script choices. The main script that is usually used to write dialectal Arabic text is Arabic script. However, dialectal Arabic sometimes appears on social media and generally on the web in a non-standard romanisation, often called Arabizi (Darwish Reference Darwish2014; Habash et al. Reference Habash, Eryani, Khalifa, Rambow, Abdulrahim, Erdmann, Faraj, Zaghouani, Bouamor, Zalmout, Hassan, Al-Shargi, Alkhereyf, Abdulkareem, Eskander, Salameh and Saddiki2018).

Ambiguous orthography. Arabic script, the most common form of writing dialectal Arabic, employs optional diacritical marks for short vowels and consonantal gemination (Habash et al. Reference Habash, Eryani, Khalifa, Rambow, Abdulrahim, Erdmann, Faraj, Zaghouani, Bouamor, Zalmout, Hassan, Al-Shargi, Alkhereyf, Abdulkareem, Eskander, Salameh and Saddiki2018). The absence of diacritical marks is the main source of ambiguity in Arabic NLP (Habash Reference Habash2010). It is also the main source of ambiguity when trying to recognise Arabic dialects in the written text. That is, pronunciation of vocabulary sometimes differs from one Arabic dialect to another while the written vocabulary is identical because diacritical marks, which are used to reflect differences in pronunciation, are always omitted in the written text. For example, the word (, bAb,‘door’)Footnote ^a is written the same way in all Arabic dialects but pronounced differently (e.g., /ba:b/Footnote ^b in Gulf dialects and /be:b/ in Maghrebi dialects).

Inconsistent orthography. Unlike MSA, Arabic dialects have no official orthographies and generally have no well-defined orthographic standards. Therefore, there is a degree of variety in dialectal spelling. That is, a single word in any dialect may be written in a large number of possible ways based on independent decisions. For example, the Egyptian word (, mAbyqulhAš,‘he does not say it’) was found written 27 different ways according to Google Search (e.g., ) (Habash et al. Reference Habash, Eryani, Khalifa, Rambow, Abdulrahim, Erdmann, Faraj, Zaghouani, Bouamor, Zalmout, Hassan, Al-Shargi, Alkhereyf, Abdulkareem, Eskander, Salameh and Saddiki2018).

3.1 Morphological analysis and POS tagging

Morphological analysis has gained significant attention in dialectal Arabic NLP. Diab et al. (Reference Diab, Habash, Rambow, Altantawy and Benajiba2010) described the Cross-Lingual Arabic Blog Alerts (COLABA) project, a great effort to create resources for dialectal Arabic blogs covering Egyptian, Levantine, Iraqi and Moroccan Arabic. Their study focused on harvesting online sources, such as blogs and forums, to create resources for dialectal Arabic. For each dialectal surface word, the annotators were asked to provide a lemma, identify the various possible POS tags and provide a detailed morphological profile for nouns, adjectives and verbs. For example, they were required to enter the feminine singular form of the noun, if available. Al-Sabbagh and Girju (Reference Al-Sabbagh and Girju2012) presented YADAC, a multi-genre and multi-dialectal Arabic corpus collected from Twitter, online knowledge market services and blogs. The corpus provides linguistic analysis at the POS tagging and base-phrase chunking levels.

Al-Shargi et al. (Reference Al-Shargi, Kaplan, Eskander, Habash and Rambow2016) presented morphologically annotated corpora for Moroccan and Sanaani Yemeni Arabic. The corpora data were collected from both online and print materials such as internet comments, forums, oral interviews, folktales, sermons, textbooks, blogs and Facebook posts. They transcribed the Roman alphabet sentences from the textbooks into the Arabic alphabet using Conventional Orthography for Dialectal Arabic (CODA), which is a consistent orthographic system for writing Arabic dialects developed by Habash, Diab, and Rambow (Reference Habash, Diab and Rambow2012a). As the authors processed Moroccan and Yemeni dialects, they also created specific CODA guidelines for Moroccan and Yemeni inspired by the work of Habash et al. (Reference Habash, Diab and Rambow2012a, 2018). Furthermore, the Curras corpus, developed by Jarrar et al. (Reference Jarrar, Habash, Alrimawi, Akra and Zalmout2017), is the first morphologically annotated corpus of the Palestinian Arabic dialect. The corpus data were collected from a variety of resources including Facebook, Twitter, forums, blogs and TV shows. The annotation was defined as a tuple of many tags including lemma, MSA lemma, Buckwalter POS tag and grammatical properties (i,e., person, aspect, gender and number). They took advantage of the MADAMIRA disambiguation tool (Pasha et al. Reference Pasha, Al-Badrashiny, Diab, El Kholy, Eskander, Habash, Pooleery, Rambow and Roth2014), which includes the CALIMA analyser (Habash et al. Reference Habash, Eskander and Hawwari2012b), and the DIWAN dialectal word annotation interface (Al-Shargi and Rambow Reference Al-Shargi and Rambow2015). The Levantine Arabic Treebank (LATB) of Jordanian Arabic was developed by Maamouri et al. (Reference Maamouri, Bies, Buckwalter, Diab, Habash, Rambow and Tabessi2006) at the Linguistic Data Consortium (LDC). The corpus was collected from Levantine conversational telephone speech and was morphologically and syntactically annotated based on the guidelines of the MSA Penn Arabic Treebank (PATB) (Maamouri et al. Reference Maamouri, Bies, Buckwalter and Mekki2004) with some extensions.

The Egyptian Arabic Treebank corpus was collected from online informal texts (e.g., discussion forum posts) at the LDC (Maamouri et al. Reference Maamouri, Bies, Kulick, Tabessi and Krouna2012, Reference Maamouri, Bies, Kulick, Ciul, Habash and Eskander2014). The corpus was developed and annotated using an early versions of the CALIMA analyser (Habash et al. Reference Habash, Eskander and Hawwari2012b), as well as annotators who were allowed to manually annotate words. More recently, the BOLT Egyptian Arabic Treebank corpus was developed by the LDC (Maamouri et al. Reference Maamouri, Bies, Kulick, Krouna, Tabassi and Ciul2018). The corpus contains data collected from discussion forums and annotated with information for part-of-speech (POS) category, morphology, gloss and syntactic tree. Khalifa et al. (Reference Khalifa, Habash, Eryani, Obeid, Abdulrahim and Al Kaabi2018) developed a large-scale morphologically annotated corpus of Emirati Arabic. The contents of the corpus were selected from eight novels of the Emirati Arabic variety of the Gumar corpus (Khalifa et al. Reference Khalifa, Habash, Abdulrahim and Hassan2016). The corpus contains about 200,000 words with POS categories, lemmatisation, English glosses and dialect identification annotations.

Samih et al. (Reference Samih, Attia, Eldesouki, Abdelali, Mubarak, Kallmeyer and Darwish2017) used a corpus, containing 350 tweets with more than 8000 words including 3000 words written in the Egyptian dialect. They manually annotated each word to provide: stem, lemma, CODA-compliant writing (Habash et al. Reference Habash, Diab and Rambow2012a), segmentation and POS tags, as well as the corresponding MSA word, MSA segmentation and MSA POS tags. Furthermore, a collection of 1400 segmented and annotated tweets was developed by Eldesouki et al. (Reference Eldesouki, Samih, Abdelali, Attia, Mubarak, Darwish and Laura2017) for Egyptian, Levantine, Gulf and Maghrebi dialects. Also, Darwish et al. (Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018) developed a new dataset composed of 350 POS-tagged Arabic tweets for each of the 4 major dialects: Egyptian, Levantine, Gulf and Maghrebi.

3.2 Sentiment analysis

Arab people usually write in their native dialects to express personal opinions on social media and in discussion forums. In fact, a large amount of online contents are user-generated and written in dialects. Therefore, dialectal Arabic NLP plays a vital role in sentiment analysis (SA) and opinion mining research. Many studies developed the required resources for their SA on various Arabic dialects. Abdul-Mageed and Diab (Reference Abdul-Mageed and Diab2012) manually developed SIFAAT, a sentiment lexicon that contains 3325 adjectives annotated as positive, negative or neutral. The SIFAAT lexicon was automatically extended using machine translation and statistical formulation to create SANA (Abdul-Mageed and Diab Reference Abdul-Mageed and Diab2014). SANA contains 224,564 entries, covering MSA as well as Egyptian and Levantine dialects. Developing SANA involved the use of many Arabic lexical resources such as THARWA, a large-scale dialectal Arabic – standard Arabic – English lexicon (Diab et al. Reference Diab, Al-Badrashiny, Aminian, Attia, Elfardy, Habash, Hawwari, Salloum, Dasigi and Eskander2014).

Ameur, Jamoussi, and Hamadou (Reference Ameur, Jamoussi and Hamadou2016) proposed a new method to develop a dictionary for emotion analysis. They used emotion symbols as sentiment polarity indicators to collect the data from Tunisian Facebook pages. NileULex is a sentiment lexicon containing single-word terms and compound phrases for MSA and Egyptian dialects. The terms were automatically extracted from social media, but they were manually annotated to ensure the quality (El-Beltagy Reference El-Beltagy2016). During implementation of the WeightedNileULex, an extension of NileULex, presented by El-Beltagy (Reference El-Beltagy2017), an automatic method was proposed and used to assign strength scores or weights to NileULex entries.

A considerable number of corpora in the literature have been developed intending to serve SA for Arabic dialects. For instance, Siddiqui, Dahab, and Batarfi (Reference Siddiqui, Dahab and Batarfi2015) developed a multilingual annotated corpus for SA, consisting of 191K reviews of hotels in Saudi Arabia written mostly in Arabic and English, and very few reviews in other languages. The annotation used for SA contains 11 categories ranging from exceptional to very poor. AraSenti-Tweet is another SA corpus, containing 17,573 tweets written mainly in the Saudi dialect. The corpus was annotated with four sentiment labels: positive, negative, neutral and mixed (Al-Twairesh et al. Reference Al-Twairesh, Al-Khalifa, Al-Salman and Al-Ohali2017). MIKA (Ibrahim, Abdou, and Gheith Reference Ibrahim, Abdou and Gheith2015) is a multi-genre SA corpus manually annotated (positive, negative or neutral) at sentence level for MSA and the Egyptian dialect. MIKA contains 4000 reviews collected from Twitter, product reviews, hotel reviews and TV programs reviews.

Hamdi, Shaban, and Zainal (Reference Hamdi, Shaban and Zainal2018) developed a multifaceted corpus, containing 15,274 reviews from various sources such as surveys, Facebook comments and Twitter posts related to governmental services. The different facets of the corpus are domains, dialects, linguistic issues and sentiment polarity for which the annotation was carried out manually by trained annotators from different Arab countries. Medhaffar et al. (Reference Medhaffar, Bougares, Estève and Hadrich-Belguith2017) created the Tunisian Sentiment Analysis Corpus (TSAC), consisting of 17K Facebook comments. The corpus was manually annotated with either positive or negative labels.

3.3 Machine translation

It is well known that machine translation systems require parallel corpora for source and target languages. With regard to Arabic dialects, little effort has been made to handle machine translation by enriching the field with necessary parallel corpora. This section illustrates examples of those attempts. For example, Zbib et al. (Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012) developed Egyptian-English and Levantine-English parallel corpora, consisting of 380k words and 1.1M words, respectively. The sentences of the corpus were extracted from web texts and translated using a crowdsourcing platform,Footnote ^c more specifically Amazon’s Mechanical Turk (MTurkFootnote ^d ). The data were utilised for building dialectal machine translation systems.

Additionally, Jehl, Hieber, and Riezler (Reference Jehl, Hieber and Riezler2012) created a parallel Twitter corpus using MTurk. That is, 1022 Arabic messages were collected randomly from Twitter, cleaned and then used in the MTurk task to be translated by workers into English. Their study covers three Arabic dialects: Egyptian, Gulf and Levantine, as well as MSA. Meftouh et al. (Reference Meftouh, Harrat, Jamoussi, Abbas and Smaili2015) created PADIC, a parallel Arabic dialect corpus, covering five dialects: Annaba’s dialect (spoken in the east of Algeria), Algiers’s dialect (spoken in the capital of Algeria), Sfax’s dialect (spoken in the south of Tunisia), Syrian and Palestinian dialects, as well as MSA. The authors examined the PADIC parallel corpus by conducting cross-lingual machine translation between all language pairs.

3.4 Other tasks

Many studies in the literature have developed annotated dialectal Arabic resources amidst work on domains whose contents are written mainly in Arabic dialects in order to tackle various NLP problems. Examples of such tasks include sentence boundary detection (Diab et al. Reference Diab, Habash, Rambow, Altantawy and Benajiba2010; Zribi et al. Reference Zribi, Ellouze, Belguith and Blache2015, Reference Zribi, Kammoun, Ellouze, Belguith and Blache2016), orthography (Habash et al. Reference Habash, Diab and Rambow2012a; Saadane and Habash Reference Saadane and Habash2015; Shoufan and Alameri Reference Shoufan and Alameri2015; Turki et al. Reference Turki, Adel, Daouda and Regragui2016; Khalifa et al. Reference Khalifa, Habash, Abdulrahim and Hassan2016; Jarrar et al. Reference Jarrar, Habash, Alrimawi, Akra and Zalmout2017) and automatic diacritics restoration (Harrat et al. Reference Harrat, Abbas, Meftouh and Smaili2013).

4.1 Sources of the collected data

Dialectal Arabic varieties are not typically written, but spoken. Consequently, the main sources from which dialectal Arabic can usually be derived are daily conversations, radio broadcasts, podcasts, movies and TV shows. Therefore, many studies in ADI research relied on transcripts of these audio sources, as well as telephone conversations and transcribed speech, in order to build annotated dialectal Arabic corpora (Gadalla et al. Reference Gadalla, Kilany, Arram, Yacoub, El-Habashi, Shalaby, Karins, Rowson, Mac-Intyre, Kingsbury, Graff and McLemore1997; McNeil and Faiza Reference McNeil and Faiza2011; Meftouh et al. Reference Meftouh, Harrat, Jamoussi, Abbas and Smaili2015; Malmasi et al. Reference Malmasi, Zampieri, Ljubešić, Nakov, Ali and Tiedemann2016).

Regarding their written form, Arabic dialects have an increased presence on the web, as social media allows users to engage in more informal conversations (Shao Reference Shao2009). Fortunately, excellent content is now available on the web, either in ‘pure’ dialectal Arabic, or in a middle language which could be considered a ‘conversational’ register of MSA or a ‘high’ register of dialects. Many studies exploit the Arabic web text as a source to collect dialectal material (Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012; Almeman and Lee Reference Almeman and Lee2013; Bouamor, Habash, and Oflazer Reference Bouamor, Habash and Oflazer2014).

Blogs and online discussion forums have also proved to be a productive source of dialectal data. The blog entries and forum postings usually represent important everyday topics such as sports, technology and lifestyle. Therefore, they are written in a very informal, conversational style. Many studies examined the contents of forums and blogs when building annotated dialectal Arabic datasets (McNeil and Faiza Reference McNeil and Faiza2011; Elfardy and Diab Reference Elfardy and Diab2012; Sadat, Kazemi, and Farzindar Reference Sadat, Kazemi and Farzindar2014; Khalifa et al. Reference Khalifa, Habash, Abdulrahim and Hassan2016; Abu Kwaik et al. Reference Abu Kwaik, Saad, Chatzikyriakidis and Dobnik2018). In addition, the reader commentary from online Arabic newspapers were used to build corpora for Arabic dialects by Zaidan and Callison-Burch (Reference Zaidan and Callison-Burch2011, Reference Zaidan and Callison-Burch2014), and Cotterell and Callison-Burch (Reference Cotterell and Callison-Burch2014).

Social media platforms such as Twitter, YouTube and Facebook contain posts, messages and users’ comments that are written in Arabic dialects. A considerable number of researchers have considered social media to build the required linguistic resources for ADI. Therefore, Twitter has been exploited, especially in recent years, to build a number of annotated corpora for different sets of Arabic dialects (Cotterell and Callison-Burch Reference Cotterell and Callison-Burch2014; Mubarak and Darwish Reference Mubarak and Darwish2014; Alshutayri and Atwell Reference Alshutayri and Atwell2017; Alsarsour et al. Reference Alsarsour, Mohamed, Suwaileh and Elsayed2018; Mubarak Reference Mubarak2018; Zaghouani and Charfi Reference Zaghouani and Charfi2018; Abu Kwaik et al. Reference Abu Kwaik, Saad, Chatzikyriakidis and Dobnik2018; Abdul-Mageed, Alhuzali, and Elaraby Reference Abdul-Mageed, Alhuzali and Elaraby2018; Bouamor, Hassan, and Habash Reference Bouamor, Hassan and Habash2019). User comments on YouTube videos have also been utilised to build dialectal corpora for a number of Arabic dialects (Salama et al. Reference Salama, Bouamor, Mohit and Oflazer2014; Abidi, Menacer, and Smaili Reference Abidi, Menacer and Smaili2017). Facebook messages, user comments and posts have also been used to build the dialectal corpora (Younes and Souissi Reference Younes and Souissi2014; Huang Reference Huang2015; Younes, Achour, and Souissi Reference Younes, Achour and Souissi2015).

Traditional written sources including folk poetry, song lyrics, collections of proverbs, and scripts for movies, plays, and television shows serve as solid sources of written Arabic dialects for building corpora (McNeil and Faiza Reference McNeil and Faiza2011). Likewise, SMS messages are typically written in an informal and conversational style, qualifying them as valuable sources for building dialectal Arabic datasets, as in the study of Younes and Souissi (Reference Younes and Souissi2014).

The translation of available existing sources from one language and/or specific dialect to another is another way to build resources for dialects. For example, the Multidialectal Parallel Corpus of Arabic (MPCA) (Bouamor et al. Reference Bouamor, Habash and Oflazer2014) contains 2000 parallel sentences, covering 5 dialects: Egyptian, Tunisian, Jordanian, Palestinian and Syrian Arabic, as well as MSA. The corpus was created by translating sentences from the Egyptian part of the Egyptian-English corpus built by (Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012). They asked four translators (native speakers of Jordanian, Palestinian, Syrian and Tunisian) to translate 2000 sentences written in Egyptian into their dialects. A fifth translator also translated the sentences into MSA. The contents of the corpus were harvested from the web, largely from weblog and online user groups. The MADAR Arabic dialect parallel corpora were created by translating English and French sentences from the Basic Travelling Expression Corpus (BTEC) (Takezawa et al. Reference Takezawa, Kikui, Mizushima and Sumita2007) into multiple Arabic dialects (Bouamor et al. Reference Bouamor, Habash, Salameh, Zaghouani, Rambow, Abdulrahim, Obeid, Khalifa, Eryani, Erdmann and Oflazer2018).

4.2 Methods of collecting and annotating data

The methods implemented to collect the dialectal data depend primarily on the sources from which the data are collected. Web sources, such as blogs and discussion forums, are essential when it comes to dialectal written texts. Many studies collected dialectal data on the web by manually navigating the web and downloading dialectal written materials or by performing web crawls of specific web pages or for distinctive dialectal keywords (McNeil and Faiza Reference McNeil and Faiza2011; Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2011; Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012; Almeman and Lee Reference Almeman and Lee2013; Cotterell and Callison-Burch Reference Cotterell and Callison-Burch2014; Khalifa et al. Reference Khalifa, Habash, Abdulrahim and Hassan2016). The various Application Programming Interfaces (APIs) that provide access to social media sites were also employed to collect Arabic dialectal contents from these social media sites such as Twitter, Facebook and Youtube (Mubarak and Darwish Reference Mubarak and Darwish2014; Salama et al. Reference Salama, Bouamor, Mohit and Oflazer2014; Younes et al. Reference Younes, Achour and Souissi2015; Huang Reference Huang2015; Abidi et al. Reference Abidi, Menacer and Smaili2017; Alshutayri and Atwell Reference Alshutayri and Atwell2017; Alsarsour et al. Reference Alsarsour, Mohamed, Suwaileh and Elsayed2018; Zaghouani and Charfi Reference Zaghouani and Charfi2018; Salameh, Bouamor, and Habash Reference Salameh, Bouamor and Habash2018).

Some sources are unavailable in written form, such as recordings of daily conversations, movies and TV shows. Some studies employed native speakers who helped either to transcribe the recordings and audio resources to make them available in written form (McNeil and Faiza Reference McNeil and Faiza2011; Meftouh et al. Reference Meftouh, Harrat, Jamoussi, Abbas and Smaili2015) or to manually translate existing corpora of other languages or dialects into the target dialects in order to create parallel corpora (Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012; Bouamor et al. Reference Bouamor, Habash and Oflazer2014; Meftouh et al. Reference Meftouh, Harrat, Jamoussi, Abbas and Smaili2015; Mubarak Reference Mubarak2018).

After collecting the data, the annotation task required to classify the collected data into a set of Arabic dialects was carried out in various ways. Most studies employed workers on online crowdsourcing platforms (e.g., MTurk and CrowdFlower) to perform annotations (Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2011; Cotterell and Callison-Burch Reference Cotterell and Callison-Burch2014; Alsarsour et al. Reference Alsarsour, Mohamed, Suwaileh and Elsayed2018) or to translate existing specific dialectal corpora into other Arabic dialects and/or different languages, such as English and French, in order to create parallel corpora of different Arabic dialects and languages (Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012; Bouamor et al. Reference Bouamor, Habash and Oflazer2014; Mubarak Reference Mubarak2018).

The semi-supervised learning methods such as co-training and self-training were examined to build dialectal Arabic corpora from Facebook posts (Huang Reference Huang2015). Self-training involved applying a supervised classifier to the unlabelled data to obtain automatically annotated dialect data that can be used to improve the performance of the classifier. The supervised classifier used in Huang’s study was trained on a previously manually annotated dialectal Arabic dataset. The built classifier then was used to automatically annotate Arabic Facebook posts containing 646M words. Co-training involved using two classifiers to annotate the same unlabelled data. A sentence was labelled and used to train the model only when the two classifiers agreed. In the study of Huang (Reference Huang2015), 476M words from Arabic Facebook posts were annotated using the co-training method. Huang (Reference Huang2015) also automatically created a dialectal corpus, relying only on users’ country information in their profiles on Facebook. As a result, the 66M word corpus is not highly accurate since the users’ country information may be missing or unreliable. In many studies, the geographical location provided in user profiles on social media sites, such as Twitter, YouTube and Facebook, were employed to classify the collected data (e.g., posts, comments, tweets and messages) from these sites (Salama et al. Reference Salama, Bouamor, Mohit and Oflazer2014; Mubarak and Darwish Reference Mubarak and Darwish2014; Younes et al. Reference Younes, Achour and Souissi2015; Alshutayri and Atwell Reference Alshutayri and Atwell2017; Salameh et al. Reference Salameh, Bouamor and Habash2018; Abu Kwaik et al. Reference Abu Kwaik, Saad, Chatzikyriakidis and Dobnik2018).

On the other hand, some studies classified collected data into a set of dialects using a degree of supervision in the form of lists of distinctive regional keywords and/or phrases. These words become ‘seed words’ to be used in the process of collecting and annotating the data for each dialect. For example, Almeman and Lee (Reference Almeman and Lee2013) used 1043 dialect words (430 from Gulf dialects, 200 from North African dialects, 274 from Levantine dialects and 139 from Egyptian dialects) as keywords to automatically collect links for each dialect from the web using Bing API. The web pages for each dialect were downloaded and saved as html files. Finally, they cleared noise from the files (e.g., removing unwanted symbols, tags and underscores) and normalised the cleared texts (e.g., removing repeated words and phrases) to obtain a corpus for each dialect.

Many studies also adopted the traditional annotation by employing a group of annotators to perform manual annotation according to given annotation guidelines (Khalifa et al. Reference Khalifa, Habash, Abdulrahim and Hassan2016; Zaghouani and Charfi Reference Zaghouani and Charfi2018).

4.3 Covered dialects

Spoken Arabic is most commonly broken down coarsely into five main dialect groups: Levantine, Gulf, Egyptian, Maghrebi and Iraqi (Habash Reference Habash2010). A large portion of the research on dialectal Arabic resources has been devoted to the creation of annotated corpora for the five aforementioned dialect groups (Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2011; Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012; Almeman and Lee Reference Almeman and Lee2013; Cotterell and Callison-Burch Reference Cotterell and Callison-Burch2014; Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2014; Salama et al. Reference Salama, Bouamor, Mohit and Oflazer2014; Huang Reference Huang2015; Mubarak Reference Mubarak2018). Since further division of the dialect groups is possible, especially in large regions, many country-level Arabic dialects have been addressed in the ADI field. For example, the four sub-dialects (Syrian, Lebanese, Palestinian and Jordanian) of spoken Levantine have received attention in the study of Abu Kwaik et al. (Reference Abu Kwaik, Saad, Chatzikyriakidis and Dobnik2018) which constructed the Shami corpus (SDC), including data from the four sub-dialects spoken in the Levant region. Furthermore, the Egyptian dialect has been studied extensively in a considerable number of works (Zaidan and Callison-Burch Reference Zaidan and Callison-Burch2011; Zbib et al. Reference Zbib, Malchiodi, Devlin, Stallard, Matsoukas, Schwartz, Makhoul, Zaidan and Callison-Burch2012; Almeman and Lee Reference Almeman and Lee2013; Salama et al. Reference Salama, Bouamor, Mohit and Oflazer2014; Mubarak and Darwish Reference Mubarak and Darwish2014; Bouamor et al. Reference Bouamor, Habash and Oflazer2014; Diab et al. Reference Diab, Al-Badrashiny, Aminian, Attia, Elfardy, Habash, Hawwari, Salloum, Dasigi and Eskander2014; Mubarak Reference Mubarak2018), as has been the Tunisian, a sub-dialect of Maghrebi (McNeil and Faiza Reference McNeil and Faiza2011; Boujelbane et al. Reference Boujelbane, Khemekhem, BenAyed and Belguith2013; Bouamor et al. Reference Bouamor, Habash and Oflazer2014; Younes and Souissi Reference Younes and Souissi2014; Bouchlaghem, Elkhlifi, and Faiz Reference Bouchlaghem, Elkhlifi and Faiz2014; Younes et al. Reference Younes, Achour and Souissi2015). More specifically, the dialect of Sfax city in the southeast of Tunis has been the focus of a number of studies (Meftouh et al. Reference Meftouh, Harrat, Jamoussi, Abbas and Smaili2015; Bouamor et al. Reference Bouamor, Habash, Salameh, Zaghouani, Rambow, Abdulrahim, Obeid, Khalifa, Eryani, Erdmann and Oflazer2018). Spoken Algerian belonging to the Maghrebi group of dialects has been covered in selected studies, such as (Abidi et al. Reference Abidi, Menacer and Smaili2017), while others specifically addressed the Algerian city dialects of Algiers and Annaba (Meftouh et al. Reference Meftouh, Harrat, Jamoussi, Abbas and Smaili2015; Bouamor et al. Reference Bouamor, Habash, Salameh, Zaghouani, Rambow, Abdulrahim, Obeid, Khalifa, Eryani, Erdmann and Oflazer2018). In addition, the regional dialects of Morocco, Libya, Sudan and Yemen have been handled in a more recent study by Zaghouani and Charfi (Reference Zaghouani and Charfi2018). Meanwhile, dialects of Saudi Arabia, United Arab Emirates, Kuwait, Oman, Qatar and Bahrain have been covered in a corpus created specifically for spoken dialects of the Arabian Peninsula (Khalifa et al. Reference Khalifa, Habash, Abdulrahim and Hassan2016). Another study carried out by Bouamor et al. (Reference Bouamor, Habash, Salameh, Zaghouani, Rambow, Abdulrahim, Obeid, Khalifa, Eryani, Erdmann and Oflazer2018) has established a corpus that encompasses a wide range of fine-grained dialects of 25 cities including Rabat, Fes, Algiers, Tunis, Sfax, Tripoli, Benghazi, Cairo, Alexandria, Aswan, Khartoum, Jerusalem, Amman, Salt, Beirut, Damascus, Aleppo, Mosul, Baghdad, Basra, Doha, Muscat, Riyadh, Jeddah and Sana’a. A recent paper by Abdul-Mageed et al. (Reference Abdul-Mageed, Zhang, Bouamor and Habash2020) has presented the findings of the First Nuanced Arabic Dialect Identification Shared Task (NADI). The corpus for the shared task covers a total of 100 provinces from 21 Arab countries and is collected from the Twitter domain, containing 30,957 tweets (Aloraini, Poesio, and Alhelbawy Reference Aloraini, Poesio and Alhelbawy2020). Despite the availability of the aforementioned resources, a great need for resources remains for a considerable number of fine-grained Arabic dialects and for more examples from different topics, domains and genres.

Althobaiti (Reference Althobaiti2020a, Reference Althobaiti2020b) has presented a recent comprehensive survey of the ADI task including available ADI annotated corpora. It is obvious that several studies have begun to address the shortage of annotated dialectal Arabic data by way of manual labelling, which costs effort and time, while others have utilised seed words to start collecting and annotating the data in a semi-automatic way. Indeed, these methods of annotation require previous knowledge of the target dialect. In our paper, we propose a novel approach that does not require deep previous knowledge of any Arabic dialect.

5.1 General framework

Our approach to automatically build an annotated dialectal Arabic corpus and word lists depends primarily on the iterations of two main components: automatic creation of lists for dialectal words and automatic creation of annotated ADI corpus as illustrated in the general framework shown in Figure 1.

Figure 1.

General framework used to automatically build an annotated corpus and word lists for Arabic dialects.

The first component, automatic creation of dialectal word lists, aims to provide a list of distinctive keywords for each dialect with no need for previous knowledge about that dialect. The distinctive word/keyword in the context of ADI is defined as the word used primarily in one Arabic variety, and so distinguishing it from others. For example, the distinctive Egyptian word (, çAwz, ‘I want’) sets the Egyptian dialect apart from others. Distinctive dialectal words/keywords can be defined as the words used in one or more Arabic dialects but never used in MSA, thereby distinguishing Arabic dialects from MSA. Our algorithm starts with two corpora: the first one contains MSA sentences, which we call “MSA corpus” and the second contains mixed texts of MSA and different dialects (i.e., sentences written in various Arabic varieties such as MSA, Gulf dialects, Levantine dialects, Egyptian dialects, …etc). The second corpus can be collected easily and randomly from the web and social media sites, since their contents are informally generated by users and are currently considered the active resources providing dialectal Arabic texts. We call the second corpus ‘mixed corpus’. Given these two corpora, as well as Arabic WordNet, the first component in our framework extracts only distinctive dialectal keywords and classifies them into their corresponding dialects based on some text processing steps and weighting factors that will be explained in detail shortly.

The second component, automatic creation of an annotated ADI corpus, selects the top distinctive dialectal words based on weighting factors and uses them as seed words for collecting dialectal data. The iteration between these two components ensures the diversity of the collected dialectal corpus and its coverage of the maximum amount of vocabulary used in different topics and genres, while maintaining accuracy as much as possible.

6.1 Dialectal words extraction

This module is used to automatically extract a list of dialectal words from mixed texts (e.g., data collected randomly from web texts, Facebook posts, Tweets and forums discussions). It starts with a set of three pre-processing steps: (a) text cleaning, (b) normalisation and (c) tokenisation. Text cleaning indicates the process of converting raw texts found on the web and social media sites into a form usable by NLP algorithms by removing diacritical marks, links, mentions from tweets, excessive spaces, emoticons, emojis, words less than two characters and stretched characters. Secondly, normalisation indicates handling the inconsistent variations when typing some Arabic letters. For example, different forms of alif may be written interchangeably. Another example is taa’ marbutah and haa’ which are usually used interchangeably in the final position of the word. The alif maqsurah and regular dotted yaa’ are also usually used interchangeably. Normalisation reduces noise and data sparsity by normalising different forms of alif into a bare alif (alif without hamzah), normalising taa’ marbutah to haa’ and normalising the dotless yaa’ (alif maqsurah) to yaa’. Tokenisation indicates simple tokenisation, which only splits off punctuation and non-alphanumeric characters from words.

Despite the fact that Arabic dialectal vocabulary sets differ from MSA and from each other, MSA words account for a considerable number of words in any Arabic dialect. For example, Harrat et al. (Reference Harrat, Meftouh, Abbas, Jamoussi, Saad and Smaili2015) analytically studied PADIC, a parallel corpus that covers five Arabic dialects, as well as MSA, and found that 51.68% of Palestinian dialect is actually MSA words, and the percentage of common words between the Syrian dialect and MSA is 37.36%. Therefore, the distinctive dialectal Arabic words (i.e., words used in one or more Arabic dialects but never used in MSA) can simply be extracted from any mixed texts by comparing words that exist in the mixed corpus with those in the MSA corpus. The words that are not found in the MSA corpus can be initially considered distinctive dialectal words. We selected two MSA resources for the purpose of comparing MSA contents to mixed contents. The first MSA resource is Arabic WordNet and the second is the MSA Wikipedia corpus. Therefore, after pre-processing the mixed texts and tokenising them, each word permeates through four filters to determine whether it is a dialectal Arabic word or MSA word, as shown in Figure 2. The comparison of MSA texts and mixed texts is conducted in the WordNet-based filter and Wikipedia-based filter. Normalisation is conducted in most ‘filters’, as illustrated in Figure 2, to help search for words in textual resources and extract sentences that contain them.

Figure 2.

Algorithm of automatic extraction of dialectal Arabic words from texts.

In the WordNet-based filter, we utilised Arabic WordNet, a lexical resource for standard Arabic that contains synsets including verbs, nouns, adjectives, adverbs and named entities (NEs) (Abouenour, Bouzoubaa, and Rosso Reference Abouenour, Bouzoubaa and Rosso2013; Black et al. Reference Black, Elkateb, Rodriguez, Alkhalifa, Vossen, Pease and Fellbaum2006). Since Arabic WordNet is in lemma space and most words will not match, we employed a light stemmer to reduce inflected words to their lemmas. We utilised ‘Light Stemmer 10’ from AraNLP libraryFootnote ^e (Althobaiti, Kruschwitz, and Poesio Reference Althobaiti, Kruschwitz and Poesio2014a) with a slight modification. Light Stemmer 10 finds the word’s lemma by following many steps: (a) removing the conjunction () if the remainder of the word is three or more characters long, (b) removing any of the definite articles () if this leaves two or more characters and (c) removing any of the suffixes () that are found at the end of the word if this leaves two or more characters. We modified Light Stemmer 10 by adding a fourth step: removing any of the prefixes (), if the remainder of the word is three or more characters long. The word will not be included in the list of dialectal Arabic words, once its lemma is found in WordNet. Even common words (in their inflected forms), which are used in both dialectal Arabic and MSA, disqualify as candidates for the list of dialectal Arabic words, once their lemmas are found in Arabic WordNet. This filter helps to exclude many MSA words, including common words used in both MSA and dialectal Arabic.

The contents of Arabic WordNet are manually verified, ensuring it is a reliable resource, but with limited size. The limited size of Arabic WordNet, especially in terms of adjectives and adverbs, led us to the second filter in which we used Arabic Wikipedia articles as an MSA corpus. That is, each word in mixed texts that can be found in Arabic WordNet are rejected for being MSA words, while those words that cannot be found in Arabic WordNet are further examined to confirm they are not MSA words by searching for their existence in another MSA resource (i.e., Arabic Wikipedia corpus). We prefer Arabic Wikipedia because it is an MSA corpus that is an open domain and represents different topics/genres. The words that cannot be found in the Arabic Wikipedia corpus are automatically included in the list of dialectal Arabic words. Table 2 shows examples of dialectal Arabic words that could not be found in either Arabic WordNet or the Arabic Wikipedia corpus, and therefore, were automatically added to the list of dialectal Arabic words.

Table 2.

Examples of Arabic words included in the dialectal Arabic lists after passing the WordNet and Wikipedia-based filters

The comparison of the MSA corpus to mixed texts is usually made in order to eliminate words shared between MSA and Arabic dialects, so as to extract only dialectal words that do not exist in MSA texts. Although this simple manner of distinguishing MSA words from dialectal ones in mixed texts is a straightforward intuitive solution, many available corpora for MSA contain dialectal words. According to Mubarak and Darwish (Reference Darwish2014), the commonly used MSA GigaWord corpus contains many dialectal words. They also found in their investigation that Aljazeera news articles contain many dialectal words, because these articles often contain quotes by people speaking in their own Arabic dialect. Aljazeera news articles also contain transcripts of interviews in which the interviewees occasionally use dialects. The corpus of Arabic Wikipedia articles is no exception, as it also contains some dialectal Arabic words within MSA sentences. Therefore, not every word found in the MSA Wikipedia articles is truly an MSA word.

The existence of dialectal Arabic words in MSA Wikipedia articles can be attributed to many factors. Firstly, some words are mistaken for MSA words, because their intended meaning is lost when taken out of context. For example, the dialectal Arabic word (, mtAç), which is a particle indicating ownership or possession in Tunisian dialects, also functions as a noun meaning ’belongings’ in MSA, taking into account that the word is written without diacritical marks. Hence, the meaning of the word and the Arabic variety to which the word belongs cannot be predicted when it is out of context and written without diacritical marks, the usual case of written Arabic texts today. Secondly, some Wikipedia articles contain written dialogue and quoted speech in which the speakers/interlocutors and quoted people converse in Arabic dialects. Table 3 shows some examples from Arabic Wikipedia where the dialectal Arabic words appear within dialectal Arabic dialogue or quoted speech included in the MSA texts. In addition, we found that some dialectal Arabic words appear in Wikipedia articles as proper names (i.e., NEs). Table 4 shows examples of the dialectal Arabic words functioning as NEs in Arabic Wikipedia. Therefore, the existence of a word in MSA Wikipedia articles cannot be used as an efficient indicator that it is an MSA word.

Table 3.

Examples of dialectal Arabic contents in the Arabic Wikipedia corpus

Table 4.

Examples of dialectal Arabic words used as NEs in the Arabic Wikipedia corpus

The next two filters, MSA POS tagging and MSA Named Entity Recognition, deepen the examination of words from mixed texts that are found in MSA Wikipedia articles by determining whether the words are dialectal or not by using MSA POS tagging and MSA Named Entity Recognition. Firstly, we reimplemented the MSA NE recogniser in the study of Althobaiti, Kruschwitz, and Poesio (Reference Althobaiti, Kruschwitz and Poesio2014b) by adopting the same features which include (a) the words immediately before and after the current word in their raw and light-stemmed forms, (b) the first 1, 2, 3 and 4 characters in a word, (c) the last 1, 2, 3 and 4 characters in a word, (d) the appearance of the word in the gazetteer, and (e) the light-stemmed form of the word. The gazetteer employed contains around 5000 entries and was built by Benajiba, Diab, and Rosso (Reference Benajiba, Diab and Rosso2008). To find the light-stemmed form of the word, we removed the same affixes removed by Althobaiti et al. (Reference Althobaiti, Kruschwitz and Poesio2014b), namely conjunctions, prepositions and definite articles. Conditional random fields (CRFs)Footnote ^f were utilised to train the NE recogniser on an NE-labelled Wikipedia corpus called (WDC), which was made freely available by the aforementioned study. The use of the built NE recogniser allowed us to exclude many detected NEs in the mixed texts from the list of dialectal Arabic words. However, some real dialectal words, such as those illustrated in Table 4, have been mistakenly prevented from being added to the dialectal Arabic word list. The purpose of building a list of dialectal words is to classify its entries to create a list of distinctive words for each Arabic dialect and then use them as seed words to build corpora for Arabic dialects. Therefore, the precision of entries on the dialectal Arabic words list is more important in our study than the coverage of them. Thus, any word that is recognised in the MSA Wikipedia corpus to be a proper name is rejected from the list of dialectal Arabic words.

Regarding the dialectal Arabic words of the spoken dialectal Arabic dialogues that appear in written form within MSA sentences of Wikipedia articles as oulined in Table 3, we took into account the contextual information of the text by using MSA POS tagging. It is well known in the field of Arabic NLP applications that tools created specifically for MSA have a high likelihood of yielding poor results when applied on texts written in Arabic dialects (Duh and Kirchhoff Reference Duh and Kirchhoff2005; Habash et al. Reference Habash, Roth, Rambow, Eskander and Tomeh2013; Elgabou and Kazakov Reference Elgabou and Kazakov2017). We aim in our proposed method to exploit the poor performance of the MSA POS tagger on dialectal texts for the sake of extracting dialectal words. Hence, we investigated the behaviour of the MSA POS tagger on dialectal Arabic sentences. We found that dialectal words are often tagged as NNP by the MSA POS tagger, although they are not proper names and are not considered NEs by the NE recogniser. Further, in case, it is assigned more than one word class (POS tags) by the MSA POS tagger, then the dominant POS tag will be ‘NNP’. Also, we determined that if the dialectal word is not tagged as NNP by the MSA POS tagger, then the MSA POS tagger behaves inconsistently when it runs on a set of sentences that contain a dialectal Arabic word. The inconsistent behaviour is revealed when the MSA POS tagger assigns more than two possible POS tags to the dialectal word without an obvious dominant (most frequent) POS tag; this is unusual behaviour when applying the MSA POS tagger on the MSA contents. That is, most MSA words are tagged with one word class by the MSA POS tagger, and if it is tagged with more than one word class according to its role (position) in the sentence, then there will be a dominant (most frequent) POS, and the relative frequency of that dominant POS is usually greater than about 65%.

According to the investigation of the behaviour of the MSA POS tagger on dialectal Arabic contents, we set up a filter in our algorithm to extract dialectal words and distinguish them from MSA words as shown in Figure 2. Therefore, for each word from mixed text that is found in the MSA Wikipedia corpus and not considered an NE in the previous filter, we extracted the sentences in which that word appears from both MSA and mixed corpora, and then we applied an MSA POS tagger. The algorithm analyses and counts the possible word classes (POS tags) assigned to the word in all various sentences. Then, it applies the rules set in the ‘MSA POS tagger filter’ of our algorithm, which includes the following: (a) having more than one word class (POS tag) assigned to a word by the MSA POS tagger with a relative frequency of the dominant POS tag less than 65%, or (b) if the dominant POS tag assigned to the word is ‘NNP’, then this word has a high probability of being a dialectal Arabic word and therefore is regarded as a dialectal word. Details about the MSA POs tagger used and the investigation we have conducted to examine the behaviour of the MSA POS tagger on dialectal Arabic texts can be found in Appendix A.

6.2 Dialectal words classification

In this section, we explain the technique used to classify the list of dialectal Arabic words, created in Section 6.1, into 15 Arabic dialects at the country level. Table 5 shows the Arabic dialects covered in our study in order from eastern to western dialects. We used two factors in the classification process, the first of which is the geographical frequency factor that classifies the dialectal words in the list based on their appearances on Twitter. That is, dialectal words that appear in tweets generated by users living in Algeria, based on longitude and latitude of the tweets’ location, will be automatically classified as Algerian words and will be added to an Algerian word list. If the dialectal word appears in tweets generated by users in two different countries representing two different dialects (e.g., Tunis & Algeria), then the dialectal words should be added to the lists of both dialects. This is necessary because Arabic dialects share many common words. For example, the dialect word (, šlwn, ‘how’) is used in Syrian, Iraqi and many Gulf dialects. However, people who live outside of their native country may set their location information on Twitter to indicate their current location. In addition, the geographical features of their tweets’ location (i.e., longitude and latitude information) may also indicate the users’ current location and not their native country. Therefore, words of their native dialect appear in their tweets and consequently, the use of geographical Twitter information as the only factor to classify any dialectal word into its dialectal category is insufficient. That is, some dialectal words appear in the tweets generated by expatriates representing their native dialects, even though the geographic information of the tweets represents the current country of residence. For example, the Yemeni distinctive word (, Ašty, ‘I want’) appears in tweets of Yemeni Twitter (based on the longitude and latitude of the tweets’ location) but also appears in tweets generated by expatriates in Gulf countries. Hence, we cannot consider this dialectal word a common one between Gulf and Yemeni dialects. Native dialects tweeted from outside of their typical range cause words to be mistakenly added to the false words list. For this reason, we need a measurement that measures the significance value of each dialectal word to other dialectal Arabic categories. Therefore, our algorithm of classification uses the pointwise mutual information (PMI) measure as the second factor when the same dialectal word appears in tweets of different countries. Strong association between the dialectal word and the two dialectal categories indicates that the word is a common one between these dialectal Arabic categories and can be added to the word lists of both of them. Otherwise, the word will be added only to the dialectal Arabic category to which it has a strong association, according to the PMI score.

Table 5.

Arabic dialects covered in our study

6.2.2 PMI factor

PMI is a statistical measure of association strength between two events used in information theory (Manning and Schütze Reference Manning and Schütze1999; Su et al. Reference Su, Xiang, Wang, Sun and Yu2006; Isola et al. Reference Isola, Zoran, Krishnan and Adelson2014). It has proved to be an efficient association measure in a considerable number of NLP applications (Van de Cruys Reference Van de Cruys2011). Mathematically, the PMI of a pair of outcomes x and y belonging to discrete random variables X and Y measures the discrepancy between the probability of their coincidence given their individual distributions and joint distribution as follows:

\begin{equation*} pmi(x,y)=\log{\frac{p(x,y)}{p(x)*p(y)}} = \log{\frac{p(x|y)}{p(x)}} = \log{\frac{p(y|x)}{p(y)}}\end{equation*}

In our study, we utilised PMI in order to evaluate how important a word is to a certain Arabic dialect by measuring the association between the word and that Arabic dialect. Mathematically, it is the log of the probability of the word occurring in a certain Arabic dialect divided by the product of the probability of the word and the probability of that Arabic dialect. Regarding the dataset on which we depend to calculate the probabilities, we used the initial data of tweets collected from Arabic Twitter, based on the list of dialectal Arabic words, and classified based on geographic information, as described in the previous section. So, let $pmi(w;<dialect\_category>_{Tweets})$ be the PMI between a word w and tweets of a certain dialect (e.g., $Egyptain_{Tweets}$ , $Saudi_{Tweets}$ ) as follows:

\begin{equation*} pmi(w;<dialect\_category>_{Tweets})=\log{\frac{p(w,<dialect\_category>_{Tweets})}{p(w)*p(<dialect\_category>_{Tweets})}}\end{equation*}

where the probabilities p(w) can be computed by counting the occurrences of the word w in the dataset, while $p(<dialect\_category>_{Tweets})$ is the number of the tweets written in a certain category of Arabic dialect (e.g., Egyptian (EG), Saudi (SA), Moroccan (MA), Tunisian (TN), Iraqi (IQ), …etc) in the dataset. The $p(w,<dialect\_category>_{Tweets})$ represents the co-occurrences of the word w and dialectal tweets $<dialect\_category>_{Tweets}$ in the dataset. The words with the highest PMI scores are more likely to appear in tweets of that specific category of Arabic dialect. That is, using PMI facilitates the ranking of the dialectal Arabic list according to their association’s degree to each Arabic dialect.

Furthermore, we ascertained that a distinctive word used primarily in a certain dialect yields a high PMI scores when computed with that particular dialect. For example, the Syrian distinctive word (, bnwb, ‘never’) has a relatively high PMI when it is computed for the Syrian dialect, while the PMI of the word with other Arabic dialects, such as Egyptian or Gulf, is very low. Table 6 shows the PMIs for some dialectal Arabic words and the Egyptian dialect.

Table 6.

Examples of dialectal Arabic words and their PMI values for the Egyptian dialect

We can see that the Egyptian words have high and positive PMI scores, indicating the strong association between these words and the Egyptian dialect. The dialectal Arabic words of other categories such as Syrian, Tunisian and Iraqi words show negative PMI scores, which reflect the lack of relatedness between these words and the Egyptian dialect. To confirm the quality of the word lists of Arabic dialects, we decided to remove all dialectal Arabic words from a dialect’s list, if the PMI scores for these words were negative. So, in the case of the Egyptian dialect, as seen in Table 6, words with negative PMIs that appear in its list will be removed. In fact, we realise that all of the words with negative PMIs are not from the Egyptian dialect. For example, (, mnyH, ‘good’) is a Syrian word that was tweeted from a location in Egypt. Words shared between Arabic dialects have relatively low PMI scores but never yield a negative score. For example, the dialectal Arabic word (, mAfyš, ‘there is no’) is used in Egypt, Libya and Yemen. Therefore, the PMI of the word when computed for the Egyptian dialect $pmi(w, EG_{Tweets})$ is low, as seen in Table 6. We decided to set a threshold for PMI to help specify the extent of relatedness for dialectal words. Setting a high minimum PMI value limits each dialect’s list of words only to distinctive keywords of that dialect, while also avoiding common words shared between one specific dialect and others as much as possible. The low minimum PMI value, on the other hand, would represent the opposite.

8.1 Characteristics of the dialectal Arabic corpus

This section presents some features of our automatically annotated corpus of dialectal Arabic ‘Twt15DA’. The corpus consists of 311,785 tweets containing 3,858,459 words in total, each of which is labelled as 1 category from the 15 Arabic dialect categories. Table 7 shows the distributions of each dialect in the corpus in terms of the number of tweets and the number of words.

Table 7.

Distribution of Arabic dialects in Twt15DA corpus

8.1.1 MSA dialect overlap

The analysis of the contents of the Twt15DA corpus shows that it contains many MSA words. These common words between MSA and dialectal Arabic make up 51% of the Twt15DA corpus. The Jordanian and Saudi dialects contain a large number of words in common with MSA, making up 53.44% and 53.08%, respectively, while the lowest percentage of common words is between the Moroccan dialect and MSA (40.77%). Table 8 shows overlap percentages between each Arabic dialect on the one hand and MSA on the other hand.

Table 8.

MSA dialect overlap percentages in the Twt15DA corpus

The words shared among MSA and dialectal Arabic can be divided into four types:

• Complete similarity (same written form, same pronunciation and same meaning): These words are written in the same way in MSA and dialectal Arabic. They also have the same meaning and they sound alike. For example, the word (, maç, ‘with’) exists in MSA and all Arabic dialects. It is also pronounced the same way in all Arabic varieties including MSA .

• Partial similarity (same written form, different pronunciations and same meaning): These words have the same written form and the same meaning. However, they have different vocalisation, leading to different pronunciation. For example, the word (, , ‘He lied’) in Arabic dialects and MSA have the same written form and the same meaning ‘He lied’. However, the word is pronounced differently. Hence, in some Arabic dialects, the word is pronounced , while in MSA it is pronounced as . Indeed, the pronunciation of the word is obvious with diacritical marks (e.g., in Arabic dialect while in MSA, ). As diacritical marks are rarely used when writing Arabic, especially in online contents such as social networking sites, the non-vocalised words (i.e., words without diacritical marks) cannot be used to determine the Arabic variety of the word.

• Complete homonyms (same written form, same pronunciation and different meanings): These words are written in the same way in MSA and Arabic dialects, sound alike, but have different meanings. For example, the word (, bAhy) is written the same in both MSA and the Libyan dialect and is pronounced the same. However, the word () in Libyan dialect means ‘yes/OK’ while in MSA, it means ‘bright/brilliant’.

• Partial homonyms (same written form, different pronunciations and different meanings): These words have the same written form, but different meanings and pronunciations. For example, the word (, mtAç) without diacritical marks means ‘belongings/goods’ in MSA, while in the Tunisian dialect this word is a particle that is used to express ownership or possession (e.g., (‘the book is mine’). In addition to different meanings of the word in MSA and the Tunisian dialect, the word is pronounced differently in the Tunisian dialect than in MSA.

8.1.2 Inter-dialect overlap

Common words are not found only between dialectal Arabic and MSA, but also among various Arabic dialects on different levels. Figure 3 shows the percentages of common words among Arabic dialects (i.e., the inter-dialect overlapping results for each pair of 15 Arabic dialects), taking into account these overlap percentages are in type (unique) space. That is, some Arabic varieties do not have online morphological analysers at all and other Arabic varieties have premade morphological analysers, but they are often neither available online nor freely available, as discussed in Section 3.1. Therefore, when matching the words of two Arabic varieties to identify which overlap, we did not analyse these words morphologically beforehand. That is, we kept clitics and affixes attached to the words. This must have affected the percentage of overlap between Arabic varieties, as the matching also included the clitics and affixes. However, the proportionality between the obtained results are logical and represents the reality. Clearly, the results in Figure 3 indicate that the less the geographical distance between the Arabic dialects, the greater the number of common words between them. Thus, Arabic dialects spoken in the Maghreb region share many common words among them but have less words in common with dialects spoken in the Middle East. For example, the percentage of overlapping words between the Tunisian and Algerian dialects (spoken in Tunis and Algeria, two adjacent countries in North Africa) is 19.50%, while the percentage between Tunisian dialect and Omani dialect (spoken in Arabian Peninsula) is 9.73%.

Figure 3.

Overlap percentages among dialects in the Twt15DA corpus.

The common words among various Arabic dialects can be one of four types: (a) complete similarity where two words from two different Arabic dialects have the same written form, the same pronunciation and the same meaning, (b) partial similarity that indicates the state of similar written form and meaning, but different pronunciations, (c) complete homonyms where two words are written in the same way in two Arabic dialects and sound alike but have different meanings, and (d) partial homonyms where the words shared between two Arabic dialects have the same written form, but different meanings and pronunciations. Table 9 presents some examples of different types of common words among Arabic dialects.

Table 9.

Types of common words among various Arabic dialects

8.1.3 Tweet and word sizes

In the Twt15DA corpus, tweets contain on average 12 words. The longest tweet in the corpus contains 46 words, while the shortest consists of 2 words. Additionally, the average word in the Twt15DA corpus contains four characters in tweets that include both MSA and dialectal Arabic. On the other hand, in dialectal Arabic lists, which contain only dialectal Arabic words, the average dialectal words consists of six characters. Detailed information regarding tweet size and word size for each Arabic dialect in the corpus is presented in Table 10.

Table 10.

The tweet size and word size for Arabic dialects in Twt15DA corpus