1. Introduction
Part-of-speech (POS) tagging is the task of automatically assigning syntactic category labels to tokens in text and is an important preprocessing step for higher order Natural Langauge Processing (NLP) tasks such as syntactic parsing (Jurafsky and Martin Reference Jurafsky and Martin2009). Most Arabic POS tagging work has focused on Modern Standard Arabic (MSA), which is used in formal communication, while work on POS tagging of dialectal Arabic (DA), which is ubiquitous on social media and informal communication, has lagged behind. DA’s informality leads to the prevalence of spelling variations (often creative), transliterated foreign words, and the use of social media artifacts such as emoticons and hashtags. Since Arabic speakers typically use dialects in their daily interactions, dialects became their natural choice in online conversations, replacing or complementing MSA. Arabic dialects are broadly classified into five major dialect groups, namely Egyptian, Levantine, Maghrebi, Iraqi, and Gulf. Each dialect group comprises a number of sub-dialects. For instance, the Maghrebi group covers Libyan, Moroccan, Tunisian, Algerian, and Mauritanian, creating a continuum rather than discrete dichotomous variations. Dialects often differ much in lexical choices and may exhibit syntactic and morphological differences. The larger and the more diverse the lexical, morphological, phonetic, and syntactic differences between dialects, the less mutually intelligible they become. Prior POS tagging work on such informal and dialectal social media text has been scant due in large part to the scarcity of annotated data. Available work is mostly uni-dialectal with Egyptian receiving the most attention (Duh and Kirchhoff Reference Duh and Kirchhoff2005; Habash et al. Reference Habash, Roth, Rambow, Eskander and Tomeh2013; Khalifa, Hassan, and Habash Reference Khalifa, Hassan and Habash2017).
This work introduces robust multi-dialectal POS tagging trained on a annotated data set of Arabic tweets in four major Arabic dialects: Egyptian (EGY), Levantine (LEV), Gulf (GLF), and Maghrebi (MGR) (Darwish et al. Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018). We employ two different approaches. The first uses conditional random fields (CRF), while the second combines word- and character-based representations in a deep neural network (DNN) architecture. We also exploit millions of unlabeled tweets to obtain word clusters and clitic-level embeddings, and we employ a variety of linguistic features, such as clitic metatypes and stem templates. We develop robust joint models that tag multi-dialectal tweets and outperform uni-dialectal models. We achieve an average word-level accuracy of 92.4% across all dialects. The success of the joint models hinges on effective automatic dialect identification that we develop using a DNN model, which is at par or better than the current state of the art, distinguishing between five dialects with 86.3% accuracy at tweet level. The contributions of the paper are as follows:
-
• We compare the use of linear CRF sequence labeling and DNN with a variety of features to leverage limited training data.
-
• We build joint POS tagging models that can effectively tag tweets in multiple dialects. In the process, we develop effective dialect identification that we embed within our models. We plan to publicly release our source code and models.
-
• We show that state-of-the-art results can be achieved for POS tagging using a small annotated data set.
2. Background
Some recent papers have focused on POS tagging of English social media text, particularly tweets. Gimpel et al. (Reference Gimpel, Schneider, O’Connor, Das, Mills, Eisenstein, Heilman, Yogatama, Flanigan and Smith2011) used a CRF-based sequence labeler in conjunction with a variety of features, such as word distributional similarity and phonetically normalized forms. In follow-on work, they improved their POS tagging accuracy from 89.3% to 92.2% by making use of Brown clusters (Brown et al. Reference Brown, Desouza, Mercer, Pietra and Lai1992), which are hierarchical clusters of words based on the contexts in which they appear. The usefulness of Brown clusters for POS tagging was also demonstrated in Owoputi et al. (Reference Owoputi, O’Connor, Dyer, Gimpel, Schneider and Smith2013) and Stratos and Collins (Reference Stratos and Collins2015). Similarly we show the effectiveness of using Brown clusters in the context of our CRF and DNN models. Derczynski et al. (Reference Derczynski, Ritter, Clark and Bontcheva2013) attempted to improve POS tagging of tweets by specifically targeting low-frequency words which are often misspellings or creatively spelled words.
Work on POS tagging of Arabic social media text is scant. The scarcity of dialectal resources has hampered research on DA. Few resources were made available by programs such as TIDES, GALE, and BOLT and distributed by LDC, which are not widely accessible due to their license requirements. As such, researchers used ad-hoc resources or small data sets that were curated locally and not widely accessible. Graja, Jaoua, and Hadrich Belguith (Reference Graja, Jaoua and Hadrich Belguith2010) created the Tunisian Dialect Corpus Interlocutor (TuDiCoI) with 893 utterances and 3404 words from dialectal conversations between Tunisian railway staff. Bouamor, Habash, and Oflazer (Reference Bouamor, Habash and Oflazer2014) used a collection of 2000 sentences in Egyptian dialect as a seed to build a multi-DA corpus. The seed sentences were translated by native speakers into their own dialects to create a parallel multi-dialectal corpus in addition to English. Cotterell and Callison-Burch (Reference Cotterell and Callison-Burch2014) extended the work of Al-Sabbagh and Girju (Reference Al-Sabbagh and Girju2010) and Zaidan and Callison-Burch (Reference Zaidan and Callison-Burch2011) to build a collection of commentaries from five Arabic newspapers and tweets that was used for automatic dialect identification. Duh and Kirchhoff (Reference Duh and Kirchhoff2005) used CallHome Egyptian Colloquial Arabic to build a POS tagger for Egyptian and achieved an accuracy of 69.83%. Habash et al. (Reference Habash, Roth, Rambow, Eskander and Tomeh2013) released a new adaptation for MADA (Roth et al. 2008) that can also process dialectal Egyptian. Darwish et al. (Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018) recently released a data set of POS-tagged tweets that cover four different Arabic dialect groups. We use this data set in this paper, and we describe it in greater detail in the next section.
Dialect and tweet-specific POS tags
3. Data description
We used the POS-tagged DA data set that was released by Darwish et al. (Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018). The data set includes 350 tweets for four major Arabic dialects that were manually segmented and POS tagged as is without applying any spelling standardization (Darwish et al. Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018), such as CODA (Habash, Diab, and Rambow Reference Habash, Diab and Rambow2012). For example, the word mtbSlw$ Footnote a “do not look at him” is segmented as m+tbS+l+w+$ and tagged as follows: PART+V+PREP+PRON+NEG_PART. The data are split into fivefold partitions for cross-validation with 70/10/20 train/dev/test splits for each dialect. For comparison, we manually segmented and POS tagged an additional 350 MSA tweets which differ in dialect but match in genre. To find MSA tweets, we used 30 very strong MSA words to filter millions of Arabic tweets. These words are mainly function words such as relative pronouns Al*yn and Al*y (who), demonstrative pronouns h*A and tlk (this, that), question words mA*A and lmA*A (what, why), and adverbs bynmA and TAlmA (while, so long as). We randomly selected 100 tweets containing each word. Then from those, we randomly selected 350 tweets as a sample of MSA tweets. The data set size is as follows:
The data set is tagged using the Farasa POS tagset (Darwish et al. Reference Darwish, Mubarak, Abdelali and Eldesouki2017), which has 18 tags for MSA, 2 dialect-specific tags (PROG_PART and NEG_PART), and 4 tweet-specific tags (HASH, EMOT, MENTION, and URL) (Darwish et al. Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018). Table 1 shows examples of the dialectal and tweet-specific tags.
Words are white-space and punctuation separated while hashtags, emotions, mentions, and URLs are considered as single units without internal segmentation. Data are formatted in CoNLL format: Words are split into tokens or clitics which are syntactic units (such as prepositions, conjunctions, determiners, pronouns, and particles) that happen to attached to words. For example, the word وأحب w¿Hb “and I like” is split into two tokens where the conjunction ð “and” is separated from the word. POS is provided in the data at the token level. In our annotation scheme, tokens, words, and sentences are separated by token boundary (TB) tag, word boundary (WB) tag, and end of sentence (EOS) tag, respectively, as shown in Table 2.
Data format for segmentation and POS tagging
Figure 1 shows that dialects display more lexical diversity than MSA, with more clitics having more than one possible tag. Joining dialects increases ambiguity with 48% of tokens having more than one POS tag.
Ambiguity in POS tags: percentage of clitics with one or more tags per dialects and in combination.
DNN architecture applied on the word mdxltw$ “you did not enter.”
4. Learning: Methods and features
4.1 Learning methods
We used two learning methods with different paradigms, namely CRFs and DNNs.
Linear chain CRF: The effectiveness of CRFs (Lafferty, McCallum, and Pereira Reference Lafferty, McCallum and Pereira2001) was shown by Darwish et al. (Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018) in POS tagging on this data set. We replicated their setup, which uses CRF++ implementation of a CRF sequence labeler with L2 regularization and default value of 10 for the generalization parameter “C.”Footnote b CRFs combine state-level and transition features and are simple, well-understood, and usually provide efficient models with close to state-of-the-art results.
Deep neural network: We used the DNN model depicted in Figure 2, which is well suited for sequence tagging. It is a variant of the bi-LSTM-CRF architecture proposed by Reimers and Gurevych (Reference Reimers and Gurevych2017), Ma and Hovy (Reference Ma and Hovy2016), Lample et al. (Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016), and Huang, Xu, and Yu (Reference Huang, Xu and Yu2015).Footnote c It combines a double representation of the input words by using word embeddings and a character-based representation (with CNNs). The input sequence is processed with bi-LSTMs, and the output layer is a linear chain CRF. The model uses the followings:
Clitic-level embeddings allow the learning algorithms to use large unlabeled data to generalize beyond the seen training data. We explore randomly initialized embeddings based on the seen training data and pre-trained embedding.
Character-level CNNs have proved effective for various NLP tasks due to their ability to extract sub-word information (ex. prefixes or suffixes) and to encode character-level representations of words (Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011; dos Santos and Guimarães Reference dos Santos and Guimarães2015; Chiu and Nichols Reference Chiu and Nichols2016).
Bi-LSTM recurrent neural networks are well suited for modeling sequential data, achieving ground-breaking results in many NLP tasks (e.g., machine translation). Bi-LSTMs (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997; Schuster and Paliwal Reference Schuster and Paliwal1997) are capable of learning long-term dependencies and maintaining contextual features from both past and future states while avoiding the vanishing/exploding gradients problem. They consist of two separate bidirectional hidden layers that feed forward to the same output layer.
CRF is used jointly with bi-LSTMs to avoid the output label independence assumptions of bi-LSTMs and to impose sequence labeling constraints as in Lample et al. (Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016).
The architecture of our model, shown in Figure 2, is applied to the example word “mA+dxl+tw+$” (you did not enter) to predict its POS tags. For each clitic, the CNN computes the character-level representation with character embeddings as inputs. Then the character-level representation vector is concatenated with both clitic embeddings vector and feature embedding vectors to feed into the bi-LSTM layer. Finally, an affine transformation followed by a CRF is applied over the hidden representation of the bi-LSTM to obtain the probability distribution over all the POS labels. Training is performed using stochastic gradient descent with momentum of 0.9 and batch size equal to 5. Given the relatively small POS data sets, we employ dropout (Hinton et al. Reference Hinton, Srivastava, Krizhevsky, Sutskever and Salakhutdinov2012) and early stopping (Caruana, Lawrence, and Giles Reference Caruana, Lawrence and Giles2000) to mitigate overfitting. We tuned our hyperparameters on the development data set using random search. We used the following hyperparameters:
4.2 Features
For both learning methods, we experimented with three features: two features that were shown to be effective for dialectal POS tagging by Darwish et al. (Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018), namely token metatypes and stem templates, and the third is Brown clusters.
Metatypes (MT) include 10 types of tokens that are heuristically determined, namely #Hashtag; @Mention; URL; Emoticon/emoji (dictionary-based); Retweet (“RT”); Foreign (all Latin characters); Number (numerals or spelled out numbers); Punctuation; Arabic (all Arabic letters); and Other.
Stem templates (STs) are morphological patterns that are used to derive stems from roots, such as the stem “lAEb” (player) which is derived from the root “lEb” using the stem template “CACC.” We used Farasa (Abdelali et al. Reference Abdelali, Darwish, Durrani and Mubarak2016) to determine stem templates.
Brown clusters (BCs). Brown clustering is a hierarchical clustering of words based on their context (Brown et al. Reference Brown, Desouza, Mercer, Pietra and Lai1992) and produces a kind of word embeddings. Similar words, particularly those with the same POS tag, tend to appear in similar contexts. BCs can be learned from large unlabeled texts and have been shown to improve POS tagging, especially for small training sets (Owoputi et al. Reference Owoputi, O’Connor, Dyer, Gimpel, Schneider and Smith2013; Stratos and Collins Reference Stratos and Collins2015). To obtain BCs, we first collected a set of 5 million tweets for each dialect by filtering tweets based on Twitter users’ stated locations. For instance, to obtain Maghrebi tweets, we collected those that matched geographical locations such as Morocco, Casablanca, Algeria, Tunisia using their Arabic, English, and French names. Similarly, we filtered tweets that match locations in which the other dialects are spoken. Though geographical filtration does not guarantee a specific dialect, we assumed that a substantial part of tweets would be in the desired dialect. For MSA, we filtered tweets using the aforementioned strong MSA words (pronouns and particles). We segmented MSA tweets using Farasa (Abdelali et al. Reference Abdelali, Darwish, Durrani and Mubarak2016) and the dialectal tweets using the DNN segmenter of Samih et al. (Reference Samih, Eldesouki, Attia, Darwish, Abdelali, Mubarak and Kallmeyer2017). We obtained BCs using the implementation of Liang (Reference Liang2005). To illustrate the effectiveness of BCs, the top tokens in the cluster that includes the emoticon “;)” include “:d”, “xd,” and “:-d.” We used this data set to pre-train embeddings.
5. Experiments and results
5.1 Conditional random field
We conducted two different sets of experiments. In the first set, we trained uni-dialectal models that are trained and tested on the same dialect. For our baseline (
BL
) model, given a sequence of clitics
$c_n ... c_{-2}, c_{-1}, c_0, c_1, c_2 ... c_m$
, where we assumed perfect segmentation, we used a combination of clitic unigram features {
$c_{-1}$
;
$c_0$
;
$c_1$
} and bigram features {
$c_{-2}^{-1}$
;
$c_{-1}^{0}$
;
$c_{0}^{1}$
;
$c_{1}^{2}$
;} (Darwish et al.
Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018). We also experimented with stem templates (
STs
), clitic metatypes (
MTs
), and
BCs
. For BCs, we varied the number of clusters to be 50, 100, 200, or 400, and we used cluster paths as features. We conducted side experiments where we used different prefix lengths of cluster paths, but we did not observe any improvements.
Table 3 shows the effect of using each of the different features individually or collectively. All the results reported in the paper are at word level (not clitic level). Since words are composed of one or more clitics, clitic-level results are almost always higher than word-level results. To illustrate the difference, consider the utterance m+Akl+t+$ Al+
$>$
kl (I did not eat the food) where the correct POS tags would be PART+V+PRON+NEG DET+NOUN. If the system erroneously generated PART+V+PRON+NEG DET+V, accuracy at word level would be 50% (first word-level composite tag is correct, while the second is not) while clitic-level accuracy would be 83.3% (5 out of 6).
Training and testing on the same dialect with different features with n brown clusters. Bold values are the best results achieved
Effect of ST+MT compared to BC200
As the results show, stem templates, metatypes, and Brown clusters improve upon the baseline model. Combining all features improves POS tagging accuracy with ST+MT+BC200 providing the best results (92.0 versus a baseline of 85.4). Though we tried to faithfully reproduce the results of Darwish et al. (Reference Darwish, Mubarak, Abdelali, Eldesouki, Samih, Alharbi, Attia, Magdy and Kallmeyer2018), which constitute our baseline, our results were consistently higher by roughly 2%.
We also compared the effect of ST+MT together with BCs to using BCs alone on clitics that were seen or unseen during training. As Table 4 shows, BCs were far more effective than the combination of MT and ST in generalizing to unseen clitics. Using all features yielded the best overall results. The effect was less pronounced for seen clitics.
In the second set, we trained joint models using the training data for all dialects and we tested on individual dialects. We wanted to determine if dialectal POS tagging for one dialect can benefit from the data of another dialect. We used all the aforementioned features and we varied the number of BCs. Table 5(a) reports the results of joint training. As shown, joint training yields lower results than uni-dialectal models.
Joint training on all dialects and testing on individual dialects with all features and dialect ID
DNN: Training and testing on the same dialect with different features
Dialect ID. The lower results of the joint model prompted us to experiment with dialect ID as a feature. The rationale for this is that uni-dialectal models fundamentally assume that dialect IDs are known. We obtained dialect IDs using two methods. First, we used the gold dialect IDs. Second, we trained an automatic dialect classifier using the training and dev parts of each of the folds and we tested on the test part of the fold. We trained the fastText DNN classifier (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2016) using character 2, 3, 4, and 5 grams as inputs, a 40-dimensional embedding vector for each input, a learning rate of 0.1, and 50 training epochs. The resultant classifier achieved an average accuracy of 86.3% across all folds. Arabic dialect detection is nontrivial due to high lexical overlap between dialects. Our dialect identifier is competitive with state-of-the-art classifiers, where reported results for MSA versus Egyptian range between 88.5% (Elfardy and Diab Reference Elfardy and Diab2013) and 94.4% (Darwish, Sajjad, and Mubarak Reference Darwish, Sajjad and Mubarak2014) with even lower results (74%) for multi-dialect identification (Malmasi, Refaee, and Dras Reference Malmasi, Refaee and Dras2015).
We added the dialect ID as three features: dialect ID, combination of dialect ID and clitic, and combination of dialect ID and Brown clusters. Table 5 reports on the results of using either gold dialect IDs, which provides the maximum attainable gain from using dialect IDs, and automatic dialect ID. As can be seen, using dialect ID, either gold or automatic, as a feature yielded results that surpassed all our previous results. Using gold dialect IDs gave 0.4% higher than using the automatic dialect IDs. As seen before, using either 200 or 400 Brown clusters yielded the best overall results.
Significance test. We ran a paired two-tailed t-test and a Wilcoxon signed-rank test to ascertain if differences between results are statistically significant or not. While the t-test is a parametric test, the Wilcoxon test is not. When comparing two setups, we compared the results they produce for every fold for every dialect (25 different values) in a paired manner. We ran
$n \times n$
comparisons between baseline; baseline with each individual feature (ST, MT, BC); baseline with all features; and joint system with all features with no dialect ID, automatic dialect ID, and gold dialect ID. When BCs were used, we looked at systems using 200 BCs.
Aside from the comparison between the baseline with +ST and the baseline +MT, where the p-values were 0.745 and 0.711 for the t-test and Wilcoxon test, respectively, all other differences were statistically significant with p-values
$\leq$
0.01 for both tests. This shows that the BCs feature yields better results than either of the two other features. Combining features leads to statistically significant improvements. Joint training without dialect IDs degrades results significantly, while joint training with automatic or gold dialect IDs improves results significantly. Lastly, improving automatic dialect identification is likely to lead to statistically significant improvements.
5.2 Deep neural network
We conducted two sets of experiments using DNN. First, we trained uni-dialectal models. We conducted four experiments with different layers stacked on top of each other, making use of linguistic features, word embeddings, and unsupervised clustering. The experiments were as follows:
Baseline (BL). We used clitics only with randomly initialized embeddings. In this setup, we use bi-LSTM with Chain CRF classifier.
Baseline+Chars. We added randomly-initialized embeddings and character representations. We add a one-dimensional CNN layer for characters. These two layers and the subsequent layers are merged (concatenated) together before being passed on to the Bi-LSTM and the CRF classifier.
BL+Chars+Embed. We used pre-trained embeddings for clitics and characters, which were trained on the aforementioned tweets corpus.
BL+Chars+Embed+Features. We used clitics and characters with pre-trained embeddings and all features (ST+MT+BC200).
The results in Table 6 show that the DNN model receives significant boosts from adding: (a) a CNN characters layer (79.3%–87.3%); (b) a pre-trained embeddings layer (87.3%–90.7%); and (c) the features (90.5%–91.7%). We conducted side experiments to ascertain the relative effectiveness of the different features. We found that the model achieves the most gain from the Brown clusters (+0.9%) and does not get any significant gain from either metatypes (–0.2%) or stem templates (+0.1%). We assume that the reason for that is that character-specific features are already encoded in the CNN characters layer. One interesting observation is that though pre-trained embedding and Brown clusters are similar in the sense that both try to learn from unlabeled data, their combination is better than either of them alone.
Joint DNN training w/ & w/o dialect IDs (Features: Chars+Embed+ST+MT+BC200)
In the second set of experiments, we trained joint models and tested on individual dialects. We used the best uni-dialectal configuration, which uses clitic and character inputs with pre-trained embeddings that represents all the dialects and all features. The results in Table 7(a) show that the model does not benefit from joint training with results dropping by 0.3%. This is consistent with results that we observed for CRFs (Table 5(a)). Thus, we also experimented by providing the dialect ID as a feature to our DNN. Table 7(b) shows the results of using gold dialect IDs. The DNN model does not benefit significantly from dialect IDs, and the results of the joint model (with and without dialect ID) are comparable to the best uni-dialectal model.
For significance testing, again we used the t-test and the Wilcoxon test to compare all the setups in Tables 6 and 7. The best uni-dialectal setup (Chars+Embed+ST+MT+BC200) and the joint training setups with and without dialect ID were all statistically indistinguishable with p-values
$\geq$
0.05 using both tests. All other differences were statistically significant with p-values
$\leq$
0.01 for both tests. This indicates that (a) adding a character-level CNN and pre-trained embeddings yielded statistically significant improvements; (b) adding features led to significant improvement; (c) the drop of the joint model, with and without dialect IDs, compared to the best uni-dialectal model was not statistically significant; and d) adding the dialect ID did not lead to significant improvement.
5.3 Discussion
We compared the results of using the CRF and DNN setups. We specifically compared the following:
-
• CRF baseline (85.4%, Table 3) and DNN with character representations and pre-trained embeddings (90.7%, Table 6). Both uni-dialectal models involve no feature engineering. Results show that the DNN setup outperforms the CRF baseline, as the DNN is able to learn features automatically. The difference is statistically significant with p-values
$\leq$
0.01 for both the significance tests. -
• CRF using all features (+ST+MT+BC200) (92.0%, Table 3) and DNN using all features (+Chars+Embed+ST+MT+BC200) (91.7%, Table 6). Though the CRF results are slightly higher (+0.3%), the difference is not significant with p-values
$\geq$
0.05 using both significance tests. -
• Joint learning with automatic dialect ID with CRF (92.4%, Table 5(b)) and DNN (91.5%, Table 7(b)). CRF yields statistically significantly better results than DNN for joint training with +0.9% improvement (absolute).
-
• We were hoping that by merely performing joint training, the results would improve overall. That was not the case. As seen in Figure 1, joining the tweets from different dialects increases ambiguity with 48% of clitics having at least two different possible POS tags compared to 30%–36% for individual dialects. However, as Table 8 shows, adding dialect ID for the CRF setup improved tagging for clitics with multiple possible POS tags.
-
• Lastly, we looked at the effectiveness of our best CRF and DNN systems in handling words that were unseen during training. The CRF system was correct 85.4% of the time compared 86.0% for the DNN system. The difference is rather small with no clear advantage for either one.
Tables 9 and 10 show the most common word- and clitic-level errors for both CRF and DNN approaches with joint training and how often they appear. The error distributions of both approaches seem to closely match, with verb–noun and noun–adjective confusion being the most common. As the examples in Tables 9 and 10 show, many errors stem from (a) dialectal or foreign words (marked with
$\phi$
in tables); (b) ambiguous words that can assume either POS tag given different contexts (marked with
$\xi$
); (c) letter substitutions between different forms of alef, ta marbouta and ha, and alef maqsoura and ya (marked with
$\psi$
); and (d) nonconventional dialectal spellings (marked with
$\lambda$
). Other errors include named entities and misspelled words.
Effect of dialect ID on improving tagging of clitics with different number of possible tags
Most common word-level errors
Most common clitic-level errors
5.4 Comparison to existing tools
We compared our results against two state-of-the-art Arabic POS taggers, namely Farasa (Darwish et al. Reference Darwish, Mubarak, Abdelali and Eldesouki2017) and MADAMIRA (Pasha et al. Reference Pasha, Al-Badrashiny, Diab, El Kholy, Eskander, Habash, Pooleery, Rambow and Roth2014). Since neither were tuned for tweets, we edited their outputs to assign tweet-specific tags as not to penalize them for missing hashtags, mentions, URLs, and emoticons. Farasa is tuned for MSA only. Its accuracy on MSA tweets was 89.3%, which is much lower than its results on news (96.2%) (Darwish et al. Reference Darwish, Mubarak, Abdelali and Eldesouki2017) and also significantly lower than the results obtained by our CRF with joint training (94.5%). This suggests that the tweets genre is rather different from news stories, which behooves the need for training taggers specifically for tweets. As for MADAMIRA, which is tuned for MSA and EGY, achieved 88.0% and 90.5% on MSA and EGY tweets, respectively. Again, these results are much lower than MADAMIRA on news stories (95.3%) (Darwish et al. Reference Darwish, Mubarak, Abdelali and Eldesouki2017) and much lower than our CRF (94.5% and 95.4%). Training on limited in-domain data can yield better results than a POS tagger that is trained on lots of out-of-domain data. It is noteworthy that both Farasa and MADAMIRA could have benefited from gold segmentations. However, altering either system to supply gold segmentations is beyond the scope of this work.
6. Conclusion
In this paper, we present state-of-the-art POS tagging of multi-DA tweets using CRF and DNN approaches, and based on a small tagged data set of Arabic tweets. The data set covers MSA and the four major dialect groups. We explore uni-dialectal and joint models. While uni-dialectal CRF and DNN models yield statistically indistinguishable results, our CRF model outperforms DNN for joint training. We show that using a mere 350 tweets per dialect can lead to word-level accuracy of 92.4% on average across dialects. The CRF model obtains the best results when using linguistic features (stem templates and clitic metatypes), word clusters from a large unlabeled tweet corpus, and automatic dialect identification. These three elements boost the accuracy from 85.4% to 92.4% (i.e., a relative error reduction of 48%). For the DNN approach, we combine clitic- and character-level inputs with the features we used for CRF to obtain the best DNN results. Though both serve similar purposes, combining pre-trained embeddings and word clusters yields the best results. Linguistic features and dialect identification marginally affect DNN results.
