4.1 Ill-formed word detection (IWD)

As stated earlier, since it is not feasible and efficient to use a lexicon lookup table for storing and checking all possible surface word forms in an MRL, one needs to have an LV that will verify whether a given word belongs to that language. In the case of an MRL, a high-coverage morphological analyzer may be used for this purpose. The aim of a morphological analyzer is to analyze the word surface forms and output their possible lemmas, part-of-speech tags and inflectional/derivational structures in the case of an MRL. If the morphological analyzer is capable of producing a valid output for a given input, this will certify that the given surface form is a valid word in that language. One should keep in mind that, with this approach, all foreign wordsFootnote ⁴ will also be detected as ill-formed and will be automatically normalized, although this may not be preferable in some cases. Although the language identification task has been considered an almost solved problem for many applications, language detectors fail in the social media context due to phenomena called code-switching (i.e., the usage of multiple languages within the same context or even the same sentence). This article treats the detection and normalization of these as out of scope and only focuses on the normalization of Turkish words. Solorio et al. Reference Solorio, Blair, Maharjan, Bethard, Diab, Ghoneim, Hawwari, AlGhamdi, Hirschberg, Chang and Fung2014 make an overview of the first shared task on language identification of code-switched data. As future work, our approach may be extended by adapting one of the recent code-switching approaches, such as Alex, Dubey and Keller Reference Alex, Dubey and Keller2007, Lui, Lau and Baldwin Reference Lui, Lau and Baldwin2014, Eskander et al. Reference Eskander, Al-Badrashiny, Habash and Rambow2014, Jhamtani, Bhogi and Raychoudhury Reference Jhamtani, Bhogi and Raychoudhury2014, Das and Gamback Reference Das and Gambäck2013 among many others.

In this stage, the erroneous letter case usages (all UPPER CASE, mIXed case writings) should also be detected and sent to the candidate generation stage for normalization. For the IWD of Turkish, we use a high-coverage morphological analyzerFootnote ⁵ Eryiğit (Reference Eryiğit2014) and a letter case checker, which detects erroneous letter case usages. All tokens that are selected as in-vocabulary (IV) words (labeled with a +NC (No Change) label in the example below) are left untouched and output in their original form. The filtered out-of-vocabulary (OOV) words are transferred to the candidate generation stage. Mentions (@user_name), hashtags (#topic), emoticons (:D), vocatives (ahahahaha) and keywords (RT) are also assumed to be OOV words, since we would like to detect these and tag them with special labels to be later used in higher level NLP modules (e.g., POS tagging or syntactic analysis).

The following example shows an input sentence (bgn Twitter çok eglenceliiiiiiiiii dimi ? :D) and the labels assigned to each of its components after being processed by the IWD stage. The expected normalized form of the sentence is ‘Bugün Twitter çok eğlenceli değil mi ? @smiley[:D]’ that means ‘Twitter is very entertaining today, isn’t it ? :D’. The words that are passed directly from IWD are marked in the example below with a check mark (✓). The ones that are filtered to be transferred to the candidate generation ( ) stage are marked with a cross (✗).

Example 2

4.2 Candidate generation

In order to design an appropriate model for the candidate generation stage, the main nonstandard spellings in Turkish social media texts are investigatedFootnote ⁶ and the following trends are observed: the usage of slang words, abbreviations and initialisms, character repetitions in interjections, logograms, social media specific words, wrong letter cases, direct transcription of words as if they were pronounced with an accent, vowel and diacritic omissions and misspelling of words. In order to normalize such nonstandard spellings, seven main candidate generation layers are designed and implemented. These are (refer to Section 5.1 for the distribution of these categories on our test sets) as follows:

1. (1) replacement rules & lexicon lookup (e.g., 2g2bt → ‘too good to be true’);

2. (2) letter case transformation (e.g., JOHN’S → ‘John’s’);

3. (3) proper noun detection (e.g. potter → ‘Potter within the context Harry Potter’);

4. (4) diacritic restoration (e.g., facade → ‘façade’);

5. (5) vowel restoration (e.g., Twttr → ‘Twitter’);

6. (6) accent normalization (e.g., gonna → ‘going to’);

7. (7) spelling correction (e.g., acheive → ‘achieve’).

The first two components and the accent normalization module (normalization of phonetic writing in different accents) are mostly solved by rule-based techniques (sometimes making use of statistics); proper noun detection and diacritic and vowel restoration modules are created using a machine learning technique (namely CRFs) and the spelling corrector using a statistical model. Traditional spelling correction techniques rely on the fact that most spelling errors are within a short edit-distance of their correct form Kukich (Reference Kukich1992); Max & Wisniewski (Reference Max and Wisniewski2010). That is why spelling correction needs special treatment in case of MRLs, which by nature consist of longer words resulting in errors in longer edit distances and mostly due to the wrong spellings outside the word lemma (within the affixes) Ingason et al. (Reference Ingason, Jóhannsson, Rögnvaldsson, Loftsson and Helgadóttir2009); Pirinen & Lindén (Reference Pirinen and Lindén2014); Pirinen et al. (Reference Pirinen and Lindén2010). In case of languages having rather shorter words than MRLs, the complete omission of diacritics and vowels would not be a severe problem and could be jointly solved with spelling correction. But with the entrance of social media into our daily lives, diacritic and vowel restoration as well as accent normalization need special treatment for MRLs. In our proposed architecture, we treat these tasks as separate layers.

Since no data resources for Turkish text normalization were available for training purposes during the implementation of this study, we tried to minimize the need for gold-standard normalization data during the design phase by either proposing rule-based techniques or by using machine learning approaches where it would be possible to create the training data automatically. During the analyses of the rule-based modules, a validation set (see Section 5.1 for details) is used to compose the relevant rules. For machine learning (i.e., vowel and diacritic restoration) and statistical models (i.e., spelling correction), the automatically created training data are explained later in the following subsections.

After the individual creation of each module, they are integrated in a cascaded structure (Section 4.2.8) in order to be able to normalize the words which could not be corrected independently by these. The outputs of each component are checked by the LV, and then, if the normalized form from a component becomes an IV word, then the process is terminated (except for the Letter Case Transformation, Replacement Rules & Lexicon Lookup and Diacritization components, where the input is replaced by the modified output although it is still not an IV word; see Section 4.2.8 for details). Otherwise, the candidate generation process continues with the next component of the original input.

4.2.1 Replacement rules & lexicon lookup

The usage of slang words and initialisms (e.g., kib for kendine iyi bak (take care of yourself) or nbr for ne haber (what’s up), character repetition in interjections (e.g., lütfeeeennnn instead of lütfen (please) or çooooooook instead of çok (very)), Web 2.0 specific words, such as hashtags (e.g., #topic), mentions (e.g., @user_name), emoticons (e.g., :p), vocatives (e.g., ahahhah or hööööö) and keywords (e.g., RT), logograms (e.g., $eker 4you instead of şeker senin için (sweety, it’s for you)), emails and URLs are very common in social media texts. In this module, the normalization of these types are realized by the use of replacement rules designed for catching and normalizing the most frequent patterns and by the use of a lexicon composed of most frequently used slang words and initialisms. Our lexicon contains 286 noncanonical slang words and initialisms, and their normalized forms. To give an example, the following words ltf, pls, ltfn, piliz all refer to lütfen (please) in Turkish. In this module, once the input token is looked up and found within the lexicon, it is directly replaced by its normalized form.

Replacement rules are written as regular expression patterns and handle the normalization of character repetitions, logograms,Footnote ⁷ the detection and tagging of Web 2.0 specific words, emails and URLs. For the normalization of character repetitions, we reduce repeated characters into a single character when the consecutive occurrence count is greater than two. There are additional replacement rules to catch and normalize logograms. Examples include $ → ~ş, ε~ → ~e, 3 → ~e, @ → ~a, ! → ~i and ß→ ~b. The following token types are tagged by the given specific labels for future usage in higher level applications:

• • Mentions: Nicknames that refer to users on Twitter are labeled as @mention[@dida].

• • Hashtags: Hashtags that refer to trending topics are labeled as @hashtag [#geziparki].

• • Vocatives: Vocatives are labeled as @vocative[hehe].

• • Smileys: Emoticons are labeled as @smiley[:)].

• • URLs: URLs are labeled as @url[http://www.cse.unt.edu/~rada/jnle].

• • E-mails: E-mails are labeled as @email[texline@cup.cam.ac.uk].

• • Keywords: Keywords like RT, DM, MT, Reply etc. are labeled as @keyword [RT].

Figure 2 shows the normalized version of a tweet in informal Turkish that could be translated as ‘@dida what’s up, why don’t you call #offended :(’, before and after being processed by this component. Although the word aramion also needs to be normalized as aramıyorsun (you don’t call), this transformation is not realized within the current component and normalized later in the accent normalization phase described in Section 4.2.4.

Fig. 2.

Normalization with replacement rules & lexicon lookup.

4.2.3 Proper noun detection

The primary aim of this component is to detect and correct proper nouns erroneously written in lowercase with an inappropriate surface form or that have been lower-cased by our previous model (Section 4.2.2). To give an example, the words ahmetten and ahmetden are both erroneously written on behalf of the ablative inflected form of a male name Ahmet and should be actually written as Ahmet’ten. Since these words in their current state would be rejected by the LV, they would be detected as an OOV by IWD and transferred to CG for correction.

Turkish proper nouns are very frequently selected from common nouns. Some samples are given below. The name umut that means hope in English is a commonly used person name in Turkish. Since the task could not be easily accomplished by just checking the occurrence of such words within gazetteers, it is quite difficult to identify such words as proper nouns when they are written in lowercase.

Example 3

Umut (hope) – a unisex name or a surname

Çiçek (flower) – a female name or a surname

Şeker (sugar) – a female name or a surname

İpek (silk) – a female name or a surname

NER can be basically defined as identifying and categorizing predefined types of named entities, such as person, location and/or organization names. Although NER approaches can achieve nearly human annotation performance on well-formed text, noisy user-generated text presents serious challenges for this task and performance on this domain is still very behind that obtained on well-formed text Baldwin et al. (Reference Baldwin, Kim, de Marneffe, Ritter, Han and Xu2015). The task of detecting and correcting ill-formed proper nouns on social media data may actually be regarded as a version of NER where named entities should be first detected and then normalized into a valid form.

In this module, we adapt a NER model (from Şeker and Eryiğit Reference Şeker and Eryiğit2012, Reference Şeker and Eryiğit2017 reporting the highest results for Turkish NER in the literature) and use it for the normalization of proper nouns. The adapted model based on sequential classification (i.e., CRFs) uses lexical, morphological and contextual features for modeling the morphologically rich nature of the Turkish language. The model uses a morphological analyzer (producing all possible analyses for a given token) and a disambiguator (selecting the most probable analysis from the outputs of the morphological analyzer) in order to assign the relevant morphological features for each token. The same training dataset Tür, Hakkani-Tür and Oflazer (Reference Tür, Hakkani-Tür and Oflazer2003) consisting ~500K words collected from news articles is used during the training stage. But since this dataset is of a well-formatted nature, in order to remove the influence of the letter case feature (e.g., uppercase, lowercase) used in the original study and to adapt it for our normalization module, we lowercased all the named entities during data preparation before the training stage. As the second step of our adaptation, we developed two new gazetteers for person names and surnames in addition to the original gazetteers used in the original study. These two new gazetteers are the filtered versionsFootnote ⁸ of the original person name and surname gazetteers using some threshold values calculated (formula given in Equation 1) over two additional corpora: the METU Corpus Say et al. (Reference Say, Zeyrek, Oflazer and Özge2002) and the web corpus of Sak, Güngör and Saraçlar Reference Sak, Güngör and Saraçlar2011. The names having a zero denominator or no occurrence count at all are assigned a very large ratio value so that they are all treated as proper nouns. Table 4 gives the occurrence ratios for three sample words. One may observe from the table that ahmet occurred 40 times in proper case form and 20 times in lower case form within the two corpora resulting in a ratio value of 2.0. The aim of this calculation is to detect nouns that are more likely to occur as a proper noun rather than a common noun. With this purpose, we select a threshold value of 1.5 (after an investigation of the gazetteers) above which a ratio would denote that the corresponding noun should be treated as a proper noun. For example, the ratio value for umut is only 0.4 (which is below the threshold) and it would not be converted to proper case. A similar situation holds for the word sağlam (healthy). Although it is a quite frequent Turkish family name, it is observed mostly as a common noun in our corpora with a ratio value of 0.09.

(1) $$\begin{equation} ratio(w_n)=\frac{OccurrenceInPropercase(w_n)}{OccurrenceInLowercase(w_n)} \end{equation}$$

Table 4.

Example of ratio values

During the testing stage, each token is looked up within the gazetteers. Since the gazetteers contain only lemmatic information, the surface form of an inflected proper noun would never exist in them. In order to derive the appropriate lemma for such words (especially if not written with appropriate orthographic rules), we make use of an unknown-word analyzer (which is distributed with the used morphological analyzer Eryiğit (Reference Eryiğit2014)) also capable of analyzing the inflected forms of unknown lemmas according to Turkish morphological rules. The unknown-word analyzer has more flexible vowel-consonant harmony rules for unknown words so that it strips all possible suffixes even if they do not phonetically conform to the unknown lemma. For our initial example, both the inputs ahmetten and ahmetden would be analyzed the same way since the suffixes -den and -ten are possible surface forms of the ablative case suffix according to consonant harmony.

Example 4

ayşe +Guess+Noun+A3sg+P2sg+Abl

ayşen +Guess+Noun+A3sg+Pnon+Abl

ayşende +Guess+Noun+A3sg+P2sg+Nom

ayşenden +Guess+Noun+A3sg+Pnon+Nom

It is very probable that multiple admissible lemmas may be produced at the end of the explained process. For example, the above output (Example 4)Footnote ⁹ is obtained after the analysis of the word ayşenden. The possible lemmas proposed by the unknown word analyzer are ayşe, ayşen, ayşende, ayşenden. The first two (Ayşe and Ayşen) are both frequently used Turkish female names and occur within the gazetteers with high ratio values. Both of the lemmas have the ablative case marker tag in the morphological analyzer output, but the first one ayşe is inflected with an extra +n suffix (second person possessive form), which is a rarely used suffix for proper nouns (meaning ‘from your Ayşe’). In this case, although both of the analyses are valid, a human would most probably choose the second one with the lemma Ayşen with no possessive marker The inflectional suffixes are then separated by the use of an apostrophe resulting in the corrected word Ayşen’den (from Ayşen). In order to assign the most probable lemmas and morphological analyses for such words, the outputs of the morphological analyzer are filtered before being transferred to the morphological disambiguator. For this filtering, the possible lemmas are looked up from the gazetteers, and the longest possible one existing within the gazetteers (e.g., ayşen for this example) is accepted as the legal one and only the analyses producing this lemma are kept. If none of the possible lemmas exist within the gazetteers, the word is left as it is and transferred to the next stage in its original form.Footnote ¹⁰ For the normalization stage, the named entities extracted as the result of the recognition layer are then capitalized and the suffixes added to the assigned lemma are separated with the use of an apostrophe sign added to the surface form.

4.2.4 Accent normalization

As defined by McKean Reference McKean2005, ‘An accent is a manner of pronunciation peculiar to a particular individual location or nation’. In social media, people generally write as they speak by transferring the pronounced versions of words directly to the written text. Eisenstein Reference Eisenstein2013a discusses the impact of phonological variation in English social media texts by giving the following Example 5. In this article, our aim is to be able to detect the most frequent accents used in Turkish social media texts and normalize them. Turkish is a language that is almost always pronounced as it is spelled. Due to its rich morphology, the accent problems of Turkish are more severe than the examples given below.

Example 5

lef → ‘left’

jus → ‘just’

wit → ‘with’

gonna → goin → ‘going’

doin → ‘doing’

kno → ‘know’

As in most MRLs, words have many inflected forms and the number of possible accents becomes very high. While the detection of accent usage is an important issue, another crucial stage is its normalization which could not be realized by just using lookup tables over stored samples. For example, while templates like going to in English could be easily resolved by storing their different accents (goin, gonna) and using them in replacement rules during normalization with any verb (as in ‘I’m gonna dance’, ‘I’m gonna sing’), the same future tense accents would be a severe issue in the case of Turkish. The tense suffix will be affixed to a stem together with many other possible suffixes (person, polarity, mood and tense markers, including compound tenses) and will conform to many vowel-consonant harmony rules according to the stem to which it is affixed. One should first catch all possible accented forms and then normalize the word by reconstructing the phonological harmony. In the below example, the verb git (to go) takes the following suffixesFootnote ¹¹ : (1) the negative marker +mA, (2) the future tense suffix +(y)AcAk and (3) the first person agreement marker +(H)m. The morphological analysis of the resulting regular word gitmeyeceğim (git+me+yecek+im) is git +Verb+Neg+Fut+A1sg. It is seen in the example that the accented version gidmiycem differs severely from the normalized form gitmeyeceğim. Even the stripped stems of the two words (gid and git) are different from each other due to the consonant lenition.Footnote ¹² One needs to use a more complex approach in order to normalize such words.

Example 6

gidmiycem instead of gitmeyeceǧim

(I will not go)

In this component, we focus on normalizing the most common verb accents that Turkish people use in spoken language since these are the ones which have the largest number of different surface forms. As the first step of our solution, we create a replacement rule table similar to the one given in Table 5, consisting of the most frequent verb accent endings, such as -cem, -yom, -çaz and their appropriate morphological analyses if they were written in appropriate manner as -ceğim, -yorum, -cağız. We then construct regular expressions in order to detect these endings on the input tokens and then replace them with the matched morphological analysis given in the table. As an example, the input gidmiycem becomes gid +Verb+Neg+Fut+A1sg.

Table 5.

Examples of accent normalization replacement rules

The morphological tags in the table stand for: +Pos: Positive, +Prog1: Present continuous tense, +A2sg: second person singular, +Fut: Future tense, +A1sg: first person singular, +A1pl: first person plural.

As a second step, we use a morphological generator Eryiğit (Reference Eryiğit2014), to reconstruct the normalized form of the verb. The morphological generator produces a valid Turkish word by applying all the rules of a morphological analyzer in the reverse order (from lexical form through surface form). There exist cases where it is not possible to obtain the correct verb stem by just removing the accented suffixes given in the table. For example, gid is the surface form of the verb stem git (to go) only after being affixed a morpheme starting with a vowel. For such cases, we also apply the phonological harmony rules to the stem.

There exist also more complex cases where the normalized form is not a single token. In the below Example 7, the proper form of the word gidiyonmu is actually gidiyor musun (are you going) and in formal writing the interrogative enclitic mi (in this case, with the second person singular suffix -sun) is written separately from the verb itself. On the other hand, the example shows that the person suffix is directly attached to the verb in the accented version: gidiyo N mu. For such cases, we apply the generation process twice (for the verbal stem and the enclitic separately). At the end of this process, if a valid output is produced, it is directly accepted as the normalization result. Otherwise, the input word is transferred to the next normalization stage in its original form.

Example 7

‘geliyonmu?’ instead of ‘geliyor musun?’

(Are you coming?)

4.2.5 Vowel restoration

Twitter played an April Fool’s jokeFootnote ¹³ announcing that they were not including the vowels in Tweets. Although this was only a joke, nowadays, people have already developed this habit of writing without vowels Eisenstein (Reference Eisenstein2013b), which was started mainly with the need to fit their messages into 140 characters Tweets or 160 character SMS messages. As a result, the vowel restoration problem is no longer limited to some specific language families such as Semitic languages Zitouni, Sorensen and Sarikaya (Reference Zitouni, Sorensen and Sarikaya2006); Gal (Reference Gal2002), but rather applicable to social media text normalization in general.

Vowel restoration in Turkish was recently studied by Adalı and Eryiğit Reference Adalı and Eryiğit2014. In this work, the authors reported an accuracy of 54.07 per cent over a test set synthetically generated by removing all the vowels in existing valid Turkish texts and using their original form as gold-standard. They use CRFs to train their model again over a synthetically generated training set. The deficiency of the approach is that it only considers the words’ consonants and adds vowels between these as in Twttr ⇒ ‘Twitter’, ignoring the already existing vowels within the given input. For example, the input svyrm is correctly vowelized as seviyorum (I love), but the input savyrm is again vowelized as seviyorum instead of savıyorum (I avoid) by ignoring the occurrence of the letter a in the given input.

In order to improve the vowel restoration system, we propose a new partial vowel restorer which will be constrained on the existing vowels in the input. With this purpose, we present an approach similar to the sequence classifier using CRFs introduced by Lafferty, McCallum and Pereira Reference Lafferty, McCallum and Pereira2001 but this time using a constrained Viterbi algorithm at the decoding stage for CRFs. We use the freely available NLP library MALLET McCallum (Reference McCallum2002) and reimplement its decoding stage in order to implement partial vowel restoration.

Since manual compilation of new training data for partially missing vowels is a costly process, we trained our improved model with the same automatically created training data from Adalı and Eryiğit Reference Adalı and Eryiğit2014. Although it would be possible to create such datasets (both for training and testing phases) synthetically by randomly removing some vowels from well-written texts, we prefer not to do so as it would not reflect real world usage statistics.

Table 6 gives the feature representation of the word okuldan (from school) for the training stage. As can be seen from the table, the task is treated as a character level sequence classification task where the class labels are the eight possible Turkish vowels (a, e, ı, i, o, ö, u, ü). In this approach, each position between the consonants needs to be classified under one of these classes (plus one null class for the case of no vowel). As a result, we have nine possible class labels (last column of Table 6) to assign to each position. Since the occurrence of two consecutive vowels (such as in the English word ‘book’) is very rare in Turkish and they are mostly not written without vowels due to produced ambiguities, each position is limited to either 0 or 1 possible vowel addition.Footnote ¹⁴ This choice also reduces the complexity of the search space and the errors caused by the extra vowel addition. We adopt the same features as the previous workFootnote ¹⁵ : the consonants of the input token and the neighboring characters within a window of ±3 characters. The table displays the values of these features for each instance.

Table 6.

Instance representation for the word okuldan

At the decoding stage, n possible positions will produce 9 ⁿ possible output words. For the input kldn, many of these would be valid Turkish words such as koldan (from the arm), kuldan (from the slave), kıldan (from the hair). The existence of some vowels within the input would undoubtedly help the disambiguation and to find the most probable output. Figure 3 shows the constructed trellis (9⁵=59,049 possible paths) for the input okuldn. By using the existing vowels as constraints during Viterbi decoding, one may reduce the possible paths to 9^{(5 − 2)}=729 (specified with continuous lines on the trellis). In addition to this, we make further pruning after investigating the vowel removal trends used in Turkish social media data. We observe that people do not miss the first and the last letter in the case that it is a vowel (for example, aşkım (my love) is never written as şkm but it is rather written as aşkm). In other words, the omission of vowels occurs most of the time within the word and not in the boundaries. Thus, differing from the previous work, we do not try to add vowels to the frontiers. However, if the word starts or ends with a vowel, we use this information at the decoding stage as our constraints. This further reduces the possible number of paths to 9²= 81 in the given example (concentrating only on locating the vowels between the consonants l - d and d - n).

Fig. 3.

Trellis for the input okuldn.

4.2.6 Diacritization

Sometimes, because of the nonconformity of the input methods in computers and mobile devices (unlocalized keyboards, inconsistent or badly designed user input software), or due to users’ typing habits, many texts in social media are missing diacritical marks. Diacritization (also called ‘deasciification’ for Turkish) is the task of reproducing Turkish-specific characters with diacritics (ı, İ, ş, ö, ç, ğ, ü) from their ASCII-compliant counterparts (i, I, s, o, c, g, u). Deasciification is not a straightforward task because of the potential for ambiguity in the asciified form. For example, the inputs cin, kus or sok all consist of two critical Turkish-specific characters belonging to the set given above and this results in 2²= 4 possible outputs (Table 7) for each of them. The given examples carry a high degree of ambiguity so that 3/4 of their deasciified counterparts are valid Turkish words.

Table 7.

Possible diacritization outputs for the input words cin, kus and sok

Contrary to the Turkish vowel restoration problem which is a contemporary issue, the diacritization problem has always been attractive even before the raise of social media, since the first day of press usage. Thus, there are many studies on Turkish diacritization in the literature, namely Tür Reference Tür2000, Zemberek Reference Akın and Akın2007, Yüret and de la Maza Reference Yüret and Maza2006 and Adalı and Eryiğit Reference Adalı and Eryiğit2014. The last study reporting the highest diacritization performance (which is also the appearing that we used in this studyFootnote ¹⁶ ) treats the task again as a character-level sequential classification problem. Although Adalı and Eryiğit Reference Adalı and Eryiğit2014 adopt exactly the same multiclass classification approach (introduced in the previous subsection) by providing the current letter in focus as a feature to the CRF, they claim that the search space is automatically reduced to that of a binary classification problem.

For example, when the character in focus is the letter c, the system will only try to decide between the two labels (c and ç) as shown in Figure 4, since this is predefined in the system even though the possible class labels are of size 14 (the non-ASCII Turkish letters along with their ASCII counterparts as provided above). Since it is very easy for a classifier to learn this from relevant features and provided samples, this information is not provided to the system as an external constraint.

Fig. 4.

Trellis for the input CIn (the critical letters are capitalized meaning that they may either be written as c or ç and i or ı).

4.2.7 Spelling correction

Since the normalization task originates in fact from correcting spelling errors (but mostly made on purpose in the case of social media), a spelling corrector is certainly an indispensable component of a normalizer. Unfortunately, as stated previously in Torunoğlu and Eryiğit Reference Torunoǧlu and Eryiğit2014, the current performances of the available spelling correctors for Turkish fall behind those for English. The authors state that they tested the impact of two spelling correctors Akın & Akın (Reference Akın and Akın2007); Microsoft (2010) and that they obtained negative impacts on normalization performances due to the low precision scores of the correctors. The spelling corrector to be used in this task should have a special bias for precision over recall since the false positives have a very harmful effect on the normalization task; the erroneous outputs may have a radically different meaning and ruin the sense of the whole sentence.

Torunoğlu-Selamet et al. Reference Torunoğlu-Selamet, Bekar, Ilbay and Eryiğit2016 make a comparison of available spelling correctors for Turkish: the error-tolerant finite-state recognizer (ETFSR) of Oflazer Reference Oflazer1996, the spelling corrector of MSWord Microsoft (2010), the open source Turkish NLP framework Zemberek Akın & Akın (Reference Akın and Akın2007) and four new spelling correction models mainly adapted from Wang et al. Reference Wang, Xu, Li and Zhang2011. In our normalization architecture, we elect to use the best performing model (SC#4) (though this is far from perfect).

Wang et al. Reference Wang, Xu, Li and Zhang2011 propose a fast and accurate approximate string search algorithm (ASS) that keeps track of the frequent mistakes extracted from training data (by use of an Aho-Corasick search tree) and proposes the most probable correction candidates. The method is not directly applicable to an MRL since it uses a vocabulary trie for validating the generated candidates, taking hundreds of megabytes in memory even if we try to place only the most frequently occurring word surface forms. In the used system, this lookup module is replaced by a language model. Figure 5 gives the general flow of the employed system. SC#4 is inspired by Linden and Pirinen Reference Pirinen and Lindén2014, in that it uses a language and an error model together in order to generate candidates. Candidates which are generated by the error model are validated using the language model and the best proposal is the candidate with minimum rule cost and maximum unigram probability. One may refer to the aforementioned references for further details.

Fig. 5.

Adaptation of ASS into Turkish Torunoğlu-Selamet et al. (Reference Torunoğlu-Selamet, Bekar, Ilbay and Eryiğit2016).

4.2.8 The cascaded model

The subsections of Section 4.2 introduced each component used in the candidate generation layer. Although each module performs well on its targeted error types (see Section 5.4 for a detailed evaluation), there exist many cases which may not be normalized by a single module but rather their joint work. Table 8 gives some examples of such complicated errors and the modules that need to cooperate in order to normalize them.

Table 8.

Samples for complex error types

In this article, we introduce a cascaded model to integrate the presented seven normalization components. The design of the system flow is a sophisticated task since each module may affect the behavior of the other ones and a detailed analysis is needed during the design phase. Figure 6 shows the proposed integration of the seven normalization components for the case of Turkish. The figure also attempts to reflect each module’s internal structure, as explained in detail in previous sections. Accent normalization, vowel restoration and proper noun detection modules do not change their inputs while transferring to the next module in the event that the inputs are not validated by the LV. The remaining modules transfer their updates to the subsequent modules to be further normalized even if their outputs are not validated yet. Some modules operate multiple times both on the original input and the input updated from the former modules. These are the proper noun detection and accent normalization components.

Fig. 6.

Cascaded normalization model.

5.1 Datasets

Torunoğlu and Eryiğit Reference Torunoǧlu and Eryiğit2014 use a dataset of 1,200 manually normalized tweets. They use half of this data for validation purposes (as we also do in this study) and the remaining half as the test set. In order to compare our results with the pioneering work, we also test our system on this test set and provide results in the following sections. However, since the motivation in the collection of that rather small dataset was just to serve as a testbed during the design and the development of the normalization framework, these data collected from Twitter within a short time interval did not clearly reflect the natural composition of error types in real data and their usage frequencies in social media in general.

In this article, we introduce two new normalization datasets: one collected from Web 2.0 sentences and one from Twitter. ITU Web Treebank (IWT) Pamay et al. (Reference Pamay, Sulubacak, Torunoğlu-Selamet and Eryiğit2015) is a recently released corpus manually annotated in multiple layers (normalization, morphology, named entities and syntax).Footnote ¹⁷ The corpus consists of 5,010 sentences collected in a balanced manner from blogs, customer reviews, forums, social media posts and reader comments on news media. The Big Twitter dataset (BTS) (see Title page footnote for further references) constitutes 26,149 manually normalized Turkish tweets extracted between May 10th, 2012 and January 20th, 2014 with keywords related to the telecommunication domain and well-known Turkish telecom companies. The manual normalization strategy was slightly different between the two datasets: In the creation of the IWT, human annotators were asked to normalize the sentences so that they also conform to orthographic conventions such as the capitalization of a sentence’s first letter. This task is limited to the normalization of the existing tokens; the addition of punctuation and changes in sentence structure were not allowed. In this dataset, the Web 2.0 specific writings such as mentions, hashtags, etc. are also tagged manually with specific labels as explained in Section 4.2.1. For IWT, we used the updated version of our ITU Annotation Tool Eryiğit (Reference Eryiğit2007). Five annotators worked for the annotation of normalization of the new corpora. Before the annotation process started, the annotators had two weeks of supervised training in the new annotation framework, after which they started on annotating actual data. The IWT took four months to develop as the dataset was also annotated in three other areas. The annotators started from raw sentences and they first manually normalized the sentences that have been extracted from the web, and then assigned morphological tags for the tokens, before moving on to the dependency annotation. In this work, only the normalization annotation was used to test the normalization tool’s performance. Also, in the beginning of the annotation process each sentence was intended to be annotated by at least two annotators, however, due to budget-related and time-related constraints, most of the sentences could only be annotated by a single annotator. As a result, it was not possible to measure interannotator agreement.

On the other hand, during the normalization of BTS, the human annotators were motivated to correct mostly the tokens which could not be analyzed by the morphological analyzer. As a result, the orthographic rules (e.g., sentence first letter capitalization) are mostly omitted if the used annotation environment Eryigit et al. (Reference Eryigit, Cetin, Yanık, Temel and Ciçekli2013) for this dataset provided a valid morphological analysis for the input word. This mostly affected the performance of the letter case transformation module discussed in the following sections. Table 9 gives statistics on the used datasets, such as the sentence counts (tweet counts for datasets collected from Twitter), token counts and manually normalized tokens. One may notice from this table that the normalized token counts are nearly a seventh of the whole token count for our two natural datasets (BTS and IWT), whereas it was nearly one-third of Test _small and validation sets from Torunoğlu and Eryiğit Reference Torunoǧlu and Eryiğit2014.

Table 9.

Description of the datasets

The design proposed in this article (i.e., the categorization of the candidate generation task into different components) is inspired by a manual investigation of normalization errors on the validation set. Thus, in order (1) to analyze the distributions of our decided error categories on real data and (2) to be able to evaluate the success of our candidate generation subcomponents on their targeted error types, we automatically categorized the normalization errors in our test sets into different subsets. Since our aim in this automatic categorization is to obtain a rough distribution of error types, the sets are split in a mutually exclusive way, that is, an error is included in only a single set even if it would fall within the scope of more than one category. For example, the word msj is a frequently observed abbreviation for the word mesaj (message) in Web 2.0 and is included in our slang word and initialisms lexicon to be normalized directly through the lookup module, the same error might also be handled by the vowel restoration module. In such cases, the error is only included in one set, namely Set_RR for this example, explained below.

After the manual annotation, both datasets were aligned at sentence level. Therefore, as with the first task, we semiautomatically aligned the datasets at token level and extracted the normalization changes (erroneous input → normalized form). The changes are then automatically categorized into seven categories as following:

1. (1) Replacement rules and lexicon lookup set (Set_RR): In order to extract the normalization errors falling under this category, we select all the inputs existing in our normalization lexicons containing letters outside the Turkish alphabet or multiple occurrences of the same letter either consecutively or as matching our regular expressions such as for catching mentions, hashtags, smileys and vocatives. Although one may argue that the related module would perform excellently with this approach, there exist many cases where the human annotators did not normalize this type of errors as expected. For example, ‘. . . . . .’ has often been manually normalized to a single dot ‘.’, whereas the related module would normalize it to ‘. . .’. And, of course, there are also cases where the related module could not correctly normalize the incoming errors. To give an example, the input olum exists in our slang word and initialisms lexicon to be normalized as oğlum (dude), but it is normalized manually as ölüm (death) within the related context.

2. (2) Letter case transformation set (Set_LCT): In order to extract normalization errors falling under this category, we use both the input word and its manually normalized form. If two words are exactly the same except for letter cases, then the pair is collected under this category.

3. (3) Proper noun detection set (Set_PND): If the manually normalized form contains an apostrophe character whereas the original input does not, or they both contain an apostrophe character but the input is not written in proper noun case, then the sample is included in this set.

4. (4) Diacritization set (Set_DA): In order to extract normalization errors falling under this category, we first check if the original and normalized words have exactly the same length. If the lengths match and the differing letters are only Turkish-specific characters (ı, İ, ş, ö, ç, ğ, ü) and their ASCII-compliant counterparts (i, I, s, o, c, g, u), we select the sample as a diacritization error.

5. (5) Vowel restoration set (Set_VR): Similar to the previous set, we selected the samples where the only differences in words are the lack of vowels (a, e, i, ı, o, ö, u, ü) in the input word.

6. (6) Accent normalization set (Set_AN): All input words ending with the accent suffixes in our list are collected under this set.

7. (7) Rest set (Set_REST): All samples not categorized with the above rules, are collected under the ‘rest’ set. In the individual module evaluation section, we test our spelling corrector on this subset, but one should notice that Set_REST contains the noisiest data where it is really hard to obtain the correct form within short edit distances.

Table 10 provides the obtained distributions on the used datasets. DA type errors have the highest portion in BTS, whereas in IWT, it is the LCT type errors. This is due to the abundance of sentences in IWT whose initial letters are required to undergo case transformation, as explained above. DA type errors still constitute the second highest portion in this dataset. Rest type errors have the third rank in both of the datasets. As expected, VR type error ratios are higher in the Twitter-specific dataset (BTS) than the general Web 2.0 data (IWT). The error distribution in Test _small supports our concern about the inorganic nature of this small dataset having nearly half of its errors in Set_RR.

Table 10.

The distribution of normalization error types

5.2 Evaluation methodology & compared models

Evaluation on the proposed normalization architecture is performed on four separate tiers: (1) the performance of the IWD, (2) the candidate generation performances (under perfect IWD as by Han and Baldwin Reference Han and Baldwin2011) of the individual components on the automatically categorized subsets, (3) the performance of the entire candidate generation stage (with all the modules integrated) and (4) the overall system performance including IWD and candidate generation together. IWD is evaluated by providing the precision (P), recall (R) and F-measure (F). Candidate generation is evaluated by providing the accuracy scores of exact matches with human annotations. In this evaluation, the candidate generation process is applied to the manually normalized whole word set (as if IWD worked perfectly). The system performance is then similarly measured by the accuracy score; but this time the candidate generation process is applied only to the ill-formed words automatically detected by the IWD stage.

Social media text normalization of Turkish is a very recent research area and, to the best of our knowledge, this is the first academic study making in-depth analyses of the problems and providing solutions. Therefore, there is no existing normalization system that can be compared with the proposed model. In order to have an idea about the performances of the available baseline models, we compare with two Turkish spelling correctors and the Giza++ lookup model generated previously by Torunoğlu and Eryiğit Reference Torunoǧlu and Eryiğit2014:

• • MSWord: The first spelling corrector that we compare with is the usage of the MSWord spelling checker Microsoft (2010) as an automatic spelling corrector system. In order to do that, we developed an automatic spelling corrector that makes use of the MSWord API to obtain the first spelling suggestion (assumed to be the best) for a given input token. If there are no available suggestions for an input word then the input is treated as an IV word accepted by the system.

• • Zemberek: Zemberek Akın & Akın (Reference Akın and Akın2007) is an open source library for Turkish language processing. The second spelling corrector, similar to the former, again takes the top candidate from the spelling suggester of Zemberek.

• • GizaLookup: Torunoğlu and Eryiğit Reference Torunoǧlu and Eryiğit2014 use a statistical MT toolkit entitled GIZA++ Och & Ney (Reference Och and Ney2003) in order to automatically align normalization errors and their corrections within a small training set of Twitter posts. The dataset used was actually a small portion (4,049 tweets which were available at that time) of the BTS dataset introduced in this article. The aligned errors and their corrections are then stored in a lookup table to be extracted during the candidate generation stage.

5.3 Evaluation of ill-formed word detection

Table 11 gives the IWD results of the systems compared. As stated previously, the morphological analyzer Eryiğit (Reference Eryiğit2014) used has a very high coverage as might be noticed from the high precision scores (91.74 per cent on Test _small and 92.48 per cent on IWT). When we investigate the false positives (the tokens detected by IWD not manually normalized by human annotators) on all of the three used datasets and especially on the BTS dataset (producing a rather low precision score 75.05 per cent when compared to our other two datasets),Footnote ¹⁸ we see that these are mostly due to the mistakenly left out normalization errors.

Table 11.

Ill-formed word detection evaluation results

As explained previously, for MSWord and Zemberek, each automatically modified word is considered an ill-formed word detected by that system. In order to make a fair comparison, while calculating the performances of MSWord and Zemberek, the special token types explained in Section 4.2.1 (labeled with special labels starting with ‘@’ in our system) are excluded from the evaluations in their favor. The last two lines in Table 11 present the scores of the newly proposed IWD system both by excluding (IWD_@excl ) and including these tokens. In these experiments, we do not provide the results of the ‘GizaLookup’ model since the IWD part of it is exactly the same as that used in the proposed model.

Normalization errors missed by the IWD module are due to the ambiguity caused by the emergence of words that are still valid (i.e., IV words) even after mistyping. Table 12 gives some examples of such real-word errors for Turkish. The ratio of the errors resulting in an actual Turkish word being measured between 8.86 per cent (for IWT) and 16.69 per cent (for Test _small ) is still very low compared to the interval 25–40 per cent reported by Kukich Reference Kukich1992 as the ratio of errors resulting in an actual English word over the entire spelling errors. This may be due to the morphologically rich nature of the Turkish language where affixation may sometimes help decrease the ambiguities Eryiğit & Adalı (Reference Eryiğit and Adalı2004). Although not as severe as in English, more sophisticated models (e.g., a noisy channel model) would be useful for detecting this type of error as in the case of spelling error detection systems.

Table 12.

Different examples of real-word normalization errors

The proposed straightforward approach in this article obtains an F-measure of 82.47 per cent on BTS and 90.41 per cent on IWT. The system is also evaluated on Test _small and obtained an F-score of 87.33 per cent with a 16.33 percentage point improvement on previously reported results (P = 71 per cent, R = 72 per cent, F = 71 per cent) by Torunoğlu and Eryiğit Reference Torunoǧlu and Eryiğit2014. This increase is due to the improved IWD system introduced in Section 4.1 and also to the used morphological analyzer Eryiğit (Reference Eryiğit2014), which has higher coverage compared to the previously used analyzer Şahin, Sulubacak and Eryiğit (Reference Şahin, Sulubacak and Eryiğit2013).

Table 11 reveals that the spelling checker of MSWord performs very close to the proposed model, whereas Zemberek produces very low precision scores by producing a large amount of false positives. The last two lines reveal that the two different evaluations of the proposed model (IWD_@excl and IWD) are very close to each other for BTS and IWT datasets. The difference is large on Test _small since this dataset has a large ratio of specially labeled tokens as given in Section 5.1.

5.4 Evaluation of candidate word generation and integrated system performance

The results of evaluated candidate generation and system performance on OOV words are provided in Table 13 for the overall datasets and in Table 14 (for BTS’s categorized error subsets) and Table 15 (for IWT’s categorized error subsets) for separate modules. In these tables, the column Per-Module Success provides the performances (in terms of accuracy scores) of the individual candidate generation components introduced in Section 4.2 on each related subset (Section 5.1), that is, the Replacement Rules & Lexicon Lookup module on Set_RR, the Letter Case Transformation on Set_LCT, the Accent Normalization module on Set_AN, the Vowel Restoration module on Set_VR, the Diacritization module on Set_DA, the Proper Noun Detection on Set_PND and, finally, the Spelling Correction module on Set_REST. The IWD recall values for each subset are also provided under the column Ill-formed Detection. The performance of the integrated candidate generation system is again evaluated on each subset as well as the entire dataset (the last rows of the tables). The results of this evaluation are provided under the column Candidate Generation (in terms of accuracy scores). In this evaluation, the system is applied directly to the input words without passing them through the IWD filtering in order to see the candidate generation performance alone. The overall system performances together with IWD and CG is then provided under the last columns of the tables as accuracy scores. As stated previously in Section 5.1, since BTS manual annotation differs from IWT in the sense of capitalization and special labeling of vocatives, mentions, keywords, etc., a case insensitive evaluation accepting all the specially labeled tokens as correct is carried out for all evaluations of BTS. For IWT, a stricter evaluation is performed for each step by calculating the exact match of automatic normalization output with human annotation. The system performance for IWT is provided by both the strict evaluation and the lighter evaluation of BTS (System Perf.) in order to compare with the BTS results. The overall system performance (Table 13) is 70.40 per cent on BTS and 67.37 per cent for IWT (77.07 per cent with the lighter evaluation). As expected, the system performance results are lower than the CG alone due to the false negatives of the IWD process.

Table 13.

Overall performances on OOV words

Table 14.

Evaluation results of different modules on big twitter set (BTS)

Table 15.

Evaluation results of different modules on ITU Web treebank (IWT)

When the per-module successes are investigated, the most successful component is the diacritization module, which yields a score of 96.84 per cent on BTS’s Set_DA and 94.46 per cent on Set_DA of IWT. Due to the lighter evaluation of BTS, LCT’s performance is 100 per cent as expected for this dataset. As a Twitter-specific trend, the samples in Set_VR are much higher in BTS than IWT (1,756 compared to 83). The vowel restoration module is observed to be more successful on BTS (84.62 per cent).Footnote ¹⁹ As expected, the spelling corrector performances are very low on the subsets Set_REST containing the most difficult normalization errors. The success of the replacement rule and lexicon lookup module is higher for IWT (85.68 per cent) than for BTS (78.49 per cent) due to the higher number of specially labeled tokens during manual annotation. The performances of accent normalization are similar for both of the datasets (64.49 per cent for BTS and 62.39 per cent for IWT).

The proper nouns related to the telecom brands in BTS are less prone to be confused with Turkish common nouns. That is noticeable from the higher per-module success of proper noun detection on BTS (42.61 per cent) than on IWT (39.57 per cent) as well as the IWD performance of 88.44 per cent for BTS’s Set_PND versus 61.19 per cent for IWT’s Set_PND. We also tested our newly proposed proper noun detection model over the gold standard named entity annotation of IWT Şeker & Eryiğit (Reference Şeker and Eryiğit2017). For measuring the success of the proposed approach, we synthetically created a test dataset from IWT by lowercasing the named entities and then trying to automatically detect and normalize them. As a baseline model, we also tested with a token based simple lookup strategy by replacing the NER recognition layer by a direct lookup from the newly added gazetteers (with threshold approach – Section 4.2.3) and application of the same normalization stages to the words with a recognized lemma. While we obtain an F-measure of 17 per cent with the described baseline model, the precision, recall and F-measure scores of the proposed approach are measured as P= 90 per cent, R=57 per cent and F=70 per cent. The positive impact of the addition of the new gazetteers to the proposed model is reflected a 4 percentage point improvement.

As may be observed from Table 14 and Table 15, integrated candidate generation (column 5) provides an improvement on some of the stand-alone usages of submodules (column 3). These improvements are due to successful collaboration of different modules in solving complex error types as exemplified in Table 8. For example, in BTS, accent normalization module success on Set_AN improves from 64.49 per cent to 74.36 per cent with the help of the other modules. There exist also cases where the performances drop with the integration of the modules. For example, for Set_DA in IWT, the performance drops from 94.46 per cent to 90.76 per cent. This is due to the cases where the previous modules in the cascaded flow (Figure 6) ruin the input before the diacritization module takes action. To give an example, the input token simdi which is manually normalized to the word şimdi (now) is correctly solved by the stand-alone application of the diacritization module. On the other hand, in the integrated model, it is first the vowel restorer which takes the action and converts the input to simidi (simit in accusative case)Footnote ²⁰ which is also a valid Turkish word. Another example is for Set_LCT and Set_PND in BTS. Since Set_LCT contains only letter case usage errors and we make a case-insensitive evaluation for BTS, the per-module success on this set is 100 per cent. The decrease (100 per cent → 77.21 per cent) observed when these words are directly pushed to the candidate generation stage is apparent, since CG will most probably change these inputs by treating them as ill-formed words by default. That is why we see again a slight increase on the system performance in the same line (77.21 per cent → 78.00 per cent) by the addition of IWD which filters out some of the words (100−86.81 = 13.19 per cent) as in-vocabulary words.

As the next step of our investigation, we measure the impact of each candidate generation submodule on the overall system performance. For this investigation, we use an ablation test which measures the overall system performance by leaving out a single CG module one at a time. Table 16 gives the results of these experiments. The first line of the table restates the overall system performance with the entirety of modules in action. The performances resulting from leaving out each module compared with that of the integrated system is given in the following lines. We use McNemar’s paired test in order to measure the statistical significance of the differences given in the table. The results reveal that all of our modules have significant impact on the overall system (with p < 10⁻⁵) except for the letter case transformation module for the IWT dataset whose absence causes a slight increase in the scores which are not statistically significant. The spelling correction, although not performing very well on Set_REST, has a positive impact on the overall system and its absence causes a drop of almost 1 percentage point in IWT and 3.5 in BTS on the performances. The vowel restoration module having the lowest impact still causes a statistically significant decrease (with p < 10⁻⁵) with its absence. The effect is obvious even with case-insensitive evaluation on BTS and is due to the early normalization of proper nouns such as john’dan (example given in Section 4.2.2) in the cascaded flow, in which case the input word is not touched by further modules.

Table 16.

The impact of individual modules on whole system performance

Table 17 gives the comparison of system performances with the introduced baseline systems. As expected, the proposed approach outperforms all the baselines by large margins. It may be noticed from the table that, although MsWord results are very low on Test _small (36.64) and BTS (41.98), the result is much higher for IWT (55.34). This is due to the high proportion of sentence-initial letter corrections in IWT (Table 10 nearly 80 per cent of 2,226 LCT type errors are of this type). Since this type of error is very rare in BTS, the performance remains only at 41.98 per cent on this dataset compared to the 70.40 per cent success of the proposed model. Since the GizaLookup model is composed of a small subset of BTS, its success is higher on this dataset than the others. The success on Test _small is 81.76 per cent, which provides an improvement of 11.28 percentage points over the previously reported results (70.48 per cent) Torunoǧlu & Eryiğit (Reference Torunoǧlu and Eryiğit2014).

Table 17.

Comparison of system performances with related works