1 Introduction
SMS language, also known as texting language, is a new way of communication developed in the last fifteen years as a result of the massive use of electronic communications, mainly by teenagers, all around the world (Power and Power Reference Power and Power2004; Miraflores and Hernández Reference Miraflores and Hernández2006). In Spain only 12,000 million SMS messages are sent per year; in 2011, 3,31,000 million SMS messages were sent in the European Economic Area (data extracted from the ‘European Mobile Industry Observatory 2011’Footnote 1), and besides there exists an incalculable amount of messages written with this kind of language in chat rooms and instant messaging programs. Given the great importance of electronic communication nowadays, there is not much work about SMS normalization (see Aw et al. Reference Aw, Zhang, Xiao and Su2006; Choudhury et al. Reference Choudhury, Saraf, Jain, Mukherjee, Sarkar and Basu2007; or Kobus, Yvon and Damnati Reference Kobus, Yvon and Damnati2008 for French). However, this massive use of SMS language makes important to develop normalization systems that help to process all this information, mainly to adult people that are not familiar with this new kind of language. There exist many applications for which SMS normalization could be really useful, for example, search engines for noisy text documents such as e-mails or blogs. As stated before, SMS language is very common over the web. So given a query ‘skate’, a search engine is expected to return web pages with words like ‘sk8’ or ‘skat’. Other applications such as text correction over the web or text-to-speech systems could be improved by using SMS normalization. Given the wide range of possible applications, it is important to make a system generalizable to other domains not restricted to the electronic communications of young people. To our knowledge, all the methods proposed to date have been focused on that domain. However, it would be interesting to test SMS normalization methods in different domains to test how general is the normalization system. Moreover, it would be interesting to test SMS normalization systems on general text normalization tasks. Both areas are related and the application of techniques of one domain to other can be the source of new ideas and solutions, so both areas can benefit from each other (see Sproat et al. Reference Sproat, Black, Chen, Kumar, Ostendrof and Richards2001 for a good review on general text normalization).
Another motivation to implement this kind of normalization systems is the preservation of native languages. SMS language does not take into account many orthographic and grammatical rules, so the uncontrolled use of this kind of language could lead to the deterioration of original languages. SMS language is not only a marginal and informal variant of the standard language but also a process of language evolution that has the potential to modify the standard language (see Crystal Reference Crystal2001; af Segerstad Reference af Segerstad2002; Baron Reference Baron2004). Finally, the methodology used in the SMS language normalization could be a source of ideas to be used in general machine translation systems, since SMS normalization could be treated as a subproblem of machine translation.
This paper presents a method for the normalization of Spanish SMS language texts into Spanish natural language texts that could be easily adapted to other languages. The method proposed carries out the lexical and semantic disambiguation making use of a combination of lexical resources such as SMS and Spanish dictionaries or the lexical database WordNet. To our knowledge, this method is the first one proposed for Spanish SMS normalization (note that Spanish is the third most used language in the Internet with more than 160 million users). Moreover, this method has two main advantages: It does not rely on annotated corpora, which are difficult to build, making the method easily adaptable to other languages. Also, its ability to perform well in different domains makes it suitable to be used in lots of applications. To our knowledge, our system is the first one that has been tested in a domain different from the electronic communications of young people. The results obtained suggest that our method can be applied not only to pure SMS normalization but also to many other tasks. The main contributions of this work are not only the whole normalization system but also a new phonetic-based distance between SMS words and Spanish words. This distance in combination with a phonetic Spanish dictionary built ad hoc for this application enables the extraction of all the Spanish words phonetically similar to SMS words.
To our knowledge, there exist no Spanish SMS corpora. For many reasons, it is not easy to build an extensive corpus of SMS. SMS are usually kept by private companies that are reluctant to make them available for research purposes, so the main method for collecting SMS data is by voluntary contributions of different users (Delany, Buckley and Greene Reference Delany, Buckley and Greene2012). Given that no Spanish SMS corpus exists, a data set was built with ninety-two SMS messages to evaluate the proposed method. The results obtained show that our method outperforms some of the existing methods of SMS normalization despite special features of the Spanish language and that of the corpus, such as a higher rate of unknown words or a lower Bilingual Evaluation Understudy (BLEU) score (Papinieni et al. Reference Papinieni, Roukos, Ward and Zhu2001) of raw text (a well-known metric that compares the translation given by a system and the real translation), that complicate the normalization task. We also used a second data set to test our method with a large corpus and in a domain different from a typical electronic communications domain. This second data set consists of a Spanish book on translation to SMS language. Our method had a good performance on both data sets suggesting that it might be generalizable to unrestricted domains with minor changes. Again, these results are obtained despite particular features of the data set and that of the Spanish language that complicate normalization.
Next two sections review some previous work and present general features of SMS language. Section 4 discusses the system architecture in detail. Section 5 presents the evaluation process and the results obtained in two experiments. Finally, Section 6 sums up the work and points out some conclusions and future work to improve the proposed method.
2 Related work
Despite the massive use of SMS language in electronic communications and different applications mentioned, there is little work done on SMS normalization. One of the first approaches was based on conceiving SMS normalization as a subtask of machine translation. This is the idea presented by Aw et al. (Reference Aw, Zhang, Xiao and Su2006). They used a standard phrase-based machine translation model with a corpus of 5,000 SMS messages. Their results clearly outperformed the baseline. However, the system did not include any graphemic or phonetic information to estimate the probability of an unseen token. This lack of information led the system to bad performance with out-of-vocabulary words (which are typical of SMS language).
Some other authors consider the SMS normalization task to be similar to the spell-checking task. For example, Choudhury et al. (Reference Choudhury, Saraf, Jain, Mukherjee, Sarkar and Basu2007) and Cook and Stevenson (Reference Cook and Stevenson2009) tried to overcome the problems of machine translation approaches by adding graphemic and phonetic information to the Hidden Markov Model. The idea was inspired on previous noisy channel models that consider that the SMS message is a noisy version of original message. Their works try to measure the probability of finding each word represented by the given SMS word. With this information, they maximize the joint probability to obtain the most probable translation. Also, Guimier de Neef, Debeurme and Park (Reference Guimier de Neef, Debeurme and Park2007) followed the spell-checking approach by using a rule-based system composed of different modules that took into account different SMS linguistic resources. Despite some good results, these approaches usually have problems with word segmentation.
Finally, some other authors used the speech recognition approach based on the observation that SMS language is highly phonetically driven. Following that approach, Kobus et al. (Reference Kobus, Yvon and Damnati2008) proposed a system that turned written words into phonetic symbols and then tried to obtain the natural language words. Also, Beaufort et al. (Reference Beaufort, Roekhaut, Cougnon and Fairon2010) used weighted finite-state machines in a similar way.
Also, the combination of different approaches could lead to improvements in performance. For example, in Kobus et al. (Reference Kobus, Yvon and Damnati2008), a hybrid system built up by combining a machine translation model and a speech recognition model outperforms each of the individual models in terms of word error rate and BLEU score. Also, Yvon (Reference Yvon2010) presents a combination of various approaches based on finite-state transducers inspired by spell-checking and speech-recognition metaphors. Finally, Oliva et al. (Reference Oliva, Serrano, del Castillo, Iglesias, Lozano, Gmez and Moreno2011) present a modular system inspired by various of these approaches. Our method is an extended version of this work, including a more sophisticated disambiguation algorithm and preprocessing techniques. Moreover, we include a more precise evaluation system in order to correctly test the performance of our method, and compare it with the existing state-of-the-art methods.
Most of these works present two main problems that are overcome by our method. First of all, most of them rely heavily on large-scale annotated corpora, which are difficult to obtain, limiting, therefore, their adaptability to other languages or application domains. Moreover, most of these do not exploit contextual information in depth. They try to translate each word without exploiting the syntactic and semantic information of words that surround the target word.
3 SMS language features
SMS texts are not formal texts written with rigor, but a quick kind of communication based on the linguistic economy phenomenon. The main features of SMS language have been extensively studied, not only for English (Baron Reference Baron2000; Herring Reference Herring, Schiffrin, Tannen and Hamilton2001) but also for other languages such as German (Döring Reference Döring2002), Japanese (Nishimura Reference Nishimura2003), or Swedish (af Segerstad Reference af Segerstad2002). A deep study of Spanish SMS language can be found in Ferri (Reference Ferri, Posteguillo, Esteve, Gea, Insa and Renau2005). The main characteristics of SMS language are as follows:
• Phonetic abbreviations and vowel elimination are the phenomena mostly used in SMS language. Phonetic abbreviations are those which keep a similar phonetic structure with the real word: txt - text. The two main methods to build a phonetic abbreviation are the elimination of vowels and the substitution of consonants for phonetically equivalent consonants.
• Non-phonetic abbreviations are also widely used. These abbreviations have no phonetic likeness with the word they refer to. For example, xxx - kisses.
• Blank characters are sometimes omitted after a punctuation mark to save characters. For example, in SMS language can be found: Hi!hw r you?.
• In addition, the orthographic rules (in Spanish or French, particularly the accentuation rules (Simard and Deslauriers Reference Simard and Deslauriers2001)) are often not taken into account.
Based on the conclusions obtained by Ferri (Reference Ferri, Posteguillo, Esteve, Gea, Insa and Renau2005), the following two hypotheses are assumed by our method:
(1) A SMS phonetic abbreviation always presents the same consonants, or consonants phonetically equivalent, as the referred Spanish word (the only exception is letter ‘h’, which has no phonetic transcription). This seems to be a very logical assumption because the elimination of a consonant would lead to a very different word in terms of phonetics.
(2) The vowels that appear in an SMS phonetic abbreviation are always in the same order in the referred Spanish word. This is also a very logical assumption, since it has no sense to add a vowel that is not in the original word or to add it in a different position.
4 SMS normalizer: system architecture
In order to overcome the limitations of different approaches pointed in the previous section, a new hybrid method is proposed. This method combines the accuracy of spell-and-speech-recognition approaches in the dealing of out-of-vocabulary words, with the lexical and semantic information commonly used by machine translation approaches.
As stated above, SMS normalization requires the processing of special symbols and phonetic abbreviations, and the disambiguation at two main levels. In order to solve these problems, a system composed of three modules is proposed (see Figure 1): Preprocessing, translation, and disambiguation modules. In order to show the way the proposed translation method works, a Spanish sample sentence will be used: Pues tiene dos teléfonos móviles (in English: So he has two mobile phones) written in SMS language as: Pues tiene 2telfs mvls.
System modular architecture.
4.1 Preprocessing module
As stated above, SMS language presents many deviations from natural language. Besides orthographic and phonetic phenomena, there are some features that our method should address first. One of these features is the possible absence of blanks after a punctuation mark or number because of space limitations that are imposed by SMS. A maximum of 160 characters per message in mobile SMS are usually allowed. So for example, tokens like 2telfs or hola.como ests? could be found. Therefore, any normalization system has to correctly tokenize the SMS text, which implies dividing the tokens consisting of letters and signs and upper-casing the words after a dot sign. However, SMS language can present abbreviations and emoticons, composed of letters and signs, which should not be split to be translated correctly. So before splitting a token consisting of letters and punctuation marks or numbers, the module tries to translate it by using a Spanish SMS dictionary with more than 11,000 entries, provided by the Asociación Española de Usuarios de Internet (Spanish Internet Users Association). If the word is in the SMS dictionary, this module stores the possible translations, otherwise this module tokenizes it obtaining the letters’ token (or tokens) and the symbols’ token (or tokens).
In the example presented, the preprocessing module would find that the word 2telfs is formed by numbers and letters. So after trying to translate it using an SMS dictionary without success, the module would break the single word and it would outcome two tokens: 2 and telfs.
4.2 Translation module
The objective of the translation module is to find all possible translations of SMS words. The task cannot be accomplished by a simple dictionary look-up strategy, since SMS language presents three different kinds of words: phonetic abbreviations, which are obtained by removing some vowels of the original word like mvls - móviles, non-phonetic abbreviations such as xa - para, and real words. The translation module uses an SMS dictionary to deal with non-phonetic abbreviations, and a Spanish phonetic dictionary to deal with phonetic abbreviations and real words. Each SMS word is translated using the SMS dictionary and the Spanish phonetic dictionary, obtaining the output of this module as the union of lists of possible translations extracted from the SMS and the Spanish phonetic dictionary (see Figure 2). For example, suppose the word se is found in the SMS text. The translation module would find the entry se - ese in the SMS dictionary. Also, the word se is found in the Spanish phonetic dictionary because se is a real Spanish word. And the word sé is also found in the Spanish phonetic dictionary and, given that it has a high similarity with the SMS word se, it would be returned as a possible translation. So the output given by the translation module when dealing with the SMS word se would be the list: ese, se, sé.
Translation module.
4.2.1 SMS dictionary
The translation module tries to find each word in the SMS dictionary, which stores non-phonetic abbreviations. If the word is found there, all the possible translations are stored. As stated above, an SMS word could be similar to an SMS abbreviation and to a real Spanish word. So even if the SMS word is found in the SMS dictionary, the translation module has to check whether it is similar to a real Spanish word. Therefore, the module uses the Spanish phonetic dictionary explained in the following point.
4.2.2 Spanish phonetic dictionary
Phonetic abbreviations are translated by a Spanish phonetic dictionary built specifically for this task. The use of such a phonetic dictionary in combination with a new similarity metric proposed further is one of the main contributions presented in this paper. It is important to note that these two contributions could be useful not only in the SMS normalization task but also in many different applications.
This phonetic dictionary could be very useful in SMS language processing, since the continuous evolution of this kind of language and the absence of a fixed vocabulary makes impossible to simply trust on an SMS dictionary as the one described before. The dictionary consists of a special code that reflects phonetic uses for each consonant in Spanish SMS language and a string storing the vowels and their positions in the word. The consonant code groups the consonants that can be used indistinctly in Spanish SMS language because of their phonetic similarity (such as B and V). These phonetic groups are B-V, C-Q-K, G-J-W, R-’RR’, X-’CH’, and Y-’LL’. Each entry of the dictionary is formed by a consonant string, composed of codes of each consonant, and a vowels string composed of vowels and their positions in the Spanish word. For example, the entry of the word teléfonos (in English: phones) in the Spanish phonetic dictionary would be formed by the strings: ‘tlfns’ and ‘0E0é0O0O0’. The vowels string is built up with the vowels of the word and the character ‘0’ substituting the consonants to preserve the position of vowels in the word. Note that accents are preserved in the dictionary. Since this phonetic similarity depends obviously on the language, this would be one of the issues to be modified in order to adapt the proposed method to other languages.
According to two hypotheses presented in Section 3, the translation process works as follows: Given an SMS word, first of all the word is searched in a Spanish phonetic dictionary to get the entries that have the same coded consonant string. Then for each of the entries, the translation module computes the similarity of the associated vowel strings.
The similarity between vowel strings is computed by using a new metric proposed here as a modification of the Levenshtein (Reference Levenshtein1966) distance inspired by the Double-Metaphone algorithm (Philips 2004). The Levenshtein distance gives a cost to the operations of insertion, elimination, or substitution of characters. By taking into account the two hypotheses explained above, no consonants or vowels could be removed from the SMS word, so the elimination operation is not allowed. Insertion of a vowel has a cost of 0.004 but the substitution of a character has a much higher cost (0.02) because of the assumption that vowels are always in the correct position. Only the substitution of a vowel for the same vowel with an accent supposes an exception and it is weighted with 0.001. This is a consequence of the mentioned fact that SMS language does not take accentuation rules into account. Once the similarity for each word has been computed, the algorithm selects among all possible translations those words with a similarity higher than a threshold of 0.98. The cost values for each operation and the threshold have been fixed empirically to maximize the accuracy of the translation. We used for this setting a subset of twenty messages not included in the evaluation data set.
The output of the translation module would be a list of sets of words in which each set represents the possible translations of the corresponding word in the original sentence. Following our example, we would get the results as shown in Table 1.
Input and output of the translation module; the corresponding English words are in parentheses
Note the three kinds of words we talked about at the beginning of this section: non-phonetic abbreviations, translated using an SMS dictionary, such as 2 - Dos (in English: two) and 2 - Tú (in English: you); phonetic abbreviations, translated using a Spanish phonetic dictionary, such as Mvls - Móviles (in English: mobile), Mvls - amables (in English: kind), Mvls - muebles (in English: furniture), and real words such as tiene (in English: have). There are also words such as pues (in English: so) that could be an abbreviation or a real word.
4.3 Disambiguation module
The goal of this module is to choose among the possible translations of each word the most suitable option. The input to this module are the lists of possible translations of each word, and the expected output is the final sentence. SMS normalization systems have to deal with ambiguity. The following examples adapted from Rosetta (Reference Rosetta1994) illustrate the problem.
(1) Jane will coach our team.
(2) The two teams were trained by the same coach.
(3) The coach was heading for the station.
The key to find the correct meaning of the word coach is in its context. The presence of word team, in the second sentence, or station, in the third one, was determining to choose the correct meaning of the word coach. So this module carries out a semantic disambiguation process calculating semantic similarities of each pair of possible translations of the words in the context of the ambiguous word. However, the ambiguity of some words is sometimes so high that the efficiency of the method is really harmed. In order to avoid this, a lexical disambiguation step precedes the semantic disambiguation, reducing significantly the ambiguity. The lexical disambiguation process lies in a shallow parsing algorithm, which computes the most probable sequence of part-of-speech (POS) tags, eliminating this way many possible translations. With this sequence of lexical and semantic disambiguation, the module gives the final result of the translation.
4.3.1 Lexical disambiguation
In order to do the lexical disambiguation task, the system uses the open source suite of language analyzers ‘Freeling 2’Footnote 2 (Carreras et al. Reference Carreras, Chao, Padró and Padró2004) with some modifications to deal with the owned particular problem. The input of this module consists of a sentence that does not have a single word in each position but a list of possible words. Each list is composed of all possible translations for each word received from the translation module. For the list associated to each position in the sentence, the module stores all the possible POS tags using the dictionary provided by Freeling and also uses Freeling to determine the most probable combination of these POS tags.
Once Freeling obtains the selected POS tag for each position, the disambiguation module selects the words of the list for each position that have this POS tag among its possible POS tags. This way the ambiguity is reduced making easier the semantic disambiguation task or even removing this ambiguity when there is only one word that matches the selected POS tag.
The process followed by this submodule can be seen in Table 2. The output of this submodule is formed by the words whose POS tag is the same as the selected POS tag given by Freeling. Note that the lexical disambiguation process has two positive effects: at the very best, lexical disambiguation is enough to completely eliminate ambiguity (see first and third words), and also in those cases if ambiguity still remains, the number of options is reduced (in our example, muebles (furniture) is removed from the possible translation list), making easier the semantic disambiguation task.
Lexical disambiguation process; the corresponding English words are in parentheses
* These abbreviations mean the following: V – verb, SC – subordinate conjunction, DT – determiner, PN – pronoun, N – noun, and ADJ – adjective.
4.3.2 Semantic disambiguation
The semantic disambiguation module uses WordNet 2.1Footnote 3 (Fellbaum Reference Fellbaum1998), which, because of its hierarchical structure and the existence of term definitions in each sense, enables the design and use of many kinds of semantic similarity measures between words. In order to access the WordNet, we used JWordNet-SimilarityFootnote 4, a Java interface that implements a variety of semantic similarity and relatedness measures, including the one used by the system proposed here, the Extended Gloss Overlap Measure (Banerjee and Pedersen Reference Banerjee and Pedersen2003) based on information placed in the WordNet. Besides, we used the Spanish WordNet-based semantic information provided by FreeLing to construct an English–Spanish dictionary based on semantics. FreeLing provides for each Spanish word its corresponding English WordNet synsets (extracted from EuroWordNetFootnote 5), so the English–Spanish dictionary was made up by inversely translating the synsets for its associated words in the WordNet. This kind of translation is supposed to be very suitable to our task (semantic disambiguation) because the semantic WordNet codes are the key issues for translation.
The semantic disambiguation module uses a novel adaptation of the Maximum Relatedness Disambiguation (MRD) algorithm (Pedersen, Banerjee and Patwardhan Reference Pedersen, Banerjee and Patwardhan2005) to make disambiguation among possible translations instead of possible senses. The MRD algorithm uses a measure of semantic relatedness to get a score for each sense of the target word (i.e., the word to be disambiguated) based on the relatedness score obtained by each pair of senses of this word and the words in a context window. The adaptation presented here gets a score for each possible word (remember that the problem is not sense disambiguation but word disambiguation) taking the most related pairs of sensesFootnote 6 of the target word and each one of the words in the context window.
The algorithm processes the input sentence word by word from left to right. When the first ambiguous word (target word) is found, the context window is built. This window is formed by the words placed just before and after the target word present in the WordNet (nouns, verbs, adjectives, or adverbs). The window size used in our system was 3, which included the target word and one word to its left and right, following the claim done by Michelizzi (Reference Michelizzi2005) that words farther away from the target word are less likely to be related to words close to the target word. Once the context window is built, the algorithm takes each possible translation of the target word and computes the relatedness of all its senses with all the senses of all the words in the context window. Note that words to the left in the context window cannot be ambiguous either because they are not ambiguous or because they have already been disambiguated. However, words to the right could be ambiguous, so the algorithm has to compute relatedness with all possible words. For each word in the context window, the algorithm gets the most related sense-pair and computes the score of each of the possible translations of the target word as the sum of the scores of the most related sense-pairs. Finally, the algorithm selects the option with the highest score.
In our example, the unique ambiguous word is the last one, so when the disambiguation algorithm reaches this word, the context window is built. The context window only included the target word and the word Teléfonos. Note that if the target word is at the beginning or the end of the sentence, the context window is not expanded to contain three words. Then the algorithm would compare the words Móviles and Amables to the word Teléfonos, measuring the similarity for each sense-pair. Given that Móviles has one unique sense (mobile) in the English–Spanish dictionary and Teléfonos has four senses (phone, telephone, telephony, and number), the algorithm would measure the similarity of mobile - phone, mobile - telephone, mobile - telephony, and mobile - number to get the similarity of the words Móviles and Teléfonos as the maximum of the similarities of each sense-pair. In the same way, the algorithm would get the similarity between Amables and Teléfonos, and would choose the word with the highest similarity. In this case, as shown in Table 3, Móviles would get a similarity score of 2.89, while Amable would get a similarity score of 2.0. So Móviles would be selected, getting in this way the real translation: Pues tiene dos teléfonos móviles.
Semantic similarity of sense-pairs of words Teléfono and Amable, and Móvil
5 Evaluation
In order to get a deep evaluation of the proposed system, we studied its global performance and the particular performances of each of the modules. This way we can identify the points that should be improved. The evaluation process works as follows: First of all, when each of the modules finishes its work, the output (i.e., the list of possible translations for each word) is stored. When the translation is done, for each word, if the real word is not included in the list of possible translations given by the module, the number of errors of the corresponding module is increased. When an error is detected on a step, the evaluation of next steps is not done because the next modules will obviously fail. Suppose that in our example, the translation module tries to translate the word mvls and the list of possible translations given by this module is muebles, amables, and the correct translation is móviles. Since the correct translation is not included in the list of possible translations, it makes no sense to evaluate the disambiguation module because it would try to find the correct word in a list of words that do not contain the correct word. To obtain the BLEU scores shown in Tables 5 and 7, we select the first option of the possible translation lists generated by each module.
To evaluate the method proposed, we used the word error rate. Also, BLEU metric is given to allow comparisons with other similar studies like the ones proposed by Aw et al. (Reference Aw, Zhang, Xiao and Su2006), Guimier de Neef et al. (Reference Guimier de Neef, Debeurme and Park2007), and Kobus et al. (Reference Kobus, Yvon and Damnati2008). In addition to word error rate, we used the sentence error rate, which provides information about the distribution of errors among sentences.
We used two data sets to test the system. The first one consists of ninety-two messages written by undergraduate students. Despite the little size of the dataset, it covers all the main features of an SMS language and reflects how young people use it. In order to test our system with a larger dataset, we also used a Spanish book translated to SMS language. With this dataset, we checked whether the conclusions obtained in the first experiment were generalizable to larger sentences and a different kind of language style.
5.1 Experiment 1
5.1.1 Data set
As stated in the Introduction, it is not easy to build an extensive corpus of SMS. Private companies are reluctant to make SMS messages available for research purposes, so the main method for collecting SMS data is by voluntary contributions of different users (Delany et al. Reference Delany, Buckley and Greene2012). Hence, there are not many studies about SMS normalization, particularly in Spanish, so there is not a suitable benchmark data set for the evaluation of Spanish SMS normalization systems. In spite of the existing of a project to study SMS language all over Europe (see www.sms4science.org), only a great corpus of 30,000 normalized French SMS (Fairon and Paumier Reference Fairon and Paumier2006) and a corpus of around 900 normalized English SMS (Choudhury et al. Reference Choudhury, Saraf, Jain, Mukherjee, Sarkar and Basu2007) were available. Thus, a small data set was built with messages written by undergraduate students of the University of Extremadura to the magazine Extremadura Universa.Footnote 7 The SMS collected were sent to the magazine by ninety-two undergraduate students between January 2003 and March 2008, and all of them were anonymous.
The ninety-two messages included in the corpus were selected by their high density of words that must be translated and disambiguated, resulting in more than 1,500 words to translate and more than 650 to be disambiguated.
Table 4 shows some relevant statistics of the corpus used to evaluate the method. Unknown tokens are those that do not correspond to a Spanish word, and ambiguous tokens are those that have more than one possible translation. The number of ambiguous tokens is an important measure that complements the number of unknown tokens. If the number of unknown tokens is very high but they have only one possible translation, a simple dictionary-based method could obtain great results. Also, BLEU 3-gram score of raw text is given to compare the a priori difficulty of the task. It is important to note that the corpus used to evaluate our method seems to have some features that make the task more complicated than the one faced in other related papers. The corpus used by Kobus et al. (Reference Kobus, Yvon and Damnati2008) presents only 32.7% of unknown words, which is much lower than 78.71% of our corpus, making easier the normalization task. Also, Aw et al. (Reference Aw, Zhang, Xiao and Su2006) use a corpus with an initial BLEU 3-gram score of 0.5784, which shows that the source messages are much more similar to the actual translations than the ones used in this paper. Moreover, the Spanish language presents some additional features such as conjugated verbs and different word forms to distinguish different genders. These features are not present in English and, as we will see in the Discussion section, they add some difficulty to the normalization process. So this fact should be taken into account in the comparison of this method with the methods designed to work for English.
Statistics of the evaluation corpus
5.1.2 Results and discussion
Table 5 shows the overall precision of the method. Moreover, we show the performance of each of the modules and submodules in order to find the strengths and weaknesses of our algorithm. As a baseline experiment, we consider a simple dictionary-based system. The baseline system preprocesses and translates the input words in the same way as the preprocessing and translation modules presented in this paper. However, for obtaining the final normalization of each message, the baseline system takes the first of the words in the possible translations list of each token. The baseline results show a word error rate of 0.6481 and a BLEU score of 0.4694. Note that the word error rate of this baseline system is not the same as the one obtained by the preprocessing and translation modules of our system (shown in the first row of Table 5) because the baseline system obtains the resulting message using the first word in the possible translation list of each token in the original message.
Number of words processed, errors, word error rate, and BLEU score of each module and the whole system
The sentence error rate obtained by the system is 0.6087, i.e., fifty-six sentences of the total ninety-two sentences contain an error. This value is much better than the 0.75 reported by Guimier de Neef et al. (Reference Guimier de Neef, Debeurme and Park2007) and is similar to the approximate value of 0.60 reported by Kobus et al. (Reference Kobus, Yvon and Damnati2008). As we have mentioned above, errors in early steps of the normalization process make new errors in next steps.
Table 5 shows good and promising results of the normalization system proposed here. The global BLEU score of 0.8054 is similar to the score of 0.8070 reported by Aw et al. (Reference Aw, Zhang, Xiao and Su2006) for English, and the approximately 0.8 reported by Kobus et al. (Reference Kobus, Yvon and Damnati2008) for French. Furthermore, our system outperformed the method proposed by Guimier de Neef et al. (Reference Guimier de Neef, Debeurme and Park2007), which obtains a BLEU score of 0.68. It is important to note that these results are obtained with a corpus with some features, such as a higher rate of unknown words or a lower BLEU score of raw text, that complicate the normalization task. However, these results can be improved by making a deeper analysis of the reasons behind the errors done by each module.
Errors in preprocessing and translation are mainly due to the words that are not contained in dictionaries, such as diminutives, nicknames, or proper nouns, which are commonly used in SMS language. The problem with diminutives could be eliminated by using dictionaries that include diminutive forms or by some method capable of detecting these forms. For example, a list of suffixes that denote a diminutive form and its corresponding POS could be used. If the SMS word ends with a suffix similar to the one in the list, then the translation module should add the diminutive form to the possible translations list and take into account the corresponding POS in the lexical disambiguation stage. Regarding proper nouns, maybe an entity recognition algorithm could detect the words related to a person or place and so improve the translation of this kind of words. The evaluation of the preprocessing module also showed a feature that is sometimes used in SMS: Some users usually write two words together when this union is similar to the way they pronounce these two words. For example, some users write menknta, whose translation is me encanta (in English: I love it) and the preprocessing module proposed is not able to divide this word correctly. The problem is that a dictionary cannot contain all possible unions that users can do, so this phenomenon should be handled with some kind of phonetic preprocessing module that divides words attending to phonetics. Diminutives and proper nouns cause a total of twenty-three errors, which means that 20.54% of errors made by the preprocessing and translation modules are 10.41% of overall errors. However, errors caused by the contraction of two words are supposed to be only 1.88% of overall errors.
Lexical disambiguation obtains great results. Observing the errors done by this module, it seems that this is not a critical point to improve. Mainly, errors are detected in the disambiguation of words that have some previous or posterior words wrongly translated, leading the disambiguation to fail.
Errors in semantic disambiguation are due to two main reasons. Sometimes, messages contain very short sentences, in which the context window for disambiguation is impossible to be built (remember that the context window is built with the words of the same sentence of the target word), so the word cannot be disambiguated. Maybe, the construction of the context window with words of near sentences could improve disambiguation. The other main source of errors in the semantic disambiguation module is only present in languages, like Spanish, that present conjugated verbs. Typically, many verb tenses share a very similar form, so in many cases, the semantic disambiguation module has to deal with many forms of the same verb. See, for example, the tenses paro, para, pare, paré . . . all of them being different forms of the Spanish verb parar (in English: to stop). Since the WordNet stores only the infinitive form of each verb, the disambiguation module has no way to choose the correct tense and the output is the first word of the list. This way there are many errors made by the semantic disambiguation module that are not due to a problem of the proposed disambiguation method but to a lack of another algorithm capable of disambiguating between different forms of the same verb. In fact, in many cases it is not possible to obtain the correct tense of the verb because there is no information in the message about the correct tense or person. Also, many errors of this module are present only in languages, like Spanish, that have different forms for different genders of a word. Sometimes this error is impossible to solve because there is no information in the sentence to disambiguate the gender of a word. Suppose we have the following message: Xa mi xic wap, in Spanish: Para mi chico guapo (In English: For my beautiful boy). The SMS words xic and wap could be translated as xic - chico, chica and wap - guapo, guapa, and in order to know the correct gender of these words, the algorithm should know the gender of the recipient of the message.
The total number of errors caused by a bad conjugation of a verb is fourteen, which is supposed to be 29.17% of the errors made by this module, and 6.51% of the total errors made by the system. Talking about gender, the proportion of errors caused by this problem is supposed to be 35.42% of the errors made by the semantic disambiguation module and 7.91% of the overall errors of the system. It is important to remark that it is impossible to correct some of these errors because the election of the correct form depends on external circumstances that are not contained in the message (for example, the gender of the recipient of the message). Also, it is important to note that these problems are only present in certain languages such as Spanish or French, but not in English. So this fact should be taken into account in the comparison of this method with methods designed to work with English.
5.2 Experiment 2
5.2.1 Data set
The data set used in this experiment is the book En patera y haciendo agua, a bilingual book (Spanish–SMS language) that can be found online at http://www.adiccionesdigitales.es/libro. The only flaw in the data set used in the previous experiment was its small size. Therefore, we used this second data set to prove our method on a larger data set (29,597 tokens) that keeps the difficulty of normalization of the first data set.
It is important to note that the SMS input text is the normalization (by a teenager) of the Spanish book. Therefore, the language used in this data set is different from the one used by teenagers in everyday electronic communications. So besides its size, this data set helps us to check the generalization of the normalization system to other domains different from the typical ‘young people communication’ domain.
Table 6 shows some relevant statistics of the corpus used to evaluate the method. Like in the previous experiment, unknown tokens are those that do not correspond to a Spanish word, and ambiguous tokens are those that have more than one possible translation. Also, BLEU 3-gram score of the raw text is given to compare the a priori difficulty of the task. Once again it is important to note that the corpus used to evaluate our method seems to have some features that make the task more complicated than the one faced in other related papers. The 82.29% of unknown words is much higher than the 32.7% of the corpus used by Kobus et al. (Reference Kobus, Yvon and Damnati2008). And the 0.09 BLEU 3-gram score of this raw text is much lower than the 0.5784 score of the corpus used by Aw et al. (Reference Aw, Zhang, Xiao and Su2006). Also, as mentioned above, in Spanish there are some additional features such as conjugated verbs and different word forms to distinguish different genders. These features are not present in English and, as we will see in the discussion section, they add some difficulty to the normalization process. So this fact should be taken into account in the comparison of this method with methods designed to work with English.
Statistics of the evaluation corpus
5.2.2 Results and discussion
The results obtained by each of the modules of the normalization system proposed, and the overall precision of the method are shown in Table 7. We used the same baseline of Experiment 1, a simple dictionary-based system. It is similar to the preprocessing and translation module of our system but it takes the first possible translation as the result of the translation process. The baseline results show a word error rate of 0.7881 and a BLEU score of 0.5162. These results are significantly different from the ones obtained by the overall system.
Number of words processed, errors, word error rate, and BLEU score of each module and the whole system
The performance of our system with this data set is poorer than the one obtained in Experiment 1. The features of the two data sets are similar. However, the vocabulary and expressions present on this data set are quite far from the ones used by young people in SMS language. Despite this fact, the performance is still quite good and shows that our method can be straightforwardly used to deal with SMS language in an unrestricted domain.
These good results are also confirmed when comparing with other similar methods. The 0.75 BLEU score is better than the 0.68 score presented by Guimier de Neef et al. (Reference Guimier de Neef, Debeurme and Park2007) and is slightly worse than the scores presented by Aw et al. (Reference Aw, Zhang, Xiao and Su2006) for English, and by Kobus et al. (Reference Kobus, Yvon and Damnati2008) for French. It is important to note the extra difficulties that imply the normalization of this data set. Like the one used in the previous experiment, this data set has some features, such as a higher rate of unknown words or a lower BLEU score of raw text, that complicate the normalization task. Moreover, as stated before, the vocabulary present in this data set is not a typical one for electronic communications.
In general, the results of the overall system and each of the different modules are similar to the ones reported in the previous experiment. The preprocessing module presents the same kind of errors. Proper nouns, diminutives, and conjugated verbs are not present in the Spanish dictionary, so they cannot be translated. These errors are 11.36% of the total number of errors. We discussed some possible solutions in Section 5.1.2. In addition to these out-of-vocabulary words, this data set presents some technical words that are also out of scope of a dictionary. This fact was not a problem in the previous experiment, since young people use very frequent words and expressions. However, in order to apply this method to a very specific domain, it should be considered to increase the vocabulary with technical words of that particular domain. Besides these errors, there is an extra source of errors in this data set. The SMS dictionary is not used with the same frequency as in the previous experiment. The SMS dictionary is made by users with the expressions that they frequently use in mobile phone communications. As mentioned above, this vocabulary and expressions are very different from the ones found in the formal language of a text book. Therefore, a lot of words that should be translated using the SMS dictionary are not translated, thereby increasing the number of errors in this module.
The lexical disambiguation module also obtains similar results. As stated before, errors in this module are mainly due to words that have some previous or posterior words wrongly translated, leading the disambiguation to fail.
The semantic disambiguation module presents a similar performance to the one of the previous experiment. The only difference in the performance is due to non-translated words caused by fails in the translation module. The rest of the errors (up to 18.02% of the total number of errors) have the same reasons as stated before: very short sentences, in which the context window for disambiguation is impossible to be built, and verbs and nouns marked for time, aspect, gender, etc. There are two important points to mark here. First of all, some of these errors are only present in languages like Spanish that present explicit markers for mood, time and aspect in verbs, or gender and number in nouns and adjectives. This fact should be taken into account when comparing with methods for other languages that do not present these features (like English). Also, it is important to note that in many cases these errors are impossible to solve because there is no information in the sentence to disambiguate those features (for example, sometimes it would be necessary to know the gender of the recipient of the message).
To sum up, our method obtained good results with a large data set in a domain different from a typical electronic communications domain. These good results point that our method is generalizable to unrestricted domains. It is important to note that to our knowledge this method is the first one that has shown good performance in an unrestricted domain. As stated in the Introduction, the generalization of a system is of great importance, given the wide range of potential applications of these systems.
6 Conclusions and future work
In this paper we presented a new method for SMS normalization. The method proposed here deals with the normalization of Spanish messages, but one of its main characteristics is easy adaptability to other languages. The system is based on a three-module architecture built up by a preprocessing module, a translation module, and a disambiguation module. The preprocessing module divides the words correctly, taking into account some special characteristics of SMS language such as the possible absence of blanks. The translation module tries to obtain all possible translations of each word, thereby making use of two different dictionaries to deal with phonetic and non-phonetic abbreviations. Finally, the disambiguation module chooses the correct translation for each word between all possible translations given by the translation module. In order to make this choice, the disambiguation module carries out processes of lexical disambiguation and semantic disambiguation, taking into account context information and semantic knowledge stored in the WordNet. The main features of the proposed method are as follows. First of all, to our knowledge, this method is the first one proposed for Spanish SMS normalization (note that Spanish is the third most used language in the Internet with more than 160 million users). Despite the great importance of Spanish in electronic communications, the method is easily adaptable to other languages as it does not depend on great annotated corpora. And finally, it showed good results on two different domains showing its easiness of adaptation across domains. Moreover, we presented a novel phonetic-based similarity measure that in combination with a Spanish phonetic dictionary could be used in many other tasks that require word phonetic processing.
In order to evaluate the proposed method, we used two data sets. The first data set was made of ninety-two selected messages written by undergraduate students. This data set contains more than 1,500 words to translate and more than 650 to be disambiguated. In the evaluation of the system, not only the overall performance was measured but the performance of each module was also evaluated. The system proposed obtains an overall 0.195 word error rate, a 0.61 sentence error rate, and a 0.8054 BLEU 3-gram score. These results outperform some of the existing methods of SMS normalization despite special features of the Spanish language and that of the corpus, such as a higher rate of unknown words or a lower BLEU score of raw text, which complicate the normalization task. The second data set consists of a book written in Spanish and SMS language. This data set allows us to test our method with a large data set and in a domain different from a typical electronic communications domain. The results show a good overall performance of the system and also of each of the particular modules. These good results show that our method is generalizable to unrestricted domains with minor changes. It is important to note that, to our knowledge, no other similar method has been tested in an unrestricted domain different from the electronic communications domain. As stated in the Introduction, the generalization of a system to different domains is of great importance, given the wide range of potential applications of these systems. Again, these results are obtained despite the particular features of the data set and the Spanish language that complicates the normalization.
Future work includes improvement of each of the modules and submodules of the system. The preprocessing module could be improved by adding an algorithm to divide words that use phonetic clues. The translation module fails when dealing with diminutives and proper nouns. Thus, using dictionaries that include this kind of words or some algorithms capable of detecting these words (for example, named entity recognition algorithms, to deal with proper nouns) would improve the performance of this module. Finally, the disambiguation module could be improved by adding an algorithm capable of disambiguating between different conjugated forms of the same verb. In order to increase the performance of the method in any particular thematic domain, it would be useful to test the method in different domains with specialized dictionaries.
Given the absence of suitable benchmark data sets for the evaluation of Spanish SMS and many other languages, it would be really useful to construct a bigger corpus of Spanish messages with its actual normalizations so that the evaluation and comparison with other methods could be standardized. Also, the adaptation of the system to other languages is another natural trend. This adaptation would allow a more suitable comparison with the related state-of-the-art methods.
