1 Introduction
The Web 2.0 has become a channel where users exchange, explain, or write about their lives and interests, give opinions and comments on other people’s opinions, most of the time using a casual language. This language often presents peculiarities that make it much closer to transcriptions of oral language than to standard edited text. Opinion mining techniques, just to mention an example, are becoming an important source of information for market research, social network analysis, or learning analytics (Pang and Lee Reference Pang and Lee2008). In order to mine data or extract information from Web 2.0 we first need to understand its contents. Shortened or misspelled words, which are frequent in the social media informal style, increase the ambiguity and interpretation possibilities for the same word form, and pose a challenge to the achievement of certain Natural Language Processing (NLP) analysis tasks such as tokenization, part-of-speech (PoS) tagging, or Named Entity Recognition, as reflected in the literature (Foster et al. Reference Foster, Wagner, Roux, Hogan, Nivre, Hogan and Van Genabith2011; Aminian et al. Reference Aminian, Avontuur, Azar, Balemans, Elshof, Newell, van Noord, Ntavelos, van Zaanen and Melero2012; Maynard, Bontcheva and Rout Reference Maynard, Bontcheva, Rout and Melero2012; Muñoz-García and Navarro Reference Muñoz-García, Navarro and Melero2012).
Natural Language Processing techniques, which are used to provide linguistic representations from unstructured data, are typically developed to deal with standard language and may not yield the expected results on User-Generated Content (UGC) text. As we discuss later in this section, several approaches have been tried to tackle this problem by adapting either the tools to the text or else the text to the existing tools through a process generally known as ‘text normalization’.
The structure of the paper is as follows. We start by explaining our motivation and reviewing the related works. We then present our findings on what characterizes UGC text in Spanish, based on a corpus study. We then explore the problems caused by UGC text to the performance of a pre-existing NLP tool for the analysis of Spanish text by comparing the results of parsing two versions of the same UGC text: as-is and manually corrected. Finally, we present and evaluate an approach to text normalization that uses a module for the automatic selection of correction candidates; this module is built on top of a pre-existing spell-checker.
1.1 Motivation
Pang and Lee (Reference Pang and Lee2008) show the relevance that the analysis of UGC text has for the analysis of product and service reputation and political trends. While the processing of large amounts of UGC data on the basis of frequencies of (mainly) words or correlated structured data (e.g. the use of star-based recommendation scales) have successfully characterized global opinion and reputation trends, depending on the granularity of the task at hand a more fine-grained characterization is required (e.g. Pang and Lee Reference Pang and Lee2008; Gianfortoni et al. Reference Gianfortoni, Adamson and Rosé2011). The nonstandard characteristics of UGC text seem to be one of the main obstacles and challenges in this respect (Ritter et al. Reference Ritter, Clark, Mausam and Etzioni2011; Maynard et al. Reference Maynard, Bontcheva, Rout and Melero2012; Muñoz-García and Navarro Reference Muñoz-García, Navarro and Melero2012).
It is often said that UGC text, as found in social media sites, resembles more oral language than regular edited text, such as news – which is the kind of text on and for which most NLP tools have been trained. However, studies of computer-mediated communication in the field of descriptive linguistics suggest that social media text is a language mode in itself (Herring Reference Herring, Tannen and Tester2012a). The linguistic characterization of UGC text is a research line that might provide interesting insights into the automatic analysis of UGC (see Bender et al. Reference Bender, Morgan, Oxley, Zachry, Hutchinson, Marin, Zhang and Ostendorf2011; Gouws et al. Reference Gouws, Metzler, Cai and Hovy2011; or Eisenstein Reference Eisenstein2013).
Researchers in opinion mining and sentiment analysis often attribute the inaccuracies of their NLP-based systems to the lower performance of the standard NLP tools when applied to noisy, or ill-formed, language – though Bermingham and Smeaton (Reference Bermingham and Smeaton2010) argue that text length and noise might be less of a disadvantage when performing sentiment analysis on Twitter texts. Several studies have investigated the effect of UGC characteristics on the performance of standard NLP tools. Some have found that UGC text characteristics affect the performance of tasks such as topic detection (Muñoz-García and Navarro Reference Muñoz-García, Navarro and Melero2012), constituency parsing and dependency parsing (Foster Reference Foster2010; Foster et al. Reference Foster, Wagner, Roux, Hogan, Nivre, Hogan and Van Genabith2011), Named-Entity Recognition (NER) (Ritter et al. Reference Ritter, Clark, Mausam and Etzioni2011; Maynard et al. Reference Maynard, Bontcheva, Rout and Melero2012; Muñoz-García and Navarro Reference Muñoz-García, Navarro and Melero2012), and tokenization (Aminian et al. Reference Aminian, Avontuur, Azar, Balemans, Elshof, Newell, van Noord, Ntavelos, van Zaanen and Melero2012). Different text cleansing or normalization strategies have been pursued to improve the performance of standard NLP tools for their application to social media text.
The contributions of this paper are (i) the presentation and characterization of a UGC corpus in Spanish, and (ii) the description of a normalization strategy that, exploiting the power of pre-existing spell and grammar correction tools, allows for broad-coverage normalization, including the normalization of tokenization of nonstandard words. In this respect, this paper supposes a novelty in both the strategies for normalization and the inclusion of tokenization in the normalization task.
1.2 Related work
In this section, we review research on three different topics: the definition of text normalization as a task, the different approaches to text normalization, and the strategies found in the literature for ranking correction candidates in an automatic correction task, as opposed to a human–machine interactive correction task.
1.2.1 Text normalization as a task
Text normalization is a term that has been used in the field to refer to a task that consists of transforming an original piece of text into a different piece of text in which some words have been converted to a normal form. The definition of norm, or normal form, depends on the task to be performed after normalization. Thus, Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001), in one of the classical papers on the topic, present work on the normalization of text to be given as input to a text-to-speech synthesis engine, and claim that a similar approach could be applied to speech recognition and information extraction tasks.
However, the norm of a text-to-speech synthesis is not necessarily the same as the norm for information extraction – to the extent that misspellings are not tackled in Sproat et al.’s (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) paper, for speech synthesis requires a word form to be obtained that can be ‘said aloud’, while information extraction requires a word form that can be interpreted or (lexically) analyzed. This poses the interesting question of how linguistic ‘norm’ is actually defined, a topic that we briefly take up below.Footnote 1 A recent paper by Eisenstein (Reference Eisenstein2013) addresses the issue of how the different ways in which social media text deviates can be influenced by different aspects: keyboard used, writer’s literacy, writer’s intentions – pragmatics –, medium’s length, and actual social variables. This paper interestingly suggests how UGC, and again Twitter in particular, can be seen as a real-time observation platform of language as an evolving social communication tool.
Another interesting issue is the type of text on which the normalization task is performed. While Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) work with four different types of text (newspaper text, real estate ads, and servlist texts on the topics of palmtop computers and cooking recipes), most later papers deal either with Short Message Service (SMS) texts – see, for instance, Choudhury et al. (Reference Choudhury, Saraf, Jain, Mukherjee, Sarkar and Basu2007), Kobus, Yvon and Damnati (Reference Kobus, Yvon and Damnati2008), or Cook and Stevenson (Reference Cook and Stevenson2009) – or Twitter text – see, for instance, Brody and Diakopoulos (Reference Brody and Diakopoulos2011), Clark and Araki (Reference Clark and Araki2011), Foster et al. (Reference Foster, Wagner, Roux, Hogan, Nivre, Hogan and Van Genabith2011), Han and Baldwin (Reference Han and Baldwin2011), Hassan and Menezes (Reference Hassan and Menezes2013), or Eisenstein (Reference Eisenstein2013). Also, Liu, Weng and Jiang (Reference Liu, Weng and Jiang2012) have worked on both SMS and Twitter datasets.
1.2.2 Approaches to text normalization
We find two main different ways of dealing with UGC text processing, which can be complementary. The first one focuses on adapting tools and algorithms to this type of text by annotating UGC corpora and training the corresponding tools from scratch (Michelson and Knoblock Reference Michelson and Knoblock2005; Ritter et al. Reference Ritter, Clark, Mausam and Etzioni2011), while the second one focuses on transforming the source UGC text into a normalized version before handling it over to the processing tools (Kobus et al. Reference Kobus, Yvon and Damnati2008; Agarwal et al. Reference Agarwal, Xie, Vovsha, Rambow and Passonneau2011; Liu et al. Reference Liu, Weng and Jiang2012). There are some attempts at combining both the methods by adapting or extending the training models while, at the same time, preparing the input text by normalizing some of the UGC-related phenomena (Foster Reference Foster2010; Foster et al. Reference Foster, Wagner, Roux, Hogan, Nivre, Hogan and Van Genabith2011).
Kobus et al. (Reference Kobus, Yvon and Damnati2008) present an interesting discussion on three different ‘metaphors’ or ways of looking at SMS language, a type of text that has some features in common with UGC text. Each of these views motivates a different approach to the normalization task. In the first approach, each input token is taken as a deviation of the correct word form, and normalization is thus viewed as a spell checking task. The second metaphor considers SMS language as a different language, and so normalization can be viewed as a machine translation task. Finally, it is possible to consider normalization as a speech recognition task because some people consider SMS as being closer to oral productions than to regular written texts. In fact, SMS spellings tend to be a closer approximation to the phonemic representation of a word than to its normative spelling, although in this context ‘typography and orthography take over the functions of sound’ (Herring Reference Herring and Chapelle2012b: 1).
Most of the attempts at text normalization are based on empirically obtained lists of frequently observed phenomena, as in Kobus et al. (Reference Kobus, Yvon and Damnati2008), Foster (Reference Foster2010), and Agarwal et al. (Reference Agarwal, Xie, Vovsha, Rambow and Passonneau2011), although some research applies more generalist approaches (Alonso Reference Alonso2010). Zhu et al. (Reference Zhu, Tang, Li, Ng and Zhao2007) present an approach to text normalization (of e-mails and blog posts) that uses a Conditional Random Field algorithm with a lot of features and compare it with two other classical normalization methods. Zhu et al.’s (Reference Zhu, Tang, Li, Ng and Zhao2007) approach tackles the normalization task as a tagging problem, where the different normalization transformations required are performed on each token depending on its type (line break, space, punctuation, word, and special). Clark (Reference Clark2003) and Clark and Araki (Reference Clark and Araki2011) use rule-based approaches, while Henríquez Q. and Hernández (Reference Henríquez and Hernández2009) approach the task using a statistical machine translation system trained on original texts and their semi-automatically corrected version. More recently, Hassan and Menezes (Reference Hassan and Menezes2013) have shown how a version of Markov Random Walks can be used to train a normalizer on a small manually corrected corpus – without any further labeling.
Finally, Liu et al. (Reference Liu, Weng and Jiang2012) propose what they call a broad-coverage normalization algorithm, meaning it should be domain-independent, which includes a module grounded on visual priming theories that favor the occurrences of words closer or related to words that have already been seen in the text. They combine this strategy with letter sequence modeling –that to a certain extent resonates as a simplification of Zhu et al.’s (Reference Zhu, Tang, Li, Ng and Zhao2007) approach and standard spell checking techniques to generate lists of alternatives to nonstandard tokens, although they assume normalized tokenization.
Our aim in the present work is to seize the power of already existing spell and grammar correction engines and endow them with automatic normalization capabilities to pave the way for the application of standard NLP tools to typical UGC text. In this respect, our research is closer to Liu et al.’s (Reference Liu, Weng and Jiang2012) research, since they aim at the implementation of a broad coverage normalization strategy.
1.2.3 Correction candidate selection
N-gram models have been used for the detectionFootnote 2 and correctionFootnote 3 of misspellings in isolated words since the late 1980s (Kukich Reference Kukich1992). Church and Gale (Reference Church and Gale1990) demonstrate the potential of word bigrams to improve the accuracy of isolated word correction. Mays, Damerau and Mercer (Reference Mays, Damerau and Mercer1991) used trigram models and obtained 76% accuracy in detection and 73% accuracy in correction. Hodge and Austin (Reference Hodge and Austin2003) integrate Hamming distance and n-gram algorithms that have high recall for typing errors and a phonetic spell checking algorithm in a single architecture. Ahmed, Luca and Nürnberger (Reference Ahmed, Luca and Nürnberger2010) propose a spell-checker that works by selecting the most promising candidates from a ranked list that is derived from n-gram statistics and lexical resources. Other approaches that correct spelling include rule-based techniques (Mangu and Brill Reference Mangu and Brill1997), a noisy channel model (Brill and Moore Reference Brill and Moore2000; Toutanova and Moore Reference Toutanova and Moore2002), and a ternary tree search (Martins and Silva Reference Martins and Silva2004). As far as we know, little work has been made to date on the subject for Spanish, with the exception of Alonso (Reference Alonso2010). Alonso (Reference Alonso2010) uses a normalization approach based on a strategy to group words into clusters by computing the Levenshtein edit distance between each new (unknown) word and a series of word clusters – her work is rather a classification task for out-of-vocabulary (OOV) words, more than a candidate selection task.
2 Corpus-based characterization of UGC text in Spanish
As a reference corpus to study UGC-related phenomena, we have collected a sample of texts in Spanish from the following sources: blogs (collected using Google Blog Search), hotel reservations (booking.com), consumer reviews in three different domains (ciao.es), and Twitter. In the following sections, we describe the corpus size, source distribution, and the annotation rationale.
2.1 Corpus size and distribution
The total size of the corpus is 7,583 sentences, or 192,417 words. Table 1 shows the size of the corpus in terms of sentences and words, and the corresponding percentage of each source. The Ciao site comprises texts from three different domains: cars (61%), mobile operators (12%), and banking (26%). The last row shows the ratio between the number of running words (total number of words) and the vocabulary size (number of different words appearing at least once, i.e. word types). The figures indicate that Twitter and booking texts present a greater lexical variation – which can be accounted for by these being less topic-focused types of texts compared with ciao.es texts.
Corpus size in sentences and words, and ratio of word types (the lower the ratio, the higher is the number of word types per total number of words)
2.2 Annotation rationale
The normalization of ill-formed text presupposes a definition of norm. The concept of norm may vary from one linguistic community to the other. For example, norm is reached by consensus in the English-speaking world, but it is dictated by a prescriptive institution for Spanish (Real Academia Española de la Lengua, RAE) or French (Académie Française). In addition, within a particular community or corporation it can be further restricted with arbitrary norms, as is the case of controlled languages for machine-translated manuals or corporate guidelines in publishing houses. In the English-speaking world of NLP, one could argue that the norm for tagging and parsing is the Wall Street Journal, since it has become the de facto standard.
In our case the norm is the one underlying our dictionary, which mostly follows the RAE, and thus includes the most common Latin American spellings and words. For instance, our dictionary includes both futbol (strength on the o) and fútbol (strength on the ú). From the perspective of NLP, the norm is further restricted by the arbitrary decisions regarding the linguistic apparatus. If the processing pipeline is not able to recognize one particular word, it will not identify it as belonging to the language; and it will fail to assign the corresponding lexical and morphosyntactic features.
As suggested in the Introduction, the normalization of text for text-to-speech (TTS) is also different from the normalization of text for text mining. For instance, in a TTS task the highway name m40, as written in the tweet captured in Figure 1, would eventually result into eme cuarenta [em forty], while in a text mining text it would stop in M-40 – which might be an intermediate state for the TTS task.
Real twitter text containing the highway name m40 for M-40.
We agree with Eisenstein (Reference Eisenstein2013) in that we argue that ‘[n]ormalization is often impossible without changing the meaning of the text’ given the social and pragmatic implications that UGC text entails.
2.3 Annotation procedure
The reference corpus was manually corrected. Textual deviations were reviewed, corrected, and classified by three different annotators (none of which is involved in the research work). Each annotator annotated a different set of texts, so inter-annotator agreement cannot be calculated. Texts were not randomly assigned to annotators, but all annotators ended up annotating at least two different text genres or domains. Each deviation from standard language norms was identified, classified, and corrected. They were instructed to perform their task in the following two steps:
Step 1 Localization and correction
(1) Search for deviating forms
(2) Mark the span of the deviating form using square brackets
(3) Write the alternative normalized version of the text within the brackets
(4) Write the number of tokens originally involved in the deviation
Step 2 Classification
(5) Classify the deviating form according to the following typologies:
(a) Linguistic type
(b) Transformation type
Annotators, all of whom had a degree in either theoretical or applied linguistics, were given the instructions in print with no examples. Weekly meetings were organized where annotators could share their findings and doubts and check their progress with us. The annotation process itself lasted for about three weeks with the annotators working part-time. An example of the annotation format and how this format was later converted into an XML format is described in Section 2.3.3.
2.3.1 Annotation of linguistic type
On the basis of an exploratory inspection of the data, we classified norm deviations into the following types:
Capitalization. The text is capitalized for emphasis or emotive purposes, or else proper nouns are not capitalized: ‘y NO es broma’ [NOT kidding] for ‘y no es broma’ or ‘me recorro españa’ [I go around spain] for ‘me recorro España’.
Accentuation. Graphical accents are omitted: ‘en numeros rojos’ for ‘en números rojos’ [in the red].
Punctuation. Punctuation signs are omitted or reduplicated; this also includes omission of blank spaces: ‘Te quiero!!!!!!!’ for ‘!Te quiero!’ [I love you!], ‘aver’ for ‘a ver’ [let’s see].
Informal spelling. All systematic shortcuts and character substitutions intentionally made by the user: ‘pq’ for ‘porque’ [because], ‘t kiere muxo’ for ‘te quiere mucho’ [he loves you so much].
Spelling errors. All spelling errors not included in the previous categories, including conventional misspellings, such as ‘oie’ for ‘oye’ [listen] or ‘targetas’ for ‘tarjetas’ [cards]; typos, such as ‘dicindo’ for ‘diciendo’ [saying]; and intentional or unintentional reduplication of characters: as in ‘cooordenadas’ for ‘coordenadas’ [coordinates], ‘alistarmeeeee’ for ‘alistarme’ [join up], or ‘frrrrrrío’ for ‘frío’ [cold].
Other errors. Lexical, syntactic), e.g. agreement errors or missing prepositions: ‘delante mi casa’ for ‘delante de mi casa’ [in front of my house], ‘mucho gente’ for ‘mucha gente’ [lots of people].
Arguably some of the above categories could have been divided into finer-grained ones. For instance, the two types of capitalization phenomena that we exemplify are quite distinct in nature: one can be attributed to an intentional communication goal to emphasize some idea, and the other is rather a relaxation of standard spelling norms. Similarly, spellings like ‘frrrrrrío’ could be included in the category of informal spellings instead of spelling error. However, we could not distinguish the reduplication of letters owing to slips of the keyboard – presumably like ‘cooordenadas’ – from those due to an intentional communication goal. We decided to keep the annotation process as simple as possible at this stage of the research – see also Brody and Diakopoulos (Reference Brody and Diakopoulos2011), a study exclusively devoted to the analysis and exploitation of letter reduplication for sentiment analysis of Twitter, which suggests that the semantic interpretation of certain UGC characteristics is a task in itself.
2.3.2 Annotation of transformation operations
Annotations included also the type of transformation operation with respect to the normalized version that a given deviation involves. The transformation operations that we adopted are derived from Damerau (Reference Damerau1964):
Addition: A character or word is added, as in ‘dijo de que vendría’ for ‘dijo que vendría’.
Omission: A character or word is omitted, as in ‘dicindo’ for ‘diciendo’ or in ‘delante mi casa’ for ‘delante de mi casa’.
Substitution: One or more characters or words are replaced by others, as in ‘t kiere muxo’ for ‘te quiere mucho’, where ‘k’ replaces ‘qu’, and ‘x’ replaces ‘ch’.
Transposition: One character or word is placed before or after the expected position, as in ‘uatobús’ for ‘autobús’.
Duplication: A character or a word is repeated one or several times, as in ‘frrrrrrío’ for ‘frío’.Footnote 4
2.3.3 Annotation format
With the instructions described in Section 2.3, annotators would correct a fragment such as the one shown in (1) into something like the fragment shown in (2).
(1) (. . .) estéticamente a la mayoria dela gente no gusta por su forma (. . .)
(2) (. . .) estéticamente a la [1 mayoría] [1 de la] gente no gusta por su forma (. . .)
Manual annotations were then automatically extracted and a table was generated. In the second step, they were manually classified by annotators into one of the categories described above. As a result, we obtained a table of error–correction pairs like the ones exemplified in Table 2.
Table for deviation categorization completed by annotators in step 2 of the annotation process
The initial manual annotation was mapped into an XML annotation scheme. This scheme is scalable and compatible with the Text Encoding Initiative (TEI) conventions. It is conceived as a stand-off annotation: instead of mixing the data with the metadata, the original text is preserved as-is, while the annotation forms a separate layer, linked to the original text through offset indicators.
Figure 2 shows the annotation for the sentence ‘concerteza amooor=)’, a real Twitter message. The normalized version is ‘|Con| |certeza| |amor| |=)|’, where the three original tokens, including a space, result in four normalized tokens, including three spaces. The word ‘con’ appears capitalized – because it is the beginning of a sentence – and the word ‘amooor’ is reduced to ‘amor’. Both instances of annotation show how multiple annotation of deviation types is possible, since more than one < dev > -tag can be assigned to the same normalized sequence of characters.
XML annotation sample.
2.4 Distribution of deviation types across different text sources
Overall, the rate of deviated input in our UGC reference corpus is quite high: over a fifth of the word types (20.8%) contain some error or deviation. This rate varies depending on the types of texts ranging from 4.62% in more edited text, such as blog posts, to over 25% in Twitter and informal consumer reviews. Note that this percentage is calculated on word types, not on word instances. The actual rate of error in word instances, as computed with the total number of words in the corpus, is around 5.7%.
Table 3 shows the percentage of word types that were manually corrected with respect to the total number of word types in each corpus, classified by error or deviation type. With minor exceptions, the frequency distribution of the deviation types does not exhibit significant variations across the different text sources.
Percentage of deviations (in terms of word-types) according to its linguistic type across genres
Due to their relative frequency, three types of deviations clearly stand out over the rest: spelling errors, capitalization, and accentuation. Even though the ‘Spelling errors’ class in our classification includes reduplications with expressive or emotive purposes, most of the instances are ‘ordinary’ orthographic errors, which, together with accentuation problems, are well handled by conventional spell-checkers. This fact has partly motivated the strategy we have chosen to deal with UGC text normalization, as we explain in Section 4.
2.4.1 Characteristics of Spanish UGC and English UGC
There is relatively little research on the characterization of the specificities of UGC. However, Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) and Han and Baldwin (Reference Han and Baldwin2011) offer some interesting information. Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) find that the percentage of ‘nonstandard words’ (NSW) in a newspaper text is 8.8%, 43.4% in real state ads, 27.3% in servlist emails on the topic palmtop computers, and 22% in servlist emails on cooking recipes. Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) include in the NSW category phenomena such as abbreviations (adv for advertisement, N.Y. for New York, or gov’t for government), any textual items, including numbers (from phone numbers to car models), non-spoken elements (certain punctuation or token boundaries), and spelling errors or funny spellings (sic).
Han and Baldwin (Reference Han and Baldwin2011 : 370–371) provide information on the OOV rate found in newspaper text, SMS, and Twitter text. Their findings show that both Twitter and SMS present a larger percentage of OOV words independent of the size of the percentage of the corpus looked at, while newspaper text presents a more Zipfian distribution of OOV words – that is, the more text you look at, the fewer newer or unknown words are found. One other interesting finding is that OOV words in SMS tend to be personal names. Last but not least, Han and Baldwin (Reference Han and Baldwin2011) also find that the most frequent reason for ‘ill-formed’ words are missing or extraneous letters, 72.44%, while the second most frequent reason is ‘slang’ – words found in a slang dictionary available on the Internet.
Alhough our findings are not fully comparable with those in the two previous references, we can see that our blog corpus is the one that presents the lowest deviation rate – comparable with newspaper text if we take into account that Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) were looking to a wider variety of nonstandard forms. In contrast, our other corpora present very high rates of deviations – which are in line with the findings of both Sproat et al. (Reference Sproat, Black, Chen, Kumar, Ostendorf and Richards2001) and Han and Baldwin (Reference Han and Baldwin2011) in their less formal types of texts.
3 Processing UGC text
Our hypothesis is that the percentage of deviations present in UGC text will have an impact on the performance of standard NLP tools. In a similar experiment, Foster (Reference Foster2010) detects problems with the handling of long coordinated sentences (mainly in the presence of erratic punctuation usage), domain-specific fixed expressions, and unknown words.
To gauge the impact of deviations on the linguistic processing of UGC text, we processed the original version of the corpus and the manually corrected one, using a conventional linguistic processing pipeline for Spanish (Rodríguez et al. Reference Banchs, Codina and Grivolla2010). The pipeline consists of general-scope, state-of-the-art linguistic tools, not specifically adapted to UGC text, integrated on a UIMA platform.
We then compared the outcome in terms of changes in the resulting annotation. According to our analysis, the differences in the results of processing the two versions amount to 30% to 100% of the tokens, with a varying percentage depending on the type of error, task, and domain, as we see below.
3.1 Impact on lexical coverage
As we had expected, the normalization of the input increases lexical coverage (see Gouws et al. Reference Gouws, Metzler, Cai and Hovy2011 on the effects of normalization on lexical coverage). Table 4 shows the percentage of words – in terms of individual instances and word types – covered by the system’s lexical resources in the original and in the manually normalized version. These values are notably lower for Twitter than for the rest of the sources.
Percentage of word coverage (instances, I, and types, T) in both original and corrected versions
The increase in coverage after normalization is shown in brackets. This increase turns out to be more evident in the comparison of word types than in the comparison of word instances, perhaps as a side effect: The normalization of deviated forms is actually decreasing the number of hapax (words occurring only once).
3.2 Impact on the performance of three basic NLP processing tasks
We investigated the effect of normalization on three basic NLP processing tasks: (i) lemmatization, (ii) PoS tagging (short-tag or syntactic category), and (iii) assignment of morphosyntactic features (gender, number, tense, etc.). These tasks are at the root of more complex or higher level processing tasks, and errors at this level are likely to propagate and affect other tasks such as constituent analysis, dependency relation identification, NERC, and other NLP tasks (see, for instance, Grefenstette and Tapanainen Reference Grefenstette and Tapanainen1994; Chung and Gildea Reference Chung and Gildea2009; or Mohamed Reference Mohamed2011).
Table 5 shows the percentage of words for which a change in the resulting linguistic analysis is found after normalization.Footnote 5 In general, results are fairly uniform across the three genres. On the one hand, we observe that lemmatization is very sensitive to the presence of error. The percentage of wrong lemma assignment to erroneous, or non-normalized, word forms is 90% overall, and between 85% and 95% depending on the corpus.
Percentage of deviating words incorrectly analyzed by domain
On the other hand, assignment of syntactic category is generally quite robust to deviation, since only less than half of the non-normalized word forms (around 46%) change their category assignment after having been corrected or normalized, while the proportion of wrong assignment of morphosyntactic features is a little higher, around 64%, across genres.
Table 6 presents the same information broken down by deviation type, for the four most frequent types. Capitalization turns out to be the least detrimental across these basic tasks, while accentuation is particularly harmful, even for a robust task such as PoS tag assignment (main grammatical category).
Percentage of deviating words incorrectly analyzed by type of error
As we can see in Table 6, lemmatization is likely to be inaccurate for most deviated words. These results are consistent with previous findings in the field, where the analysis of lexical items was found to be particularly sensitive (e.g. Gouws et al., Reference Gouws, Metzler, Cai and Hovy2011: 28; Han and Baldwin, Reference Han and Baldwin2011 : 370–371). Lemmatization errors increase the variability for the same concept and thus are likely to affect most semantic-related tasks.
4 Building a normalizer on top of an existing spell-checker
The large presence of deviated forms in UGC text and its costly impact on the performance of NLP tools convinced us of the need for a specific solution that addresses the problem of processing of this type of text. As discussed in Section 1, we find two approaches dealing with this issue in the literature: either transform the input text (i.e. normalize it) or adapt the tools (e.g. by retraining the models). While the second option is feasible mostly for statistically trained tools (the case for Foster Reference Foster2010), the first option should work for any tool, statistical and rule-based, and it is the course we have followed in this work.
Among the different approaches to normalization presented in Section 1, we have chosen to view normalization as a spell checking task, particularly motivated by the high rate of typical orthographic errors (including accentuation) in UGC text. For this purpose we have built an automatic normalizer on top of the Spanish version of COTiG, a spell and grammar checker first developed for Catalan (Quixal et al. Reference Quixal, Badia, Benavent, Boullosa, Domingo, Grau, Massó, Valentín, Chair, Choukri, Maegaard, Mariani, Odjik, Piperidis and Tapias2008).
A key difference between a regular spell-checker and a normalizer is interactivity with an end-user. The lack of interactivity in the normalization task has an important implication: A specific strategy has to be put in place to rank possible correction candidates and decisively choose the best one over the whole set. The rest of the paper deals with the development of a dedicated selector module that ranks the list of correction candidates proposed by the spell-checker and selects the highest ranked one.
There are other important aspects that need to be addressed to effectively turn a regular spell-checker into a normalizer. A main concern is the rate of false positives. If over-correcting may be annoying for the user of a spell-checker, over-normalizing, i.e. introducing unwanted changes to the original text, can be invalidating for a normalizer. OOV words are one of the main sources of false positives. A straightforward way to deal with this is using larger dictionaries or dictionaries adapted to the domain in question. Moreover, typical UGC phenomena, such as informal spellings or emoticons, may not be appropriately dealt with by using standard spell checking procedures, such as edit distance algorithms; thus, ad hoc or specific strategies to approach these phenomena may also be required.
5 Selection of the right candidate using language models
The output of the spell-checker consists of an unordered list of correction candidates obtained through the application of its own algorithms for the generation of correction proposals based on simple editing-distance criteria. For instance, given a sentence such as aunque mire otros coxes de la misma categoría [even if I look at other cars of the same type], the list generated by the spell-checker for the word coxes (coches [cars] intended) looks as shown in Figure 3.
Output of the normalizer in XML format containing a list of correction candidates.
To select the most probable correction among a set of candidates, we experimented with the use of different trigram models trained on an in-domain corpus described in Section 5.1, each model conveying a different level of information.
Inspired by the so-called factored models employed in statistical machine translation (Bilmes and Kirchhoff Reference Bilmes and Kirchhoff2003; Koehn and Hoang Reference Koehn and Hoang2007), our selector module enables the integration of different language models trained on different automatically generated linguistic annotations at the word level:
• Truecase form model (TC). This model was trained on the original unmodified text, where uppercase and lowercase instances of the same form are different words. It is the most specific – or the least abstract one – since it is the model with the largest vocabulary.
• Lowercase form model (LC). This model has been trained on the lowercase version of the original corpus. Uppercase and lowercase versions of the same form are now the same word.
• Lemma model (Lemma). This model has been trained on the lemmatized version of the original corpus, where each inflected form has been substituted by its root or lemma. Plural and singular versions of the same noun are now the same word; the same happens with variations in person, tense, and number of verbal forms.
• Part-of-Speech model. This model has been trained on the PoS tags version of the corpus. Tags are PAROLE-style PoS long tags, which for each word include its syntactic category and morphosyntactic features (e.g. AQ0CP0, NCFP000, VSIC3P0; for details see Villegas, Brosa and Bel Reference Villegas, Brosa and Bel1998). This model is the most general – or the one that abstracts at a higher level – since it has the smallest vocabulary.
5.1 Building the models
The corpus used to build the models is a 24 million-word corpus collected from the Web that comprises texts from the same domains and genres included in the reference corpus, namely: banking, cars, mobile, Twitter, and blogs.Footnote 6
The lemma and morphosyntactic labels were assigned using our in-house linguistic pipeline (cf. Section 3). We then built four trigram models with the IRSTLM toolkit (Federico, Bertoldi and Cettolo Reference Federico, Bertoldi and Cettolo2008) using the ‘modified shift-beta’ as smoothing method, also known as ‘improved Kneser-Ney smoothing’.
5.2 Querying the models
At run time we query the models using a sliding window over the proposed correction of at most five words. For the example in Figure 3, eleven different five-word candidate strings are generated, one for each candidate correction, surrounded by its immediate context.
For each candidate strings Si, we generate four parallel strings:
• STC with all words in truecase: mire otros boxes de la
• SLC with all words in lowercase: mire otros boxes de la
• SLemma with lemmas only: mirar otro box de la
• SPoS with PoS only: VMSP3S0 DI3MP0 NCMP000 SPS00 DA3FS0
With C(M, S) being the cost of string S (i.e. logarithm of the probability) according to model M, we query each of the four models for the cost value of each one of the strings. For example, C(TC, STC) is the cost of the truecase string computed against the TC model. That means that for each candidate, we get four cost values: C(TC, STC), C(LC, SLC), C(Lemma, SLemma), and C(PoS, SPoS).
5.3 Combining the models
The aim of building the four different models is to evaluate which of them is more discriminative at ranking the different candidates. We also experiment with the combination of models to find out whether such a strategy can improve over the results of the best performing model used in isolation. The linear equation for the parametrization of the combination of different models is shown in (1):
\begin{equation}
\begin{split} C(total,S)= \lambda _{TC} C(TC, S_{TC}) + \lambda _{LC} C(LC, S_{LC}) \\
+\ \lambda _{Lemma} C(Lemma, S_{Lemma}) + \lambda _{PoS} C(PoS, S_{PoS}) \end{split}
\end{equation}
5.3.1 Linear interpolation of language models
Linear interpolation of language models is one of the most popular model combination techniques. Given a set of n-gram language models, linear interpolation consists in computing the weighted average of the component model costs as shown in (1). The interpolation weight λi, where {i = TC, LC, Lemma, PoS} satisfies ∑iλi = 1 and it is typically tuned to optimize the perplexity of the development set. We empirically adjusted the combination weights by following the standard procedure for linear interpolation in language modeling in order to build a unique optimized model. This procedure uses the expectation-maximization (EM) algorithm to minimize the perplexity over a specific development set (Stolcke Reference Stolcke2002). To apply this procedure, we previously calculate the probability of the same random variable, which in the TC model is a word and in the other three models (LC, Lemma, and PoS) is a class. For a class model C, we calculate:
\begin{equation}
P_C(\omega _n|\omega _{n-k+1}. . .\omega _{n-1})=\sum _i P(\omega _n|c^i_{\omega _n}|c_{\omega _{n-k+1}}\ldots c_{w_{n-1}})
\end{equation}
where ci ωn are different classes of the word ωn. Using this equation for the three models where the variable is a class, we can use expectation-maximization to calculate the mentioned interpolation weights λi, where {i = TC, LC, Lemma, PoS}.
The development set was extracted out of the text of the reference corpus (described in Section 2), as detailed in Section 5.3.2. This development set of around 50K words was used in the optimization phase. Table 7 shows the vocabulary sizes for our training and development sets for the four models: TC, LC, Lemma, and PoS. Even though the number of words is the same for the four training sets – i.e. 24 million words – the size of the respective vocabularies varies considerably, going from 525,742 in the case of the TC model to only 243 in the case of the PoS model. The same applies to the development set.
Vocabulary size (unigram counts) for the training and development corpus
The language model weight optimization was computed using the compute-best-mix function of the SRILM toolkit (Stolcke Reference Stolcke2002). We used the option of expand-classes to normalize the vocabularies in classes to words (for LC, PoS, and Lemma language models). The perplexity of each language model is shown in Table 8. The PoS language model is the one with the highest perplexity, which seems reasonable because it has fewer classes and, therefore, a higher degree of ambiguity. In contrast, the TC model, where each word is a class, has the lowest perplexity.
Perplexity in the development set for each language model
The resulting optimization assigns a sizeable weight to the TC model, a small weight to the LC model and practically overlooks the Lemma and PoS models. The final set of weights is presented in (3):
\begin{equation}
\begin{split} C(total,S) = 0.961861\;*\;C(TC, S_{TC})\;+\; 0.0309452\;*\;C(LC, S_{LC})\; \\
+\; 0.00690778\; *\; C(Lemma, S_{Lemma})\; +\; 0.00000003\; *\; C(PoS, S_{PoS}) \end{split}
\end{equation}
Note that this is theoretically a methodology to optimize the weights over language models with different word classes. However, in practice, the task of expanding classes may be too ambiguous, which directly affects the interpolation procedure. For example, when expanding the lowercase word class of zapatero we have only two possible surface forms zapatero and Zapatero, which is quite an easy task. But in the case of expanding the PoS word class of Prepositon, we have twenty-three possible word surface forms. At the end, the weight given to the language model with the most generic classes (in this case PoS) is quite low, which may be a direct consequence of an improper expansion of classes.
5.3.2 Optimization on the task
The combination of the models can be further optimized by tuning the parameters of the four language models to minimize errors directly on the selection task. Optimization on the task is possible if a gold standard can be used to iteratively measure improvement. For this we used the reference corpus described in Section 2. We processed the original non-corrected section of the reference corpus, and found a total of 4,235 deviation annotations for which the underlying spell-checker correctly detects an error and the normalization of reference is within the set of candidates proposed by the selector module. Therefore, the number of deviations that we are going to optimize in this experiment is the number of deviations (correctly detected by the spell-checker) for which the candidate ranked first by the selector module coincides with the normalization of reference according to our gold standard. To evaluate our results, we use the manually normalized reference corpus. We divide it in two parts: 30% for the development set (the same one as for the linear interpolation) for optimizing on the task and 70% for evaluation.
To iteratively minimize errors we used a modified version of the Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall Reference Spall1992; Lambert and Banchs Reference Lambert and Banchs2006) as an optimization algorithm.
SPSA Algorithm. The SPSA procedure is in the general recursive stochastic approximation form, as shown in (4):
\begin{equation}
\skew 3\hat{\mathbf {\lambda }}_{k+1}=\skew 3\hat{\mathbf {\lambda }}_k-\mathbf {a}_k\hat{\mathbf {g}}_k(\skew 3\hat{\mathbf {\lambda }}_k)
\end{equation}
where 
$\hat{\mathbf {g}}_k$(
$\skew 3\hat{\mathbf {\lambda }}_k$) is the estimate of the gradient g(λ) ≡ ∂E/∂λ at the iterate $\skew 3\hat{\mathbf {\lambda }}_k$, and ak denotes a positive number that usually decreases as k increases. The gradient was computed with a one-sided approximation which, given E($\skew 3\hat{\mathbf {\lambda }}_k$), requires the evaluation of E($\skew 3\hat{\mathbf {\lambda }}_k$ + perturbation). In the simultaneous perturbation approximation, all elements of $\skew 3\hat{\mathbf {\lambda }}_k$ are randomly perturbed togetherFootnote 7 and the approximated gradient vector is:
\begin{equation}
\hat{\mathbf {g}}_k(\skew 3\hat{\mathbf {\lambda }}_k)=\frac{E(\skew 3\hat{\mathbf {\lambda }}_k+c_k\mathbf {\Delta _k})-E(\skew 3\hat{\mathbf {\lambda }}_k)}{c_k} {\left[\begin{array}{*10c}\ 1/\Delta _{k1}\\
\ 1/\Delta _{k2}\\
\ \vdots \\
\ 1/\Delta _{kN}\\
\end{array}\right]}
\end{equation}
In (5), Δk is a perturbation vector of the same dimension N as λ, whose values Δi are computed randomly. ck denotes a small positive number that decreases as k increases. We took for each component Δi of Δk a Bernoulli ± 1 distribution with probability of 1/2 for each ± 1 outcome, or a 0, ±1 distribution with probability of 1/3 for each outcome. Full details of our implementation of the algorithm are given in Lambert and Banchs (Reference Lambert and Banchs2006). The main change with respect to the original algorithm is that the parameters are not necessarily updated at each iteration. If the new set of parameters 
$\skew 3\hat{\mathbf {\lambda }}_{k+1}$ is worse than the current one, it is updated with a probability, which depends on how much worse it is. The algorithm typically converged after fifty to sixty iterations. Note that, in general, SPSA converges to a local maximum.
Task optimization. Figure 4 exemplifies the optimization process for the candidate selection task with the goal of maximizing precision (minimizing the number of times the correct proposal was not ranked first in the list). With the initial set of weights, 
$\skew 3\hat{\mathbf {\lambda }}_0$, we normalize the original part of the development corpus and calculate the precision against the manually corrected reference of the same corpus. This gives us E($\skew 3\hat{\mathbf {\lambda }}_0$). We perturb the $\skew 3\hat{\mathbf {\lambda }}_0$ vector with c 0Δ0, normalize the development corpus with the perturbed set of weights, and calculate its precision, which gives E($\skew 3\hat{\mathbf {\lambda }}_0$ + c 0Δ0). We can now update the set of weights according to (4) and (5), giving 
$\skew 3\hat{\mathbf {\lambda }}_1$. We evaluate E($\skew 3\hat{\mathbf {\lambda }}_1$) and according to the result, we take $\skew 3\hat{\mathbf {\lambda }}_1$ or keep $\skew 3\hat{\mathbf {\lambda }}_0$ as the current set of weights. Then we perturb again the current set of weights with the ck Δk vector, and so on.
Language model weight optimization according to the selection precision.
The equation resulting from the optimization assigns equal moderate weight to TC and LC models, a small weight to the Lemma model, and zero to the PoS model, as shown in (6):
\begin{equation}
\begin{split} C(total,S) = 0.4519\;*\;C(TC, S_{TC})\;+\; 0.4519\;*\;C(LC, S_{LC})\; \\
+\; 0.0962\; *\; C(Lemma, S_{Lemma})\; +\; 0\; *\; C(PoS, S_{PoS}) \end{split}
\end{equation}
5.4 Evaluation of candidate selection
As indicated in the previous section, we used for evaluation purposes the unused portion of the annotated reference corpus, i.e. 70% of the total. We tested each of the four models individually and the two different model combinations presented in Section 5.3: linear interpolation and optimization on the task. As a baseline we ran the normalizer without the selector module, i.e. taking the first item in the list as proposed by the underlying spell-checker – which has an almost nonexistent ranking strategy (Quixal et al. Reference Quixal, Badia, Benavent, Boullosa, Domingo, Grau, Massó, Valentín, Chair, Choukri, Maegaard, Mariani, Odjik, Piperidis and Tapias2008). Moreover, since Twitter text is known to present specific features that make it different from other UGC text and that may affect performance, we also evaluate Twitter and the rest of the corpus sources separately. We repeated the experiment of optimization on the task with specific development and test sets for Twitter and non-Twitter.
The results, in terms of percentage of instances where the reference correction was ranked first, are shown in Table 9. The model built on a lowercase version of the corpus turns out to be the more discriminating model in isolation, with a precision of 86.59%, as measured on the test set (second column of the table). On the same set, we see that neither model combination improves the precision achieved by the LC model, although they perform better than the rest of the models used in isolation. Noticeably, PoS obtains results that are below the baseline. Both model combination techniques, linear interpolation and optimization on the task, obtain quite comparable results on the test set, although the results of the latter are slightly better (around 0.4 points improvement). The first column in Table 9 shows the results for the development set of the corpus, used to obtain the optimized set of weights described in Section 5.3.2. We observe that in this case the weighted combination obtained by the optimization on the task technique improves on the result of the best performing model in isolation, i.e. LC model, in the development set, but not in the test set and that linear interpolationFootnote 8 and the TC model achieve the same precision. Remarkably, the baseline system (without a selector module) obtains a much lower result on the development set (10 points below the test set), and in this case it is clearly surpassed by the worst performing model, i.e. PoS. Table 10 shows separated results for Twitter and non-Twitter text. In this table we put in brackets the negative or positive difference of each value with respect to the values in Table 9. All in all the precision measure of the correction selection drops 4.5 points on average when tested on Twitter text. The ranking of the models does not change for the separated results.
Precision values for each model, their combination, and baseline
Precision values for each model, their combination, and baseline, separated results for Twitter and non-Twitter text, from the test set
5.5 Discussion of accuracy, precision and recall, and related work
The research that is closer to the experiment presented in this paper is that by Han and Baldwin (Reference Han and Baldwin2011) and Liu et al. (Reference Liu, Weng and Jiang2012), since they both work on data, including text other than Twitter text, and they both work on the normalization of domain-unspecific text – Liu et al. (Reference Liu, Weng and Jiang2012) call their approach a broad-coverage system.
In terms of spell checking detection and generation of proposals, both Han and Baldwin (Reference Han and Baldwin2011) and Liu et al. (Reference Liu, Weng and Jiang2012) work with systems generated by themselves, rather than using existing resources. Particularly Liu et al. (Reference Liu, Weng and Jiang2012) present proposal generation strategies inspired by visual priming and in word similarity measures. With the former they can favor the generation and ranking of spelling corrections on the basis of their frequency in social text and the most economic transformation operation – the fewer characters that are changed, the better. With the latter they generate lists of representatives that one particular word can have in a text. For instance, for a word like please they can generate the set {pleas, pleeas, pleaas, pleeaas, pleeeaas}. Han and Baldwin (Reference Han and Baldwin2011) use an algorithm based on morphophonemic similarity to generate correction proposals. These two approaches allow for higher accuracy in the percentage of errors for which a correction proposal is generated – not selected.
In addition to the differences in the proposal generation strategies, there is also an important difference, which is that we include the normalization of tokens as part of the normalization process, as opposed to what the above-mentioned works do. For these reasons results are hardly comparable among the different experiments. With this caveat in mind, our restricted experiment, in which we are using only those cases where the spell-checker has been able to correctly detect an error and propose a set of correction candidates containing the reference correction, shows a higher precision in terms of the first ranked correct proposals.Footnote 9 Compare our 86.59% versus Liu et al.’s (2012) 84.13% when using a Twitter-based language model or their 79.12% when using a web language model. Their recall is respectively 78.38% and 77.11%, while Han and Baldwin (Reference Han and Baldwin2011) reach a precision and recall of 75.30%.
6 Conclusions
In this paper we presented and analyzed a Spanish corpus of UGC, including text from five different sources. We have seen that UGC text presents some particular features that set it apart from standard text. On the one hand, it contains specific phenomena that add expressiveness, such as emoticons, informal spellings, nonstandard capitalization, and reduplication. On the other hand, the rate of typical orthographic errors is much higher than in more edited types of text.
We analyzed the impact that norm deviations have on standard NLP processing tools by comparing the results of processing it with the results of processing its manually corrected version. As a result, we observed that UGC text has a negative impact on the performance of three basic NLP tasks: lemmatization, assignment of the main syntactic category, and assignment of morphosyntactic features.
We propose to address the problem by normalizing UGC text before linguistic processing. For this, we have used a conventional spell-checker on top of which we have built a module to automatically select the best correction candidate. For the selection task, we wanted to experiment with the use of linguistic information with various levels of fine-grainedness. Therefore, we have trained four language models on a large domain-specific corpus, each model containing a different degree of information in a trade-off with its generalization capabilities. The model based on the lowercase version of the corpus has turned out to be the most predictive, with a reasonable precision value of 86.59% as measured on the test set. This figure is well above the baseline and is slightly higher than the highest accuracy obtained by the related work of Liu et al. (Reference Liu, Weng and Jiang2012) – 84.13% in the ranking of correction proposals in the first position of the list. We have also experimented with two methods for combining the four models: model optimization directly on the final task, using a modified version of the SPSA algorithm, and a linear combination of the models optimized independently of the task. Both methods achieve comparable results, although the weights obtained for each language model are quite different. This is an interesting conclusion since optimizing independently for the task is much easier than the other way round. Similar conclusions have been obtained in different tasks (e.g. in a phrase-based statistical machine translation). Results of the combination of models are close to the best performing model, i.e. LC model, but do not perform better, at least on our test set of data. The evaluation for the additional experiment using only Twitter text shows a significant drop in precision, compared with non-Twitter text. The experiment of using different language models has been useful to obtain better results using the LC model instead of the baseline TC model.
Acknowledgments
The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. The authors would also like to thank Barcelona Media Innovation Center (where a part of the work was conducted) and Institute for Infocomm Research for their support and permission to publish this work.
The research leading to these results has received funding from the CDTI through the CENIT project Social Media and from the Seventh Framework Program of the European Commission through the International Outgoing Fellowship (IMTraP-2011-29951) and the Intra-European Fellowship (CrossLingMind-2011-300828) Marie Curie Actions.
