1 Introduction
People in virtual communities use words on screens to exchange pleasantries and argue, engage in intellectual discourse, conduct commerce, exchange knowledge, share emotional support, make plans, brainstorm, gossip, feud, fall in love, find friends and lose them, play games, flirt, create a little high art and a lot of idle talk.
Rheingold Howard (Reference Howard1993)The centrality of emotions in social life is manifested by the rich history of theories and debates about emotions and their nature (Solomon Reference Solomon1973; Ekman Reference Ekman and Blacking1977; Frijda Reference Frijda1986; Ortony, Clore and Collins Reference Ortony, Clore and Collins1988; Izard Reference Izard1993; Lazarus Reference Lazarus, Ekman and Davidson1994; Roseman and Smith Reference Roseman, Smith, Scherer, Schorr and Johnstone2001). Social interaction among people is an essential aspect of every society, a strong foundation for the development and self-actualization of a person, and for the establishment of genuine interpersonal relationships and communities. In computer-mediated communication world people tend to interact in a social way: they virtually remain in touch with their relatives and friends to exchange experiences, share opinions and feelings and satisfy their social need of interpersonal communication. Researchers (Peris et al. Reference Peris, Gimeno, Pinazo, Ortet, Carrero, Sanchiz and Ibanez2002) argue that online communication may stimulate rather than inhibit social relations, and chat users find it a medium for rich, intense and interesting experiences.
In our research we address the tasks of recognition and interpretation of affect communicated through written language in online communication environments, specifically, in Instant Messaging (IM) and diary-like blog posts, where people tend to use an informal and abbreviated style of writing, often accompanied by the use of emoticons (short symbols that resemble facial displays). The informal nature of language (e.g. use of acronyms and emoticons, underuse of apostrophes, etc.) pose a challenge for syntactic and dependency parsers employed as part of a pipeline in a variety of NLP tasks, including affect sensing in online communication media (see recent work of Foster (Reference Foster2010), who investigated the parser performance on discussion forum posts and attempted to handle problematic phenomena by transforming the input sentence or the material, on which the parser was trained). To address this challenge in the task of affect recognition from text, we propose a rule-based linguistic approach designed to deal with not only grammatically and syntactically correct textual input, but also informal messages written in an abbreviated or expressive manner (Neviarouskaya, Prendinger and Ishizuka Reference Neviarouskaya, Prendinger and Ishizuka2007a).
Our Affect Analysis Model (AAM) performs fine-grained classification of sentences using ten categories: nine emotions (‘Anger’, ‘Disgust’, ‘Fear’, ‘Guilt’, ‘Interest’, ‘Joy’, ‘Sadness’ (‘distress’), ‘Shame’ and ‘Surprise’) defined by Izard (Reference Izard1971) and neutral. The proposed rule-based approach processes each sentence in stages, including symbolic cue processing, detection and transformation of abbreviations, sentence parsing and word/phrase/sentence-level analyses. Each analysed sentence is automatically annotated with emotion (or neutral) label and numerical intensity (the strength of emotion).
We evaluate the performance of AAM on two data sets of sentences from diary-like blog posts (our data set, and data set provided by Aman and Szpakowicz Reference Aman and Szpakowicz2007), on sentences from fairy tales (Alm's 2008 data set) and news headlines (corpus provided by Strapparava and Mihalcea Reference Strapparava and Mihalcea2007). The AAM shows promising results regarding its capability to accurately recognize fine-grained emotions reflected in sentences from diary-like blog posts (averaged accuracy is up to 77 per cent), fairy tales (averaged accuracy is up to 70.2 per cent) and news headlines (our algorithm outperformed eight other systems on several measures).
The remainder of the paper is structured as follows. In Section 2, we report on related work. The special features of online communication are described in Section 3. Section 4 discusses the development of Affect database. Our AAM, its evaluation and experimental results are detailed in Sections 5 and 6. Finally, in Section 7, we discuss the performance of the proposed method and conclude the paper.
2 Background and related work
Language is about something, does something, and is something in itself; the content and conduct of emotional communication are integrally related.
Donald Brenneis (Reference Brenneis, Lutz and Abu-Lughod1990: 114)
Issues of recognition, interpretation and representation of affect have been extensively investigated by researchers in the field of affective computing (Picard Reference Picard1997). A wide range of modalities has been considered, including affect in speech, facial display, posture and physiological activity. Recently, textual information has been gaining increased attention of researchers interested in studying different kinds of affective phenomena, including sentiment, subjectivity and emotions. In order to analyse affect communicated through written language, researchers in the area of natural language processing have proposed a variety of approaches, methodologies and techniques.
Various approaches to subjectivity, sentiment or affect analysis on different textual composition levels have been proposed:
(1) word level: Subasic and Huettner (Reference Subasic and Huettner2001); Kamps and Marx (Reference Kamps and Marx2002); Riloff, Wiebe and Wilson (Reference Riloff, Wiebe and Wilson2003); Turney and Littman (Reference Turney and Littman2003); Baroni and Vegnaduzzo (Reference Baroni and Vegnaduzzo2004); Andreevskaia and Bergler (Reference Andreevskaia and Bergler2006); Strapparava, Valitutti and Stock (Reference Strapparava, Valitutti and Stock2007);
(2) synset level: Esuli and Sebastiani (Reference Esuli and Sebastiani2006); Wiebe and Mihalcea (Reference Wiebe and Mihalcea2006);
(3) phrase level: Wilson, Wiebe and Hoffmann (Reference Wilson, Wiebe and Hoffmann2005);
(4) clause or sentence level: Olveres et al. (Reference Olveres, Billinghurst, Savage and Holden1998); Boucouvalas (Reference Boucouvalas2003); Liu, Lieberman and Selker (Reference Liu, Lieberman and Selker2003); Yu and Hatzivassiloglou (Reference Yu and Hatzivassiloglou2003); Mulder et al. (Reference Mulder, Nijholt, den Uyl and Terpstra2004); Read (Reference Read2004); Kim and Hovy (Reference Kim, Hovy, Dale, Wong, Su and Kwong2005); Neviarouskaya, Prendinger and Ishizuka (Reference Neviarouskaya, Prendinger, Ishizuka, Paiva, Prada and Picard2007c); Moilanen and Pulman (Reference Moilanen and Pulman2007); Alm (Reference Alm2008); Aman and Szpakowicz (Reference Aman and Szpakowicz2008); Choi and Cardie (Reference Choi and Cardie2008);
(5) paragraph or document level: Subasic and Huettner (Reference Subasic and Huettner2001); Turney (Reference Turney2002); Pang, Lee and Vaithyanathan (Reference Pang, Lee and Vaithyanathan2002); Mishne (Reference Mishne2005); Kim and Hovy (Reference Kim and Hovy2006); Leshed and Kaye (Reference Leshed and Kaye2006); Mihalcea and Liu (Reference Mihalcea and Liu2006); Nadeau et al. (Reference Nadeau, Sabourin, De Koninck, Matwin and Turney2006).
2.1 Lexical resources
To support applications relying on the recognition of textual subjectivity, semantic orientation and affective language, researchers have created different lexical resources: subjective (Wilson et al. Reference Wilson, Wiebe and Hoffmann2005), affective (Strapparava and Valitutti Reference Strapparava and Valitutti2004), appraisal (Argamon et al. Reference Argamon, Bloom, Esuli and Sebastiani2007) and polarity (Hatzivassiloglou and McKeown Reference Hatzivassiloglou and McKeown1997; Esuli and Sebastiani Reference Esuli and Sebastiani2006; Neviarouskaya, Prendinger and Ishizuka Reference Neviarouskaya, Prendinger and Ishizuka2009) lexicons.
Hatzivassiloglou and McKeown (Reference Hatzivassiloglou and McKeown1997) created a list of 1,336 adjectives manually labelled as either positive or negative. The subjectivity lexicon developed by Wilson et al. (Reference Wilson, Wiebe and Hoffmann2005) is comprised by over 8,000 subjectivity clues annotated by type (strongly subjective/weakly subjective) and prior polarity (positive/negative/both/neutral). Motivated by the assumption that ‘different senses of the same term may have different opinion-related properties’, Esuli and Sebastiani (Reference Esuli and Sebastiani2006) developed a SentiWordNet lexicon based on WordNet (Miller Reference Miller1990) synsets comprised from synonymous terms. Three numerical scores (Obj(s), Pos(s) and Neg(s), which range from 0.0 to 1.0 and in sum equal to 1.0), characterizing to what degree the terms included in a synset are objective, positive and negative, were automatically determined based on the proportion of eight ternary classifiers assigning the corresponding label to the synset. The SentiFul database, which contains an extensive list of sentiment-conveying adjectives, adverbs, nouns and verbs annotated by sentiment polarity, polarity scores/intensities and weights, was introduced in Neviarouskaya et al. (Reference Neviarouskaya, Prendinger and Ishizuka2009). The Appraisal lexicon developed by Argamon et al. (Reference Argamon, Bloom, Esuli and Sebastiani2007) contains adjectives and adverbs annotated by attitude type (affect, judgment, appreciation) and orientation. Aimed at introducing the hierarchy of “affective domain labels”, Strapparava and Valitutti (Reference Strapparava and Valitutti2004) created WordNet-Affect, a lexicon of affective concepts, based on the subset of WordNet synsets. Affective labels for the concepts related to emotional state, moods, traits, situations evoking emotions or emotional responses were assigned to the WordNet-Affect entries (e.g. ‘happy’ – EMOTION, ‘aggressiveness’ – TRAIT, etc.).
2.2 Sentiment analysis in text
2.2.1 Machine-learning approach
An unsupervised statistical method for the task of separating opinions from facts and classifying opinions as positive, negative or neutral, using Naïve Bayes classifier, proposed by Yu and Hatzivassiloglou (Reference Yu and Hatzivassiloglou2003), resulted in a high accuracy (up to 91 per cent) at a sentence level. The best performance was observed when words, bigrams, trigrams, part-of-speech and polarity were included in the feature set. The decision on the polarity of a sentence was based on the number and strength of semantically oriented words in the sentence. Kim and Hovy (Reference Kim, Hovy, Dale, Wong, Su and Kwong2005) built a classifier that identified all sentences expressing polarity in a given text based on strong markers of opinion such as certain modal verbs, adjectives and adverbs. To automatically distinguish prior and contextual polarity of individual words and phrases in sentiment expressions, Wilson et al. (Reference Wilson, Wiebe and Hoffmann2005) employed a machine learning method with not only lexical (e.g. word, modification) and polarity (e.g. negation, polarity shifter) features, but also syntactic structure features.
In order to overcome the problem of strong dependency of machine learning techniques on domain, topic and time, Read (Reference Read2005) constructed a corpus of text marked-up with emoticons and developed the emoticon-trained classifier aimed at sentiment classification. While this classifier performed well (up to 70 per cent accuracy) on the articles extracted from the constructed corpus, it was not very effective in predicting the polarity of movie reviews and news. Read (Reference Read2005) inferred that there exists the language-style dependency in sentiment classification.
The model of integration of machine learning approach with compositional semantics was proposed by Choi and Cardie (Reference Choi and Cardie2008). A dependency tree-based method for sentiment classification of Japanese and English subjective sentences using conditional random fields with hidden variables was recently introduced by Nakagawa, Inui and Kurohashi (Reference Nakagawa, Inui and Kurohashi2010). This approach relies on the lexicon (sentiment polarity expressions and polarity reversing words), dependency parser and a probabilistic model to handle interactions between hidden variables.
2.2.2 Rule-based linguistic approach
To analyse contextual sentiment (polarity) of a phrase or a sentence, rule-based linguistic approaches (Nasukawa and Yi Reference Nasukawa and Yi2003; Mulder et al. Reference Mulder, Nijholt, den Uyl and Terpstra2004; Moilanen and Pulman Reference Moilanen and Pulman2007; Subrahmanian and Reforgiato Reference Subrahmanian and Reforgiato2008) were proposed.
There is a strong tie between our approach to affect analysis from text with the work of Moilanen and Pulman (Reference Moilanen and Pulman2007) on sentiment composition. In their work, Moilanen and Pulman (Reference Moilanen and Pulman2007) propose a theoretical composition model employing deep dependency parsing, sentiment propagation, polarity reversal and polarity conflict resolution within various linguistic constituent types at various grammatical levels. The experiments with the developed lexical system revealed the crucial dependency on a wide-coverage lexicon, accurate parsing and sentiment sense disambiguation in a compositional approach to sentiment analysis. The significant differences between our approaches lie in (1) the levels of classification (fine-grained classes, or nine emotions, and neutral, in our work versus polarity-based classes, or positive/negative/neutral in Moilanen and Pulman Reference Moilanen and Pulman2007); (2) the use of content words playing the role of reversers of a non-neutral polarity of related words (e.g. ‘reduce’, ‘prevention’) in their algorithm. However, it is not trivial to find pairs of opposite emotions in the case of a fine-grained classification, so such reversers cannot be applied here in a straightforward manner.
2.3 Affect analysis in text
2.3.1 Lexical approach
An approach to analysing affect content in free text using fuzzy logic techniques was proposed by Subasic and Huettner (Reference Subasic and Huettner2001). Some researchers employed a keyword-spotting technique to recognize emotion in text (Olveres et al. Reference Olveres, Billinghurst, Savage and Holden1998; Strapparava et al. Reference Strapparava, Valitutti and Stock2007) or expressed in a multi-modal way (e.g. speech signals along with textual content, Chuang and Wu Reference Chuang and Wu2004). However, the use of a purely word-level analysis model cannot cope with cases where affect is expressed by phrases requiring complex phrase/sentence-level analyses, as words are interrelated and influence each other's affect-related interpretation (as in the sentence ‘I use the ability to breathe without guilt or worry’), or when a sentence carries affect indirectly through underlying meaning (e.g. ‘I punched my car radio, and my knuckle is now bleeding’).
2.3.2 Machine-learning approach
With the aim to classify blog sentences by six basic emotions (Ekman Reference Ekman1993), Aman and Szpakowicz (Reference Aman and Szpakowicz2008) used machine learning model that utilized corpus-based features (unigrams) and the following emotion lexicons: Roget's Thesaurus (Jarmasz and Szpakowicz Reference Jarmasz and Szpakowicz2001) and WordNet-Affect (Strapparava and Valitutti Reference Strapparava and Valitutti2004). The text-based emotion prediction problem in the domain of children's fairy tales was explored by Alm, Roth and Sproat (Reference Alm, Roth and Sproat2005) using a supervised machine-learning approach. As researchers did not have sufficient training data to classify sentences according to fine-grained distinct emotions, in their preliminary study, Alm et al. (Reference Alm, Roth and Sproat2005) focused only on three categories: neutral, positive emotion and negative emotion. In her dissertation, Alm (Reference Alm2008) described the refined and improved feature set, and presented the results of experiments on fine-grained emotion classification of text using a hierarchical sequential model. To automatically recognize emotions in news headlines, Katz, Singleton and Wicentowski (Reference Katz, Singleton and Wicentowski2007) employed a supervised system based on unigram model, and Strapparava and Mihalcea (Reference Strapparava and Mihalcea2008) proposed several methods using Latent Semantic Analysis (LSA) technique and Naïve Bayes classifier trained on the corpus of blog posts annotated by emotions. Researchers also applied statistical language modelling techniques to analyse moods conveyed through online diary-like posts (Mishne Reference Mishne2005; Leshed and Kaye Reference Leshed and Kaye2006; Mihalcea and Liu Reference Mihalcea and Liu2006).
The weak points of these methods include: large corpora required for meaningful statistics and good performance; neglect of some prepositions, negation, modal and condition constructions (although the work of Sokolova and Szpakowicz (Reference Sokolova, Szpakowicz, Kegl and Lapalme2005) on analysis and classification of strategies of successful and unsuccessful electronic negotiations is not closely related to our work on fine-grained classification of emotion sentences, we acknowledge here that they used negations and modal verbs in their machine-learning algorithm); disregard of syntactic relations and semantic dependencies in sentences; and long processing time.
2.3.3 Rule-based linguistic approach
Advanced rule-based linguistic approaches targetting textual affect recognition at the sentence level are described in Boucouvalas (Reference Boucouvalas2003), Liu et al. (Reference Liu, Lieberman and Selker2003) and Chaumartin (Reference Chaumartin2007). Boucouvalas (Reference Boucouvalas2003) developed the Text-to-Emotion Engine based on word tagging and analysis of sentences. The proposed system uses a small set of emotions, the six basic types defined by Ekman (Reference Ekman1993). The emotion extraction engine can analyse input text from a chat environment, identify the emotion communicated and deliver the parameters necessary to invoke an appropriate expressive image on the user's display. However, the proposed system employs a parser that generates emotional output only if an emotional word refers to the person himself/herself and the sentence is in present continuous or present perfect continuous tense. We believe that such limitations greatly narrow the potential of textual emotion recognition. As a result, sentences like ‘Onion pie is disgusting’ and ‘It was the most joyous feeling!’ are disregarded by the parser despite the fact that they evidently carry affect. An approach for understanding the underlying semantics of language using large-scale real-world commonsense knowledge was proposed by Liu et al. (Reference Liu, Lieberman and Selker2003), who incorporated the affect sensing engine into an affectively responsive email composer called EmpathyBuddy. Chaumartin (Reference Chaumartin2007) developed a rule-based system relying on the lexicon from WordNet-Affect (Strapparava and Valitutti Reference Strapparava and Valitutti2004) and SentiWordNet (Esuli and Sebastiani Reference Esuli and Sebastiani2006), and applied it to affect sensing in news headlines.
The weakness of most affect recognition systems integrated with chat (Olveres et al. Reference Olveres, Billinghurst, Savage and Holden1998; Boucouvalas Reference Boucouvalas2003) or e-mail (Liu et al. Reference Liu, Lieberman and Selker2003) browsers, or analysing diary-like blogs (Aman and Szpakowicz Reference Aman and Szpakowicz2008), is that they do not take into account crucial aspects of informal online conversation such as its specific style and evolving language. In order to account for the peculiarity of online messaging and blogs, and to ensure satisfactory results on real examples, we investigated style, linguistic and interactional features of online communication (see Section 3 for details), and took these into consideration in constructing our AAM relying on rule-based linguistic approach.
3 Features of online communication media
Many Internet users adopt online communication not only to conduct business but also to keep in touch with their family and friends, to seek emotional support or to search for new interesting relationships. Nowadays, IM has proven to be one of the most popular online applications. As stated by Shiu and Lenhart (Reference Shiu and Lenhart2004), younger Internet users employ IM in greater numbers and more ardently than older generations.
In order to construct a practical and usable system, we investigated the style of communication and the linguistic and interactional features of real-time conversations. Linguistic features of online communication media (chats, IMs, blogs, discussion forums, social networks), such as emoticons, unconventional spellings, representations of spoken language features, regional dialect features, etc. have been extensively studied by the linguists and sociolinguists (Androutsopoulos Reference Androutsopoulos2006; Herring Reference Herring and Donsbach2008). The main problem in messaging is that people cannot easily keep up with the evolving language. Although some of the abbreviations, such as ASAP (‘as soon as possible’), FYI (‘for your information’) or TIA (‘thanks in advance’), are widely known, most of the acronyms are only used within the context of online environments. Examples include: BC (‘because’), 2l8 (‘too late’), CUL (‘see you later’), etc. Participants often use different levels of abbreviations, and hence find it annoying when abbreviations are used without surrounding context to help the correct understanding of their meaning. During the study conducted by Grinter and Eldridge (Reference Grinter and Eldridge2001), the teenager subjects reported using several different abbreviations for the same words, which makes text messages difficult to parse.
Successful computer-mediated communication, particularly within the IM environment or diary-like blogs, depends on the use of various symbolic conventions, such as emoticons (to portray emotion states or communicative behaviour), capital letters or asterisks (to emphasize words), special symbols, etc. Trends show that IM users are increasingly turning to such expressive textual cues to supplement the lack of non-verbal (visual and aural) cues (Hu et al. Reference Hu, Wood, Smith and Westbrook2004; Derks Reference Derks2007). In her dissertation, Derks (Reference Derks2007) examined the use of emoticons (short symbols that resemble facial displays) in text-based computer-mediated communication, and observed that online messages are often replete with emoticons to fill the conversational gaps and to give additional social and emotional meaning. The study showed that: (1) the most common motives for emoticon use are ‘expressing emotion’, ‘strengthening a message’ and ‘expressing humour’; (2) most emoticons are used towards friends, as compared to strangers; (3) more emoticons are used in positive than in negative contexts (spontaneously as well as intentionally) (Derks Reference Derks2007: 112).
4 Affect database
In this section, we describe Affect database created to support the handling of abbreviated language and the interpretation of affective features of emoticons, abbreviations and words by an automatic emotion recognition system. Our Affect database includes the following tables: Emoticons, Abbreviations, Adjectives, Adverbs, Nouns, Verbs, Interjections and Modifiers.
Three independent annotators (non-native English speakers studying at the Graduate School of Information Science and Technology, the University of Tokyo) were asked to manually label the entries of the database using nine emotion categories (‘Anger’, ‘Disgust’, ‘Fear’, ‘Guilt’, ‘Interest’, ‘Joy’, ‘Sadness’, ‘Shame’ and ‘Surprise’) and intensities. Emotion intensity values range from 0.0 to 1.0, and describe the intensity degree of affective states from ‘very weak’ to ‘very strong’. Annotators conformed to our guidelines with the description of emotional state gradation within intensity levels. For example, ‘cheerful’, ‘glad’, ‘happy’, ‘joyful’ and ‘elated’ all correspond to the ‘Joy’ emotional state, but to a different degree of intensity (0.2, 0.4, 0.6, 0.8 and 1.0, correspondingly). In the resulting intensity estimation for each affect-related entry, variance of data from the annotator mean was taken into consideration. If the variance was less than or equal to 0.027, the resulting intensity was measured as the average of intensities given by three annotators. Otherwise, the intensity value responsible for exceeding the variance threshold was removed, and only the remaining values were taken into account.
For the accumulation of relevant and most often used emoticons and abbreviations (along with their transcriptions), we employed five online dictionaries dedicated to and describing such data (please see the examples of such dictionaries of emoticonsFootnote 1 and abbreviationsFootnote 2). Only entries that occurred in at least three sources were selected. In this way, we collected 364 emoticons, both of American and Japanese style (e.g. ‘:’‘>’ and ‘=∧_∧=’ for ‘blushing’), and the 337 most popular acronyms and abbreviations, both emotional and non-emotional (e.g. ‘BL’ for ‘belly laughing’, ‘gj’ for ‘good job’ and ‘4U’ – ‘for you’). As interjections, such as ‘alas’, ‘wow’, ‘yay’, ‘ouch’, etc. are specific indicators of communicated emotion caused by unexpectedness, a long-awaited joyful event or pain, they were collected as well.
Emoticons and abbreviations were transcribed and related to named affective states (with intensity), whereby each entry was assigned to only one category. The inter-rater agreement on the assigned category was calculated using Fleiss' Kappa statistics (Fleiss Reference Fleiss1971). The measured Kappa coefficients for emoticons and abbreviations are 0.94 and 0.93, respectively, showing strong annotation reliability (supposedly, due to unambiguous transcriptions provided along with these symbolic cues). The percentage distributions of emoticons and abbreviations according to resulting affective labels (majority vote) are as follows (in descending order):
(1) Emoticons: ‘Joy’ – 45, ‘Sadness’ – 23, ‘Fear’ – 11, ‘Anger’ – 7, ‘Surprise’ – 7, ‘Disgust’ – 3, ‘Shame’ – 3, ‘Interest’ – 1 and ‘Guilt’ – 0 per cent;
(2) Abbreviations: ‘Joy’ – 74, ‘Guilt’ – 7, ‘Surprise’ – 6, ‘Disgust’ – 4, ‘Fear’ – 3, ‘Anger’ – 2, ‘Sadness’ – 2, ‘Shame’ – 2 and ‘Interest’ – 0 per cent.
By way of example, some emoticons and abbreviations extracted from the developed database are listed in Table 1.
Examples of emoticons and abbreviations extracted from Affect database
The next category consists of words conveying affective content. From WordNet-Affect (Strapparava and Valitutti Reference Strapparava and Valitutti2004), we have taken 1,627 words – adjectives (635), nouns (521), verbs (274) and adverbs (197) – that refer directly to emotions, moods, traits, cognitive states, behaviour, attitudes and sensations. Moreover, we added to our database 434 words that carry the potential to elicit affective states in humans (e.g. ‘beautiful’, ‘disaster’, ‘break’, ‘deceive’, ‘violate’, etc.). These words are considered as indirect emotion words describing the objects and situations that lead to some emotional reactions.
Considering the fact that some affective words may express more than one emotion state, annotators could relate words to more than one category. For instance, in the annotation of the word ‘frustrated’, both ‘Anger’ and ‘Sadness’ emotions are involved, with intensities 0.2 and 0.7, respectively (see Table 2).
Examples of words taken from Affect database
Assignments of emotion labels to the same word might differ among annotators. We faced the difficulty of employing Fleiss' Kappa coefficient (Fleiss Reference Fleiss1971) to measure inter-rater agreement here, as the important requirement of using it is that each entry needs to be assigned to only one of possible categories. For the resulting labelling, we only considered emotion categories that occurred in the assignments of at least two annotators. The most frequent emotion labels in resulting sets were ‘Joy’ and ‘Sadness’ (34.3 per cent and 30.0 per cent of overall number of affective words, respectively) whereas the least frequent was ‘Guilt’ (3.1 per cent). The distribution of affective words with one, two and three emotion labels is 67 per cent, 29 per cent and 4 per cent, respectively. Only one word (adjective ‘aggravated’) was annotated by four resulting emotion labels (‘Anger:0.5’, ‘Disgust:0.5’, ‘Sadness:0.3’, ‘Fear:0.1’).
Regarding the emotion intensity annotations of affective words, we observed interesting statistics within each of the nine emotion categories. The percentages of cases with valid (not exceeding the threshold) variance of given intensities within each emotion category are as follows (in descending order): ‘Shame’ – 57.8, ‘Guilt’ – 51, ‘Anger’ – 49.8, ‘Fear’ – 42.7, ‘Disgust’ – 39.2, ‘Surprise’ – 27.8, ‘Sadness’ – 26.6, ‘Joy’ – 18.8 and ‘Interest’ – 8.6 per cent. The annotators easily agreed in intensity assignments to ‘Shame’, ‘Guilt’ and ‘Anger’ categories, in contrast to frequent disagreement in cases of ‘Interest’, ‘Joy’ and ‘Sadness’. We can only speculate that disagreement is related to the huge diversity of ‘joyful’ and ‘sad’ synonymous words with different emotional colorations, and due to the fuzziness of the ‘interest’ concept (some of psychologists do not consider ‘interest’ as an emotional state at all).
As to the indirect affective words that possibly induce emotional states through indication of emotional causes or responses (notions of direct and indirect affective words were used by Strapparava, Valitutti and Stock Reference Strapparava, Valitutti and Stock2006), about 300 nouns from WordNet-Affect (Strapparava and Valitutti Reference Strapparava and Valitutti2004) along with the categories, to which they correspond, were kindly provided by Dr. Alessandro Valitutti. Some examples are given in Table 3.
Examples of indirect affective nouns from WordNet-Affect and their annotations
In order to label indirect emotion words using our fine-grained categories, the annotations of direct affective words that (1) represent emotion labels in WordNet-Affect and (2) are already included in Affect database (see annotations in brackets [] in the middle column of Table 3) were automatically analysed. The maximum intensity within the same emotion label from Affect database was taken as the resulting intensity for that emotion state in final annotations (see some results in the last column of Table 3).
Further, we included 112 modifiers (e.g. ‘very’, ‘extremely’, ‘slightly’, ‘hardly’, ‘less’, ‘not’, etc.) into our database, as they influence the strength of related words and phrases in a sentence. Adverbs of degree have an impact on neighbouring verbs, adjectives or another adverb, and are used to mark that the extent or degree is either greater or less than usual (Biber et al. Reference Biber, Johansson, Leech, Conrad, Finegan and Quirk1999). In Benamara et al. (Reference Benamara, Cesarano, Picariello, Reforgiato and Subrahmanian2007), the authors use adverbs of degree to modify the score of adjectives in sentiment analysis. In our work, such adverbs along with some of the prepositions constitute the set of modifiers. Two annotators gave coefficients for intensity degree strengthening or weakening (from 0.0 to 2.0) to them, and the result was averaged (see Table 4).
Examples of modifiers with coefficients of intensity degree strengthening or weakening
5 The Affect Analysis Model
We have developed a novel algorithm for analysing affect expressed by text messages (Neviarouskaya et al. Reference Neviarouskaya, Prendinger and Ishizuka2007a, Reference Neviarouskaya, Prendinger, Ishizuka, Paiva, Prada and Picard2007c). The proposed algorithm consists of five main stages: (1) symbolic cue analysis; (2) syntactic structure analysis; (3) word-level analysis; (4) phrase-level analysis; (5) sentence-level analysis. The architecture of the AAM is presented in Figure 1.
Architecture of the Affect Analysis Model.
5.1 Symbolic cue analysis
In the first stage, the sentence is tested for occurrences of emoticons, abbreviations, acronyms, interjections, ‘question mark’ and ‘exclamation mark’, repeated punctuation and capital letters. First, punctuation marks in a sentence are delimited from words in order to disambiguate sentence punctuation marks from those belonging to emoticons. The detected ‘exclamation mark’, repeated punctuation and capital letters are considered as an emphasis of the communicated emotion.
If there is an emoticon or abbreviation related to an emotional state, no further analysis of affect in text is performed based on the simplifying assumption that the emoticon (or abbreviation) dominates the affective meaning of the entire (simple or compound) sentence. It is known that people type emoticons and emotional abbreviations to show actual feeling (e.g. ‘I have taken the exams timetable already: S [worry; Fear:0.4]’), or to avoid misleading the other participants, for instance, after irony, joke or sarcasm (e.g. ‘Thank you so much for your kind encouragement: -( [Sadness:0.8]’ or ‘If you miss the meeting, I will hunt you down and murder you:) [Joy:0.6]’). In a face-to-face communication sarcasm is conveyed by a positive tone or a smile and a negative message (Planalp and Knie Reference Planalp, Knie and Fussell2002). Similarly, emoticons ‘can create ambiguity and express sarcasm online by varying the polarity of the emoticon and the polarity of the message’ (Derks Reference Derks2007: 63). On the other hand, if there are multiple emoticons or emotion-relevant abbreviations in the sentence, we determine the prevailing (or dominant) emotion based on the following two (simplifying) rules.
(1) When emotion categories of the detected emoticons (or abbreviations) are the same (e.g. ‘G [grin; Joy:0.6] it is nice song too;-) [winking; Joy:0.3]’), the higher intensity value is taken for this emotion.
(2) When they are different (e.g. ‘I did not save that song: S [worry; Fear:0.4], please send it once more;”> [blushing; Shame:0.5]’), the category (and intensity) of the emoticon occurring last is considered dominant.
As interjections are added to text to reflect the author's feelings, as in the sentences ‘Oh no, I forgot that the exam was today!’ and ‘But anyways, yay!’, they are analysed as well. In case of an interrogative sentence, we process it further at subsequent stages in order to identify whether the question expresses a strong emotion or not. While some researchers (Boucouvalas Reference Boucouvalas2003) ignore such sentences, we believe that questions, like ‘Why do you irritate me so greatly?’ may carry emotional content.
It is important to emphasize here that we distinguish two ways of assigning an emotional value to the sentence. In one case (as described above), the affective information is provided by emotion-related emoticons or abbreviations, and in the other one by the lexical meaning propagated through rules to the sentence level. If there are no emotion-relevant emoticons or abbreviations in a sentence, we prepare the sentence for parser processing by replacing non-emotional abbreviations and acronyms by their proper transcriptions found in the database (e.g. ‘I m [am] stressed bc [because] i have frequent headaches’). In such a way, the issue of correct processing of abbreviated text by syntactic parser is resolved.
5.2 Syntactic structure analysis
The second stage of AAM algorithm is devoted to syntactic structure analysis, and it is divided into two main subtasks: (1) sentence analysis by the syntactic parser, Connexor Machinese Syntax,Footnote 3 developed by the Connexor Oy company; (2) parser output processing. Connexor Machinese Syntax provides a full analysis of texts by showing how words and concepts relate to each other in sentences, with competitive speed and accuracy. This tool assigns meaning-oriented syntactic structure to text, thus helping analytic applications understand text beyond the level of words, phrases and entities. The parser returns exhaustive information for analysed sentences, including word base forms (lemmas), parts of speech, dependency functions representing relational information between words in sentences, syntactic function tags and morphological tags.
When handling the parser output, we represent the sentence as a set of primitive clauses (either independent or dependent). Each clause might include Subject formation, Verb formation and Object formation, each of which may consist of a main element (subject, verb or object) and its attributives and complements. The developed algorithm can detect not only subjects represented by noun phrases, but also subjects represented by gerund (non-finite verb form) as in the sentence ‘Walking on the beach is a pleasure’, by an infinitive as in the sentence ‘To offend the youngest child is an obscene action’ or by a full clause, introduced by ‘that’, itself containing a subject and a predicate like in the sentence ‘That tomorrow weather will be sunny is great’. For the processing of complex or compound sentences, we build a so-called ‘relation matrix’, which contains information about dependences that the verbs belonging to different clauses have.
5.3 Word-level analysis
After handling the result from the previous analysis stage, the system transfers the data to the third stage, word-level analysis. For each word (found in Affect database) of a sentence, the affective features of a word are represented as a vector of emotion state intensities e = [Anger, Disgust, Fear, Guilt, Interest, Joy, Sadness, Shame, Surprise]. Here are three examples: e(‘rude’) = [0.2,0.4,0,0,0,0,0,0,0]; e(‘brotherly’) = [0,0,0,0,0,0.2,0,0,0]; and e(‘love’) = [0,0,0,0,0.8,1.0,0,0,0]. In the case of a modifier, the system identifies its coefficient (e.g. coeff(‘barely’) = 0.4).
Adjectives and adverbs have two forms that indicate degrees of comparison: comparative form and superlative form. As our Affect database contains words only in their dictionary form, one important system function at this stage is to increase the intensity of the emotion vector of an adjective (e.g. e(‘glad’) = [0,0,0,0,0,0.4,0,0,0]), or emotional adverb, if it is in comparative or superlative form, by multiplication by values 1.2 or 1.4, respectively (e.g. e(‘gladder’) = [0,0,0,0,0,0.48,0,0,0] and e(‘gladdest’) = [0,0,0,0,0,0.56,0,0,0]). Two persons were involved in the procedure of defining these multipliers. After annotators had manually assigned intensities to the set of words (e.g. ‘good’, ‘better’, ‘best’), multipliers were derived from the averaged assignments.
5.4 Phrase-level analysis
In the fourth stage, phrase-level analysis is performed. The purpose of this stage is to detect emotions involved in phrases, and then in Subject, Verb and Object formations (for definitions, see Section 5.2). Words in a sentence are interrelated and, hence, each of them can influence the overall meaning and affective bias of a statement. We have defined rules for processing general types of phrases with regard to affective content:
(1) Adjective phrase: modify the vector of adjective (e.g. e(‘extremely doleful’) = coeff(‘extremely’) * e(‘doleful’) = 2.0 * [0,0,0,0,0,0,0.4,0,0] = [0,0,0,0,0,0,0.8,0,0]).
(2) Noun phrase: output vector with the maximum intensity within each corresponding emotional state in analysing vectors (e.g. e1 = [0..0.7..] and e2 = [0.3..0.5..] yield e3 = [0.3..0.7..]). For instance, e(‘brotherly love’) = [0,0,0,0,0.8,1.0,0,0,0] where e(‘brotherly’) = [0,0,0,0,0,0.2,0,0,0] and e(‘love’) = [0,0,0,0,0.8,1.0,0,0,0]. In the rare case of words with opposite polarities, the resulting vector will contain mixed emotions (e.g. e(‘annoying care’) = [0.3,0,0,0,0.2,0.2,0,0,0] where e(‘annoying’) = [0.3,0,0,0,0,0,0,0,0] and e(‘care’) = [0,0,0,0,0.2,0.2,0,0,0]).
(3) Verb plus adverbial phrase: output vector with the maximum intensity within each corresponding emotional state in analysing vectors (e.g. e(‘to shamefully deceive’) = [0,0.4,0,0,0,0,0.5,0.7,0] where e(‘shamefully’) = [0,0,0,0,0,0,0,0.7,0] and e(‘to deceive’) = [0,0.4,0,0,0,0,0.5,0,0]).
(4) Verb plus noun phrase: if verb and noun phrase have opposite polarities (e.g. ‘to deceive hopes’, ‘to enjoy bad weather’), consider vector of verb as dominant (for instance, e(‘to deceive hopes’) = [0,0.4,0,0,0,0,0.5,0,0] where e(‘to deceive’) = [0,0.4,0,0,0,0,0.5,0,0] and e(‘hope’) = [0,0,0,0,0.1,0.3,0,0,0]); if polarities are the same (e.g. ‘to celebrate victory’, ‘to hate crying’), output vector with maximum intensity in corresponding emotional states.
(5) Verb plus adjective phrase (e.g. ‘is very kind’, ‘feel bad’): output vector of adjective phrase, as adjectives can come only after ‘stative’ verbs, which do not express actions, and they always refer to and qualify the subject of the sentence.
The rules for modifiers are as follows:
(1) Adverbs of degree increase or decrease emotional intensity values.
(2) Negation modifiers such as ‘no’, ‘not’, ‘never’, ‘any’, ‘nothing’ and connector ‘neither. . .nor’ cancel (set to zero) vectors of the related words, i.e. ‘neutralize the emotional content’ (e.g. positive vector of ‘exciting’ is neutralized due to ‘nothing’ in ‘Yesterday I went to a party, but nothing exciting happened there’). We use this rule as an initial heuristic, as it is problematic to find pairs of opposite emotions (except for ‘Joy’ and ‘Sadness’), in contrast to straightforward reversing of the polarity in polarity-based classification.
(3) Prepositions such as ‘without’, ‘except’, ‘against’, ‘despite’ cancel vectors of related words (e.g. the phrase ‘despite his endless demonstrations of rude power’ and the sentence ‘I climbed the mountain without fear’ are neutralized due to prepositions).
Statements beginning with words like ‘think’, ‘believe’, ‘sure’, ‘know’, ‘doubt’ or with modal verbs such as ‘can’, ‘may’, ‘must’, ‘need’, ‘would’, etc. are not considered by our system, as they express a modal attitude towards the proposition. Conditional clause phrases beginning with ‘after’, ‘although’, ‘as if’, ‘as though’, ‘before’, ‘even if’, ‘even though’, ‘if’, ‘if only’, ‘unless’, ‘whether’, ‘when’, ‘whenever’, etc. are disregarded as well (e.g. ‘I eat when I'm angry, sad, bored. . .’, or ‘If only my brain was like a thumbdrive, how splendid it would be’).
Each of the Subject, Verb or Object formations may contain words conveying emotional meaning. During this stage, we apply the described rules to phrases detected within formation boundaries. Finally, each formation can be represented as a unified vector encoding its emotional content.
5.5 Sentence-level analysis
In the fifth and final stage, the overall emotion of a sentence and its resulting intensity degree are estimated. Our algorithm enables processing of different types of sentences, such as simple, compound, complex (with complement or relative clauses) or complex–compound.
5.5.1 Emotion vector of a simple sentence (or a clause)
The emotion vector of a simple sentence (or a clause) is generated from Subject, Verb and Object formation (SF, VF and OF, respectively) vectors resulting from phrase-level analysis. The main idea here is to first derive the emotion vector of Verb-Object formation relation. It is estimated based on the ‘verb plus noun phrase’ rule described above. In order to apply this rule, we automatically determine polarities of Verb and Object formations using their unified emotion vectors (particularly, non-zero-intensity emotion categories). For instance, polarity of ‘to calm disobedient child’ is positive based on polarity of a verb, which dominates negative polarity of object ‘disobedient child’. The estimation of the emotion vector of a clause (Subject plus Verb–Object formations) is then performed in the following manner:
(1) If polarities of Subject formation and Verb formation are opposite (e.g. Subject formation = ‘my darling’, Verb formation = ‘smashed’, Object formation = ‘his guitar’; or Subject formation = ‘troubled period’, Verb formation = ‘luckily comes to an end’), we consider the vector of the Verb-Object formation relation as dominant. For example, negative Subject formation ‘mother's disapproval’ and positive Verb formation ‘calmed’ in a sentence ‘Mother's disapproval calmed disobedient child’ yield domination of positive emotion vector of ‘calmed disobedient child’.
(2) Otherwise, we output the vector with maximum intensities in corresponding emotional states of vectors of Subject and Verb-Object formations.
Let us consider the processing of Subject formations themselves containing Verb–Object formation (e.g. ‘To offend the neighbor’ in ‘To offend the neighbor is an unfriendly behaviour’) or a full clause, Subject plus Verb–Object formations (e.g. ‘tomorrow weather will be sunny’ in ‘That tomorrow weather will be sunny is great’). In such cases, first we estimate the emotion vector of main Subject formation, formed by Verb–Object formation or an embedded clause, using rules described above. Then, we estimate the resulting emotion vector of a whole sentence.
It is important to note that our system enables the differentiation of the strength of the resulting emotion depending on the tense of a sentence and availability of first person pronouns. We introduce this idea based on our findings from the literature on psychology studies. Taking tense into account is very important, as ‘emotions typically occur in response to an event, usually a social event, real, remembered, anticipated or imagined’ (Ekman Reference Ekman1993: 386) (emphasis added by authors). As Ekman states, ‘sometimes when people give an account of an emotional experience they unexpectedly begin to re-experience the emotion’ (Ekman Reference Ekman1993: 392). The genuine emotion expressions display that an emotion is now felt, whereas so-called referential expressions occur most often when people talk about past or future emotional experiences. Therefore, we assume that the strength of emotions conveyed by text depends on tense (e.g. strongest emotion for present tense, weakened emotion for past tense and the weakest emotion for future tense).
As to first person pronouns, people tend to use them to ‘more directly portray the speaker as the experiencer of the emotion’ (Lutz Reference Lutz, Lutz and Abu-Lughod1990), and to underline the strength of an emotion. Many researchers neglect these phenomena. They ignore the difference between ‘I am charmed by the cherry blossoms of Japan’ versus ‘The cherry blossoms of Japan are charming’ (we think that emotion conveyed through the first sentence is stronger than in the case of the second one), and some of them completely disregard sentences in past or future tense and without first person pronouns (Boucouvalas Reference Boucouvalas2003). The analysis of personal pronouns as contributors to the immediacy of disclosure in electronic negotiations (please see details in Sokolova and Szpakowicz Reference Sokolova, Szpakowicz, Lamontagne and Marchand2006) revealed the importance of their use in language patterns for the tactical moves of influence strategies.
According to our proposal, the emotion vector of a simple sentence (or of a clause) is multiplied by the corresponding empirically determined coefficient of intensity correction (see Table 5).
Coefficients of intensity correction
The dominant emotion of the sentence is determined according to the emotion state with the highest intensity within the emotion vector. However, if there are several emotion states with the same maximum intensity in the resulting vector, we use a function that selects the prevailing emotion randomly. Let us consider the example of processing the following simple sentence: ‘My darling smashed his favourite guitar without regret’ (see Figure 2), where emotion vector e = [Anger, Disgust, Fear, Guilt, Interest, Joy, Sadness, Shame, Surprise]; SF, VF and OF mean Subject, Verb and Object formations, respectively; the superscripts 0, − and + indicate ‘neutral’, ‘negative’ and ‘positive’ polarities, respectively.
Example of affect sensing in a simple sentence.
5.5.2 Emotion vector of a compound sentence
A compound sentence is composed of at least two independent clauses, but no dependent clauses. The clauses are joined by a comma and coordinate connector, or a semicolon with no conjunction. In order to estimate the emotion vector of a compound sentence, first, we evaluate the emotion vectors of its independent clauses. Then, we define the resulting vector of the compound sentence based on the following rules:
(1) With comma and coordinate connectors ‘and’ and ‘so’ (e.g. ‘It is my fault, and I am worrying about consequences’, ‘Exotic birds in the park were amazing, so we took nice pictures’), or with a semicolon with no conjunction: output the vector with the maximum intensity within each corresponding emotional state in the resulting vectors of both clauses.
(2) With coordinate connector ‘but’ (e.g. ‘They attacked, but we luckily got away!’, ‘It was hard to climb a mountain all night long, but a magnificent view rewarded the traveler in the morning.’): the resulting vector of a clause following after the connector is dominant.
5.5.3 Emotion vector of a complex sentence
A complex sentence is a sentence with an independent clause and at least one dependent (embedded or subordinating) clause. The dependent clause is introduced by either a subordinate conjunction (e.g. ‘as’, ‘because’, ‘if’, ‘since’, ‘that’, etc.) or a relative pronoun such as ‘who’ or ‘which’. Some subordinating conjunctions, when used to introduce a phrase instead of a full clause become prepositions with identical meanings. In Section 5.4 we mentioned that in our AAM conditional clause phrases are neutralized due to specific prepositions or conjunctions. Therefore, the emotion vector of a dependent clause starting with one of these conjunctions represents a zero vector, and the vector of the independent clause forms the resulting emotion vector of such a complex sentence. If the subordinating clause in the complex sentence is connected to an independent clause through conjunctions such as ‘as’, ‘because’, ‘since’, we take the maximum intensity within each corresponding emotional state in the resulting vectors of both clauses for the estimation of the resulting vector of the complex sentence.
We can distinguish two types of embedded clauses:
(1) Complement clauses.
(2) Relative clauses.
Let us first look at the case of sentences with complement clauses. Special subordinating conjunctions, so-called complementizers (e.g. ‘whether’, ‘that’, etc.), introduce complement clauses (e.g. ‘I wonder whether we will go to the amusement park next weekend’ and ‘We hope that you feel comfortable’). There are basically three complementizers in English language: ‘that’, ‘for-to’ (‘for’ precedes the complement sentence and the ‘to’ precedes the auxilliary constituent of the complement sentence) and what is known as ‘POSS-ing’ (‘POSS’ means the possessive suffix, which is affixed to the noun, and the ‘-ing’ means the suffix attached to a verb stem) (Cairns and Cairns Reference Cairns and Cairns1976: 58–62). Here are some examples below:
(1) With ‘that’: ‘Sam preferred that John take the blame’.
(2) With ‘for-to’: ‘Sam preferred for John to take the blame’.
(3) With ‘POSS-ing’: ‘Sam preferred John's taking the blame. John resented Sam's telling the truth’.
In order to process a sentence with a complement clause, first we derive the emotion vector of the complement clause (e.g. ‘John take the blame’, ‘John to take the blame’, ‘John's taking the blame’ or ‘Sam's telling the truth’), then create Object formation for the main clause using this vector, and finally estimate the resulting emotion vector of the main clause with added Object formation. In brief, we represent such sentence as a simple one, using the following pattern: ‘who-subject does-verb what-object’, where object is represented as a complement clause.
Our program is also able to process the complex sentences containing adjective (relative) clauses introduced by ‘who’, ‘whom’, ‘whose’, ‘that’, ‘which’ or ‘where’. An adjective clause is a dependent clause that modifies a noun. Depending on the role (subject or object) that the relative pronoun plays in the embedded clause, sentences are called ‘subject relatives’ (see examples 1 and 2 below) or ‘object relatives’ (see example 3) (Cairns and Cairns Reference Cairns and Cairns1976). The followings are examples of complex sentences with relative clauses:
(1) ‘The wolf who ate the grandmother scared Little Red Riding Hood.’
(2) ‘The wolf who ate the grandmother who lived in the cottage scared Little Red Riding Hood.’ (a case of multiple embedding)
(3) ‘The wolf who the woodman killed scared Little Red Riding Hood.’
In our algorithm, the sentences of such type are analysed in the following manner:
(1) First, the emotion vector of adjective clause is estimated.
(2) Then, this emotion vector is added to the Subject or Object formation of the main clause depending on the role of the word to which the adjective clause relates. For example, in a sentence ‘The man who loved the woman robbed the bank’, the adjective clause ‘who loved the woman’ relates to the subject ‘man’; and in the sentence ‘The man robbed the bank where his beloved wife was working’, the adjective clause ‘where his beloved wife was working’ relates to the object ‘bank’.
(3) Finally, the emotion vector of the whole sentence is estimated.
Figure 3 illustrates the steps by way of a complex sentence with multiple embedding of relative clauses: ‘The policeman who loved his job ruined the life of the art student who had stolen famous painting, which was created by his favourite artist’.
Example of affect sensing in a complex sentence with multiple embedding of relative clauses.
In Figure 3, the emotion vector is denoted by e = [Anger, Disgust, Fear, Guilt, Interest, Joy, Sadness, Shame, Surprise]; SF, VF and OF represent Subject, Verb and Object formations, respectively; the superscripts 0, − and + indicate ‘neutral’, ‘negative’ and ‘positive’ polarities, respectively; main and dep mean belonging to ‘main’ and ‘dependent’ clauses, respectively.
5.5.4 Emotion vector of a complex–compound sentence
Sentences with at least two independent clauses and one or more dependent clauses are referred to as complex–compound sentences (e.g. ‘Max broke the china cup, with which Mary was awarded for the best song, so he regretted profoundly’). While processing such type of sentences, first we generate emotion vectors of dependent clauses, then of complex sentences and finally, we analyse the compound sentence formed by the independent clauses.
6 Evaluation of the Affect Analysis Model algorithm
In order to evaluate the performance of the AAM and to compare out method with related work, we conducted a set of experiments on the data sets created in different domains: diary-like blogs, fairy tales and news headlines.
6.1 Experiments with our collection of sentences extracted from diary-like blogs
6.1.1 Data set description
As it is difficult to access logs of real IM sessions (due to privacy concerns), we initially investigated a collection of diary-like blog posts provided by BuzzMetrics.Footnote 4 Here, we focused on online diary or personal blog entries, which are typically written in a free informal style and are rich in emotional colourations (Neviarouskaya, Prendinger and Ishizuka Reference Neviarouskaya, Prendinger and Ishizuka2007b). Our observations suggest that every author practises a different style of writing. The most noticeable aspects of diary-like text are privacy, naturalism and honesty in the expression of the author's thoughts and feelings. We concluded that the nature of such blog entries is reasonably close to online IM conversations (with the evident difference in the size of messages, however), and extracted 700 sentencesFootnote 5 from this Weblog Data Collection in order to evaluate the emotion recognition algorithm.
Three independent annotators labelled the sentences with one of nine emotion categories listed in Section 1 (or neutral) and a corresponding intensity value. Additionally, we interpreted these fine-grained annotations using three polarity-based categories (positive emotion, negative emotion and neutral) by merging ‘Interest’, ‘Joy’ and ‘Surprise’ in positive emotion category, and ‘Anger’, ‘Disgust’, ‘Fear’, ‘Guilt’, ‘Sadness’ and ‘Shame’ in negative emotion category. The reliability of the human raters' annotations was measured using the Fleiss' Kappa coefficient. The level of agreement on 700 sentences was moderate (0.47 in case of original annotations, and 0.59 in case of polarity-based annotations), and suggests that persons' comprehension, interpretation and evaluation of emotions are individualistic and might depend on personality type and emotional experience. As Davitz (Reference Davitz1969: 85) stated, ‘In some respects, the experiences of each subject are undoubtedly unique. In fact, for any one person, even though experiences at different times are labelled by the same [emotional] term, these experiences are likely to differ somewhat from one another.’
To the best of our knowledge, there is no data set of sentences annotated with such an extensive set of labels (nine emotions and neutral); so there is no possibility to compare the agreements. It is obvious that the level of agreement on coarse-grained annotations (e.g. factual/subjective; positive/negative/neutral) is higher than on fine-grained annotations. As manual fine-grained annotations were interpreted using polarity-based categories by merging the emotions, this procedure could influence the agreement on polarity-based annotations. For example, annotators could assign ‘neutral’ label to non-emotional but having strong polarity sentences. If annotators were asked to provide polarity-based annotations for the sentence ‘That place is one of the best places in Rochester for Mexican food, no lie’, they would completely agree on ‘positive’ label, whereas fine-grained manual annotations were ‘neutral’/‘neutral’/‘Interest’.
For the evaluation of algorithm performance, we created gold standards, on which at least two out of three human raters completely agreed: 656 sentences with fine-grained annotations (Fleiss' Kappa coefficient is 0.51) and 692 sentences with polarity-based annotations (Fleiss' Kappa coefficient is 0.6). The percentage distributions of labels across gold standard sentences are as follows (in descending order):
(1) ‘Joy’ – 28, ‘Sadness’ – 22, ‘Neutral’ – 11, ‘Anger’ – 9, ‘Fear’ – 7, ‘Interest’ – 7, ‘Surprise’ – 7, ‘Disgust’ – 5, ‘Guilt’ – 3 and ‘Shame’ – 1 per cent;
(2) ‘Negative’ – 50, ‘Positive’ – 39 and ‘Neutral’ – 11 per cent.
6.1.2 Results based on gold standard sentences, on which at least two annotators agreed
To analyse the importance of words of different parts-of-speech in affect recognition, first we evaluated the performance of the AAM with adjectives only, then we cumulatively added adverbs, verbs and nouns to the algorithm. Averaged accuracy, precision, recall and F-score at each step of this experiment are shown in Table 6 for each category. As was expected, the obtained results indicate that consideration of all content parts-of-speech plays a crucial role in emotion recognition from text. Two-tailed t-tests with significance level of 0.05 showed that the differences in accuracy between the preceding and the following algorithms are statistically significant (p > 0.001) in fine-grained as well as coarse-grained classifications, with the exceptional case of insignificant difference after adding the adverbs to the algorithm relying purely on adjectives.
Accuracy across sentences from blogs in the experiment with words of different parts-of-speech
*AAM stands for Affect Analysis Model; ADJ, ADV, V and N refer to adjectives, adverbs, verbs and nouns, respectively.
The baseline for comparison (last raw in Table 6) is represented by the results of a simple method that selects the emotion with maximum intensity from the annotations of sentence tokens found in Affect database. Our AAM outperformed the baseline method (the difference is statistically significant in fine-grained and coarse-grained classifications: p < 0.001), thus demonstrating the contribution of the sentence parsing and our hand-crafted rules to the reliable recognition of emotions from text.
Next, we conducted a functional ablation experiment that aimed at evaluating our AAM with selectively removed functionality components: negation, neutralization due to modality, neutralization due to conditionality, modification by adverb-intensifiers and intensity correction. We compared AAM with all functionalities, AAM without additional functionalities and five approaches in which one specific functionality component was ablated from AAM. We believe that the remaining five AAM configurations would show the enhancement that each functionality adds to complete AAM rather than what is missing when each is removed. Table 7 includes the results of this experiment, showing that AAM mostly benefits from rules on negation and conditionality. Although no statistically significant differences in accuracy were found between the AAM with all functionalities and AAM algorithms with single additional functionality component removed, the statistical testing with significance level of 0.05 showed that the accuracy of AAM is significantly higher (p < 0.01 in fine-grained classification and p < 0.05 in coarse-grained classification) than the accuracy of AAM without all additional functionalities.
Averaged accuracy across sentences from blogs in functional ablation experiment
The analysis of errors in the assignment of polarity-based categories for the AAM (see Table 8) revealed that system requires common sense or additional context to deal with 28.5 per cent of all errors. As human annotators labelled sentences only using fine-grained emotion categories and could assign ‘neutral’ to non-emotional but having strong polarity cases, we can consider the next type of error in the table (21.0 per cent) as a non-strict one in the experiment with merged labels, where gold standard was based on fine-grained emotion annotations. In nine per cent of cases, where the system result did not agree with the gold standard due to the rule of neutralization of negated phrases, the solution would be to reverse the polarity of a statement; however, finding the pairs of opposite emotions might be problematic. The errors resulting from neutralization due to ‘cognition-related’ words comprise about seven per cent of errors. The failures also include some exceptional cases with connector ‘but’, errors caused by the lack of relevant terms in Affect database and incorrect results from the syntactic parser.
Distribution of errors of AAM in experiment on sentences from blogs with merged labels (gold standard annotations and AAM results are given in the form ‘Emotion – Polarity-based label for merged categories’ in a last column)
We also evaluated the system performance with regard to intensity estimation. The percentage of emotional sentences (not considering neutral ones), on which the result of our system conformed to the fine-grained gold standard, according to the measured distance between intensities given by human raters (averaged values) and those obtained by the AAM is shown in Table 9. As seen in the table, our system achieved satisfactory results in emotion intensity estimation.
Percentage of high agreement sentences according to the range of intensity difference between human annotations and output of algorithm
6.2 Experiment with the emotion blog data set developed by Aman and Szpakowicz (Reference Aman and Szpakowicz2007)
6.2.1 Data set description
This data set was developed and kindly provided by Aman and Szpakowicz (Reference Aman and Szpakowicz2007). It includes sentences collected from blogs, which are characterized by rich emotional content and good examples of real-world instances of emotions conveyed through text. To directly compare the AAM with the machine learning methods proposed by Aman and Szpakowicz (Reference Aman and Szpakowicz2008), as the gold standard we considered their benchmark, which includes sentences annotated by one of six emotions (‘Happiness’,Footnote 6 ‘Sadness’, ‘Anger’, ‘Disgust’, ‘Surprise’ and ‘Fear’) or neutral, on which two annotators completely agreed. The distribution of labels across sentences from the benchmark used in the experiment is shown in Table 10.
Distribution of labels across sentences from benchmark used in the experiment
6.2.2 Results
As AAM is capable of recognition of nine emotions, and methods described in Aman and Szpakowicz (Reference Aman and Szpakowicz2008) classify text to six emotions, in order to compare the results of our approaches we decided to reduce the number of our labels by mapping ‘Interest’ to ‘Joy’, and ‘Guilt’ and ‘Shame’ to ‘Sadness’. The results of experiments are shown in Table 11, where AAM is compared to two classifiers trained using Support Vector Machines (the results of these classifiers are taken from Aman and Szpakowicz Reference Aman and Szpakowicz2008): (1) ‘ML with unigrams’, which employs corpus-based features, namely, all unigrams that occur more than three times in the corpus, excluding stopwords; (2) ‘ML with unigrams, RT features and WNA features’, which combines corpus-based features with features based on the following emotion lexicons: Roget's Thesaurus (Jarmasz and Szpakowicz Reference Jarmasz and Szpakowicz2001) and WordNet-Affect (Strapparava and Valitutti Reference Strapparava and Valitutti2004).
Results of AAM compared to machine learning algorithms proposed by Aman and Szpakowicz (Reference Aman and Szpakowicz2008)
The obtained results (precision, recall and F-score) revealed that our rule-based system outperformed both machine learning methods in automatic recognition of ‘Joy’, ‘Sadness’, ‘Anger’, ‘Disgust’ and ‘neutral’. In case of ‘Surprise’ and ‘Fear’ emotions, ‘ML with unigrams’ resulted in higher precision, but lower recall and F-score than our AAM.
6.3 Experiment with sentences from fairy tales
6.3.1 Data set description
In our next experiment, we wanted to compare the performance of the AAM with Alm's (Reference Alm2008) system that reportedly outperformed Liu et al. (Reference Liu, Lieberman and Selker2003) system on sentences from fairy tales. Following the same evaluation scenario as Alm (Reference Alm2008), we considered three hierarchical levels of affect labels in our experiment (see Figure 4).
Affect hierarchy and set of labels.
We ran the experiment on the subset of 1,207 sentences marked by high agreementFootnote 7 (indicating that affect labels assigned by four human annotators for the sentence were identical), and a subset of sentences with neutral label. As we did not have the subsets of neutral sentences used by Alm in her experiments, we randomly extracted them from the whole corpus of sentences that was labelled by human annotators as neutral (differences in data sets, however, might add some incomparability to the results). The size of a sample of neutral sentences varied at each hierarchical level and was determined based on the number of affective labels at each level by Equation 1 (taken from Alm Reference Alm2008),
![{equation}
\left[ {\frac{{\left| {HA} \right|}}{{\left| {Ai} \right| - 1}}} \right],
\end{equation}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20151022070247454-0856:S1351324910000239_eqn1.gif?pub-status=live){kind=small}
such that HA is the set of high-agreement affect sentences in the whole corpus; Ai is the set of affect labels at a specific level i in the affect hierarchy.
6.3.2 Results
We compared the results of the AAM with Alm (Reference Alm2008) LOOHAsnowtag method (supervised machine learning approach) and two baselines (see Table 12 partially taken from Alm Reference Alm2008). The ratio of neutrally labelled sentences was considered as first baseline (N-BL), while the second baseline (Freq-BL) was represented by the ratio of the most frequent affect label.
Accuracy across sentences from fairy tales in high agreement experiment (span of mean accuracy given for LOOHAsnowtag method)
As seen from Table 12, AAM resulted in a similar accuracy level as the LOOHAsnowtag method, outperforming both baselines. The observed increase in accuracy is inversely proportional to the number of possible labels at the hierarchical levels (the highest accuracy is at the TOP level with only two possible categories). The two classes of errors of AAM method (first-class errors in the case of high-agreement affect sentences at the ALL level; second-class errors in the case of neutral sentences at the TOP level) are analysed in Table 13.
Distribution of errors of AAM in high agreement experiment on sentences from fairy tales
The largest group of all AAM errors (52.7 per cent of first-class errors; 67.1 per cent of second-class errors) occurred in the cases where additional common sense knowledge or context are required for correct interpretation of conveyed emotion. The ambiguity of word senses was responsible for about 20 per cent of second-class errors, highlighting the importance and necessity of word sense disambiguation. About 14 per cent of first-class errors were so-called ‘conflict’ errors, meaning that the correct emotion was in the final vector of AAM, but it was not dominant, taking into account its intensity. The errors carrying grammatical character and caused by the parser include about 11 per cent of first-class errors and 4.5 per cent of second-class errors. The rules for negations, neutralizations and combinations of the SF, VF and OF together caused about 14.5 per cent of the first-class errors.
As our approach relies on the affect lexicon, and its recall depends on the coverage of the lexicon used (about seven per cent of errors in high agreement affect sentences at the ALL level were due to the lack of affect entries in database), we improved the performance of our system (see results of AAM_LEX in Table 12, where LEX means ‘lexicon’) by expanding the Affect database with some sample words from experimental data. The decision for further improvement (see results of AAM_LEX_WI in Table 12, where WI means ‘without Interest’ or ‘neutral label instead of Interest’) was based on the fact that about 8.5 per cent of errors in neutral sentences at the TOP level (se Table 13) were comprised of the cases where AAM system resulted in ‘Interest’ emotion that was not considered as a label in the Alm's (Reference Alm2008) set of affect labels.
6.4 Experiment with news headlines
In addition to the experiments on sentences from diary-like blog posts and fairy tales, we have also evaluated AAM algorithm on news headlines.
6.4.1 Data set description
This data set was created for the SemEval-2007 task on ‘Affective Text’ (Strapparava and Mihalcea Reference Strapparava and Mihalcea2007). The test data set consists of 1,000 news headlinesFootnote 8 independently labelled by six annotators using scores of six emotions (‘Anger’, ‘Disgust’, ‘Fear’, ‘Joy’, ‘Sadness’ and ‘Surprise’) by means of web-based interface with six slide bars. The emotion score interval is [0, 100], where 0 means the emotion is missing from the given headline, and 100 represents the maximum emotional load. The annotators were instructed to select the appropriate emotional scores for each headline based on the presence of words or phrases with emotional content, as well as the overall feeling invoked by the text, ensuring thus annotations of cases where multiple emotions are involved. In this gold standard, the distribution of headlines according to the number of non-zero scores of particular emotions assigned is as follows: one emotion – 1.1 per cent of headlines; two emotions – 19.1 per cent of headlines; three emotions – 17.7 per cent of headlines; four emotions – 21.6 per cent of headlines; five emotions – 27.2 per cent of headlines; and six emotions – 13.3 per cent of headlines. Therefore, 62.1 per cent of all headlines were annotated by at least four emotions.
6.4.2 Results
In this experiment, we evaluated the performance of our system based on the fine-grained and coarse-grained evaluation metrics proposed in Strapparava and Mihalcea (Reference Strapparava and Mihalcea2008). Fine-grained evaluations were based on Pearson measure of correlation between the system scores and the gold standard scores, averaged over all the headlines in the data set. In order to produce more or less comparable results, we (1) considered the final emotion vector resulting from AAM as the overall annotation for each headline; (2) reduced the number of our labels to six by mapping ‘Interest’ to ‘Joy’, and ‘Guilt’ and ‘Shame’ to ‘Sadness’; (3) scaled intensities in interval [0.0, 1.0] to scores in interval [0, 100]. However, it is important to note here that the concepts and functions of our ‘emotion intensities’ and ‘emotion scores’ used in the gold standard differ significantly, as intensity shows the strength of emotion involved while score in the gold standard indicate how much particular emotion is involved in the headline. For the coarse-grained evaluations, each emotion was mapped to a 0/1 classification (0 = [0, 50), 1 = [50, 100]), and precision, recall and F-score were calculated for each emotion.
We compared the performance of our method with the systems participating at the SemEval-2007 task on ‘Affective Text’. The results of our AAM and other systems (reported in Strapparava and Mihalcea Reference Strapparava and Mihalcea2008) are shown in Table 14, where (1) AAM is our Affect Analysis Model; (2) WN-A is ‘WordNet-Affect presence’ method, which computes the scores based on the frequencies of the direct affective words found in the headlines; (3) LSA SW is ‘LSA single word’ method, which measures the similarity between the given text and each emotion, where an emotion is represented as the vector of the specific word denoting the emotion (e.g. ‘Joy’); (4) LSA ES is ‘LSA emotion synset’ method, which uses the synonyms from the WordNet synsets in addition to the word denoting an emotion; (5) LSA AEW is ‘LSA all emotion words’ method, which extends the previous set by adding the words from all the synsets labelled with a particular emotion in WordNet-Affect; (6) NB BLOG is a Naïve Bayes classifier trained on the corpus of blog posts annotated by emotions (methods (2)–(6) were developed by Strapparava and Mihalcea Reference Strapparava and Mihalcea2008); (7) SWAT (Katz et al. Reference Katz, Singleton and Wicentowski2007) is a supervised system, which is based on unigram model; (8) UA (Kozareva et al. Reference Kozareva, Navarro, Vazquez and Montoyo2007) is a system, which calculates emotion scores using Pointwise Mutual Information; (9) UPAR7 (Chaumartin Reference Chaumartin2007) is a rule-based system, which is based on linguistic approach using SentiWordNet (Esuli and Sebastiani Reference Esuli and Sebastiani2006) and WordNet-Affect (Strapparava and Valitutti Reference Strapparava and Valitutti2004).
Results of AAM compared to knowledge-based and corpus-based systems participating in the task ‘Affective Text’ at SemEval-2007
In the fine-grained evaluation, our system achieved the highest results in recognition of ‘Anger’ and ‘Sadness’ emotions, while SWAT was more successful in case of ‘Disgust’ and ‘Joy’, and UPAR7 in case of ‘Fear’ and ‘Surprise’. In terms of averages of Pearson's correlation coefficients for all emotions, UPAR7 showed the best performance (28.38), followed by our AAM system (26.99) and SWAT (25.41). These results indicate that our AAM system showed good results in detecting emotions in news headlines, in spite of the facts that it was not initially developed for this particular task and its results are not completely comparable to the gold standard. The important point is that the annotators of headlines assigned emotion scores based on words in the sentence (e.g. ‘Tsunami fears ease after quake’ was annotated in the gold standard by Anger:0, Disgust:0, Fear:79, Joy:13, Sadness:13, Surprise:0, with predominant ‘Fear’ emotion); in contrast, our system analysing the sentence in consecutive stages outputs unified emotion vector, which for this example sentence does not contain ‘Fear’ emotion involved in the object ‘tsunami fears’, as positive vector of the verb “ease” dominates, therefore the final vector has Fear:0 and Joy:8 (‘Joy’ with intensity 0.08). Probably, to get more comparable results from AAM, one have to sum emotion vectors from ‘Word-level analysis’ stage (see Section 5.3 for details) and ignore ‘Phrase-level analysis’ and ‘Sentence-level analysis’ stages, so that the final vector includes all possible emotions. Such approach would perhaps result in a higher correlation coefficient between the AAM scores and gold standard; however, there would not be much intelligence.
In the coarse-grained evaluation, AAM ensured the best F-scores for ‘Anger’, ‘Fear’ and ‘Sadness’, as well as in case of average results, while the highest F-scores for ‘Disgust’, ‘Joy’ and ‘Surprise’ were achieved by LSA SW, NB BLOG and UA systems, correspondingly.
In summary, the evaluation of the AAM algorithm showed promising results regarding its capability to recognize affective information in sentences from different domains: blogs, fairy tales and news titles. However, the current main limitations of the developed affect recognition module must be noted, namely: strong dependency on the resource of lexicon (Affect database) and the commercially available syntactic parser; no disambiguation of word senses; and disregard of contextual information.
7 Conclusions
This paper introduced a novel rule-based linguistic approach for affect recognition from text. Typically, researchers in the sentiment analysis field deal with grammatically and syntactically correct textual input. In contrast, our analysis of affect expressed through written language is inspired by the evolving language, style and specifics of IM conversations and diary-like blog posts. For textual input processing, our AAM copes with not only correctly written text, but also informal messages written in an abbreviated or expressive manner. In order to support the handling of abbreviated language and the interpretation of affective features of emoticons, abbreviations and words, a special Affect database was created.
The proposed rule-based algorithm for affect sensing from text processes each sentence in stages, including symbolic cue processing, detection and transformation of abbreviations and acronyms, sentence parsing and word/phrase/sentence-level analyses. Our method is capable of processing sentences of different complexity, including simple, compound, complex (with complement and relative clauses) and complex–compound sentences. Affect in text is classified into nine emotion categories (or neutral). The strength of the resulting emotional state depends on emotion vectors of words, relations among them, tense of the analysed sentence and availability of first person pronouns.
The salient features of the AAM are the following:
(1) Analysis of nine emotions on the level of individual sentences: this is an extensive set of labels if compared to six emotions mainly used in related work.
(2) The ability to handle the evolving language of online communications: to the best of our knowledge, our approach is the first attempt to deal with informal and abbreviated style of writing, often accompanied by the use of emoticons.
(3) Foundation in database of affective words (each term in our Affect database was assigned at least one emotion label along with emotion intensity, in contrast to annotations of one emotion label or polarity orientation in other approaches), interjections, emoticons, abbreviations and acronyms, modifiers (which influence the degrees of emotion states).
(4) Vector representation of affective features of words, phrases, clauses and sentences.
(5) Consideration of syntactic relations and semantic dependences between words in a sentence: our rule-based method accurately classifies context-dependent affect expressed in sentences containing emotion-conveying terms, which may play different syntactic and semantic roles.
(6) Analysis of negation, modality and conditionality: most researchers ignore modal expressions and condition prepositions, therefore, their systems show poor performance in classifying neutral sentences, which is, indeed, not an easy task.
(7) Consideration of relations between clauses in compound, complex or complex–compound sentences: to our knowledge, AAM is the first system comprehensively processing affect reflected in sentences of different complexity.
(8) Emotion intensity estimation: in our work, the strength of emotion is encoded through numerical value in the interval [0.0, 1.0], in contrast to low/middle/high levels detected by some of other methods.
Our system showed promising results in affect recognition on real examples of diary-like blog posts: (1) on data set created by us (gold standard where at least two annotators agreed), averaged accuracy was 72.6 per cent for fine-grained (nine categories and neutral) emotion classification and 81.6 per cent for polarity-based merged categories; (2) on data set provided by Aman and Szpakowicz (Reference Aman and Szpakowicz2008), averaged accuracy was 77.0 per cent for fine-grained (six categories and neutral) emotion classification, and our system outperformed the method reported in related work in terms of precision, recall and F-scores. On the sentences extracted from fairy tales (Alm Reference Alm2008), averaged accuracy of AAM was 70.2 per cent for fine-grained emotion classification and 75.7 per cent for polarity-based merged categories (these results are slightly better than those reported in related work). Comparing the performance of the AAM with eight systems from related work on the task of recognition of emotions in news headlines (Strapparava and Mihalcea Reference Strapparava and Mihalcea2008), we found that, even though there were some inconsistencies in comparing the results with gold standard, our system resulted in high level of accuracy, outperforming other methods on several measures.
Currently, the main limitations of the developed affect recognition module are: strong dependency on the lexicon resource (Affect database) and the commercially available syntactic parser; no disambiguation of word meanings; disregard of conversation history; and inability to recognize and process misspelled words in a sentence.
In our future study, we will investigate those issues and explore the possibilities to overcome the current limitations of the system. For example, the Affect database can be expanded automatically by considering (1) direct synonymy (e.g. ‘congratulate’ – ‘felicitate’) and antonymy (e.g. ‘brave’ – ‘faint-hearted’) relations and hypernym–hyponym (e.g. ‘success’ – ‘winning’) relations in WordNet; (2) compounding using known emotion-carrying base components (e.g. ‘well-wishing’, ‘ill-conditioned’, ‘terror-haunted’); (3) morphological modifications (e.g. ‘harmony’+‘-ous’ = >‘harmonious’, ‘dis-’+ ‘honest’ = >‘dishonest’).
As our system relies on the lexicon and the language of online conversations is ‘evolving’, we are also planning to implement a procedure for the automatic updating of the Affect database. In addition, we will thoroughly examine different cases of the use of modals and condition prepositions in sentences, so as to be able to automatically distinguish emotional cases from neutral ones (in contrast to simple neutralization applied in current work). With respect to the rules for composition of emotion vectors of terms comprising phrases or clauses, we believe the approach aiming at learning rules from corpora would be useful.
To enrich the user's experience in online communication, to make it enjoyable, exciting and fun, we implemented a web-based IM application, AffectIM and endowed it with emotional intelligence by integrating the developed AAM. The findings of a twenty-person study conducted with our AffectIM system are described in Neviarouskaya, Prendinger and Ishizuka (Reference Neviarouskaya, Prendinger and Ishizuka2010).
Acknowledgments
We would like to acknowledge and thank Dr. Alessandro Valitutti and Dr. Diego Reforgiato for their kind help during the Affect database creation. We wish also to express our gratitude to Dr. Dzmitry Tsetserukou, Dr. Shaikh Mostafa Al Masum, Manuel M. Martinez, Zoya Verzhbitskaya, Hutchatai Chanlekha and Nararat Ruangchaijatupon, who have contributed to annotations of Affect database entries and sentences for their efforts and time. This research was partly supported by a JSPS Encouragement of Young Scientists Grant (FY2005-FY2007), an NII Joint Research Grant with the University of Tokyo (FY2007) and an NII Grand Challenge Grant (FY2008-FY2009).
