2.1 Negation detection for English

Negation detection in English has been a productive research area during recent years in the NLP community as shown by the challenges and shared tasks held (e.g., BioNLP’09 Shared Task 3 (Kim et al. Reference Kim, Ohta, Pyysalo, Kano and Tsujii2009), i2b2 NLP Challenge (Uzuner et al. Reference Uzuner, South, Shen and DuVall2011), ^*SEM 2012 Shared Task (Morante and Blanco Reference Morante and Blanco2012) and ShARe/CLEF eHealth Evaluation Lab 2014 Task 2 (Mowery et al. Reference Mowery, Velupillai, South, Christensen, Martinez, Kelly, Goeuriot, Elhadad, Pradhan, Savova and Chapman2014)). It is worth noting that the initial solutions that arose for sentiment analysis are not accurate enough since they have relatively straightforward conceptualisations of the scope of negation and have traditionally relied on rules and heuristics. For example, Pang and Lee (Reference Pang and Lee2004) assumed that the scope of a negation keyword consists of the words between the keyword and the first punctuation mark following it (see also Polanyi and Zaenen Reference Polanyi, Zaenen, Shanahan, Qu and Wiebe2006).

The first work we are aware of detecting negation and its scope using a more robust approach is presented by Jia, Yu, and Meng (Reference Jia, Yu and Meng2009). They develop a rule-based system that uses information derived from a parse tree. This algorithm computes a candidate scope, which is then pruned by removing the words that do not belong to the scope. Heuristic rules, which include the use of delimiters (i.e., unambiguous words such as because) and conditional word delimiters (i.e., ambiguous words like for), are used to detect the boundaries of the candidate scope. Situations in which a negation cue does not have an associated scope are also defined. The authors evaluate the effectiveness of their approach on polarity determination showing that the identification of the scope of negation improves both the accuracy of sentiment analysis and the effectiveness of opinion retrieval.

Regarding the impact of negation identification on sentiment analysis using machine learning techniques, this has not been sufficiently investigated. As Díaz and López (Reference Domínguez-Mas, Ronzano and Furlong2019) point out, this is perhaps because reasonably sized standard corpora annotated with this kind of information have only recently become available. However, there is relevant work that shows the suitability of applying negation modelling to the task of sentiment analysis in other languages and that, therefore, has inspired the experimentation presented in this paper. For example, Councill et al. (Reference Councill, McDonald and Velikovich2010) develop a system that can precisely recognise the scope of negation in free text. The cues are detected using a lexicon (i.e., a dictionary of 35 negation keywords). A Conditional Random Field (CRF) algorithm is used to predict the scope. This classifier incorporates, among others, features from dependency syntax. The approach is trained and evaluated on a product review corpus. Using the same corpus, Lapponi, Read, and Øvrelid (Reference Lapponi, Read and Øvrelid2012) present a state-of-the-art system for negation detection. Their proposal is based on the application of CRF models for sequence labelling, which makes use of a wealth of lexical and syntactic features, together with a fine-grained set of labels that capture the scopal behaviour of tokens. With this approach, they also demonstrate that the choice of representation has a significant effect on performance. Cruz, Taboada, and Mitkov (Reference Cruz, Taboada and Mitkov2016) also conduct research into machine learning techniques in this field. They define a system which automatically identifies negation cues and their scope in the Simon Fraser University (SFU) Review corpus (Konstantinova et al. Reference Konstantinova, De Sousa, Díaz, López, Taboada and Mitkov2012), showing results in line with the results of other authors in the same task and domain, exactly, they yield a performance of 80.26 for the gold standard cues and 69.69 for the scopes of the predicted cues.

Another type of approach worth mentioning is the composition models that implicitly learn negation. For instance, Socher et al. (Reference Socher, Perelygin, Wu, Chuang, Manning, Ng and Potts2013) generate the Stanford Sentiment Treebank corpus and apply Recursive Neural Tensor Networks to it, improving the state of the art in single sentence positive/negative classification. This model also accurately captures the effects of negation and its scope at various tree levels for both positive and negative phrases.

Deep learning techniques have also been applied to the task of negation detection. Work by Fancellu, Lopez, and Webber (Reference Fancellu, Lopez and Webber2016) and Qian et al. (Reference Qian, Li, Zhu, Zhou, Luo and Luo2016), although not focused on the sentiment analysis domain, should be highlighted. Fancellu et al. (Reference Fancellu, Lopez and Webber2016) present two different neural network architectures, that is, a hidden layer feed-forward neural network and a bidirectional long short-term memory (LSTM) model for negation scope detection. Training, development and tests are done on the negative sentences of the Conan Doyle corpus, that is, those sentences with at least one cue annotated (Morante and Blanco Reference Morante and Blanco2012). The results show that neural networks perform on a par with previously developed classifiers. Qian et al. (Reference Qian, Li, Zhu, Zhou, Luo and Luo2016) propose a convolutional neural network-based model with probabilistic weighted average pooling to also address negation scope detection. This system first extracts path features from syntactic trees with a convolutional layer and concatenates them with their relative positions into one feature vector, which is then fed into a soft-max layer to compute the confidence scores of its location labels. It is trained on the abstract sub-collection of the BioScope corpus, achieving the second highest performance for negation scope on abstracts.

In the field of sentiment analysis, one of the latest published works proposes a multi-task approach to explicitly incorporate information about negation (Barnes, Velldal, and Øvrelid Reference Barnes, Velldal and Øvrelid2019). Similarly to Fancellu et al. (Reference Fancellu, Lopez and Webber2016), this system consists of a BiLSTM-based model, relying not only on word embeddings as input but also adding a CRF for the prediction layer. This configuration shows that explicitly training the model with negation as an auxiliary task helps improve the main task of sentiment analysis.

In all, as the related work described above reveals, rules and heuristics for detecting scope significantly improve the accuracy of sentiment analysis systems. However, full understanding of negation scope is still a work in progress. In this regard, most recent methods with deep learning show promise, but more annotated corpora are needed.

2.2 Negation detection for Spanish

Negation processing in NLP for Spanish has started relatively recently compared to English. We find systems such as those proposed by Costumero et al. (Reference Costumero, López, Gonzalo-Martn, Millan and Menasalvas2014), Stricker, Iacobacci, and Cotik (Reference Stricker, Iacobacci and Cotik2015) and Cotik et al. (Reference Cotik, Stricker, Vivaldi and Hontoria2016) aimed at automatically identifying negation in the clinical domain by adapting the popular rule-based algorithm NegEx (Chapman et al. Reference Chapman, Bridewell, Hanbury, Cooper and Buchanan2001), which uses regular expressions to determine the scope of some negation cues.

In the review domain, negation has also been taken into account for Spanish sentiment analysis. One of the first systems, developed by Brooke, Tofiloski, and Taboada (Reference Brooke, Tofiloski and Taboada2009), adapts an English lexicon-based sentiment analysis system, SO-CAL (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011), to Spanish. The original implementation uses simple rules and heuristics for identifying the scope of negation. It is this method that we improve upon for this paper.

More sophisticated is the work of Vilares, Alonso, and Gómez-Rodrguez (Reference Vilares, Alonso and Gómez-Rodríguez2013; Reference Vilares, Alonso and Gómez-Rodríguez2015), which incorporates dependency parses to better pinpoint the scope of several intensifiers and negation cues.Footnote 2 Their results show that taking into account the syntactic structure of the text improves accuracy in the review domain, whether the sentiment analysis system uses machine learning or lexicon-based approaches. They do not, however, analyse the gain obtained using negation individually (separate from intensification) and, therefore, it is not possible to determine the relative contribution of negation by itself to the improvement obtained.

Jiménez-Zafra et al. (Reference Jiménez-Zafra, Martínez-Cámara, Martín-Valdivia and Molina-González2015) study the most important cuesFootnote 3 according to La Real Academia Española (Bosque, Garc a de la Concha, and López Morales 2009) and propose a set of rules based on dependency trees for identifying the scope of these negation cues. Later, Jiménez-Zafra et al. (Reference Jiménez-Zafra, Martín-Valdivia, Martínez-Cámara and Ureña-López2017) apply the detection negation module for sentiment analysis on Twitter, incorporating some changes to address the peculiarities of the language used in this social medium. They also use a lexicon-based system and statistically demonstrate that the results obtained considering the negation module are significantly higher than those obtained without taking negation into account. Moreover, they compare the proposed method with the method mostly used to determine the scope of negation in English tweets (Potts Reference Potts2011b), showing that the classification with their approach is better.

Other work that has clearly demonstrated the benefits of negation detection for Spanish sentiment analysis includes Mitchell et al. (Reference Mitchell, Becich, Berman, Chapman, Gilbertson, Gupta, Harrison, Legowski and Crowley2004) or Amores, Arco, and Barrera (Reference Amores, Arco and Barrera2016). In most works (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011; Vilares et al. Reference Vilares, Alonso and Gómez-Rodríguez2013, Reference Vilares, Alonso and Gómez-Rodríguez2015; Jiménez-Zafra et al. Reference Jiménez-Zafra, Martínez-Cámara, Martín-Valdivia and Molina-González2015; Miranda, Guzmán, and Salcedo Reference Miranda, Guzmán and Salcedo2016; Amores et al. Reference Amores, Arco and Barrera2016; Jiménez-Zafra et al. Reference Jiménez-Zafra, Martín-Valdivia, Martínez-Cámara and Ureña-López2017), negation detection is applied, but without a detailed error analysis or without ablation experiments to determine the gains of negation and how errors in negation detection affect the accuracy of the sentiment analysis system. This is, in part, due to the lack of an annotated corpus for negation in the review domain.

However, after the annotation of the SFU Review $_{\text{SP}}$ -NEG corpus (Jiménez-Zafra et al. Reference Jiménez-Zafra, Taulé, Martín-Valdivia, Ureña-López and Martí2018) and the organization of the 2018 and 2019 editions of NEGación en ESpañol (NEGES) (Jiménez-Zafra et al. Reference Jiménez-Zafra, Díaz, Morante and Martín-Valdivia2019a,b), the Workshop on Negation in Spanish, we find some systems for processing negation in the review domain. The aim of this workshop is to promote the identification of negation cues in Spanish and the application of negation for improving sentiment analysis. For the negation cue detection task, six systems have been developed (Fabregat, Martínez-Romo, and Araujo Reference Fabregat, Martínez-Romo and Araujo2018; Loharja, Padró, and Turmo Reference Loharja, Padró and Turmo2018; Giudice Reference Giudice2019; Beltrán and González 2019; Domínguez-Mas, Ronzano, and Furlong Reference Domínguez-Mas, Ronzano and Furlong2019; Fabregat et al. Reference Fabregat, Duque, Martínez-Romo and Araujo2019). All of them address the task as a sequence labelling problem using machine learning approaches. Deep learning algorithms and CRF algorithm are predominant, with CRF performing best.

Existing works addressing the clinical domain provide methods for the identification of negated entities and negated findings and those focusing on the review domain detect negation cues. None of them, however, focuses on the identification of the scope. To address this gap, we have recently developed a system for the identification of negation cues and their scopes in Spanish texts in which the SFU Review $_{\text{SP}}$ -NEG corpus is also used (Jiménez-Zafra et al. Reference Jiménez-Zafra, Morante, Blanco, Martín-Valdivia and Ureña-López2020), which will be explained in detail in Section 4.

5.1 SO-CAL

SO-CAL, The Semantic Orientation CALculator,Footnote 7 is a lexicon-based sentiment analysis system that was specifically designed for customer reviews but has also been shown to work well on other texts such as blog posts or headlines (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011). It contains dictionariesFootnote 8 classified by part of speech (nouns, verbs, adjectives and adverbs), for a total of about 5000 words for English and just over 4200 for Spanish. SO-CAL takes into account intensification by words such as very or slightly, with each intensifier having a percentage associated with it, which increases or decreases the word it accompanies.

Negation in the standard SO-CAL system for both English and Spanish takes the shift method, that is, any item in the scope of negation sees its polarity shifted by a fixed amount, 4 points in the best-performing version of the system. Thus, excellent (a $+5$ word in the dictionary) becomes not excellent, $+1$ , and sleazy, which is a $-3$ word also becomes $+1$ when negated.

Negation in SO-CAL is handled by first identifying a sentiment word from the dictionaries. If a word is found, then the system tracks back to the previous and searches for a negation keyword. If a negation keyword is present before the sentiment word, then negation is applied to the sentiment word. Scope is not explicitly identified, that is, the system assumes that a sentiment-bearing word is in the scope of negation if it is after the negation keyword in the same sentence. The system may continue to track back and keep looking left for negation keywords if a ‘skipped’ word is present, such as adjectives, copulas, determiners and certain verbs. Skipped words allow the system to look for keywords in cases of raised negation, for example, I don’t think it is good, where the system would keep skipping backwards through the words is, it and think to find the raised negation that affects the sentiment of good.

Sentiment for a text is calculated by extracting all sentiment words, calculating intensification and negation for relevant phrases, and then averaging the values of all the words and phrases in the text. The accuracy of the original system is 80% for English (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011) and about 72% for Spanish (Brooke et al. Reference Brooke, Tofiloski and Taboada2009). Our goal in this paper is to investigate whether a more accurate method for negation detection can improve the Spanish results.

5.2 Experiments

We conduct experiments on the corpus with negation annotations, the SFU Review $_{\text{SP}}$ -NEG (Jiménez-Zafra et al. Reference Jiménez-Zafra, Taulé, Martín-Valdivia, Ureña-López and Martí2018). The experimentation was organised in the following phases:

1. • Phase A: Negation cue detection

   1. – Prediction of the negation cues using the system provided by Jiménez-Zafra et al. (Reference Jiménez-Zafra, Morante, Blanco, Martín-Valdivia and Ureña-López2020) and 10-fold cross-validation in order to classify all the reviews.

2. • Phase B: Scope identification

   1. – Identification of the scopes corresponding to the predicted cues in Phase A—1.

3. • Phase C: Sentiment analysis

   1. 1. Classification of the reviews using the SO-CAL system without negation.

   2. 2. Classification of the reviews using the SO-CAL system with built-in negation, that is, using the rule-based method that incorporates the detection of cues and scopes in Spanish that is built in the SO-CAL system.

   3. 3. Classification of the reviews using the SO-CAL system with the output of the negation processing system applied in Phase A and Phase B.

5.3 Evaluation measures

The output of the systems for negation cue detection and scope identification (Phases A and B) is evaluated with Precision (P), Recall (R) and F-score (F1) measures using the script (https://www.clips.uantwerpen.be/sem2012-st-neg/data.html) provided in the ^*SEM 2012 Shared Task ‘Resolving the Scope and Focus of Negation’ (Morante and Blanco Reference Morante and Blanco2012). It is based on the following criteria:

1. • Punctuation tokens are ignored.

2. • A True Positive requires all tokens of the negation element (cue or scope) to be correctly identified.

3. • A False Negative is counted either by the system not identifying negation elements present in the gold annotations or by identifying them partially, that is, not all tokens have been correctly identified or the word forms are incorrect.

4. • A False Positive is counted when the system produces a negation element not present in the gold annotations.

For the evaluation of the sentiment analysis experiments (Phase C), the traditional measures used in text classification are applied: P, R and F1. They are measured per class (positive and negative) and averaged using the macro-average method.

5.5 Error analysis

In this section, we conduct an analysis of the SO-CAL system using our algorithm for negation detection, compared to SO-CAL’s built-in detection system, which simply traces back until it finds a negation cue, without explicitly detecting scope (see Subsection 5.1). As expected and as shown in previous work, the performance of the systems that integrate negation (SO-CAL with built-in negation and SO-CAL with negation processing system) outperforms the baseline (SO-CAL without negation) in terms of overall precision, recall, F1 and accuracy. We are interested in studying how this improvement takes place and, in particular, in the cases where it hinders rather than helps the sentiment prediction.

In general, SO-CAL without negation is biased towards positive polarity in Spanish, with the F1-score for positive reviews higher than for negatives ones, that is, this configuration without negation performs better on positive reviews, presumably because they contain fewer instances of negation. We found the same result in English; see Cruz et al. (Reference Cruz, Taboada and Mitkov2016). This means that ignoring negation has an impact on the recognition of negative opinion in reviews. It is also the case, however, that the negative class has a lower overall performance, mostly due to low recall. It is well established that detecting negative opinions is more difficult than detecting positive ones (Ribeiro et al. Reference Ribeiro, Araújo, Gonçalves, Gonçalves and Benevenuto2016), for a host of reasons, including a possible universal positivity bias (Boucher and Osgood Reference Boucher and Osgood1969).

The configuration of SO-CAL with the negation processing system achieves the best performance, improving on the baseline by 3.3% and the search heuristic by 2.5% in terms of overall accuracy. These results can be explained by two factors. First, the negation detector that we propose benefits from a wider list of cues (the built-in search heuristics in SO-CAL include 13 different negation cues while the SFU Review $_{\text{SP}}$ -NEG corpus contains 245 different negation cues). Second, the scope detection approach goes beyond the window-based heuristic that the SO-CAL system incorporates. We illustrate these two situations with examplesFootnote 9 below. Note that in the examples, there are two identifiers, ‘NEGATED’ and ‘NEGATIVE’. The former refers to a word in the scope of negation. The latter is used for any word or phrase that is negative either from the dictionary (i.e., it has a negative value in the dictionary) or that becomes negative as a result of negation. When a negative expression is encountered, SO-CAL multiplies its value by 1.5. This accounts for the asymmetry of negation: a negative expression tends to be more saliently negative than a positive expression is positive (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011; Taboada, Trnavac, and Goddard Reference Taboada, Trnavac and Goddard2017; Rozin and Royzman Reference Rozin and Royzman2001).

1. • Case 1: Negation cue predicted by the negation processing system but not present in the SO-CAL list. For example, in Example (7), the negation cue ningãún is identified by the negation detector but it is not present in the built-in SO-CAL list. Therefore, ningun temazo is correctly classified as negative ( $-1.5$ points) by SO-CAL when we integrate the Spanish negation detector, but with the heuristic that it incorporates by default it is incorrectly classified as positive (3.0 points).

2. (7) Aqui tenemos un disco bastante antiguo de los smith… a mi gusto no cuenta con ningun temazo…

   Here we have a pretty old Smith album… to my liking it doesn’t have any hits…

   1. a. SO-CAL with built-in negation:

      temazo 3.0 = 3.0

   2. b. SO-CAL with negation processing system:

      ningun temazo 3.0 - 4.0 (NEGATED) X 1.5 (NEGATIVE) = −1.5

3. • Case 2: Scope correctly identified by the Spanish negation processing system, but not detected by SO-CAL due to its heuristic that checks if a word is negated based on looking for a negation cue in the previous word, unless the previous word is in the list of skipped words (see Section 5.1). The sentence in Example (8) is correctly classified as negative when we use the Spanish negation detector in SO-CAL, because the sentiment word buena is identified as negated by the negation cue no. However, using the search heuristic that SO-CAL incorporates by default, the sentence is incorrectly classified as positive. The search heuristic works as follows. The system detects that buena is a sentiment word and checks if the previous word is a negation cue of the list; una is not in the list, so the system checks if it is a skipped word in order to continue checking the previous words, but una is not in the skipped list either, and therefore the sentence is incorrectly classified as positive.

4. (8) Han ahorrado en seguridad, lo que no es una buena politica.

   They have saved on security, which is not a good policy.

   1. a. SO-CAL with built-in negation:

      seguridad 2.0 = 2.0; buena 2.0 = 2.0

   2. b. SO-CAL with negation processing system:

      seguridad 2.0 = 2.0; no es una buena 2.0 - 4.0 (NEGATED) X 1.5 (NEGATIVE) = −3.0

The improvement we obtained on the sentiment classification of the reviews using the Spanish negation detector system is not as high as that achieved with the English system. In English, using the system by Cruz et al. (Reference Cruz, Taboada and Mitkov2016), we saw an improvement of 5% over the baseline and about 2% over the built-in search heuristics both in terms of overall F1 and accuracy. To determine where errors occurred in the Spanish analysis, and to tease apart those that were the result of a potentially under-developed Spanish SO-CAL as opposed to faulty negation detection, we have identified different error types. We describe each error type below, with examples.

Type 1 error: Words correctly identified as scope by the Spanish negation detector that are present in the SO-CAL dictionary, but are not sentiment words in the domain under study. In Example (9), the word official is in the scope of negation and belongs to the positive dictionary. However, in this context, official is not a positive word. The application of the sentiment heuristic of SO-CAL converts this word into a very negative one, and consequently, the negative polarity of the sentence is increased in an incorrect way.

1. (9) De todas las mecánicas que puede montar, a mi la que más me gusta es el modelo de gasoil, de 1.9 cc pues creo que lo que pagas y las prestaciones que te da están muy bien, además su consumo es bastante equilibrado, si no subimos mucho el régimen de giro (por encima de las 3500 vueltas), podemos gastar unos 6 litros y poco más de gasoil, estos datos no son los oficiales, son los reales obetnidos con este modelo, aunque por supuesto, dependiendo de muchos factores, este consumo varriará.

1. a. SO-CAL with built-in negation:

   oficiales 1.0 = 1.0

2. b. SO-CAL with negation processing system:

   no son los oficiales 1.0 - 4.0 (NEGATED) X 1.5 (NEGATIVE) = −4.5

Type 2 error: Positive words in the SO-CAL dictionary whose sentiment value is low and the negation weighting factor is very high ( $-4$ ). The sentiment heuristic of SO-CAL works as follows: if a positive word is negated, 4 points are subtracted from the scoring of the positive word and if the result is a negative value, it is multiplied by 1.5 points (this helps capture the asymmetric nature of negation; see above). On the other hand, if the word is negative, it is annulled, that is, to the scoring of the word we add its opposite value. In Example (10), the positive word mejor has a value of 1 point in the SO-CAL dictionary. This is a low sentiment value, and the negation weighting factor is very high ( $-4$ ), consequently the polarity of the sub-string sin ser el mejor has a high negative value ( $-4.5$ ), causing the sentence to be incorrectly classified as negative ( $0.67 - 4.5 + 1.25 + 1 = -1.58$ ).

1. (10) Es una buena opción que sin ser el mejor ordenador del mercado, en relación calidad-precio es muy aceptable y durante un par de años (mínimo) estarás muy agusto con él, luego, quizás tengas que ampliar memoria, etc.

1. a. SO-CAL with built-in negation:

   ampliar 1.0 = 1.0; buena 2.0 X 1/3 (REPEATED) = 0.67; mejor 1.0 X 1/2 (REPEATED) = 0.5; muy aceptable 1.0 X 1.25 (INTENSIFIED) = 1.25

2. b. SO-CAL with negation processing system:

   buena 2.0 X 1/3 (REPEATED) = 0.67; sin ser el mejor 1.0 - 4.0 (NEGATED) X 1.5 (NEGATIVE) = -4.5; muy aceptable 1.0 X 1.25 (INTENSIFIED) = 1.25; ampliar 1.0 = 1.0

Type 3 error: Sentiment words not included in the SO-CAL dictionary. In Example (11), the positive word encanta is not detected by SO-CAL because it is not in the positive dictionary. Therefore, the sentence is incorrectly classified with 0 points instead of being labelled as a positive sentence.

1. (11) A todos mis amigos les encanta mi movil y ahora están pensando en comprárselo ellos también, bueno os dejo amigos de Ciao!!

1. a. SO-CAL with built-in negation: 0

2. b. SO-CAL with negation processing system: 0

Type 4 error: Negation used in an ironic way. In Example (12), the sub-string no nos íbamos a asfixiar porque tenía sus boquetitos contains the negation cue no, that is correctly identified along with its scope by the Spanish negation detector. However, in this case, negation is used in an ironic way and it should not have been taken into account as negation. Therefore, instead of being classified with 0 points, it should have been assigned a negative score.

1. (12) INCREÍBLE, el cuarto era de moqueta y no brillaba la limpieza, la iluminación era del conde drácula y a mi me daba un agobio no poder abria la ventana increible, pero claro no nos íbamos a asfixiar porque tenía sus boquetitos por el que entraba el aire perfumado por lo que adornaba la ventana.

1. a. SO-CAL with built-in negation:

   agobio −4.0 X 1.5 (NEGATIVE) = −6.0; asfixiar −5.0 X 2.0 (HIGHLIGHTED) X 1.5 (NEGATIVE) = −15.0; increible −4.0 X 1.5 (NEGATIVE) = −6.0; claro 1.0 X 2.0 (HIGHLIGHTED) = 2.0

2. b. SO-CAL with negation processing system:

   agobio −4.0 X 1.5 (NEGATIVE) = −6.0; claro no nos íbamos a asfixiar −5.0 + 5.0 (NEGATED) X 1.3 (INTENSIFIED) X 2.0 (HIGHLIGHTED) = 0; no poder abria la ventana increible −4.0 + 4.0 (NEGATED) = 0

Types 1–4 are errors of the sentiment analysis system. Now, we focus on errors in the negation detection system.

Type 5 error: Negation cue detected by the SO-CAL system, but not predicted by the negation detector. In Example (13), the negation processing system has not predicted the word falta as negation cue. Therefore, the word mejorar has been classified as positive (6 points), but it should have been classified as negative due to the presence of negation.

1. (13) Es un gran telefono por la forma, pero falta mejorar lo muchisimo para mi gusto.

1. a. SO-CAL with built-in negation:

   falta mejorar 3.0 - 4.0 (NEGATED) X 2.0 (HIGHLIGHTED) X 1.5 (NEGATIVE) = −3.0; gran 3.0 = 3.0

2. b. SO-CAL with negation processing system:

   mejorar 3.0 X 2.0 (HIGHLIGHTED) = 6.0; gran 3.0 = 3.0

Type 6 error: Scope erroneously predicted by the negation detector. In Example (14), the negation processing system has predicted the following as scope of the last negation cue, no: no dejaria de escribir sobre esta horrible experiencia. However, this scope is not correct and, consequently, the sentiment word horrible has been negated, but it should not have been negated and it should have preserved its negative polarity.

1. (14) No hace falta hablar de la calidad de dicho aparato, PESIMA increiblemente malo, resulta que al abrir la tapa se ha roto la pantalla interior y no se ve nada, fui a el servicio tecnico ya que es sorprendente que solo me durara un mes y me dijeron que se habia roto por la presion ocasionada a el abrirlo, ALUCINANTE ya que no he ejercido ninguna presion en el movil ni he dado ningun golpe, pero bueno vamos a las prestaciones que tiene que de la rabia que tengo no dejaria de escribir sobre esta horrible experiencia.

1. a. SO-CAL with built-in negation:

   horrible −4.0 X 2.0 (HIGHLIGHTED) X 1.5 (NEGATIVE) = −12.0; increiblemente malo −3.0 X 1.35 (INTENSIFIED) X 1.5 (NEGATIVE) = −6.075; solo −1.0 X 1.5 (NEGATIVE) = −1.5; FACILIDAD 3.0 X 2.0 (CAPITALIZED) = 6.0; presion −3.0 X 1.5 (NEGATIVE) = −4.5; ninguna presion −3.0 + 3.0 (NEGATED) = 0; sorprendente 3.0 = 3.0; movil 1.0 = 1.0

2. b. SO-CAL with negation processing system:

   increiblemente malo −3.0 X 1.35 (INTENSIFIED) X 1.5 (NEGATIVE) = −6.075; no dejaria de escribir sobre esta horrible −4.0 + 4.0 (NEGATED) X 2.0 (HIGHLIGHTED) = 0; solo −1.0 X 1.5 (NEGATIVE) = −1.5; FACILIDAD 3.0 X 2.0 (CAPITALIZED) = 6.0; presion −3.0 X 1.5 (NEGATIVE) = −4.5; no he ejercido ninguna presion −3.0 + 3.0 (NEGATED) = 0; sorprendente 3.0 = 3.0; movil 1.0 = 1.0

In Table 11, we provide the results of an error analysis of 224 sentences from 12 different reviews, 88 of which (39.3%) contained at least one instance of negation. In those sentences, we found 23 errors with the negation or with how the negation was incorporated into SO-CAL. Type 1 and Type 2 errors are the most common, having to do with either lack of coverage in the SO-CAL dictionaries or with how negation was computed if it was found. The next highest category is Type 6, with the negation system mistakenly identifying a scope. There are, however, only six of those cases, and they do not all affect the sentiment. It is clear that the negation processing system does its job fairly well, but it is hindered by the relatively less well-developed Spanish SO-CAL (in comparison to the English version). Better performance, therefore, can be achieved by developing the system in conjunction with adopting a state-of-the-art negation processing system.

Table 11.

Error analysis of 224 sentences

One interesting case that is rare, but indicative of the difficulty of sentiment analysis, is a case where the negation processing system detects negation accurately, and negates the right word, but the negation ends up hurting the score of the text overall because the negated word is not directly relevant to the product being discussed. In Example (15), SO-CAL’s built-in negation method misses the negation. The negation processing system detects it and changes the polarity of activa, literally ‘active’ to negative. In this case, however, the word does not refer to the phone being discussed in the review, or to the other phone the reviewer considered (PEBL U6), but to the phone company’s available phones for their ‘active pack’. We categorised this as a Type 1 error, having to do with the domain vocabulary, but it is clearly a more complex issue about word ambiguity in context (Benamara, Inkpen, and Taboada Reference Benamara, Inkpen and Taboada2018).

1. (15) yo a el principio keria el PEBL U6 negro pero era demasiado caro y no estaba en el pack activa de movistar.

Regarding the detection of negation, as identified in the work of Jiménez-Zafra et al. (Reference Jiménez-Zafra, Morante, Blanco, Martín-Valdivia and Ureña-López2020), the Spanish language has some peculiarities. Negation cues can be simple (Example (16)), continuous (Example (17)) or discontinuous (Example (18)). Moreover, some common negation cues, such as no, are also frequent in comparative (Example (19)), contrasting (Example (20)) and rhetoric structures (Example (21)), making the task more difficult. In addition, the scope of negationFootnote 10 can span before the cue (Example (22)), after the cue (Example (23)), or before and after the cue (Example (24)). In the texts analysed from the point of view of their application to sentiment analysis, the errors found are due to (i) the system is trained to identify syntactic negation and (ii) it is difficult to determine whether the subject and complements of the verb are included in the scope or not. Depending on the negation cue used, we can find different types of negation: syntactic negation (e.g., no [no/not], nunca [never]), lexical negation (e.g., negar [deny], desistir [desist]), morphological or affixal negation (e.g., ilegal [illegal] and incoherente [incoherent]). The negation detector is trained to identify only syntactic negation, so it is not able to correctly predict lexical nor morphological negation. As it is shown in Example (13)), the negation processing system has not predicted the lexical negation cue falta. This could be solved by performing the annotation of the other types of negation, task that we plan to carry out in the future. On the other hand, the identification of scopes is a difficult task. Most of the errors that we foundFootnote 11 are due to the fact that the scope does not always start with the cue and does not always end in a punctuation mark (Example (25)) and the inclusion or not of the subject (Example (26)) and the complements of the verbs within it (Example (27)). In order to solve these errors, we plan to experiment with adding more sophisticated syntactic features.

1. (16) El problema es que no saben arreglarlo.

   The problem is they don’t know how to fix it.

2. (17) Ni nunca quiso ser de nadie.

   Nor did he ever want to be anyone’s.

3. (18) No tengo nada en contra de Opel.

   I have nothing against Opel.

4. (19) No me gusta tanto como lo otro.

   I don’t like it as much as the other thing

5. (20) No hay más solución que comprar una lavadora.

   There is no other solution than to buy a washing machine

6. (21) Viniste a verlo, no?

   You came to see him, didn’t you?

7. (22) [El producto tiene fiabilidad cero].

   [The product has zero reliability].

8. (23) El problema es que [no saben arreglarlo].

   The problem is [they don’t know how to fix it].

9. (24) Aunque [las habitaciones no están mal], la atención recibida me hace calificarlo mal.

   [The rooms are not bad], but the attention received makes me rate it poorly.

10. (25) Tiene 156.25 gibabytes de disco duro, que como os podreis imaginar [{sin ser excesivo] estan muy bien a nivel usuario}, se tarda mucho, pero mucho tiempo en llenar el disco duro…

    It has 156.25 gibabytes of hard disk, which as you can imagine [{without being excessive] are very good at the user level}, it takes a lot, but a long time to fill the hard drive…

11. (26) Los plásticos resultan demasiado evidentes y [la tapicería {no es nada del otro mundo}].

    Plastics are too obvious and [upholstery is {no big deal}].

12. (27) Vamos, [por 11900 euros {yo no me lo compraba}].

    [For 11900 euros {I didn’t buy it}].

In summary, we have shown that accurate negation detection is possible and that the system that we have adopted from previous work perform well. In addition, we show that improvements in sentiment analysis can be gained from detecting negation and its scope with sophisticated negation processing systems.