2.1 Negation in sentiment models

Negation is a frequent linguistic phenomenon which has a direct impact on sentiment analysis (Wiegand et al. Reference Wiegand, Balahur, Roth, Klakow and Montoyo2010). Within the framework of lexicon-based sentiment analysis, researchers first attempted to model negation with simple heuristics, such as reversing (Hu and Liu Reference Hu and Liu2004; Polanyi and Zaenen Reference Polanyi and Zaenen2006; Kennedy and Inkpen Reference Kennedy and Inkpen2006) or modifying (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011) the polarity signal of a negated word. This approach to tackle contextual valence shifting generally assumes that the final polarity of a text is some function of the prior polarities of adjectives, verbs, and nouns found in the text. The scope of negation is determined heuristically, by finding common negation cues and assuming all words between the cue and the next punctuation are in scope (Hu and Liu Reference Hu and Liu2004) or based on the distance from the cue (Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011).

Early machine learning approaches to sentiment analysis also used heuristics, such as attaching a negation tag (“_neg”) to words assumed to be in scope (Pang et al. Reference Pang, Lee and Vaithyanathan2002; Das and Chen Reference Das and Chen2007). This approach, however, leads to an increase in sparsity and varying results, as the sentiment model is not able to explicitly connect the original and negated features. Other research has used negation detection systems to enhance the feature space of sentiment models (Councill et al. Reference Councill, McDonald and Velikovich2010; Lapponi et al. Reference Lapponi, Read and Øvrelid2012a; Cruz et al. Reference Cruz, Taboada and Mitkov2016), leading to improved results on sentiment classification. Additionally, certain negation cues contribute to higher shifts in polarity than others (Zhu et al. Reference Zhu, Guo, Mohammad and Kiritchenko2014), which indicates they should be modeled separately.

More recent advances in negation detection, both in terms of modeling and data annotation, have not been incorporated into sentiment classification models so far, to the best of our knowledge.

2.2 Negation detection

Previously reported approaches to negation analysis commonly break it down to (at least) two sub-tasks, performing (i) negation cue detection, followed by (ii) scope detection. The example in Figure 1 shows the negation annotation of the following sentence (in the first row):

1. (3) There is no flowery dialog, and time isn’t wasted.

Figure 1.

An example of negation detection annotation on a sample sentence from the SFU data set.

For our example in Figure 1, the cue detection component would locate the negation cues in the sentence, that is, the negative determiner “no”, and the copula with its negative contraction “is n’t”, whereas the scope detection module would recognize the noun phrase “flowery dialog” and the verb phrase “wasted” as the scopes of these cues, respectively. Depending on the specific annotation scheme, subjects may or may not be part of the scope of negation.

A large portion of early work on negation detection (Morante, Liekens, and Daelemans Reference Morante, Liekens and Daelemans2008; Morante and Daelemans Reference Morante and Daelemans2009; Velldal et al. Reference Velldal, Øvrelid, Read and Oepen2012) has been done within the biomedical domain due to the availability of the BioScope corpus (Vincze et al. Reference Vincze, Szarvas, Farkas, Móra and Csirik2008), which is annotated for negation cues and scopes. Interest in the task was further spurred by the ^*SEM shared task (Morante and Blanco Reference Morante and Blanco2012), which focused on the detection of negation cues and scopes, in addition to detection of negated events and their so-called focus. The shared task made available the ConanDoyle-neg corpus, which is described in Section 3. A number of systems were submitted for this task, employing a wide variety of strategies. For example, the best performing systems for the closed track and open track employed, respectively, Support Vector Machine (SVM)-based ranking of constituent (sub-)trees (Read et al. Reference Read, Velldal, Øvrelid and Oepen2012) and conditional random field (CRF)-based sequence labeling using dependency features (Lapponi et al. Reference Lapponi, Velldal, Øvrelid and Read2012b).

Traditional approaches to the task of negation detection have typically employed a wide range of hand-crafted features describing a number of both lexical, morphosyntactic, and even semantic properties of the input text. Syntactic parsing has often been used to analyze the input prior to negation detection and has been based on both constituency-based (Read et al. Reference Read, Velldal, Øvrelid and Oepen2012; Packard et al. Reference Packard, Bender, Read, Oepen and Dridan2014) and dependency-based representations (Lapponi et al. Reference Lapponi, Read and Øvrelid2012a; White Reference White2012; Enger, Velldal, and Øvrelid Reference Enger, Velldal and Øvrelid2017). It is also possible to combine an existing system (Read et al. Reference Read, Velldal, Øvrelid and Oepen2012) with an additional layer of manually defined rules over Minimal Recursion Semantics structures created by an Head-driven Phrase Structure Grammar (HPSG) parser (Packard et al. Reference Packard, Bender, Read, Oepen and Dridan2014).

There are a few previous studies that investigate neural modeling for the task of negation detection. Among these, we find a CNN model for the negation scope detection on the abstracts section of the BioScope corpus, which operates over syntactic paths between the cue and candidate tokens (Qian et al. Reference Qian, Li, Zhu, Zhou, Luo and Luo2016). Fancellu et al. (Reference Fancellu, Lopez and Webber2016) present and compare two neural architectures for the task of negation scope detection on the ConanDoyle-neg corpus: a simple feed-forward network and a bidirectional LSTM (BiLSTM). Note that these more recent neural systems disregard the task of cue detection altogether (Fancellu et al. Reference Fancellu, Lopez and Webber2016; Qian et al. Reference Qian, Li, Zhu, Zhou, Luo and Luo2016; Fancellu et al. Reference Fancellu, Lopez, Webber and He2017), relying instead on gold cues and focusing solely on the task of scope detection.

While syntactic information has often been found useful for scope resolution, the task of cue detection appears to only require simpler surface information. Velldal et al. (Reference Velldal, Øvrelid, Read and Oepen2012) present an approach to cue detection which treats the set of cue words as a closed class and apply a disambiguation-based approach to the problem of cue detection, showing that simple lexical features based on a narrow context window are sufficient to achieve good performance.

As further detailed in Section 4.1, in the current paper, we model cue detection and scope resolution concurrently as a sequence labeling task, using a BIO label encoding which is illustrated in the final row of Figure 1. BIO labeling for negation detection has been employed in previous work, following the early work of Morante et al. (Reference Morante, Liekens and Daelemans2008). Our joint modeling of cue and scope detection differs from previous approaches to negation detection as reviewed above that handle cue and scope resolution as two separate tasks (Morante and Blanco Reference Morante and Blanco2012; Lapponi et al. Reference Lapponi, Velldal, Øvrelid and Read2012b; Read et al. Reference Read, Velldal, Øvrelid and Oepen2012; Fancellu et al. Reference Fancellu, Lopez and Webber2016; Cruz et al. Reference Cruz, Taboada and Mitkov2016; Qian et al. Reference Qian, Li, Zhu, Zhou, Luo and Luo2016). Moreover, motivated by the assumption that downstream tasks like sentiment analysis only need information about which words are within the scope of some negation, regardless of which particular cue it relates to (in cases where more than one cue is present), we do not attempt to explicitly retain this coupling. Similar to Fancellu et al. (Reference Fancellu, Lopez and Webber2016), we use a BiLSTM-based model, relying not only on word embeddings as input, but also adding a CRF for the prediction layer. When it comes to incorporating information about negation to our sentiment model, our approach takes advantage of the representation learning capabilities of neural models: rather than passing on the negation predictions output by the final CRF layer, we pass on the intermediate representations learned by the BiLSTM. The details of this cascading architecture are further described in Section 4.

2.3 Sentiment analysis

Approaches to sentiment analysis have moved from lexicon-based methods (Turney Reference Turney2002; Hu and Liu Reference Hu and Liu2004; Taboada et al. Reference Taboada, Brooke, Tofiloski, Voll and Stede2011), to machine learning methods based on hand-derived features (Pang et al. Reference Pang, Lee and Vaithyanathan2002; Pang and Lee Reference Pang and Lee2008) and finally to neural networks that learn to extract useful features in an end-to-end fashion (Socher et al. Reference Socher, Perelygin, Wu, Chuang, Manning, Ng and Potts2013; Tang et al. Reference Tang, Wei, Yang, Zhou, Liu and Qin2014; Tai, Socher, and Manning Reference Tai, Socher and Manning2015). While some of these neural architectures have been tailored to suit specific tasks better (Irsoy and Cardie Reference Irsoy and Cardie2014; Lei et al. Reference Lei, Yang, Yang and Liu2018), two recent end-to-end architectures have shown competitive results on a large number of natural language processing tasks: BiLSTMs Networks (Graves and Schmidhuber Reference Graves and Schmidhuber2005) and Self-Attention Networks (SANs) (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017). Variants of these two architectures give state-of-the-art results on document-level (Howard and Ruder Reference Howard and Ruder2018), sentence-level (Peters et al. Reference Peters, Neumann, Iyyer, Gardner, Clark, Lee and Zettlemoyer2018; Devlin et al. Reference Devlin, Chang, Lee and Toutanova2018), and aspect-level (Xu et al. Reference Xu, Liu, Shu and Yu2019) sentiment analysis tasks.

The claim made by the proponents of end-to-end learning is that the models implicitly learn compositional functions (Socher et al. Reference Socher, Perelygin, Wu, Chuang, Manning, Ng and Potts2013; Irsoy and Cardie Reference Irsoy and Cardie2014), thereby removing the need to explicitly provide information about inter-word dependencies, negation, or speculation in the form of hand-crafted features. Recent research, however, challenges the idea that end-to-end learning is able to fully capture compositional effects (Verma, Kim, and Walter Reference Verma, Kim and Walter2018; Barnes et al. Reference Barnes, Øvrelid and Velldal2019a). It is therefore worth asking whether we can help the model by providing some form of explicit training on compositional phenomena in sentiment.

2.4 Multi-task learning

Multi-task learning (MTL) (Caruana Reference Caruana1993) stems from the idea that learning-related tasks simultaneously allow a machine learning algorithm to incorporate a useful inductive bias by restricting the search space of possible representations to those that are predictive for both tasks. MTL assumes that features that are useful for a certain task should also be predictive for similar tasks, and in this sense, MTL also effectively acts as a regularizer, as it prevents the weights from adapting too much to just one task. Under some circumstances, multi-task learning can also be seen as a kind of data augmentation, where an MTL model takes advantage of extra training data available in an auxiliary task to improve the main task (Kshirsagar et al. Reference Kshirsagar, Thomson, Schneider, Carbonell, Smith and Dyer2015; Plank Reference Plank2016; Fares, Oepen, and Velldal Reference Fares, Oepen and Velldal2018). MTL is particularly well-suited for neural models, given the possibilities for modular design and representation learning. Below we outline some of the different ways that multi-task learning can be set up, while also reviewing previous MTL efforts in NLP.

Hard parameter sharing (Caruana Reference Caruana1993), which assumes that all layers are shared between tasks except for the final predictive layer, is the simplest way to implement a multi-task model. When the main task and auxiliary task are closely related, this approach has been shown to be an effective way to improve model performance (Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011; Peng and Dredze Reference Peng and Dredze2017; Martínez Alonso and Plank Reference Martínez Alonso and Plank2017; Augenstein, Ruder, and Søgaard Reference Augenstein, Ruder and Søgaard2018). Other research (Søgaard and Goldberg Reference Søgaard and Goldberg2016), on the other hand, finds that it is better to make predictions for low-level auxiliary tasks at lower layers of a multi-layer MTL setup. They also suggest that under the hard-parameter framework, auxiliary tasks need to be sufficiently similar to the main task for MTL to improve over the single-task baseline.

There have also been several effective implementations of soft parameter sharing, where two models have both shared and private task-specific parameters, such as including a gating mechanism that allows a MTL model to select which information to share across tasks (Liu, Qiu, and Huang Reference Liu, Qiu and Huang2016; Ruder et al. Reference Ruder, Bingel, Augenstein and Søgaard2019). Their results suggest that hard-parameter sharing is only beneficial for low-level tasks, while for high-level tasks, it is better to learn how much to share at each layer and subspace of parameters in the network. Furthermore, they find that MTL is more beneficial when there is less training data for the main task and that modeling subspaces explicitly helps in almost all domains.

What characteristics of an auxiliary task are necessary to improve a main task is still largely unknown. Some research suggests high-level semantic auxiliary tasks generally help more than low-level auxiliary tasks as MTL tends to work when the main task learning plateaus quickly and the auxiliary task learning does not (Bingel and Søgaard Reference Bingel and Søgaard2017). Others find that auxiliary tasks with compact, uniform label distributions are preferable (Martínez Alonso and Plank Reference Martínez Alonso and Plank2017). Additionally, choosing a suitable auxiliary task is still vital, as introducing an unsuitable auxiliary task can actually hurt performance (Augenstein and Søgaard Reference Augenstein and Søgaard2017).

In this work, we propose that MTL is an appropriate framework to incorporate negation detection in a sentiment classifier. Unlike previous approaches in sentiment analysis (Councill et al. Reference Councill, McDonald and Velikovich2010; Lapponi et al. Reference Lapponi, Read and Øvrelid2012a; Cruz et al. Reference Cruz, Taboada and Mitkov2016), our method does not rely on incorporating negation information as explicit features, but rather uses a cascading architecture where the intermediate representations learned for predicting negation feeds into subsequent layers (along with skip connections) for predicting sentiment. While the final layers of the network hierarchy are dedicated to the sentiment task, the lower layers are shared and supervised by both tasks. The components of the architecture are further detailed in Section 4. Note that, for comparison, we also explore using other auxiliary tasks beyond negation.

In parallel work to this, Barnes et al. (Reference Barnes, Touileb, Øvrelid and Velldal2019b) showed that a similar cascading or hierarchical MTL architecture could be used for incorporating information from sentiment lexicons to improve models for sentence-level sentiment classification. Also in parallel work, Sanh, Wolf, and Ruder (Reference Sanh, Wolf and Ruder2019) apply hierarchical MTL to learn shared representations for a set of semantic tasks where lower-level task like named entity recognition and entity mention feeds into higher-level tasks like coreference resolution and relation extraction.

Table 1.

Overview of the data sets for sentiment and negation. Note that for SST, SFU, and Conan Doyle Neg, we show the number of sentences, while for SemEval 2013, we show the number of tweets

4.1 Negation model

We start by discussing the part of the model responsible for detecting negation cues and scopes, including how it relates to some of the previously reported approaches that are most directly relevant.

Similarly to Fancellu et al. (Reference Fancellu, Lopez and Webber2016), we use a BiLSTM network (Graves and Schmidhuber Reference Graves and Schmidhuber2005) to extract features from the embedding layer, but where Fancellu et al. (Reference Fancellu, Lopez and Webber2016) use a linear softmax layer for prediction, we use a linear-chain CRF with Viterbi decoding to find the most probable assignment of labels, as CRFs can better model sequences. Moreover, while Fancellu et al. (Reference Fancellu, Lopez and Webber2016) assume gold cues, encoded as separate cue embeddings concatenated to the word embeddings provided as input, we here let the BiLSTM to predict both cues and scopes – performed in one pass.

Note that there might be several instances of negation in the same sentence, as in the example of Figure 1. In the setup of Fancellu et al. (Reference Fancellu, Lopez and Webber2016), each instance is multiplied out into a separate example, effectively duplicating the sentence for each pair of cue and scope. In our setup, all instances will be treated in the same pass. In the CRF model of Lapponi et al. (Reference Lapponi, Velldal, Øvrelid and Read2012b) too, all scopes are predicted in one pass – although cues are there predicted in a preceding step using an SVM classifier as in Read et al. (Reference Read, Velldal, Øvrelid and Oepen2012) – but then post-processing heuristics are applied to assign the identified negation tokens to their respective cues.

A simplifying assumption made in our model is that we do not care about explicitly preserving the links between particular cues and scopes in our output; intuitively, the important information for a downstream task like sentiment analysis is whether a token is within the scope of negation, regardless of the identity of the negation cue. Additionally, since we do not incorporate sub-token information in our model, we treat any token annotated with morphological negation, for example un- or -less, as a negation cue.

As shown in Figure 2, given a sequence of tokens, our negation model first embeds these in an embedding layer and then uses a BiLSTM to create a contextualized representation of each token. This representation is then used as features in the CRF. In our experiments, we use Viterbi decoding to find the most probable assignment of labels and train the model to minimize the negative log likelihood.

4.2 Sentiment model

The sentiment model uses the same embedding layer and the first BiLSTM layer to create the contextualized representation of the input tokens. We make use of skip connections where we concatenate each of the original embeddings to the contextualized representations. This sequence then serves as input to a second sentiment-specific BiLSTM layer.

Finally, we perform a max pooling operation on the output of the sentiment-specific BiLSTM and pass this max-pooled representation to a softmax layer to compute the class probabilities. We then minimize the cross entropy loss of the sentiment predictions with respect to the true sentiment.

During training, the model alternates between training one epoch on the main task and one epoch on the auxiliary task. Preliminary experiments showed that more complicated training strategies (alternating training between each batch or uniformly sampling batches from the two tasks) did not lead to improvements. Note that we do not upsample negation data. We train the model for 10 epochs using Adam (Kingma and Ba Reference Kingma and Ba2014), performing early stopping determined by accuracy on the development set. We regularizeFootnote ^b with dropout before the BiLSTM layers ( $0.5$ ), between BiLSTM layers ( $0.3$ ), apply batch norm, and L2 regularization ( $0.0001$ ). As neural models are sensitive to the random initialization of their parameters, we perform five runs with different random seeds and show the mean and standard deviation as the final result for each model.Footnote ^c

6.1 Error analysis

A per class evaluation of the SST-binary and SST-fine tasks (Figure 3) shows what effect multi-task learning has on each sentiment class.Footnote ^f On SST-binary, the MTL model improves on both positive and negative classes. On the SST-fine task, however, the model improves only on the negative and strong positive classes, performing worse on strong negative and positive, while performing nearly the same on neutral. An analysis of the data shows that the negative class contains the largest percentage of negated sentences (27%), while the strong positive has the least (13%). It is possible that the MTL model is able to better discriminate relevant and non-relevant negation. In the example from the SST-fine task (4) below, the STL model assigned the sentence a positive label due largely to the number of positive tokens, while the MTL-CD model correctly predicted the negative label, as it was able to resolve the negation.

1. (4) Accuracy and realism are terrific, but if your film becomes boring, and your dialogue isn’t smart, then you need to use more poetic license.

The previous analysis suggests that the multi-task setup is beneficial for sentiment analysis, but does not confirm that the model is actually learning better representations for negated sentences. Here, we look at how each model performs on negated and non-negated sentences.

As we do not have access to gold negation annotations on the main task sentiment data, we create silver data by assuming that any sentence that has a negation cue (taken from SFU) is negated. We then extract the negated and non-negated sentences from the SST fine-grained (397 negated/1813 non-negated) and binary (319/1502) test sets. While this inevitably introduces some noise, it allows us to observe general trends regarding these two classes of sentences.

Table 3 shows the results of the analysis. On the binary task, the MTL model performs better on both the negated (+2.2 ppt.) and non-negated (+1.3 ppt.) subsets. On the fine-grained task, however, the STL model outperforms the MTL model on the negated subsection (−0.4 ppt.), while the MTL model performs better on the non-negated subsection (+0.3 ppt.). More detailed analysis would be needed to explain why the binary-task STL model outperforms the MTL model on our silver-standard negated sample. However, when it comes to the better performance of the MTL model on the non-negated sample – for both the binary and fine-grained task – one possible explanation is that learning about negation also enables the model to make more reliable predictions about sentiment bearing words that it has perhaps only seen in a negated context during training, but outside of negation during testing.

Table 3.

Mean accuracy and standard deviation of STL and MTL models on the negated and non-negated subsets of the SST test data. Bold results show the best model on each data sub-set.

6.2 Impact of data size

In order to better understand the effects of multi-task learning of negation detection, we compute learning curves with respect to the negation data for the SST-binary setup. The model is given access to an increasing number of negation examples from the SFU data set (from 10 to 800 in intervals of 100), and accuracy is calculated for each number of examples. Figure 4 (left) shows that the MTL model improves over the baseline with as few as 10 negation examples and plateaus somewhere near 600. An analysis on the SST-fine setup showed a similar pattern. There is nearly always an effect of diminishing returns when it comes to adding training examples, but if we were to instead plot this learning curve with a log-scale on the x-axis, that is doubling the amount data for each increment, it would seem to indicate that having more data could indeed still prove useful, as long as there were sufficient amounts. In any case, regardless of the amount of data, exposing the model to a larger variety of negation examples could also prove beneficial – we follow up on this point in the next subsection.

Figure 4.

Mean accuracy on the SST-binary task when training MTL negation model with differing amounts of negation data from the SFU data set (left) and sentiment data (right).

While the previous experiment shows that models already improve with as few as 10 auxiliary examples, here we investigate whether a sentiment model benefits from multi-task learning more when there is limited sentiment data, as previous research has shown for other tasks (Ruder et al. Reference Ruder, Bingel, Augenstein and Søgaard2019). We keep the amount of auxiliary training data steady and instead vary the sentiment training data from 100–8000 examples.

Figure 4 (right) shows that the performance of the STL model begins to plateau at around 5000 training examples (although note the comment about diminishing returns above). Until this point, the MTL model performs either worse or similarly. From 5000 on, however, the MTL model is always better. Therefore, negation detection cannot be used to supplement a model when there is lacking sentiment data, but rather can improve a strong model. This may be a result of using a relevant auxiliary task which has a different labeling unit (sequence labeling vs. sentence classification), as other research (Ruder et al. Reference Ruder, Bingel, Augenstein and Søgaard2019) suggests that for similar tasks, we should see improvements with less main task data.

6.3 Can we combine negation data despite differences in annotation?

The previous experiment suggests that more negation data will not necessarily lead to large improvements. However, the model trained on the SFU negation data set performs better on the SST-binary task, while the CD negation model is better on SST-fine. In this section, we ask whether a combination of the two negation data sets will give better results, despite the fact that they have conflicting annotations, see Section 3 above.

We train an MTL model on the concatenation of the SFU and CD train sets (MTL-Combined) using the same hyperparameters as in the previous experiments. The results in Table 4 show that MTL-Combined performs worse than the MTL-SFU model on SST-binary, while it is the best performing model by a small margin ( $p > 0.01$ with approximate randomization tests as described in Section 5) on SST-fine. This shows that simply combining the negation data sets does not necessarily lead to improved MTL results, which is most likely due to the differences in the annotation schemes.

Table 4.

Combining the SFU and CD negation data (MTL-Combined) does not lead to large improvements. Bold results show the best performing model on each dataset.

6.4 Scopes or cues

As described in the initial sections, negation is usually represented by a negation cue and its scope. One interesting question to address is whether both of these are equally important for downstream use in our multi-task setup. Here, we investigate whether it is enough to learn to identify only cues or only scopes. We train the MTL model from Section 4 to predict only cues or only scopes and compare their results with the STL model and the original MTL model which predicts both.

The results of each experiment are shown in Table 5. Learning to predict only cues or only scopes performs worse than the MTL model trained to predict both. Additionally, learning to predict only one of the two elements also performs worse than STL on the fine-grained setup. This indicates that it is necessary to learn to predict both scopes and cues. One likely explanation for this is that the cue predictions in turn benefits scope predictions. On the SST-binary, the MTL-Cues model performs better than the MTL-Scopes model, while the opposite is true for the SST-fine task, indicating that it is more important to correctly predict the negation scope for the fine-grained setting than the binary.

Table 5.

Mean accuracy and standard deviation on the sentiment task for the STL model, MTL models trained to predict only negation cues or only negation scope, and finally the MTL model trained to predict both scopes and cues. Bold results show the best performing model on each dataset.

6.5 Can models trained on phrase-level data improve with MTL?

Besides the sentence-level annotations we have used so far, the SST also contains sentiment annotations at each node of a constituent tree (i.e., its constituent phrases) for all sentences (statistics are shown in Table 6). The sentence “It was fun to watch”, for example, would contain the constituents “It”, “was”, “fun”, “to”, “watch”, “was fun to watch”, “fun to watch”, “to watch”, each with its corresponding polarity label.

Table 6.

Number of training, development, and test examples for the Stanford Sentiment Treebank phrase-level data

Although originally intended to enable recursive approaches to sentiment, it has also been shown that training a non-recursive model with these annotated phrases leads to models that are better at capturing compositionality effects (Iyyer et al. Reference Iyyer, Manjunatha, Boyd-Graber and Daume2015). This is likely because, given a sentence such as “The movie was not great”, models are explicitly shown that “great” is positive, while “not great” is negative. A relevant question is therefore whether this phrase-level annotation of sentiment reduces the need for explicit negation annotation. In this section, we compare training on these sentiment-annotated phrases to multi-task learning on negation annotations, and also the combination of these two approaches.

We train the STL model from Section 4 on the phrase-level SST data (STL-phrase) and compare with the MTL model trained with phrase-level SST data and with negation as an auxiliary task (MTL-phrase). In order to fairly compare with models trained only on sentence-level annotation, we test on the sentence-level SST data described in Section 3. The results in Table 7 show that even though the largest gains are found by training on the phrase-level data, multi-task learning (MTL) of negation still provides small but consistent gains. This indicates that while end-to-end models may learn some compositional functions implicitly when trained on phrase-level data, there is still room for further improvements by combining this with training on explicit negation annotations in addition. However, while the MTL approach is generally applicable to any sentiment data set, the phrase-level annotations are particular to the SST data.

Table 7.

Mean accuracy and standard deviation of STL, MTL, and STL-Phrase-level models. Bold results show the best performing model on each dataset.

6.6 Evaluating the negation component: A case for transfer learning?

Although our interest in negation modeling in this paper is primarily tied to its influence on sentiment analysis, we do, however, also want to evaluate negation performance in isolation, just to make sure the model is reasonable. There are a number of evaluation metrics used for negation detection. For example, $\text{F}_1$ scores can be computed with respect to cues or scopes or both, either requiring an exact match of predicted spans or allowing for partial matches, or evaluating on the token level. A range of different metrics were implemented for the ^*SEM 2012 shared task on negation, see Morante and Blanco (Reference Morante and Blanco2012) for an overview. In this section, we report $\text{F}_1$ for cues and scopes separately, both on the token level. The latter corresponds to the measure called ‘scope tokens’ in Morante and Blanco (Reference Morante and Blanco2012) and Fancellu et al. (Reference Fancellu, Lopez and Webber2016).

Table 8 compares our best performing MTL sentiment models with a setup where the negation component of the architecture – corresponding to only the first-layer BiLSTM+CRF as shown in Figure 2 – is trained as a single-task model for negation prediction. The single-task negation model achieves a token-level scope $\text{F}_1$ score of 89.23 on the SFU data and 75.38 on the CD data, while the MTL model reaches 74.69 and 63.81, respectively. As we are optimizing the MTL models for sentiment, the single-task models achieve much better token-level scope $\text{F}_1$ scores (14.5 ppt. on the SFU data, 11.6 on CD). For comparison, the best performing system (Read et al. Reference Read, Velldal, Øvrelid and Oepen2012) on CD with respect to the same metric in the ^*SEM 2012 shared task (Morante and Blanco Reference Morante and Blanco2012) achieved an $\text{F}_1$ of 85.26.

Table 8.

Token-level $\text{F}_1$ for the SFU and ConanDoyle-neg (CD) negation tasks. The single-task negation models outperform the multi-task (MTL) models. Note that the MTL models are not tuned to optimize negation, this being the auxiliary task

An analysis of the common errors shows that neither the STL nor MTL models generalize well to morphological negation cues, for example “unlikely”, that have not been seen in training. This is not surprising, given that neither model has access to subtoken information. Of course, this also affects the scopes, as the models rely on predicted cues. Additionally, the MTL model has difficulty in identifying long scopes.

Given that the lower BiLSTM(+CRF) component does not achieve strong results for the auxiliary negation task when trained in the multi-task setup, it is logical to ask if better sentiment predictions can be obtained by starting from a better performing negation model. To test this, we explore a transfer learning approach where we pre-train the negation component with a single-task negation objective as described above.

In contrast to the MTL setup, with transfer learning, we first optimize the negation parameters and afterward use these parameters to initialize the lower BiLSTM layer of the sentiment model (cf. Figure 2). These pre-trained parameters are then further fine-tuned when train the overall model on sentiment data, but using a reduced learning rate. Note that this continued training is no longer MTL, however, as the entire network is only supervised by the sentiment task.

Table 9 shows that while transfer learning based on initializing the sentiment model with a pre-trained negation model does show improvements over the STL (1 ppt. on binary and 0.7 on fine-grained), it performs worse than MTL. Counterintuitively, having a better negation detection model, in terms of performance on the negation data sets, does not lead to better results on the sentiment main task.

We also consider that the poor performance of the transfer learning approach may be due to overfitting to the training data, which only contains negated examples. However, further experiments training with balanced data (50% negated and 50% non-negated) give poorer performance overall (85.3 $\text{F}_1$ for both STL and MTL scope-level), indicating that this is not the root of the problem.

Table 9.

Mean accuracy and standard deviation of single-task, multi-task, and transfer-learning models. Bold results show the best performing model on each dataset.

6.7 Comparing negation detection to other common auxiliary tasks

In MTL for natural language processing, it is common to employ a number of auxiliary tasks, which range from simple tasks, (predicting word frequency), to morphosyntactic tasks (chunking, dependency relation classification), to semantic tasks (semantic frame detection, super-sense tagging). In this section, we compare common auxiliary tasks and their effect on sentiment analysis. Specifically, we train the MTL model from Section 4 on three additional auxiliary tasks: POS tagging, multi-word detection (MWE) (identifying multi-word expressions, i.e. by the way, cope with), and super-sense tagging (SEM) (assigning course-grained semantic types to verbs and nouns).

The data for the auxiliary tasks come from the STREUSLE data set (Schneider and Smith Reference Schneider and Smith2015), which contains sentences from the Review section of the English Web Treebank (Bies et al. Reference Bies, Mott, Warner and Kulick2012), which have been enriched with multi-word and super-sense annotations. Table 10 shows the statistics of the data sets, as well as the entropy and kurtosis of the labels. Here, entropy indicates the amount of uncertainty in the label distribution, while kurtosis indicates the skewness. These measures have been shown to correlate well with the usefulness of auxiliary tasks in previous work (Martínez Alonso and Plank Reference Martínez Alonso and Plank2017; Bingel and Søgaard Reference Bingel and Søgaard2017).

Table 10.

Train and test splits available for each auxiliary task, as well as label entropy and kurtosis

The results are shown in Table 11. POS tagging and super-sense tagging perform worse than the baseline single-task model, while multi-word detection and negation detection show improvements. The MTL negation model, however, is still the best performing model, which demonstrates the importance of negation on sentiment classification. The fact that multi-word detection is helpful may correlate with the importance of multi-word idioms in expressions of sentiment (Williams et al. Reference Williams, Bannister, Arribas-Ayllon, Preece and Spasić2015; Liu et al. Reference Liu, Qian, Qiu and Huang2017; Jochim et al. Reference Jochim, Bonin, Bar-Haim and Slonim2018; Barnes et al. Reference Barnes, Øvrelid and Velldal2019a).

Table 11.

Accuracy on SST-fine with POS tagging, multi-word detection (MWE), and super-sense tagging (SEM) as auxiliary tasks. Bold results show the best performing auxiliary task.

Both the SFU and CD data sets not only have low label kurtosis but also have relatively low label entropy. This partially aligns with previous research (Martínez Alonso and Plank Reference Martínez Alonso and Plank2017; Bingel and Søgaard Reference Bingel and Søgaard2017), which suggests that for an auxiliary task to improve the main task, the entropy of the labels should be high (implying the task should not be trivial to learn), and the kurtosis should be low (the labels should not have an overly long-tailed distribution). The fact that the multi-word task is more helpful, however, seems to indicate that the appropriateness of the auxiliary task for the main task is more important than the specific data set properties.