1 Introduction
Most of the text found in newspapers can be clearly assigned to either of the classes ‘objective’ or ‘opinion’ (notwithstanding the generally problematic nature of the term ‘objective’). Traditionally, newspapers are obliged to separate the fact from the opinion, as far as the political topics are concerned; in sports or culture sections, the situation is different, but we leave those aside for our purposes. While the first category of texts reports the news with relatively little bias, the second one states the opinion of a journalist on a current affair and, especially in case of main editorials, these texts are often taken to represent the stance of ‘the paper’ as a whole. Another group of opinion texts in newspapers are letters to the editor, which obviously represent the opinion of the particular reader who authored it. They always comment on prior news or prior comment, rather than providing objective news themselves. As texts from this category are only minimally subjected to the newspaper’s stylistic or informative standards, we expect them to present their subjectivity in the most straightforward manner compared to all news texts.
We are interested in the task of distinguishing news reports and opinion texts automatically.Footnote 1 This is a matter of genre classification if we apply standard definitions of genre, according to which it represents a class of texts that display a conventional structure and fulfil a common communicative purpose (e.g., Biber and Conrad Reference Biber and Conrad2009). For news, this purpose is to inform the reader about facts; for editorials, the purpose is to make the reader sympathetic towards a particular viewpoint, or to reflect on the pros and cons of a particular decision and possibly to suggest a particular position. We see letters to the editor as playing the same role, although they are written by nonprofessionals, can differ in terms of style, and display their own specific structure.
Classification of news versus opinion text has been performed for several years, both under the heading of genre classification and within the sentiment analysis community, as we will show in Section 2. The earlier approaches focused on bag-of-words models, text-statistics, and part-of-speech (PoS) features, and applied these to a relatively homogeneous corpus. Our work, in contrast, emphasises the goal of robustness against domain change, which we regard as a broader spectrum than the earlier work did. It includes the following:
-
• changes in topic (what the text is about);
-
• changes in text length (what portion of a text is analysed);
-
• changes in time (when the text was written);
-
• changes in newspaper (where the text was edited);
-
• changes in opinion genre (whether letters or editorials are analysed).
To demonstrate this robustness, we work with three different corpora from various newspaper sources. They contain texts from different time periods (although we did not specifically contrast temporal spans), and some have been annotated with topic domains, so we can test portability. Our basic approach is to train on one corpus (Wall Street Journal (WSJ)) and use the resulting models for testing also on the two other corpora.
Moreover, we test our approach not only on full texts but also on single paragraphs. The rationale for this step is twofold. We want the method to be applicable to long and to short texts, and also to portions taken out of longer documents: Many types of newspaper articles (e.g., in the ‘culture’ section) are essayistic and long, where only portions may express an opinion while others are of a descriptive or narrative type. (Besides, we envisage the system to be applicable also to non-newspaper text such as web documents or portions of them; but we do not test this in the present study.)
In terms of methodology, we argue that the described robustness relies on a set of linguistic features, which we expect to be stable across the various dimensions of change. Supplementing the customary lemma and PoS features, we experimented with a wide range of such linguistic features and will show the set that proved to be most successful.
After discussing related work from genre detection and subjectivity classification in the next section, Section 3 introduces our general approach and the feature set, targeting several English corpora. Then, Section 4 describes our experiments and presents the results. Finally, we offer some conclusions in Section 5.
2 Related work
The relevant related work for our classification task stems from two different research fields. The first one is, obviously, Automated Genre Classification (henceforth: AGC), which often has been applied to different genres of newspaper text. The second one is the more recent area of sentiment analysis, where Subjectivity Classification (henceforth: SC) is the subtask of deciding whether a given span of linguistic material (usually either a sentence or a full text) contains ‘subjective’ or ‘objective’ material. This distinction corresponds fairly well (though not completely) to that between ‘opinion’ and ‘news’. In this section, we discuss work from both areas.
2.1 Automated genre classification
In the 1990s, corpus-based computational linguistics moved from working with relatively small and homogeneous corpora to larger and heterogeneous collections of text documents. Motivated by this shift, early approaches to AGC used surface features describing various aspects of the documents in order to detect the genre automatically. In this vein, Karlgren and Cutting (Reference Karlgren and Cutting1994) ran a series of experiments on the Brown Corpus. For one thing, they employed the distribution of PoS tags as features; this will be discussed below. In addition, they used surface-based features requiring no tagging: length of the document in terms of characters and sentences, length of the sentences in terms of characters and words, and length of words; type/token ratio, frequency of long words, and the frequency of the words ‘therefore’, ‘I’, ‘me’, ‘it’, ‘that’, and ‘which’. The experiments were executed in variants with two, four, and fifteen categories. In the last one, some of the categories are relevant for our purposes: The classification of ‘press editorial’ yielded 30 per cent errors, and that of ‘press reportage’ 25 per cent.
Kessler, Nunberg and Schütze (Reference Kessler, Nunberg and Schütze1997) also used the Brown Corpus and worked with surface-based features, but these are not explained in detail in the paper. They include lexical features (latinate affixes, date expressions, etc.) and punctuation (number of question marks, exclamation marks, capitalised and hyphenated words, acronyms). Further, some derived features were computed: average word and sentence length, and type/token ratio. The authors reported the following accuracies for six categories: reportage 83 per cent, editorial 61 per cent, scitech 83 per cent, legal 20 per cent, nonfiction (= other expository writing) 47 per cent, fiction 94 per cent.
In their systematic study of genre/topic dependence in various AGC approaches, Petrenz and Webber (Reference Petrenz and Webber2011) reimplemented earlier proposals and tested them on common data sets. They showed that both of the above-mentioned surface-based approaches achieve an accuracy of around 80 per cent and are relatively robust against a change of topic in the text: the performance is only weakly reduced. As explained earlier, for our purposes, this robustness across topics, and furthermore across different newspapers, styles and authors, is a central goal. We will return to the experiments of Petrenz and Webber later, and compare our own work to theirs.
The first natural step beyond surface features is to employ PoS tagging. Karlgren and Cutting (Reference Karlgren and Cutting1994), in their earlier work, used the frequency of adverbs, nouns, present tense verbs, present participles, prepositions, and first and second person pronouns. A technically more elaborate and more recent study by Feldman et al. (Reference Feldman, Marin, Ostendorf and Gupta2009) compared naive Bayes classification of top-ranked PoS trigrams to a sliding-window approach. The window of size 5 is moved over the PoS sequence of the text, and afterwards the feature vector for the text is composed of (normalised) mean and standard variation of the windows. Additionally, a principal component analysis (PCA) is performed on the training features, and the top-ranked components are taken as the final feature vector. The study used six genres, with several hundred training documents for each: broadcast news, broadcast conversations, meetings, news wire, switchboard dialogues, and weblogs. Using a generative classifier quadratic discriminant analysis with full covariance matrices estimated by maximum likelihood, the authors obtained an accuracy of 98.42 per cent. In comparison, the baseline approach of PoS-trigrams, using a naive-Bayes classifier, yielded only 89.31 per cent. (Notice that both these figures cannot be directly compared to our task, though, since the six genres are much more diverse than ours.)
Besides length features, punctuation, and PoS tags, the final method that has been employed is to refrain from any linguistic analysis and use bag-of-words (Freund, Clarke and Toms Reference Freund, Clarke and Toms2006) or bag-of-character-n-grams (Sharoff, Wu and Markert Reference Sharoff, Wu and Markert2010) models. This has the obvious advantage of knowledge-freeness and yields very good results in the domains of the training data, but Petrenz and Webber (Reference Petrenz and Webber2011) showed in their experiments that this idea is extremely vulnerable to topic/domain shifting: The models largely learn from the content words in the training texts, and these can be very different from day to day, when the news and the opinions on them reflect the current affairs.
The same finding was reported by Finn and Kushmerick (Reference Finn and Kushmerick2003), who investigated three different types of features (bag-of-words, PoS statistics, and text statistics) across three domains. They found that bag-of-words performs best on single-topic data sets and worst in the cross-domain experiments, which indicates that subjectivity is conveyed by different words in different topic domains. PoS statistics yielded the best results in cross-domain experiments, as it allowed for a better abstraction.
2.2 Subjectivity classification
In sentiment analysis research, the vast majority of the SC literature deals with sentence-level subjectivity: The task is to identify those sentences from which opinions are to be extracted in the follow-up step. Sometimes, however, document-level classification is being addressed, and we mention three influential studies here.
As part of the early work on sentiment analysis, Wiebe et al. (Reference Wiebe, Wilson, Bruce, Bell and Martin2004) discussed a classification approach based on WSJ articles. Their main goal was to extract subjective expressions from annotated corpora, and they developed a system to classify newspaper articles into ‘opinionated’ versus ‘nonopinionated’ articles. To this end, some of the WSJ articles have been annotated for subjectivity, and some contain meta-information for assigning them to a genre (e.g., editorials, letters to the editor, reviews, viewpoints, and reports). In a first step, these categories were double-checked by two annotators. The task was to find texts which were labelled as reports but were still very similar to editorials or commentaries. While the annotators did not receive any training, they still achieved a kappa value of 0.94 and 0.95.
The classification is based on ‘potential subjective elements’ (words or phrases that are potentially ambiguous between subjective and objective uses). They were extracted from the data as follows. The authors first extracted all low-frequency words, arguing that low frequency is an indicator for subjectivity. All these words are regarded as potential subjective elements. Then, n-grams that are potential subjective elements are calculated from the annotations by taking the quotient of the number of times the n-gram occurred within a subjective context according to the annotations and how often it occurred in an objective context. If it exceeds a certain threshold, the n-gram is included in the set of potential subjective elements. Finally, adjectives and verbs that are distributionally similar are successively added if they exceed a certain threshold by means of the quotient described above.
The text classifier then employs k-nearest-neighbours classification, using the feature of the (normalised) number of potential subjective elements. According to Wiebe et al. (Reference Wiebe, Wilson, Bruce, Bell and Martin2004), this classifier achieves an accuracy of 0.939. They also mention a majority baseline for their classification of 0.915 due to the massive imbalance of their data set. As we describe in Section 3, we avoid this problem by balancing the training and testing sets. Further, note that these results were achieved on a section of the WSJ corpus, without aiming to transfer the approach to other data.
Yu and Hatzivassiloglou (Reference Yu and Hatzivassiloglou2003) used WSJ texts as well, but also avoided the high baseline problem by limiting their data set to a smaller size: 2,000 articles from the genre ‘editorial’ plus 1,695 texts from ‘letter to the editor’ versus 2,000 articles from ‘business’ and ‘news’, respectively. They reported on their results of classifying texts containing opinions (i.e. ‘mostly subjective’) against texts which are factual (i.e. ‘mostly objective’). Their naive-Bayes classifier with tokens as features (no stemming or stop word removal) achieved an F-score of 0.97. However, the authors trained and tested solely on WSJ tests taken from a particular section, and thus the same criticisms that we mentioned above in discussing the bag-of-words AGC approaches apply.
More recently, the classification of newspaper articles into the categories ‘subjective’ and ‘objective’ was tackled by Toprak and Gurevych (Reference Toprak and Gurevych2009). The train/test data consisted of 7,866/5,245 documents, where the training data contained more subjective articles (56 per cent). These were composed of the categories ‘letter to the editor’, ‘debate’, and ‘analysis’. Objective articles came from news in the sections of domestic and foreign politics, and economy.
The authors experimented with three types of features: (i) Word-based features included unigrams, bigrams, variants with surrounding tokens, as well as frequency-amended lemma features (using a tf*idf measure). (ii) Lexicon features exploited the Subjectivity Clues Lexicon (Wilson, Wiebe and Hoffmann Reference Wilson, Wiebe and Hoffmann2005), SentiWordnet (Esuli and Sebastiani Reference Esuli and Sebastiani2006), and a list of communication and mental verbs, as used for web-genre classification by Santini (Reference Santini2007).
It turned out that word class features outperform the other classes, with an accuracy of up to 0.857. Specifically, the tf*idf representation was successful. Such frequency-based representations are known to be effective for classical topic categorization tasks, and this study provides an indication that they may also help for related tasks (especially when the class distribution is skewed). Another finding was that plain unigrams beat the larger n-grams and context features.
Among the lexicon features, the best single one is the Wilson et al. lexicon (accuracy 0.806), but performance increases to 0.820 when the other two lexicons are being added. PoS features performed significantly worse than the others.
3 Approach
In this section, we describe our approach to the problem, which (in contrast to the SC work discussed above) is characterised by less emphasis on the detection of overt lexical expressions of opinion, and places more weight on general linguistic features. Another difference to earlier work is our experimenting with training and testing corpora covering different genres (news, editorials, letters to the editor) and several different newspapers in order to show our model’s robustness against domain changes. In the following, we present our training and testing corpora and feature sets.
3.1 Corpora
Our main data source is the BLIIP WSJ (1987–1989, Charniak et al. Reference Charniak, Blaheta, Ge, Hall, Hale and Johnson2000). It does not contain metadata that could be used for genre classification, but the corresponding metadata can be found in the parallel ACL/DCI-WSJ corpus (see Plank Reference Plank2011 for corpus alignment). We extracted a first pool of more than 74,000 reports but less than 2,000 opinion texts, the latter containing exclusively ‘letters to the editor’ (see Webber Reference Webber2009 for an analysis of the corpus genre distinctions). As explained in Section 1, we expect ‘letters to the editor’ to prototypically represent opinion texts in general and hence to provide an adequate learning base for the opinion part of our classifiers. For the balanced training corpus, we randomly selected 1,751 texts each for the news and the opinion genre, with at least 3 KB file size (approximately five sentences per file). The topics of the news texts are mostly politics and business. (Note that a specific topic selection is only possible for WSJ news reports due to the limited metadata available.) Since the amount of opinion texts at our disposal was so much smaller, we also allowed for shorter texts in this genre. Still, 89.95 per cent of these texts contain five sentences or more. (In contrast to some earlier work, text length is not included in our model’s feature set.) In addition, we randomly selected fifty texts from each genre for feature development. The raw texts were then parsed with the Stanford Parser within the Stanford CoreNLP tool (version 2015-12-09; de Marneffe, MacCartney and Manning Reference de Marneffe, MacCartney and Manning2006; Manning et al. Reference Manning, Surdeanu, Bauer, Finkel, Bethard and McClosky2014).
For evaluating our model, we conducted a series of experiments both within as well as outside the trained domain (to be reported in Section 4). First, we investigated the results of training our classifier on the WSJ corpus by tenfold cross-validation. Second, we used the NLTK Brown Corpus (Bird, Loper and Klein Reference Bird, Loper and Klein2009), which is subclassified into various genres. We ran our models on a balanced randomly selected set of fifty-four texts from the sections ‘news’ and ‘editorial’. For the third group of experiments, we used the New York Times Annotated Corpus of the Linguistic Data Consortium as a test corpus (NYTAC, 1987–2007, Sandhaus (Reference Sandhaus2008)). The NYTAC contains more than 1.8 million articles with extensive metadata annotation. However, annotation quality and coverage vary as some parts were automatically assigned, others were produced by human annotators. We closely followed the method described in Petrenz and Webber (Reference Petrenz and Webber2011) for extracting feature/topic combinations from the NYTAC and selected two of their three topics for our corresponding NYTAC test samples. Our first topic, ‘Armament, Defense and Military Forces’, is still closely related to the ‘politics’ topic covered by our WSJ news training genre. Our second test corpus topic, ‘Medicine and Health’, represents a clear topical change as a challenge for the classifier. For each topic, we randomly extracted 1,000 news reports as a first genre and a set of 1,000 opinion texts as the second genre (consisting of 500 editorials and 500 letters to the editor). In order to make our experiments directly comparable to the setup in Petrenz and Webber (Reference Petrenz and Webber2011), we additionally tested the two categories within the opinion genre, editorials, and letters to the editor, separately against the news texts, as will be described in the results section in detail. In two last experiments on the NYTAC, we used the same topic/genre constellations as in the first two NYTAC experiments, but evaluated the classifier on single paragraphs instead of full articles. Paragraph boundary markers are incorporated in the NYTAC xml data. Paragraph lengths vary widely throughout the NYTAC corpus. We restrict our experiments to paragraphs with a minimum length of at least fifty words. As NYTAC paragraphs are not separately annotated according to their content (that is, whether they express an opinion or report news), we must assume that the higher level whole text annotation applies here, too. However, the amount of opinion expressed in the single paragraphs of an article is likely to vary a lot.
3.2 Feature set
As opposed to the approaches described in Section 2 – which are often solely based on bag-of-words models or PoS tags – we decided to use linguistically motivated features in addition to token-based features. This is intended to increase the robustness of the classifier against changes of corpus and domain. Overall, our feature set can be divided into three major groups: (i) Lemma: counts of lemmatised types of an instance (unigrams); (ii) PoS-tag counts; and (iii) Ling: (linguistic) features describing various properties of the instance. The Lemma feature set in our implementation requires PoS tagging for discrimination purposes. We do not exclude stop words, numbers, or punctuation.
In the rest of this subsection, we describe the Ling features in detail. Most of them are derived from the PoS tags assigned by the parser in our pipeline. The structural output of a full parser is only required for one subcategory of the linguistic features (connectives, see feature Ling:Connectives below) and could possibly be omitted for obtaining a less resource-intensive approach; this will be discussed in Section 4. All linguistic features we implemented are presented in Table 1.
Comprehensive list of the linguistic features used in the classifier
For the first features, we calculate the average lengths of the sentences and tokens (Ling:SentLength, Ling:TokenLength; for scaling reasons, these values are inverted), the hypothesis being that argumentative texts like editorials will tend to feature longer sentences.
Due to the presence of rhetorical figures and other forms of argumentative moves, we expect to find more negations in opinion texts than in reports. We thus compute their normalised frequency (Ling:Negation) with the help of a list of lemmatised negation words, including negation particles (e.g., not), quantifiers (e.g., no), prepositions (e.g., without), etc. We do not include negation morphemes such as un- or dis-, but do consider the negation suffix n’t, since it applies to verbs and thus can be assumed to have a wide scope over the clause.
Similarly, we expect that opinion texts will make occasional use of exclamations and (rhetorical) questions, which should be much less frequent in reports. We thus determine the numbers of exclamation marks and question marks, compared to all sentence-ending punctuation symbols (Ling:Exclamations, Ling:Questions). Semicolon and comma features are established in the same manner (Ling:Semicolons, Ling:Commas).
In argumentative texts, we expect to find a more complex syntax and thus a more nested sentence-hierarchy. To approximate the measure of sentence complexity in a heuristic way (rather than computing elaborate measures over parse trees), we calculate the ratio of finite verbs over all tokens of the text (Ling:Complexity).
As a discourse-level feature, we consider the presence of certain classes of connectives. To this end, we use a set of connectives derived from the Penn Discourse Treebank (Prasad et al. Reference Prasad, Dinesh, Lee, Miltsakaki, Robaldo, Joshi and Webber2008), which provides, among others, sets of temporal, causal (contingency), comparative/contrastive, and expansive connectives. Specifically, we hypothesise that causal and contrastive connectives are more frequent in opinion texts, while temporal ones tend to occur in reports. In order to discriminate between certain ambiguous English connectives (e.g., as, for, since, before, etc.), we rely on the output of the Stanford dependency parser. Both the numbers of total connectives per text (Ling:Connectives) and those of the four subcategories are calculated.
A further discourse-level hypothesis is that the presence of direct or indirect speech plays a role for the classification task. When appearing within a quote, some particular features are therefore ignored for fear of introducing noise. However, we use these features for classification if they appear outside of quotes. This applies to the following grammatical and lexical features: first and second person pronouns (Ling:1stPerson, Ling:2ndPerson, Ling:1st/2ndPerson), tense (past/present, relative to the total of verb occurrences in the text, Ling:Past, Ling:Present), sentiment value of the tokens (see below), and modal verbs. Also, we calculate the frequency of quotes (Ling:Citations) and their average length (in number of words, Ling:CitationLength) per sentence, assuming that reports tend to contain more and longer quotes than editorials, which instead are expected to focus more on the author’s own opinion. We also seek to capture the speech contexts indirectly by examining the number of communication verbs outside quotes (verbs of saying, Ling:VoS, our list of ninety verbs, e.g., announce, claim, declare, explain, insist, mention, etc.), compared to all verb occurrences. For tracking indirect speech contexts directly, more structural information from the parser is needed so we keep this as an option for improvement at a later stage. Similarly to first and second person pronouns, modal verbs (Ling:Modals, e.g., can, must, may, etc., with the exception of will, which we put in a separate feature) are expected to be a better indicator for an opinion text rather than for a report, if measured outside quotes. In contrast, we expect digits to be more frequent in the factual contexts of reports (Ling:Digits).
Finally, the degree of sentiment is calculated using the MPQA Subjectivity Clues Lexicon (Wilson et al. Reference Wilson, Wiebe and Hoffmann2005). Since we intend to calculate the total sentiment content of a given text, while the MPQA Subjectivity Clues Lexicon only features a categorical distinction in ‘weak-’ and ‘strong-subjective’ (negative or positive prior polarity), we map these categories to real number values (‘weak-subj’: 0.1, ‘strong-subj’: 1.0). The average strength of the polarity of a certain text is thus obtained by adding all sentiment values of the tokens (except quotes), and dividing by the total number of tokens in a text (Ling:Sentiment). This is conducted both for all tokens in a text as well as separately for adjectives only (Ling:SentimentAdj). These features are motivated by the assumption that, in order to communicate judgements more clearly, opinion texts employ a less neutral language and thereby increase the usage of words with a higher prior-polarity.
4 Results and discussion
We report here on several machine-learning experiments conducted on the WSJ corpus and several test corpora as described in Section 3.1. All features described in Section 3.2 have been used in various combinations and different classifiers using the WEKA machine learning toolkit (Hall et al. Reference Hall, Frank, Holmes, Pfahringer, Reutemann and Witten2009). We employ different machine-learning algorithms in order to detect features that work across all classifiers and to employ the overall best classifier in our project. The first classifier is a support-vector machine (SVM) using sequential minimal optimisation (henceforth: SMO; Platt Reference Platt1998). SMO is used with two different kernels: a polynomial kernel K(x, y) = ⟨x, y⟩ e with e = 2, and a linear kernel. Additionally, we test ADTree (Moore and Lee Reference Moore and Lee1998), a Bayesian Network (Pearl Reference Pearl1988), logistic regression (see, for example, Hosmer, Lemeshow and Sturdivant (Reference Hosmer, Lemeshow and Sturdivant2013) for an extensive exploration), and a multilayer perceptron (see, for example, Lippmann (Reference Lippmann1987) for an overview).
The first experiment is on the WSJ corpus alone and also investigates the most effective features. The second experiment uses the classifiers trained on the WSJ corpus and evaluates them on a balanced subset of the NLTK Brown corpus, while the third group of experiments performs classification on various test corpus subsets of the NYTAC.
4.1 Intracorpus evaluation and features
Since our sample of the WSJ corpus is balanced for reports and opinion texts, a baseline system guessing one class at a time achieves an F1-score of 0.5. As can be seen in Table 2, the purely lemma-based classifier achieves a high F1-score of 0.962 after tenfold cross-validation on the same corpus and can thus compete with the classifier of Yu and Hatzivassiloglou (Reference Yu and Hatzivassiloglou2003). This result can be insignificantly outperformed by combining all three groups of feature sets (Ling+PoS+Lemma), resulting in an F1-score of 0.965. However, when lemmas are removed from the complete feature set, the F1-score drops marginally by 0.9 per cent. A classifier trained exclusively on the linguistic features performs slightly worse (again, insignificantly in a χ2-test), but it still clearly surpasses a classifier trained solely on PoS tags by almost 5 per cent (χ2 = 86.63, p < .0001). The best results are achieved by the SMO classifiers.
Results of tenfold cross-validation on the WSJ corpus for various classifiers
F1-score is the average of the F1-scores of the two classes. The polynomial SVMs were trained with a polynomial kernel with e = 2.
Table 3 shows the highest weighted features after tenfold cross-validation of training the SVM classifier with a linear kernel on the WSJ training corpus. Interestingly, some of the linguistic features even outperform the corresponding PoS tag and lemma features. For example, the top-ranking opinion-text feature is Ling:1st/2ndPerson, followed by the third-ranked Ling:1stPerson, whereas the corresponding lemmas follow on lower positions. This is due to our awareness of speech contexts, which only lets the classifier consider these features if they appear outside quotes, thus improving the performance of the features. The highest ranking lemma features for opinion texts include nouns like ‘article’ and ‘editorial’, which specifically match the composition of the opinion section in our training corpus (that is, letters to the editor) and possibly imply overfitting. The same reasoning applies to the opening and closing brackets (POS:-RRB-/POS:-LRB- and Word:(_-LRB-/Word:)_-RRB-), as many of the WSJ letters to the editor refer to the articles they comment on using bracketed insertions (e.g., ‘Thank you for the very perceptive article ‘Keeping faith’ (July 14) [. . .]’). Accordingly, the possessive pronoun Word:you_PRP$ (most often used for referencing the respective article) can be found on a top-level position. The subordinating conjunction ‘that’, on the contrary, is a lemma feature which we expect to find strongly represented in most argumentative texts, not only in letters. For reports, the most effective lemma feature is ‘say’ in past (Word:say_VBD) as well as in present tense (Word:say_VBZ). Correspondingly, the linguistic feature for typifying the verbs of saying ratio (Ling:VoS) is among the best performing features.
Top features for an SVM with a linear kernel and all features (Tokens, PoS Tags, Ling)
Negative values indicate an orientation toward opinionated and positive values toward news texts. PoS tags are labelled according to the Penn Treebank tag set.
Table 4 displays the feature weights which we attained by training the linear SVM using Ling features only. As these were designed in order to catch the particular linguistic properties of opinion texts and reports, we can now evaluate our assumptions. As expected, prominent features of opinion texts are the first and second personal pronouns (outside quotes), the sentiment calculated over the whole text (except quotes), negations, questions and exclamations, and the use of present tense. Remarkably, the sentiment value of the text is not of singular importance to the classification process, but it is merely one, and not the best, of several comparably effective Ling features (frequencies of first/second person pronouns, negations, citations, and verbs of saying).
Feature weights for a SVM with a linear kernel and only Ling features
Negative values indicate an orientation toward opinionated and positive values toward news texts. Features marked * are inverted.
In contrast to our expectations, higher frequencies of quotes (Ling:Citations) and digits are effective opinion-text features instead of report features. We consider this to be an effect of our training set’s composition of ‘letters to the editor’, which often explicitly comment on a particular article of a particular date (thereby referring to the article with quotes). While a higher overall frequency of connectives grouped as a whole (irrespective of the categories) is weakly associated with opinion texts as expected, all four distinct groups of connectives correspond to reports. This may indicate that many reports we investigate tend to focus mainly on one aspect of the news, e.g. temporal, causal, etc. Hence, those reports may show an increased rate predominantly of connectives related with that aspect, while the overall frequency of connectives is not increased.
As we supposed, the higher proportions of verbs of saying (Ling:VoS) and commas, as well as a shorter average sentence length are effective features of reports (sentence length and token length features were inverted for scaling reasons). However, the by far most potent report feature is the negation suffix n’t, quite contrary to our expectations. Manual investigation of the training corpus confirms that this type of negation is mostly used within citations or quotes. As citations in reports usually are introduced by verbs of saying and these rank high among the report features, the negation suffix is likely to receive a correspondingly high feature weight and orientation. A second feature with an unforeseen (but strong) orientation toward reports is our measurement of sentence complexity (the ratio of finite verbs over all tokens). Obviously, an increased sentence complexity does not match the shorter average sentence length of reports. A further reason for the orientation of the feature may be the composition of our training corpus, containing letters for the opinion genre. Arguably, these texts differ stylistically from argumentative texts by professional writers.
Modal verbs and past tense are less potent features than we assumed, but they each correspond to the genre we expected. As opposed to the high significance of the overall sentiment value for opinion texts, restricting the sentiment value to adjectives in the text yields a report feature (even if only a weak one). Further analysis of the learned feature weights for PoS tags indicates that adjectives, in contrast, serve as reliable features of opinion texts. The feature set containing only PoS tags mirrors the suspected overfitting performed by the Lemma features in matters of, for example, brackets and possessive pronouns for opinion features. It is worth mentioning that both past and present tense verbs here represent features of the same category (news reports), as a contrast to the distribution yielded by our Ling features (considering speech contexts).
4.2 Performance of the models on the test corpora
The models from Section 4.1 are now evaluated on subsets of the NLTK Brown Corpus and the NYTAC (as described in Section 3.1). In the first test on a balanced sample of the NLTK Brown Corpus, the best performing model is no longer the one built on the combined feature set (Ling+PoS+Lemma), as can be seen in Table 5. On the contrary, the combined model now achieves one of the lowest F1-scores of all. A model built exclusively on Lemma features only performs as well as the medium-ranked Ling models and is otherwise inferior to the best Ling model as well as the PoS tag-based model. The decreased performance of the lemma-based models (compared to their performance in intracorpus evaluation) is mainly due to a drop of recall within the opinion class. Obviously, the learned lemma distributions from our WSJ opinion class (letters to the editor) do not match the lemma distributions in the Brown opinion class (consisting of editorials). Thus, a classifier built on lemmas does not show robustness across genres and corpora, here. In contrast, relying solely on our Ling features results in a top-ranking F1-score of 0.963, if a Bayes Network classifier is used. However, PoS tags outperform the Ling features by 1.9 per cent if classification is done with an SMO classifier. Due to the small size of the Brown Corpus, none of these differences are significant.
Results of the models trained on the WSJ corpus and evaluated on the balanced NLTK Brown Corpus test set
F1-score is the average of the F1-scores of the two classes.
The second test corpus comprises news reports, editorials, and letters to the editor from the NYTAC on the topic ‘Armament, Defense and Military Forces’ (defense). In a first test, we evaluated the classifiers on a balanced sample of 1,000 reports and 1,000 opinion texts (that is, a more general set containing both 500 editorials and 500 letters to the editor). In the second and third test, 500 of the reports were separately classified against the editorials and the letters to the editor as two distinct genres. This experimental setup was then transferred to single paragraphs of news reports, editorials, and letters to the editor on the same topic (minimum length fifty words). In total, this resulted in six experiments on one topic. In all cases, the linear SMO classifier was used, usually yielding the best results at best performance rates in our experiments. The results are displayed in Table 6 for full-text classification and Table 8 for paragraph-wise genre classification. Analogously, we conducted six similar experiments on NYTAC texts on the topic ‘Medicine and Health’ (medicine) for the same genre combinations. Results are shown in Tables 7 and 9.
Results of the models trained on the WSJ corpus and evaluated on balanced test sets of full NYTAC texts on the ‘defense’ topic
Classified with linear SMO. F1-score is the average of the F1-scores of the two classes.
Results of the models trained on the WSJ corpus and evaluated on balanced test sets of full NYTAC texts on the ‘medicine’ topic
Classified with linear SMO. F1-score is the average of the F1-scores of the two classes.
Results of the models trained on the WSJ corpus and evaluated on balanced test sets of NYTAC paragraphs on the ‘defense’ topic
Classified with linear SMO. F1-score is the average of the F1-scores of the two classes.
Results of the models trained on the WSJ corpus and evaluated on balanced test sets of NYTAC paragraphs on the ‘medicine’ topic
Classified with linear SMO. F1-score is the average of the F1-scores of the two classes.
The model that was most successful in training – combining Lemma, PoS, and Ling features – has dropped to an F1-score of 0.851 when applied to the classification of opinion texts (as a broader genre) against news texts in the NYTAC ‘defense’ test corpus (see Table 6). The same effect occurs with the lemma-based model and, similarly but less strongly, with the PoS tag-based model. This drop reflects the lack of robustness of the classifiers trained on the lemmas of a specific corpus. The Ling features, on the other hand, have been modelled with the goal of separating different text types and thus are less vulnerable to domain changes. Consequently, the best results for classifying news reports against opinion texts in general (being as heterogeneous as letters and editorials) are achieved by the model using our Ling features alone or the combination Ling+PoS (differences insignificant), when tested on full texts on ‘defense’ (being a similar topic as seen in training). Including both Lemma and PoS features results in a significant decrease in F1-score by almost 5 per cent (χ2 = 20.88, p < .0001).
Splitting our broader NYTAC opinion genre into editorials and letters to the editor (the genres also used by Petrenz and Webber Reference Petrenz and Webber2011) and classifying these distinct subsets separately against news reports reveals overall better performance of all models on letters than on editorials as opinion texts (see the middle column in Table 6). Since the models were initially trained on WSJ texts containing only letters to the editor in the opinion genre, this training effect meets our expectations. However, the extent of the effect varies throughout the different classification models. Of all the models, the one based only on Ling features shows the smallest, insignificant increase when tested separately on the ‘letters’ versus ‘news’ subset. The model based only on PoS tag features achieves an increase in F1-score of almost 2 per cent. Most impressively, the models containing Lemma features each gain more than 10 per cent in F1-score, showing a strong learning effect for the letters genre (χ2 = 63.7, p < .0001).
As a consequence, when tested on the separate editorials subset against news reports (the middle column in Table 6) the lemma-based classifier is clearly outperformed by the model based only on our Ling features by 15 per cent (χ2 = 59.85, p < .0001). Combining PoS and Ling features leads to a decrease in F1-score by almost 4 per cent (χ2 = 5.35, p < .05). The model based only on PoS tags is still superior to both models including lemmas, Ling+PoS+Lemma and Lemma only, with the latter performing least successful on the NYTAC ‘defense’ topic (lead of PoS is significant with χ2 = 5.1, p < .05). A detailed analysis shows that the decrease in F1-score of the lemma-based models is mainly due to a massive drop in recall of the opinion genre (letters to the editor: precision 0.965, recall 0.928, editorials: precision 0.938, recall 0.516 for the Lemma classifier). This, again, indicates that this type of classifiers relies heavily on some highly distinctive ‘letters’ features (such as, for example, the bracketed reference to a newspaper article, month names, or the possessive pronoun) which lose their effectiveness when transferred to other opinion genres. The performance hierarchy among the different models on the ‘editorials’ versus ‘news’ NYTAC corpus is equivalent to the results on the NLTK Brown Corpus as tested above. For both test sets, a purely Ling-based classifier achieves the highest F1-score, followed by the classifiers including PoS tags but no lemmas. The last group incorporates lemma features and shows the least competitive classification results for a genre not previously included in the training data.
A change in topic leads to an F1-score decreased by almost 3 per cent for the Ling classifier (χ2 = 4.87, p < .05) when evaluated on news reports against letters to the editor. Obviously, the change in topic even affects the Ling features (see Table 7, middle column). Interestingly, the results of the Lemma- and PoS-based classifiers remain robust even throughout changes in topic (differences across topics being not significant). The Lemma classifier is (insignificantly) outperformed by the classifier combining all three feature sets, Ling+PoS+Lemma, across both topics. We consider the robustness of the lemma features to be predominantly caused by the unspecific topic of the opinion part of our training corpus. Still, since recall for the news class is not significantly altered across topics, either, we hypothesise that these classifiers are generally less vulnerable to topic transfers than to style-related variation. Accordingly, the training effect and hence the decrease in Lemma and PoS performance when transferred to the editorials genre is reproduced on the ‘medicine’ topic (the right-hand column in Table 7). Thus, the Ling classifier remains the best performing classifier after a change of both genre and topic, with a significant lead of more than 6 per cent (χ2 = 10.8, p < .01) ahead of the other classifiers.
In the similar experiments by Petrenz and Webber (Reference Petrenz and Webber2011), only test set 2 represents a similar change of topic and genre for the classifiers and can thus be compared to our results of classifying editorials (for the opinion class) against news on the ‘medicine’ topic. (This comparison, however, merely reveals a trend since Petrenz and Webber (Reference Petrenz and Webber2011) only provide the averaged accuracy of all genre/topic combinations in their test set.) Our classifier based on only Ling features surpasses the best scoring classifier in Petrenz and Webber (Reference Petrenz and Webber2011), KNS (that is, following the approach of Kessler et al. Reference Kessler, Nunberg and Schütze1997), by almost 2 per cent. Moreover, comparing the intracorpus evaluation (that is, test set 1) in Petrenz and Webber (Reference Petrenz and Webber2011) to our results after cross-validation, even our model based only on Ling features clearly exceeds the KNSPOS classifier (the best performing from the group aimed at domain robustness) by more than 5 per cent.
The same series of tests was performed on single paragraphs of only fifty words or more (Tables 8 and 9). Of course, due to the sparsity of the data contained in the paragraphs, all of the classification results are significantly lower. Whereas the Lemma classifiers achieved the best results on NYTAC full texts when no change of genre was involved, they lose their predominance even here when evaluated on small paragraphs. Instead, the combined feature set of Ling+PoS+Lemma achieves the best results across both the ‘defense’ and the ‘medicine’ topic. The differences on the separate letters/news corpus are, however, not significant. When tested after genre transfer (on editorials for the opinion class), again, the Ling classifiers perform better or at least as good as the combined one (significant lead of Ling before Lemma or PoS on ‘defense’, χ2 = 8.6, p < .01). On the more general test set (containing both letters to the editor and editorials), the Ling+PoS+Lemma classifier consequently achieves the best results (significant lead before Lemma classifier on ‘defense’ topic, χ2 = 7.01, p < .01). The classifier relying only on PoS tag distributions achieves the least competitive results in all tests on paragraphs. This strongly supports the findings by Petrenz and Webber (Reference Petrenz and Webber2011) on PoS tags, indicating that the complete set of all PoS tags does not yield good classification results. In summary, the Ling features are the essential component of a paragraph-wise genre classifier that is robust even on opinion genres not previously seen in training. Because of the sparsity of paragraphs, Lemma and PoS features may help to stabilise the performance.
Finally, we investigate the relevance of the four subcategories of connectives (Ling:ContrastiveConn, Ling:ExpansiveConn, Ling:CausalConn, Ling:TemporalConn). A linear SMO classifier that omits these four Ling features shows similarly good results after training (tenfold cross-validation, F1-score: 0.947) as well as when tested on the Brown corpus (F1-score: 0.926), but achieves slightly lower scores on all tests performed on the NYTAC test corpora. However, the differences are not statistically significant. Therefore, ignoring the PDTB categories of the connectives may be a similarly robust option for future applications.
5 Conclusions
We described a classification system for separating opinion text from news text in newspapers. Our central goal is to build a classifier that is robust against changing
-
• topics being discussed,
-
• text length,
-
• time of origin of articles,
-
• the newspaper (source of articles),
-
• specific subgenres of opinion texts.
We varied these parameters accordingly in our corpora. Our main result is that models using linguistic features clearly outperform standard bag-of-lemma approaches, when applied to reports and opinion texts which differ stylistically from the ones learnt in training. Thus, this type of classifier most effectively learns patterns of subjectivity and objectivity found in only a restricted genre and succeeds in transferring them to other genres. However, models considering only lemma distributions remain impressively robust in our experiments when applied exactly to the genres they were trained on. This effect even holds throughout changes in topics. Classifiers based solely on PoS tags or lemmas perform especially unsuccessful when applied to shorter texts. Instead, on short texts or paragraphs the best results are obtained using a combination of several types of features, including our linguistic features as an essential component.
We investigated the impact of the different features and found that most of our assumptions (for selecting them) were borne out in the experiments, with a few interesting exceptions. The most predictive features include presence of sentiment, negation, quotation, use of communication verbs and of personal pronouns. For the purpose of enhancing accuracy, we considered speech contexts (direct speech) and ignored some features if they appeared inside a citation. Connectives can also be useful, but they need to be disambiguated against nonconnective uses. This is the only step where we used a syntax parser in our approach; if one chooses to omit connective disambiguation, this knowledge-lean approach can relatively easily carry over also to less-resourced languages (but PoS tagging and some lexical information on sentiment and verb class is required).
In order to transfer our approach to other tasks, two basic requirements must be met. First, preclassified data is necessary for training the classifier. Second, the data should consist of two distinct classes, otherwise it must be mapped to a clear dichotomy. So far, we have not evaluated our approach in multiclass classification experiments.
In future work, we plan to further improve domain robustness of the linguistic model and reproduce the results in another language. Preliminary results show that most features can easily be transferred to, for example, German, yielding a similarly robust genre classifier.
Acknowledgements
Most of the research reported in this paper was done in the project e-Identity, which was funded (2012–2015) by the German Federal Ministry of Education and Research (BMBF) under grant agreement 01UG1234.
About the Authors
Katarina Ragna Krüger: Studied comparative literature at the University of Göttingen from 2003 to 2008 and, with a full student fellowship, at Åbo Akademi, Finland, from 2005 to 2006. Member of the Göttingen Work Group for Literary Theory and editorial assistant at the Journal of Literary Theory (JLT) from 2006 to 2008. Has been studying computational linguistics at Potsdam University since 2010.
Anna Lukowiak: Studied computational linguistics at Potsdam University.
Jonathan Sonntag: Studied computational linguistics at Potsdam University from 2006 to 2012. Completed internship at the SkyGrid Inc (Sunnyvale) in 2008. Student assistant at the University of Potsdam from 2007 to 2008 and 2008 to 2010. Further, student assistant at Wikimedia Germany in 2011. Since 2012, J. Sonntag has been a researcher and PhD student in the e-Identity project at Potsdam University.
Saskia Warzecha: Studied computational linguistics at Potsdam University from 2009–2013. Student assistant 2010–2013 and 2014–2015. Internship at Wikimedia Germany, 2013. Data analyst at the Amazon Development Center Germany until December 2016. Since then, language architect at Retresco, Berlin.
Manfred Stede: Studied computer science and linguistics at TU Berlin from 1985 to 1989. M.Sc. in computer science, Purdue Univ., Indiana, 1990. PhD student at the Univ. of Toronto Computer Science, 1990–1996. Research assistant in the Verbmobil MT project, TU Berlin branch, 1995–2000. Internal project leader of the DFG project KIT-Marker at TU Berlin (1998–1999). Researcher at Interprice GmbH, Berlin, 2000–2001. Has been a professor of applied computational linguistics at Potsdam University since 2001.
