3.1 Relations and discourse marker selection

As in Sporleder and Lascarides (Reference Sporleder and Lascarides2005), we chose a subset of the rhetorical relations from sdrt's inventory of relations (Asher and Lascarides, Reference Asher and Lascarides2003): contrast, result, explanation, summary and continuation. We selected these relations on the basis that (i) for each of them there are discourse markers which unambiguously signal them, and (ii) these relations also frequently occur without a discourse marker, making it beneficial to be able to determine them automatically if no cue phrase is present. An example of a relation that lacks this second property is condition, which always requires a discourse marker (e.g., if . . . then or suppose that . . .).

This set of five relations roughly overlaps with, but is not identical to, the ones used in Marcu and Echihabi (Reference Marcu and Echihabi2002) – we say “roughly” because we use sdrt relations rather than rst ones, and so the semantics of individual relations do not exactly match. We did not use exactly the same set of relations as Marcu and Echihabi (Reference Marcu and Echihabi2002) partly because we were interested to see how well their models performed on different classes, but also because some of the relations they investigated, such as condition, always require a discourse marker. As we just mentioned, we are interested in relations which often occur without such phrases, since we want to investigate whether models which train on marked examples can identify rhetorical relations in unmarked examples: that is, examples that do not contain an unambiguous discourse marker.

sdrt relations are assigned a dynamic, truth conditional semantics and therefore tend to be less fine-grained than those used in Rhetorical Structure Theory (rst) (Mann and Thompson, Reference Mann and Thompson1987). Let R(a,b) denote the fact that a relation R connects two spans a and b (or, more accurately for sdrt, a and b would be labels, labelling the content of two spans). Asher and Lascarides (Reference Asher and Lascarides2003) offer a detailed, dynamic semantic truth definition for the five relations that we will model. Roughly speaking, these semantic interpretations amount to the following. First, for each of the five relations we investigate, it holds that R(a,b) is true only if the contents of a and b are true too. In addition, contrast(a,b) entails that a and b have parallel syntactic structures that induce contrasting themes, result(a,b) entails that a causes b, summary(a,b) entails that a and b are semantically equivalent (note that it does not entail that the text span a is larger than b), continuation(a,b) means that a and b have a contingent, common topic and explanation(a,b) means that b is an answer to the question why a?, as defined by the semantics of why-questions in Bromberger (Reference Bromberger and Butler1962) and Achinstein (Reference Achinstein1980).

To create the mappings from discourse markers to the sdrt relations they signal, and in particular to identify unambiguous discourse markers, we undertook an extensive corpus study, using 10 randomly selected examples for each of 300 discourse markers listed in Oates (Reference Oates2000) (i.e., around 3,000 examples in all), as well as linguistic introspection given sdrt's dynamic semantic interpretation. The differences between the relations in sdrt vs. rst mean that some discourse markers which are ambiguous with respect to rst's taxonomy of rhetorical relations are unambiguous with respect to sdrt's. For example, in other words can signal either summary or restatement in rst but sdrt does not not distinguish these relations since the length of the related spans is irrelevant to sdrt's semantics. So in sdrtin other words signals summary only, with the rough truth conditions defined above. Similarly, sdrt does not distinguish explanation and evidence, and therefore, while because is ambiguous in rst, it is unambiguous in sdrt: if the subordinate clause is after the main clause (i.e,. the sentence is of the form A because B), then because unambiguously signals explanation; and if the subordinate clause is before the main clause (i.e., Because A, B), then because unambiguously signals result, according to sdrt's definitions. sdrt also does not distinguish contrast, antithesis and concession, making but unambiguous. From the list of discourse markers that we studied, we identified those that unambiguously signal one of the five rhetorical relations that we aim to model.

Sentences (6) to (10) below show one automatically extracted example for each relation (throughout this paper the discourse markers which were used for the extraction of the example and then removed before training are shown in bold face; spans, where relevant, are indicated by square brackets). A list of all 55 unambiguous discourse markers that we used for creating training data, together with some corresponding training examples, is given in Appendix A (Tables 9 to 13).

1. (6) [We can't win] [but we must keep trying.]

   (Contrast)

2. (7) [The ability to operate at these temperatures is advantageous,] [because the devices need less thermal insulation.]

   (explanation)

3. (8) [By the early eighteenth century in Scotland, the bulk of crops were housed in ricks,] [the barns were consequently small.]

   (result)

4. (9) [The starfish is an ancient inhabitant of tropical oceans.] [In other words, the reef grew up in the presence of the starfish.]

   (summary)

5. (10) [First, only a handful of people have spent more than a few weeks in space.] [Secondly, it has been impractical or impossible to gather data beyond some blood and tissue samples.]

   (continuation)

3.2 Automatically extracted data

We used three corpora to extract training data: the British National Corpus (bnc, 100 million words), and two corpora from the news domain – the North American News Text Corpus (350 million words) and the English Gigaword Corpus (1.7 billion words).Footnote ³ We took care to remove duplicate texts, e.g., newspaper articles that occur in both the North American News Texts and the Gigaword Corpus were included only once.Footnote ⁴ We also removed all Wall Street Journal articles contained in the rst Discourse Treebank (Carlson et al., Reference Carlson, Marcu and Okurowski2002) as we used this treebank to obtain the manually labelled data (see Section 3.3). Since we were mainly interested in written texts, we also excluded all bnc files which are transcripts of speech. For texts which were not annotated with sentence boundaries (all from the news domain), we used a publicly available sentence splitter (Reynar and Ratnaparkhi, Reference Reynar and Ratnaparkhi1997), which was pre-trained on newstexts, to automatically insert sentence boundaries.

Like Marcu and Echihabi (Reference Marcu and Echihabi2002), we extracted both intra- and inter-sentential examples (see examples (6) to (10) above). Extracting training examples comprises two steps: (i) identifying potential examples based on the presence of discourse markers and (ii) determining the span boundaries. To identify examples of our five relations, we manually wrote extraction patterns based on the 55 unambiguous discourse markers. In general, we aimed for high precision and extracted conservatively, using the linguistic context to weed out false positives wherever possible. For example, we used the phrase in short as one of our cues to identify the summary relation, but we only extracted examples in which in short occurred at the beginning of a sentence or after a punctuation mark, such as a semi-colon or a dash, as in example (11a). If in short occurs in other positions, as in example (11b), it frequently does not function as a discourse marker.

1. (11)

   1. a. Of highest priority is driving the consistent transformation of South Africa into a non-racial, non-sexist society – in short, the molding of a united nation.

   2. b. Flu vaccines are currently in short supply.

   3. c. In short order I was to fly with ‘Deemy’ on Friday morning.

However, even the restriction to sentence or clause initial occurrences of in short does not avoid false positives entirely. For example, in (11c), the phrase in short occurs sentence initially but it is not a discourse connective. Rather, it is part of the larger prepositional phrase in short order. One could avoid these examples by extracting only those sentences in which in short is followed by a comma. However, we found that this is overly restrictive as commas are frequently omitted after discourse markers. Another possibility would be to use a parser, as we have done in previous work (Sporleder and Lascarides, Reference Sporleder and Lascarides2005). Relying on a parser is problematic though, as it can result in a significant reduction in training data, because there is always a certain percentage of sentences that cannot be parsed. We found that this lack of coverage greatly outweighs any gains obtained through better filtering of false positives, as a small proportion of false positives in the training data does not do much harm to the performance of the models whereas a significant reduction in training data due to low coverage of the parser can cause noticeable damage to the accuracy of the models. Consequently, we decided not to use a parser and instead relied on our surface-based extraction rules, even if this means that the extracted data will contain a small number of false positives. (Details about the accuracy of our extraction methods are given shortly.)

The second extraction step involves determining the spans. The task consists of (i) determining the boundary of the span containing the discourse marker and (ii) identifying the second span involved in the relation signalled by the discourse marker.

A sentence can contain several spans. For example, the sentence in example (12) consists of four elementary discourse units (edus) that can function as the spans of rhetorical relations (see the discourse tree in Figure 2): the two rightmost edus (3 and 4) are related by continuation,Footnote ⁵ the resulting, larger span is then related to the second edu by explanation (as signalled by the discourse marker because) and finally the result is related to the first edu by contrast. Ideally an extraction method for the explanation relation in (12) should return edu 2 (but it would be difficult to carry out) as the left span and edus 3 to 4 (because that period has passed into history and little evidence has survived) as the right span.Footnote ⁶

1. (12) Such a study might be instructive, but it would be difficult to carry out because that period has passed into history and little evidence has survived.

Fig. 2.

Discourse tree for Example 12.

There are two potential sources of error. First, the extension of a span can be under- or over-estimated, i.e., the wrong span boundaries are assumed. For example, an extraction method that assumed that there were two edus in the sentence and that the second started with the discourse marker because (see Figure 3) would over-estimate the extent of the first span. Second, the wrong spans can be selected, i.e., the relation does not hold between the pair of spans that are identified by the model. For example, the span detection method might wrongly hypothesise that the relation signalled by because in (12) holds between the spans that period has passed into history and and little evidence has survived (see Figure 4).

Fig. 3.

Over-estimating the left span.

Fig. 4.

Choosing the wrong spans.

Identifying the correct spans automatically in cases like this is not trivial and would normally involve full discourse parsing, i.e., knowledge of the discourse tree structure. For our purposes this is obviously not an option as our aim is to automatically extract examples so as to bootstrap a tool for identifying rhetorical relations, which could then be further developed into a full discourse parser. We therefore need to develop an approximate solution to the span detection problem.

One could use a syntactic parser to help with span detection. But for reasons outlined above we decided not to rely on parsing. Another possibility would be to use an external tool for segmenting discourse into spans. Soricut and Marcu (Reference Soricut and Marcu2003) and Sporleder and Lapata (Reference Sporleder and Lapata2005) propose two such tools, both of which utilise supervised machine learning and were trained on the RST-DT. However, we found that the span detection heuristics outlined in the next paragraph work just as well for our purposes.

In cases where the relation is signalled by a pair of discourse markers, one for each span, span detection is relatively easy as the span boundaries are indicated by the discourse markers. This applies to many examples of continuation, which is often marked by a pair of cues, such as first . . .second. In all other cases we relied on the following information to determine the spans:

• • the position of the discourse marker in the sentence

• • linguistic background knowledge (e.g., the possible positions of spans that can potentially be related by the discourse marker)

• • punctuation (i.e., we assumed that the spans of an intra-sentence relation were marked by punctuation)

We further made the following two simplifying assumptions about spans, which are also, albeit implicitly, made by Marcu and Echihabi (Reference Marcu and Echihabi2002). First, we assumed that the spans involved in inter-sentential relations (e.g., in cases where but is sentence-initial) are sentences, rather than multi-sentence units or parts of sentences. This is not always true as it is conceivable that a relation (especially summary) holds between a multi-sentence span and a sentence, or even between a sentence and part of an adjacent sentence, though this latter case is rare in the rst-dt (around 5% according to Soricut and Marcu (Reference Soricut and Marcu2003)). Second, we assume that there are at most two spans in a sentence. Hence for intra-sentential relations we try to determine the border between the two spans but we do not try to identify the border between each span and a possible third or fourth span in the sentence. In the majority of cases these assumptions are justified. However, in a few cases we will over- or under-estimate the extension of a span. For instance, in sentence (12), we would assume that there are two spans, with a span boundary in front of because; we would thus over-estimate the extent of the left span. Note, that this type of error does not necessarily pose a big problem for our machine learning approach, because an example with slightly wrong span boundaries might still contain enough cues for our models to correctly learn the relation.

To determine how reliable our extraction method is, we manually corrected a small sample of automatically extracted examples (50 examples for each relation, 250 in total). There are four potential sources of error: (i) the wrong relation is predicted, (ii) a relation is predicted where there is none, (iii) one of the hypothesised spans is wrong, and (iv) the hypothesised spans are either over or under-estimated (i.e., the span boundaries are wrong). There were five errors overall in our sample. There was no case in which a wrong relation had been hypothesised. This suggests that the discourse markers which we identified as being unambiguous do indeed unambiguously signal one relation. In one case a relation had been predicted where there was none (see example (13), the hypothesised spans are indicated by square brackets).Footnote ⁷ In two cases the spans were wrong. Example (14) illustrates one of these errors. The correct analysis is shown in (14a): the sentence consists of three spans and the last two are related by result (since the because-clause precedes the clause it is related to; see Section 3.1). The example extracted by our method is shown in (14b); here the boundary between the last two spans is missed and hence it is assumed that an explanation relation holds between the first span and the rest of the sentence. The final two errors were due to over-estimating the extension of one of the spans (cf. the discussion for example (12)). The overall accuracy on the sample was thus 98%; hence our extraction method is fairly reliable.

1. (13) [During the nineteenth century, with improvements in communications, these traditions were modified by the] [consequently increased sensitivity to changing preferences in architectural styles.]

2. (14)

   1. a. [The cost of one wall was saved and] [because the buildings housing animals were close to the barn,] [they could be supplied more easily with straw.]

   2. b. [The cost of one wall was saved and] [because the buildings housing animals were close to the barn, they could be supplied more easily with straw.]

Writing the templates and code for the automatic extraction and labelling process took around a week. Using this extraction method, we were able to extract 8.3 million examples in total from our three corpora. The number of training examples that we could extract automatically for different relations varied considerably, due to the different frequencies of the discourse markers that we used in the extraction process. Taking all corpora into account, the least examples (around 8,500) were extracted for continuation, which is a common relation but rarely signalled by unambiguous discourse markers. The most examples were extracted for contrast (nearly seven million, largely thanks to the relatively frequent discourse marker but). Explanation was also fairly frequent, with over one million examples. For result just under 15,000 examples were extracted; for summary just under 17,000 (see Table 1).

Table 1.

Number of automatically extracted examples per relation

Before we trained the classifiers on the automatically extracted and labelled data, we removed from each example the unambiguous discourse markers which formed the basis for the example's extraction. Note that the reason for removing the unambiguous marker was purely technical: if we had not removed it, the classifiers would have relied on it as a predictive cue for the correct relation, at the expense of other, less predictive features. However, we wanted the classifier to learn precisely from these other features, to ensure that rhetorical relations could also be determined if no unambiguous marker was present.

Sometimes an unambiguous marker is accompanied by another, ambiguous discourse cue. For instance, in example (15) the expression it follows that unambiguously signals the result relation. It is preceded by so which can also indicate result, but not unambiguously; it can also signal summary, for example, as in (16). As so does not unambiguously signal result, there is no technical reason for removing it. If it is kept, the classifier might learn that the occurrence of the word so at the beginning of a span is a good predictor for result, but it is unlikely to rely exclusively on this cue, as the training set probably also contains examples in which so indicates a summary relation. Hence, the classifier also has to exploit other features. Keeping ambiguous discourse markers in the training examples thus does not impede a classifier's ability to determine relations in unmarked or not unambiguously marked examples. On the contrary, keeping ambiguous cues provides the classifiers with a valuable source of evidence for or against a given relation, in cases where such an ambiguous cue is present. For this reason, while we did remove the unambiguous discourse markers which were used to extract an example, we did not remove any other, ambiguous markers that might also have been present. Similarly, the manually labelled data which we tested our models on did not contain any unambiguous discourse markers, but some of them did contain ambiguous markers (see Sections 3.3 and 5.1.2).

1. (15) [These pieces are very recognizable, but there is obviously a market for them somewhere] [So it follows that they are going out of the country.]

2. (16) [John ate a fantastic salmon followed by a very nice dessert.] [So he had a great meal.]

3.3 Manually labelled data

The aim of this paper is to test the utility of automatically labelling training data for the task of classifying rhetorical relations. To determine how well a classifier that was trained on automatically labelled (unambiguously marked) examples performs on examples in which the rhetorical relation is not signalled unambiguously by a discourse marker, we need to label a set of those examples manually with the correct rhetorical relation.

Currently, the only publicly available English corpus that is annotated with discourse relations is the rst Discourse Treebank (rst-dt) (Carlson et al., Reference Carlson, Marcu and Okurowski2002).Footnote ⁸ We could not use the rst-dt directly as it is annotated in the framework of Rhetorical Structure Theory (rst) (Mann and Thompson, Reference Mann and Thompson1987) whereas the classes for our models are a subset of sdrt's taxonomy of relations (Asher and Lascarides, Reference Asher and Lascarides2003). rst and sdrt have different inventories of rhetorical relations and there is often no one-to-one mapping between the individual relations in these taxonomies (see the discussion in Section 3.1).

On the other hand, labelling new examples from scratch is a lot of work, especially since most examples will not contain any of our five relations, making it necessary to label quite a large data set to obtain a reasonably large number of examples for each of our relations. However, we could use the rst-dt as the starting point for automatically finding potential examples, which are then manually labelled with one of our five relations from sdrt or identified as not containing any of the relations. To this end, we first extracted from the rst-dt all examples which fulfilled the following three conditions: (i) they involved a relation between two adjacent sentencesFootnote ⁹ or between clauses within a sentence, (ii) they were labelled with an rst relation which could potentially be mapped to one of our sdrt relations, and (iii) they did not contain any of the 55 unambiguous discourse markers we used to create the automatically labelled training set. Note that the examples were allowed to contain discourse cues which were not in our set of unambiguous markers, for example, ambiguous markers such as so or and. The motivation for allowing these was that they may also be present in the automatically labelled examples on which the models were trained. As we mentioned in the previous section, only unambiguous markers were removed from the automatically labelled data.

We identified rst relations that could potentially map to one of the five sdrt relations by using a mixture of factors: (a) a comparison of the definitions of the rst relation (Mann and Thompson, Reference Mann and Thompson1987) with the dynamic semantic analysis of the relations in sdrt (Asher and Lascarides, Reference Asher and Lascarides2003); and (b) a comparison of an existing mapping from discourse markers to rst (Oates, Reference Oates2000) with the mapping from discourse markers to relations in sdrt that we mentioned earlier.

Having extracted examples from the rst-dt which match conditions (i)–(iii) above, we then randomly selected around 300 examples per (hypothesised) sdrt relation for manual labelling. The only exception to this was summary; the potential examples we identified for this relation via conditions (i)–(iii) were rare in the rst-dt; in fact, only around 50 such potential examples were extracted and we therefore manually labelled all of them. The manual labelling was done by an independent, experienced annotator, who saw for each example the two spans (as taken from the rst-dt annotation) together with two preceding and two following sentences as context. We also retained all paragraph boundary markings. The annotator was given a set of instructions to annotate the data with one of the five relations, or with other for those cases where he judged that none of the five relations held. As we mentioned above (Section 2), in sdrt it is possible that two relations hold simultaneously. However, we instructed our annotator to always assign a unique relation. Situations where multiple relations held simultaneously were resolved as follows. If one of our five relations held together with a relation not in our set, the relation in our set was assigned rather than other. A situation where two of our five relations hold together can only occur with continuation, which sometimes holds together with contrast, explanation or result. In these cases the annotator was instructed not to assign continuation.

The instructions included semantic definitions for each of the five relations, together with both manually constructed and naturally occurring examples which exhibit the rhetorical relation. To avoid biasing the annotator, he did not have access to the original rst-dt relation nor to the sdrt relation that was hypothesised in the automatic mapping process.

Our final manually labelled data set contained 1,051 examples: 260 examples of continuation, 266 examples of result, 44 examples of summary, 268 examples of explanation, and 213 examples of contrast. The examples labelled other were discarded since our models were not trained to classify this. Three examples of our manually labelled data for each relation are listed in Appendix B.

To test the reliability of the manual labelling, we computed intra- and inter-annotator agreement. For the former, we asked our original annotator to re-annotate 200 randomly chosen examples six months after the original labelling was done. To compute inter-annotator agreement, we asked a second annotator to annotate the same 200 examples. Table 2 shows the agreement figures, measured in terms of accuracy and the Kappa coefficient (Siegel and Castellan, Reference Siegel and Castellan1988). While the figures are not very high, they are in line with the agreement figures reported in the literature for similar discourse annotation tasks. For example, Carlson et al. (Reference Carlson, Marcu, Okurowski, van Kuppevelt and Smith2003) measured the agreement achieved for relation assignment while building the RST-DT and report accuracies between 60% and 79%, where the higher figure was only achieved after training the annotators for several months.

Table 2.

Intra- and inter-annotator agreement for manual labelling of relations

We also kept track of how long it took our annotator to label the 1,250 examples. By coincidence, annotating the data took about the same time as writing the templates for the automatic extraction and labelling process, i.e., around a week. On top of this, we spent around one day on writing the annotation instructions and training the annotator. However, because we did not label examples from scratch but started with the RST-DT, this is an optimistic estimate of the time required for labelling examples manually.

4.1 The Naive Bayes word pair model

The Naive Bayes classifier was implemented along the lines of Marcu and Echihabi (Reference Marcu and Echihabi2002). This model assumes that the relation that holds between two spans can be determined on the basis of co-occurrences between words.

Let r _i be the rhetorical relation that holds between two spans W ₁ and W ₂. The word pair model assumes that r _i can be determined on the basis of the word pairs in the Cartesian product over the words in the two spans: (w _i,w _j) ∈ W ₁ × W ₂. The model is derived as follows: Given the assumption in the word pair model, the most likely relation is given by . According to Bayes rule:

(17)

Since for any given example P(W ₁ × W ₂) is fixed, the following holds:

(18)

We estimate P(r_i) via maximum likelihood on the training set. And to estimate P(W ₁ × W ₂|r_i) we assume that all word pairs in the Cartesian product are independent. I.e.:

(19)

To estimate the probability of a word pair (w_i, w_j) given a relation r_i, we use maximum likelihood estimation and Laplace smoothing. We converted all words in the spans to lower case but – to stay faithful to Marcu and Echihabi (Reference Marcu and Echihabi2002) – we did not apply any other pre-processing, such as stemming.

4.2 The multi-feature BoosTexter model

While the Naive Bayes model uses only word pairs as features, our second model employs a variety of linguistically motivated features which are retrievable from shallow processing. To combine the linguistic-based features into a classifier we used BoosTexter (Schapire and Singer, Reference Schapire and Singer2000). BoosTexter was originally developed for text categorisation. It combines a boosting algorithm with simple decision rules and allows a variety of feature types, such as nominal, numerical or text-valued features. Text-valued features can, for instance, encode sequences of words or parts-of-speech. BoosTexter applies n-gram models when forming classification hypotheses for these features (i.e., it tries to detect n-grams in the sequence which are good predictors for a given class label).

We implemented 41 linguistically motivated features, roughly falling into six classes: positional features, length features, lexical features, part-of-speech features, temporal features, and cohesion features. These are described (and motivated) below. Some of our features make use of word stems, word lemmas and part-of-speech tags. To obtain the word stems we applied the Porter stemmer to our data (Porter, Reference Porter1980); to obtain the lemmas and part-of-speech tags, we used the RASP toolkitFootnote ¹⁰ (Minnen et al., Reference Minnen, Carroll and Pearce2001).

Positional Features We implemented three positional features. The first encodes whether the relation holds intra- or inter-sententially. The second and third encode whether the example occurs towards the beginning or end of a paragraph, respectively. The motivation for these features is that the likelihood of different relations may vary with both their paragraph position and the position of sentence boundaries relative to span boundaries. For instance, summary frequently holds between sentences, while explanation typically links two spans within a sentence. Similarly, a summary relation is probably more frequent at the beginning or end of a paragraph than in the middle of it.

Length Features Information about the length of the spans might be equally useful. For example, the average span length for some relations (e.g., explanation) may be longer than for others (e.g., summary). We therefore also encoded the lengths of the two spans in terms of the number of words contained in them.

Lexical Features Information about lexical items is also likely to provide useful cues for identifying the correct relation (cf. Marcu and Echihabi (Reference Marcu and Echihabi2002)). For example, word overlap may be evidence for a summary relation. Furthermore, while we removed the unambiguous discourse marker on whose basis a given example was labelled with a particular relation, it is possible that the example contains other cue words for the relation, which can and should be exploited by the model (see Section 3.3). For instance, in the automatically labelled example (20), but would be removed as it was used to label the example as contrast, however, still would be retained. The presence of still is an important cue for a contrast relation, though it does not unambiguously signal contrast, since it can also indicate a purely temporal relation.

1. (20) It's a long book, but still very entertaining.

We incorporated a variety of lexical features. For each of the spans, we included the sequence of lemmas and stems of all words as a text-valued feature. We also included the lemmas of all content words. Including lexical items as text-based features allows BoosTexter to automatically identify n-grams that may be good cues for a particular relation. We also calculated the overlap between the spans, i.e., what proportion of stems, lemmas, and content-word lemmas occurs in both, and added these overlap figures as numerical features.

Part-of-Speech Features We encoded the sequence of part-of-speech tags for both spans as a text-valued feature. It is possible that certain part-of-speech tags (e.g., certain pronouns) are more likely for some relations than for others. Following Lapata and Lascarides (Reference Lapata and Lascarides2004), we also kept specific information about the verbs, nouns and adjectives in the spans, since their models show that these linguistic cues can be highly informative for predicting temporal relations. In particular, we included the string of verb (noun, adjective) lemmas contained in each span as text-based features. For instance, the strings of verb lemmas in example (7), repeated as (21) below, are “operate be” (left span) and “need” (right span).

1. (21) The ability to operate at these temperatures is advantageous because the devices need less thermal insulation.

Furthermore, to ameliorate sparse data that would arise from using only the word lemmas themselves, we mapped the lemmas to their most general WordNet (Fellbaum, Reference Fellbaum1998) class (e.g., verb-of-cognition or verb-of-change for verbs, event or substance for nouns etc.), and used those as features as well. Ambiguous lemmas which belong to more than one class were mapped to the class of their most frequent sense. If a lemma was not in WordNet, the lemma itself was used. Finally, we also calculated the overlaps between the lemmas of both spans and between WordNet classes for each part-of-speech class (of both spans) and included these as numerical features.

Temporal Features The motivation for including temporal features in the model was that tense and aspect provide clues about temporal relations among events and may also influence the probabilities of different rhetorical relations (e.g., distinguishing explanation from result). We extracted simple temporal features from verbs. First, we parsed all our examples using Charniak's parser (Charniak, Reference Charniak2000), and extracted all verbal complexes from the parse trees. We then used simple heuristics to classify each of them in terms of finiteness, modality, aspect, voice and negation (Lapata and Lascarides, Reference Lapata and Lascarides2004). For example, need in example (21) maps to: present, 0, imperfective, active, affirmative.

Cohesion Features The degree of cohesion between two spans may be another informative feature. To estimate it we looked at the distribution of pronouns and at the presence or absence of ellipses (as proposed by Hutchinson (Reference Hutchinson2004) for the task of classifying discourse connectives). For the former, we encoded the number of first, second and third person pronouns in each span. We also used simple heuristics to identify whether either span ends in a VP ellipsis and included this information as a feature.

5.1 Training on automatically labelled data

In the first set of experiments, we trained the two classifiers on the automatically extracted and labelled data. We then tested the trained classifiers on both, automatically labelled, marked data (Section 5.1.1) and manually labelled, unmarked data (Section 5.1.2).

The extracted data set is highly imbalanced with 85% of the examples being labelled as contrast, 14.5% being labelled as explanation and the remaining three relations together making up just around 0.5% of the data (see Section 3.2). Learning from data that is so skewed is problematic as it will bias most learners in favour of the majority class(es). The class imbalance problem has recently gained a lot of attention in the machine learning community and several different solutions have been proposed (see Chawla et al. (Reference Chawla, Bowyer, Hall and Kegelmeyer2002) for an overview). We opted for random undersampling, i.e., randomly selecting a subset of examples from the majority classes for training.

Note that the distribution of different relations in the automatically extracted data set is to some extent artificial; it does not reflect the true distributions of the relations in the corpus, as the amount of extracted examples depends only on the (relative) frequencies of unambiguous discourse markers that signal the relations. Hence the relative frequency of a relation in our data set is not a good predictor of the relative frequency of that relation in general. In particular, it is not a good predictor of the relative frequency of that relation in the set of unmarked or ambiguously marked examples, i.e., our ultimate test set. Some relations are more likely to be signalled by an unambiguous discourse marker than others. For example, while contrast is frequently marked and thus very prevalent in the automatically extracted data set, continuation is frequently not marked by an (unambiguous) discourse marker and thus relatively rare in our data. In general, however, continuation is a very frequent relation, and the relative likelihoods of continuation vs. contrast in the set of unmarked or ambiguously marked examples will therefore be quite different.

Since we do not know the true distribution of relations in the set of unmarked or ambiguously marked examples, we decided to make our training set as uniform as possible, while not discarding too many examples of the minority classes. Hence, we used the full set of examples for continuation (8,542), result (14,978) and summary (16,718), and randomly selected 16,750 examples each for explanation and contrast. This amounts to around 72,000 training examples overall.

BoosTexter has two parameters which need to be optimised (the number of training iterations and the maximal length of n-grams for text-valued features), so we set aside 250 randomly selected examples for each relation as a development set for optimisation. We found that 200 iterations and n = 2 works best and we kept this set-up in all following experiments.Footnote ¹¹

5.1.1 Testing on unambiguously marked data

First, we tested our models on a test set of lexically unambiguously marked examples from which the discourse marker had been removed (i.e., data that is similar to the automatically labelled training data). We used 10-fold cross-validation and computed the average precision (Prec.), recall (Rec.) and F-score for each relation and for all relations together, as well as the average overall accuracy (Acc.).

The results for the Naive Bayes word pair model are shown in Table 3. The overall accuracy obtained by this model is fairly low at 42.34%, though it is significantly better than the 20% average accuracy that would be obtained by randomly guessing a relation (χ² = 2258.98, DoF = 1, p <= 0.01). Marcu and Echihabi (Reference Marcu and Echihabi2002) report an accuracy of 49.7% for their 6-way Naive Bayes word pair classifier. The results are not entirely comparable though, as they model a different set of relations and, crucially, also train on a much larger data set of just under 10 million examples compared to our 72,000 examples. Given that the word pair model is geared towards large data sets, it is likely that its performance would increase if we trained it on more data.

Table 3.

Applying the Naive Bayes word pair model to unambiguously marked data, 10-fold cross-validation

Looking at the individual F-scores, it can be observed that there is some variation regarding how well the model performs for a given relation. The relatively low F-score for continuation and result might be due to the fact that, even after random undersampling of the more frequent relations explanation and contrast, these are (still) the least frequent relations in our training set. However, the performance on contrast is also fairly low. We experimented with adding additional training examples for contrast but found that this leads to a degradation in overall performance of the model: while the F-score for contrast goes up slightly if more examples are added, this is more than outweighed by the fact that the F-scores for the other relations go down.

Table 4 shows the results of applying the BoosTexter model. It can be seen that this model generally performs better than the word pair model. This difference in overall accuracy is significant (χ² = 1019.82, DoF = 1, p <= 0.01). Hence, for relatively small data sets, this model seems to work better than the word pair model. As with the word pair model, explanation and summary are predicted most reliably, the other three relations seem to be slightly more difficult.

Table 4.

Applying the BoosTexter model to unambiguously marked data, 10-fold cross-validation

As both models perform significantly better than the random baseline on the test set, it seems that there is a certain amount of redundancy between the discourse marker and the general linguistic context in (unambiguously) marked examples. Hence, rhetorical relations are learnable – at least to some extent – from automatically labelled data. In the next section, we investigate whether these results carry over to unmarked or ambiguously marked test data.

5.1.2 Testing on Unmarked data

Testing the models on unambiguously marked data from which the discourse markers had been removed led to reasonably good results. But we are interested in whether a labeller that is trained on automatically labelled data of this type can also identify rhetorical relations in examples that are not unambiguously marked. In other words, do the semantic clues for rhetorical relations that were learnt by the classifier from unambiguously marked examples approximate those that one needs to determine the rhetorical relations in unmarked (or ambiguously marked) examples?

As in the previous experiments, the models were trained on the automatically labelled data. We could not employ proper cross-validation due to the need of keeping the test and training data disjoint (due to the fact that the former were manually labelled and the latter were automatically labelled). In order to still be able to generalise away from the potential idiosyncrasies of any given training set, we split the training data randomly into 10 equal sized, disjoint subsets and trained 10 different models on it, each time leaving out one subset and using the remaining nine for training, thus ensuring that the models were trained on the same amount of data as those in the previous experiments. Each of the ten models was then applied to the complete set of manually labelled data (1,051 instances) and the results of the 10 models were averaged.

Table 5 shows the results for the Naive Bayes word pair model. It can be seen that the performance drops quite markedly when compared to testing on data that was originally unambiguously marked: the accuracy drops by more than 38% (from 42.34% to 25.92%) and the F-score falls by more than 52% (from 40.57% to 19.15%). The decrease in performance is particularly noticeable for summary (from 41.46% F-score to 6.63%). However, in spite of the big drop in performance, the model still performs significantly above the 20% baseline of choosing one of the five relations randomly (χ² = 18.88, DoF = 1, p <= 0.01).

Table 5.

Applying the Naive Bayes word pair model to data that is not unambiguously marked, averaged over 10 training runs

Table 6 shows the results of applying the BoosTexter model. While this model outperformed the Naive Bayes word pair model when tested on examples where the relation was originally unambiguously marked by a discourse marker, it drops to a performance level that is similar to that of the word pair model when tested on naturally unmarked or ambiguously marked data. Because BoosTexter starts from a higher level, the fall in performance is even more pronounced: both accuracy and F-score decrease by around 58% (from 60.88% to 25.80% for accuracy and from 59.60% to 24.50% for F-score). As with the word pair model, the decrease is particularly stark for the summary relation, where the F-score drops from 61.16% to 15.49%, i.e., by more than 74%. However, unlike the word pair models, the BoosTexter models obtain a relatively high recall for summary. This might indicate that the bad performance for this relation has something to do with the difference in its relative frequency in the training and test sets; the distribution of relations is relatively uniform in the training set by design, whereas in the test set, summary is relatively rare (44 instances out of 1051, see Section 3.3). The uniform distribution in the training data leads BoosTexter to predict all relations with more or less the same frequency, which for summary results in a high recall but low precision.

Table 6.

Applying the BoosTexter model to unmarked data, averaged over 10 training runs

Overall the decline in performance when testing on unmarked or ambiguously marked data shows that our models, which were trained on examples where an unambiguous discourse marker was originally present, do not generalise well to examples which occur naturally without unambiguous discourse markers. However, the performance is still higher than the baseline of selecting a relation randomly.

The fact that both models show a significant drop in performance to just above chance suggests that this behaviour may be largely independent of the type of classifier used (i.e. knowledge-lean vs. complex). Instead it looks like the problem might stem from the data itself, i.e., marked and unmarked data could just be too dissimilar linguistically to allow a classifier to generalise from one to the other. We investigate this further in Section 6. In the next section, however, we explore how the performance achieved by training on automatically labelled data compares to that achieved by training on manually labelled, not unambiguously marked examples. In particular, we want to know how many examples one would have to annotate to obtain a performance similar to training on automatically labelled data.

5.2 Training on manually labelled (unmarked) data

The set of manually labelled data consists of 1,051 examples (see Section 3.3). To train and then test the models, we split this set in half, ensuring that each half contained similar proportions of each rhetorical relation, and then trained the models on one half and tested on the other. We then swapped the training and test set and repeated the procedure five times, each time with a different split of the data set (i.e., effectively using five times 2-fold cross-validation).

Table 7 shows the results of applying the Naive Bayes word pair model. The average accuracy is 12.88% and the average F-score is 12.96%. This is noticeably lower than the performance achieved by training on automatically labelled examples and testing on manually labelled ones (25.92% accuracy and 19.15% F-score). The difference in accuracy is significant (χ² = 45.83, DoF = 1, p <= 0.01). Furthermore, randomly guessing a relation would actually lead to a higher accuracy (i.e., 20%) on average. Again this difference is significant (χ² = 16.19, DoF = 1, p <= 0.01). The bad performance of the word pair model when trained on the manually labelled data set can be explained by the fact that this model requires a large training set to achieve reasonable performance levels. This explains why the model's performance when trained on over 72,000 automatically labelled examples is much better than when trained on 526 manually labelled examples. Indeed, the need for a large data set appears to outweigh the disadvantages of using a different type of data for training vs. testing; i.e., it outweighs any linguistic differences that there might be between the automatically labelled examples and the manually labelled ones, even though our prior experiments suggest that such differences are real and affecting performance.

Table 7.

Training and testing on manually labelled data, Naive Bayes word pair model, 5 times 2-fold cross-validation

Table 8 shows the results of training and testing the BoosTexter model on manually labelled examples. For this model the situation is different: training on a small set of manually labelled examples leads to a 14.50% higher accuracy and a 9.19% higher F-score than training on a large set of automatically labelled examples. The difference in accuracy is significant (χ² = 57.05, DoF = 1, p <= 0.01). The results might be slightly optimistic as the manually labelled examples all come from the set of 385 Wall Street Journal articles which make up the rst-dt (Carlson et al., Reference Carlson, Marcu and Okurowski2002), whereas the automatically labelled examples come from a variety of sources, though mainly in the news domain.Footnote ¹² However, we only used a fraction of the examples in the rst-dt for labelling and we randomised them beforehand (see Section 3.3), so one would expect a fairly wide spread across the 385 articles.Footnote ¹³ Furthermore, the difference in performance between training on automatically extracted examples and training on manually labelled examples is so large that it can hardly be explained by the relative homogeneity of our manually labelled examples alone. Hence, for this model, it looks like training on even a relatively small set of manually labelled examples leads to a significantly better performance than training on a large set of automatically extracted examples. This is in stark contrast to the word pair model, where having a large training set outweighs any reductions in performance that are caused by training and then testing on different types of examples.

Table 8.

Training and testing on manually labelled data, BoosTexter model, 5 times 2-fold cross-validation

Given that training on even a small set of manually labelled examples leads to significant improvements over training on automatically labelled data alone, we wanted to know how little manually labelled data are necessary to beat training on automatically labelled data. Therefore, we conducted a learning curve experiment for training on manually labelled data. We only tested the BoosTexter model as the Naive Bayes word pair model was found not to perform very well on small amounts of training data. For the learning curve, we randomly split the manually labelled data into 16 subsets of similar size and with similar proportions of examples per relation. One of these was set aside as a test set and the remaining 15 were used to create 15 progressively increasing training sets, with the smallest containing just one subset and the largest containing all 15. We then trained a model on each of the training sets and tested it on the test set.

Figure 5 shows the results. While there is some variation for individual training sets (due to the fact that they were randomly selected), it can be seen that all training sets, except for the smallest, lead to an accuracy of above 25%, i.e., the accuracy achieved by training a BoosTexter model on automatically labelled data. In other words, around 140 manually labelled examples are enough to beat training on automatically labelled data. This estimate might again be slightly optimistic due to the relative homogeneity of our manually labelled data. However, given that automatic data labelling is not entirely cost-neutral either, i.e., some human effort is required for writing the extraction rules, it seems that manually labelling some examples may sometimes be a better option than training on automatically labelled examples alone. In our case, writing the extraction rules took around one week, while our annotator labelled more than 220 examples per day. However, this is an optimistic estimate of the time involved in manually labelling examples, as we did not start from scratch but used the rst-dt to pre-select examples. Also, large scale labelling would also require some time for training the annotators and, because rhetorical relations are not distributed uniformly, it might be necessary to label quite a large amount of data to ensure that there are sufficient training examples for the rarer relations.

Fig. 5.

Learning curve for training and testing on manually labelled, unmarked data.