2.1 Event anchor and temporal relation extraction

The first event extraction tasks were organized as part of the Message Understanding Conferences (MUC) (Grishman and Sundheim Reference Grishman and Sundheim1996), where events were viewed as scenario templates defined for specific domains. The scenarios were predetermined (e.g., a shuttle launch) and the task was to fill the information slots for a given scenario. Early approaches and systems for extracting event-based information were predominantly rule-based (Humphreys et al. Reference Humphreys, Gaizauskas, Azzam, Huyck, Mitchell, Cunningham and Wilks1998; Aone and Ramos-Santacruz Reference Aone and Ramos-Santacruz2000). The improvements were mostly gained by including additional extraction patterns for scenarios from new domains (Aone and Ramos-Santacruz Reference Aone and Ramos-Santacruz2000). However, more recent approaches focus on automatically acquiring extraction templates (Yangarber et al. Reference Yangarber, Grishman, Tapanainen and Huttunen2000; Surdeanu et al. Reference Surdeanu, Harabagiu, Williams and Aarseth2003).

The emergence of standards for annotating event and temporal information in text, such as TimeML (Pustejovsky et al. Reference Pustejovsky, Castano, Ingria, Sauri, Gaizauskas, Setzer, Katz and Radev2003b), and the creation of the accompanying datasets, such as TimeBank (Pustejovsky et al. Reference Pustejovsky, Hanks, Sauri, See, Gaizauskas, Setzer, Radev, Sundheim, Day and Ferro2003a), triggered the transition from template-based event extraction to more robust and general approaches. The organization of shared evaluation tasks, such as ACE (2005) and TempEval (Verhagen et al. Reference Verhagen, Gaizauskas, Schilder, Hepple, Katz and Pustejovsky2007Reference Verhagen, Sauri, Caselli and Pustejovsky2010) was another step forward. Similar to Message Understanding Conferences and early work on event extraction, ACE (2005) focused on domain-specific event extraction. The tasks included the extraction of event anchors and arguments (both tasks are framed as classification into one of the predetermined domain classes) as well as event coreference resolution. Some of the evaluated systems approached the tasks with machine learning methods (e.g., Ahn Reference Ahn2006). The first TempEval campaign (Verhagen et al. Reference Verhagen, Gaizauskas, Schilder, Hepple, Katz and Pustejovsky2007) focused on three temporal relation extraction tasks: extraction of relations between events and temporal expressions, between events and document creation time, and between the main events of adjacent sentences. The second TempEval campaign (Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010), TempEval-2, introduced three additional tasks: extraction of event anchors, extraction of temporal expressions, and recognition of temporal relations between events from the same sentence, where one syntactically dominates the other. On the anchor extraction task, the best results were achieved by two rather different systems: (1) a supervised machine-learning approach using conditional random fields with a rich set of linguistic features (Llorens et al. Reference Llorens, Saquete and Navarro2010), and (2) a rule-based approach filtering head verbs and head nominalizations with WordNet-based attributes (Grover et al. Reference Grover, Tobin, Alex and Byrne2010). On the temporal relation extraction task, the best-performing system used supervised machine learning with Markov logic networks (UzZaman and Allen Reference UzZaman and Allen2010). Recently, Llorens, Saquete and Navarro-Colorado (Reference Llorens, Saquete and Navarro-Colorado2013) have demonstrated that semantic knowledge (lexical semantics and semantic roles) improves the recognition of event anchors and temporal relations.

The same tasks addressed in TempEval-2, although in a different domain, were also addressed in the i2b2 NLP challenge for clinical records (Sun, Rumshisky and Uzuner Reference Sun, Rumshisky and Uzuner2013), which focused on the temporal relations within clinical narratives. The participating systems extracted (1) clinically significant events (e.g., problems, tests, admissions, transfers), (2) temporal expressions (e.g., dates, durations, frequencies), and (3) temporal relations between the events and temporal expressions from a corpus of clinical discharge summaries. Machine learning-based approaches performed best on event detection, and rule-based methods performed best on temporal expression extraction and normalization, whereas hybrid methods that combined machine learning with heuristics performed best on the classification of temporal relations. In contrast to the work done in the i2b2 challenge, we aim to determine the temporal relations between pairs of events instead of between events and temporal expressions.

Unlike the aforementioned approaches, which consider temporal relations between events in isolation (i.e., by extracting relations between pairs of events), other approaches have focused on constructing a more complete temporal structure from documents. Kolomiyets et al. (Reference Kolomiyets, Bethard and Moens2012) extract narrative timelines of children’s stories as dependency trees using two different parsing models – a deterministic non-projective shift-reduce parser and a graph-based maximum spanning tree parser. However, children’s stories tend to have a simpler (i.e., more linear) temporal structure than other genres (e.g., news), which is also confirmed by their experiments in which a linear temporal sequence was found to be a very competitive baseline. Bramsen et al. (Reference Bramsen, Deshpande, Lee and Barzilay2006) represent the temporal structure of a document as directed acyclic graphs (DAGs). They consider only temporal precedence (relations before and after) and do not consider temporal overlap. Globally optimal directed acyclic graphs are induced using integer linear programming optimization. Compared with our work, their graphs represent a more coarse temporal structure because the nodes in their directed acyclic graphs represent textual segments rather than individual event mentions.

Extracting more complete temporal structures has also been the focus of the most recent shared temporal evaluation campaign, TempEval-3 (UzZaman et al. Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013). TempEval-3 featured the same tasks as TempEval-2 but provided an overall evaluation metric called temporal awareness as a unified measure of a system’s ability to extract temporal expressions, event mentions, and temporal relations between them. TempEval-3 also introduced new corpora: a small platinum corpus for evaluation, manually annotated by task organizers, and a large silver corpus, automatically annotated by merging the predictions of the best performing systems from TempEval-2. ClearTK (Bethard Reference Bethard2013), the most temporally aware system, employed a pipeline of linear machine-learning models in combination with simple morphosyntactic features.

A related stream of research addresses the learning of narrative chains from the text (Chambers and Jurafsky Reference Chambers and Jurafsky2008Reference Chambers and Jurafsky2009; Jans et al. Reference Jans, Bethard, Vulić and Moens2012). Chambers and Jurafsky (Reference Chambers and Jurafsky2008) define a narrative chain as a ‘partially ordered set of events related by a common protagonist’. They consider only verbal events and, similar to Bramsen et al. (Reference Bramsen, Deshpande, Lee and Barzilay2006), only the relation of temporal precedence between two events. The set of events that constitute a chain is determined based on the pointwise mutual information between pairs of events estimated from a large corpus, after which the temporal ordering is determined using a binary classifier (Before versus Other) trained on TimeBank. In their subsequent work, Chambers and Jurafsky (Reference Chambers and Jurafsky2009) introduce typed event chains, in which shared arguments are constrained to a certain semantic role. In contrast to FrameNet or PropBank, the semantic roles are not predetermined but are instead represented as sets of words induced from a corpus in an unsupervised fashion. For example, in the narrative chain (suspect X) → (hunt X) → (accuse X) → (convict X), the shared argument X should have the semantic role of a ‘criminal person,’ defined by certain words {criminal, murderer, suspect, thief}. Chambers and Jurafsky (Reference Chambers and Jurafsky2009) go on to define event schemas as sets of typed event chains as a means of modeling all the actors participating in a set of events. For example, if (hunt X) is part of one narrative chain, then (Y suspect) → (Y hunt) has to be a part of another chain, but both chains belong to the same narrative schema. Although related to our work, the extraction of narrative chains and event schemas differ from the extraction of event graphs in two important aspects. First, only verbal events are considered for participation in narrative chains, while nominal event mentions (e.g., ‘elections’, ‘murder’, ‘match’) – often informationally very relevant – are ignored. Second, because event schemas are unordered sets of narrative chains, while narrative chains model only the temporal relations between events that share a protagonist, the temporal relations between event mentions with no common protagonists (e.g., ‘Obama and Putin met only a week after the meeting between Sarkozy and Merkel in Athens’) are also ignored.

2.2 Argument extraction

In ACE, argument extraction was defined as a separate task and framed as a supervised classification problem. Ahn (Reference Ahn2006) considers the named entities, temporal expressions, and numerical expressions as argument candidates. His system pairs all argument candidates with all event anchors within the same sentence and uses supervised classification to determine whether a candidate is indeed the argument of the anchor. However, in general, event arguments may belong to classes other than named entities, temporal expressions, or numerical expressions. Furthermore, Ahn uses domain-specific argument types, not allowing for cross-domain argument extraction.

As events anchors are very often predicates, argument extraction is related to the task of semantic role labeling (SRL), which seeks to induce shallow semantic representations of sentences by extracting predicate-argument structures (Gildea and Jurafsky Reference Gildea and Jurafsky2002). SRL has been proven beneficial for many NLP tasks (Surdeanu et al. Reference Surdeanu, Harabagiu, Williams and Aarseth2003; Melli et al. Reference Melli, Wang, Liu, Kashani, Shi, Gu, Sarkar and Popowich2006; Moreda et al. Reference Moreda, Llorens, Saquete and Palomar2011; Llorens et al. Reference Llorens, Saquete and Navarro-Colorado2013). Approaches to SRL differ in the role sets they use; the two main role sets are the one based on FrameNet (Baker, Fillmore and Lowe Reference Baker, Fillmore and Lowe1998) and PropBank (Palmer, Gildea and Kingsbury Reference Palmer, Gildea and Kingsbury2005). The two role sets are of different granularity: FrameNet, which is based on semantic frame theory (Fillmore Reference Fillmore1976), defines a very detailed representation of situations (frames) along with their conceptual roles (frame elements). In contrast, PropBank is theory-neutral and defines role labels on a per-verb basis.

Bejan and Hathaway (Reference Bejan and Hathaway2007) use FrameNet frames to recognize the arguments of events. They identify arguments and their semantic roles through several processing steps, each of which introduces non-negligible performance loss: (1) identification (disambiguation) of the frame satisfied by the predicate; (2) identification of phrases that represent the arguments of a predicate; and (3) assignment of semantic roles to previously identified arguments. Predicates are typically verbs, and frame identification is similar to word sense disambiguation (WSD) (Bejan and Hathaway Reference Bejan and Hathaway2007). Since word sense disambiguation approaches generally perform significantly worse on verbs than on nouns or adjectives (Navigli and Lapata Reference Navigli and Lapata2007; Agirre and Soroa Reference Agirre and Soroa2009), the correct frame is often not selected, although it is covered by the resource. The selection of an incorrect frame in turn typically triggers incorrect argument extraction and role labeling. This error is further exaggerated by the fine-grainedness of semantic roles in FrameNet.

The lack of coverage and robustness limit the effectiveness of FrameNet-based SRL approaches for event extraction. PropBank-based SRL approaches may be more adequate as they have a wider lexical coverage. Apart from the predicate-specific semantic roles, PropBank defines a set of modifier adjuncts, i.e., general arguments that any verb can optionally take (e.g., Location, Time, Purpose, Manner). PropBank-based SRL was the topic of the CoNLL-2005-shared task, in which most participants employed supervised machine-learning models that used phrase-structure features (Haghighi, Toutanova and Manning Reference Haghighi, Toutanova and Manning2005; Surdeanu and Turmo Reference Surdeanu and Turmo2005). Unlike CoNLL-2005 SRL systems, which label the predicate arguments based on constituency parsing information, our event argument extraction is rule-based and relies on dependency parsing information.

2.3 Event coreference resolution

Coreferent event mentions are mentions that refer to the same real-world event. By identifying coreferent event mentions, we can gather all of the information about the real-world event to which these mentions refer, even though they are scattered throughout the text. The task of event coreference resolution was first addressed by Humphreys, Gaizauskas and Azzam (Reference Humphreys, Gaizauskas and Azzam1997) who arrange mentions from text in a hierarchical semantic network according to an event ontology, and coreference was determined pairwise by computing scores based on the distance of events in the ontology. Bagga and Baldwin (Reference Bagga and Baldwin1999) build summaries for each event anchor in the document by selecting all sentences in which the same anchor or one of its nominalizations appears. To determine coreference, they analyze the overlap in named entities and temporal expressions between the events’ summaries. Ahn (Reference Ahn2006) addresses event coreference resolution using a maximum entropy classifier to determine the coreference of paired event mentions. Bejan and Harabagiu (Reference Bejan and Harabagiu2010) propose a clustering approach based on non-parametric Bayesian models for both within- and cross-document event coreference resolution. Their models are based on a rich set of linguistic features, including features that compare event arguments. However, they extract arguments according to semantic frames defined in PropBank (Kingsbury and Palmer Reference Kingsbury and Palmer2002) and FrameNet (Baker et al. Reference Baker, Fillmore and Lowe1998), effectively limiting the approach to the predicates covered by those resources. Chen et al. (Reference Chen, Su, Pan and Tan2011) propose a framework that integrates five different event coreference resolvers, arguing that coreferent pairs with different parts of speech (POS) exhibit different behaviors (e.g., the coreference of verb–verb event pairs differs from coreference of verb–noun event pairs). They train one binary classifier for each pair of POS-tags and use a rich set of linguistic features, including argument-matching features. Their argument extraction approach is limited to the extraction of arguments from anchor premodifiers and prepositional attachments.

2.4 Temporal evaluation

The work on temporal evaluation originates from the need to measure inter-annotator agreement on the annotations of temporal relations (Setzer, Gaizauskas and Hepple Reference Setzer, Gaizauskas and Hepple2003; Tannier and Muller Reference Tannier and Muller2008). Setzer et al. (Reference Setzer, Gaizauskas and Hepple2003) propose performing transitive closure for a set of annotated temporal relations prior to measuring the inter-annotator agreement. Precision and recall are defined as the number of relations from the transitive closure of one annotator’s annotations that can be found in the closure of another annotator’s annotations. Tannier and Muller (Reference Tannier and Muller2008) argue that relations introduced transitively should not be matched the same way as core relations (which cannot be inferred from other relations). Thus, they evaluate only the overlap in the core relations between the gold standard temporal annotations and the system-extracted relations. UzZaman and Allen (Reference UzZaman and Allen2011) propose a middle-ground approach: They define the precision as the proportion of the system-extracted relations that can be found in the transitive closure of the gold-annotated relations and the recall as the proportion of gold-annotated relations that can be found in the transitive closure of the system-extracted relations. The proposed temporal awareness metric has already been used for the TempEval-3-shared task (UzZaman et al. Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013).

The above-mentioned metrics, each in its own way, account for the temporal structure induced by temporal relations. However, these metrics are still inadequate for the evaluation of event graphs. First, we find these metrics to be biased toward evaluating temporal relations. An event graph that contains no correct temporal relations but contains all and only correct event mentions (anchors and their arguments) would be assigned a temporal awareness score of zero, which seems inappropriate. Furthermore, because temporal awareness essentially compares the sets of temporal relations, it cannot be used to measure the degree to which the structures of two event graphs correspond. In Section 4, we introduce two evaluation metrics to address these issues.

3.1 Event graph

A graph is the representation of choice used when modeling relations between concepts. In our case, the concepts of interest are the individual real-world events, realized in text as event mentions. We define an event graph as a vertex- and edge-labeled mixed graph in which vertices represent the individual event mentions and edges represent the temporal and coreference relations between event mentions. Given that some relations between events are symmetric (temporal overlap, temporally equal, or coreference), whereas others are not (temporally before or temporally after), an event graph may contain both directed and undirected edges; thus, we define an event graph as a mixed graph. Figure 2 shows an example of an event graph and the corresponding news story.Footnote ¹

Fig. 2.

Example of a reference event graph built from a news story.

Let M denote a set of extracted event mentions and R denote the set of event relations. An event graph G is a tuple

\begin{equation*} G = (V, E, A, m, r) \end{equation*}

where V is the set of vertices, E is the set of undirected edges, A is the set of directed edges (arcs), m: V → M is a vertex labeling function mapping vertices to mentions from M, and r: E∪A → R is an edge labeling function mapping edges to relations from R. Each event mention from M is assigned a type (e.g., Occurrence, Reporting) and consists of an event anchor and a set of typed event arguments.

The above definition is general in that it does not specify any concrete sets for M and R; in practice, of course, these sets need to be specified in detail. The concrete definitions are likely to be application and domain-specific. In our work, we define the set of events M and the set of relations R based on the following assumptions.

Factuality of events. In principle, event-oriented information needs concern factual, real-world events. For example, in question answering a question about a factual event (e.g., ‘Who killed Osama bin Laden?’) is much more likely to appear than a question about a non-factual event (e.g., ‘Who did not win a medal in 2012 Olympics?’ or ‘When might Obama resign?’). Similarly, in answering temporal questions, complex queries have been addressed by decomposing them into facts, which, in most cases, relate to factual event mentions, and by retrieving the answers for each fact individually (Saquete et al. Reference Saquete, González, Martínez-Barco, Munoz and Llorens2009). Since we aim to represent real-world events as they actually occurred, in this work we focus exclusively on factual event mentions – mentions for which we can ascertain that the event to which they refer has indeed occurred or is occurring in the real world (Polanyi and Zaenen Reference Polanyi and Zaenen2006; Saurí and Pustejovsky Reference Saurí and Pustejovsky2012). Thus, we do not consider negative (e.g., ‘He did not win’), hypothetical (e.g., ‘He can win’), future (e.g., ‘He will win’), and counterfactual event mentions (‘If he had won...’), nor do we consider mentions referring to uncertain events (e.g., ‘Maybe he won’). Event factuality is also essential for the temporal ordering of events (Karttunen and Zaenen Reference Karttunen and Zaenen2005) because non-factual events can hardly be placed on a timeline.

Argument types. Aiming for robustness, we focus on extracting four generic types of event arguments: agent, target, location, and time. These types of arguments are informationally most salient as they directly answer four main wh-questions: ‘Who did what to whom, where, and when?’. By considering only these four argument types, we can use a more generic, and thus a more robust approach to argument extraction. The approach is arguably more robust as it eliminates the coverage issues present in FrameNet-based approach of Bejan and Hathaway (Reference Bejan and Hathaway2007), involves fewer processing steps, and thus reduces the chance of an error. Note that an event mention may have more than one argument of the same generic type (e.g., in ‘Low Speed Chase and Stallion rounded the South Farallon Island... ’ both ‘Low Speed Chase’ and ‘Stallion’ are the agents of the mention ‘rounded’).

Relation types. In principle, it is possible to identify many different semantic relations between events (e.g., temporal, causal, containment, semantic relations). In this work, we focus on two types of relations that intuitively appear to be most relevant for a wide range of possible applications: temporal relations (because events are temporally determined concepts; Pustejovsky et al. Reference Pustejovsky, Castano, Ingria, Sauri, Gaizauskas, Setzer, Katz and Radev2003b) and event coreference relations (because a single real-world event may be referred to by multiple mentions, potentially conveying complementary information). When resolving within-document event coreferences, we do not merge the vertices representing coreferent mentions. In other words, each graph vertex represents a single event mention and not a real-world event referred to by the mentions. For coreference resolution, we do not consider mentions other than event anchors (e.g., we do not consider pronominal anaphoric mentions).

3.2 Anchor extraction

Extraction of factual event mentions is the first step in the construction of event graphs. Apart from considering only factual event mentions, our definition of an event anchor differs from TimeML (Pustejovsky et al. Reference Pustejovsky, Castano, Ingria, Sauri, Gaizauskas, Setzer, Katz and Radev2003b) in three important respects, which we summarize below.

1. (1) We do not consider TimeML states as events. TimeML defines states as ‘situations or predicates describing states or circumstances in which something obtains or holds true’ (Pustejovsky et al. Reference Pustejovsky, Castano, Ingria, Sauri, Gaizauskas, Setzer, Katz and Radev2003b). Similarly, Smith (Reference Smith1999) characterizes states as situations that do not change over time, i.e., situations that hold over an undifferentiated interval. They are typically situations that hold true for a long period of time (e.g., ‘used’ in ‘It’s not something we’re not used to in the industry’). We adopt this definition of a state and have chosen not to extract predicates describing states. We argue that in texts reporting on real-world events (e.g., breaking news, police reports, biographies, electronic health records), states are less informative than actions.

2. (2) We consider state changes, i.e., the transitions from one state to another (e.g., ‘The CBI index rose 3.2%’) as a distinct class of events. We define a state change as an event that alters some intrinsic property of an entity participating in that event. For example, we consider the event ‘fell’ in ‘The girl fell and started crying’ to be an instance of the Occurrence class because falling does not change any intrinsic property of the girl (her position is not an intrinsic property). In contrast, we consider the event ‘fell’ in ‘The P&G stocks fell 5% in early trading’ to be an instance of the State change class because it describes a change in an intrinsic property of the stock – its value has changed. We note that state changes are mostly telic events (completed in some sense), whereas occurrences can be telic or atelic. In the TimeML standard (Pustejovsky et al. Reference Pustejovsky, Castano, Ingria, Sauri, Gaizauskas, Setzer, Katz and Radev2003b), state changes are not distinguished from other instances of the more general Occurrence class. However, we find this distinction to be useful in domains where the intrinsic properties of entities change frequently (e.g., electronic health records).

3. (3) Our event anchors are single words. Although in TimeML standard event anchors can be multiword expressions, a vast majority of event anchors (99.63%) in the TimeBank corpus are single words.

The definition of an event mention determines the complexity of the anchor extraction task. Due to the first point above, when compared with tasks based on the TimeML definition of an event (Bethard Reference Bethard2008; Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010), our task is arguably more demanding, as it also requires us to discriminate between events and states. Furthermore, our models need to discriminate between factual and non-factual events.

We address the task of event anchor extraction in two steps: anchor identification and event type classification. For each token, anchor identification predicts whether it is an event anchor. Note that we do not merge adjacent event tokens. Instead, we treat adjacent event tokens as two independent event mentions. We then classify the tokens identified as event anchors into one of the following five classes: Occurrence, Intentional action, Perception, Reporting, and State change. We adopted the first four classes from the TimeML standard and added the State change class for event mentions that refer to transitions from one state to another.

We use supervised machine learning models with the same rich set of lexical, POS, and syntactic features for both subtasks. We compute most of the features using the basic linguistic and preprocessing tools from the Stanford Core NLP library.Footnote ² We use the following features:

• • Token features: These consider only current token information. We use the following token features: word, lemma, stem, ending (the suffix of the word from the last vowel, or the penultimate vowel if the last letter of the word is a vowel), POS tag, and coarse POS tag (noun, verb, adjective, or others). Similar sets of features have been used in most previous approaches to event anchor extraction (Bethard Reference Bethard2008; Llorens et al. Reference Llorens, Saquete and Navarro2010; UzZaman and Allen Reference UzZaman and Allen2010).

• • Context features: These are all of the token features (e.g., word, lemma, stem) computed for all the tokens in a two-token symmetric window (two tokens to the left and two tokens to the right of the current token). Feature windowing for the anchor extraction task has also been proposed in previous works (Bethard Reference Bethard2008; Llorens et al. Reference Llorens, Saquete and Navarro2010; UzZaman and Allen Reference UzZaman and Allen2010).

• • Syntactic features: These include all syntactic dependency relations of a token, the chunk types (e.g., NP, VP, PP) of the token chunk and its adjacent chunks to the left and the right. Similar dependency-based syntactic features have been proposed by Bethard (Reference Bethard2008), while Llorens et al. (Reference Llorens, Saquete and Navarro2010) based their syntactic features on phrase-structure trees. We additionally employ a feature indicating whether the token has a nominal subject (i.e., it governs a nominal subject dependency relation), a feature indicating whether the token is a direct object (i.e., dependent of a direct object dependency relation), and a feature indicating whether the token has a direct object (governs a direct object dependency relation). The Stanford dependency parser (De Marneffe, MacCartney and Manning Reference De Marneffe, MacCartney and Manning2006) was used to compute the values of syntactic features.

• • Modifier features: These aim to discriminate between factual and non-factual mentions by describing different modifiers for the token: the modality modifier (e.g., ‘may’ in ‘Stallion may have rounded...’), the auxiliary verb (e.g., ‘had’ in ‘Low Speed Chase had already finished the race’), the determiner of the token (e.g., ‘The’ in ‘The race around South Farallon Island...’), and a feature indicating whether the current token is negated (e.g., true for mention ‘finish’ in ‘Stallion did not finish the race’). We also use dependency relations to compute the values for modifier features (the relation mod for modality modifiers, neg for negations, aux for auxiliaries, and det for determiners).

• • Other features: These include a feature indicating whether the word is capitalized (named events, such as ‘World Cup’ are capitalized), features indicating whether the word and lemma of the current token are more frequently annotated as an event or not event on the training set, and features indicating the most frequent event type assigned to the word and lemma on the training set (e.g., Reporting for word ‘said’ and lemma ‘say’).

Considering the large size of the feature space (approximately 100K), induced by the large number of lexical features, we chose to use linear models, L2-regularized logistic regression and linear support vector machine (SVM). We used the LibLinear implementation of the algorithms (Fan et al. Reference Fan, Chang, Hsieh, Wang and Lin2008).

3.3 Argument extraction

We use a rule-based approach for event argument extraction. As we are considering a small number of generic argument types, we can use a relatively small number of linguistically motivated extraction rules to reliably identify the argument and its type by considering the dependency relations between the argument and anchor. The underlying assumption is that certain types of arguments tend to be realized in certain syntactic roles. Each extraction rule specifies a dependency pattern and a set of possible argument types, whereas a dependency pattern defines the set of possible dependency relations between the event anchor and head word of the argument.

Argument extraction consists of two steps: (1) Argument detection, which syntactically identifies argument candidates, and (2) argument type classification. Some of the syntactic patterns are semantically unambiguous, i.e., they can be used to both identify the argument and in most cases unambiguously identify its type. For example, the syntactic role of a nominal subject is typically associated with the semantic type of an agent. Other patterns are semantically ambiguous, in the sense that the argument type is not uniquely determined by the syntactic role. For example, the prepositional object of an event anchor may be a target (e.g., ‘...competing against 48 opponents’), location (e.g., ‘...competing around South Farallon Island’), or a time specification (e.g.,‘...competing on Saturday’) of the event. To disambiguate the argument type in such cases, we use additional rules based on named entity recognition (Finkel, Grenager and Manning Reference Finkel, Grenager and Manning2005), temporal expression extraction (Chang and Manning Reference Chang and Manning2012), and WordNet-based semantic similarity (Wu and Palmer Reference Wu and Palmer1994). Next, we describe the two extraction steps in more detail.

3.3.1 Argument detection

We define the syntactic patterns used within extraction rules in terms of non-collapsed dependency relations provided by the Stanford dependency parser (De Marneffe et al. Reference De Marneffe, MacCartney and Manning2006). We designed the patterns based on the analysis of a manually annotated development set (cf. Section 6.2) by incrementally adding the rules that increased the recall without significantly decreasing the precision. To prevent overfitting, we preferred more general rules (rules whose patterns have fewer dependency relations) over the more complex rules. For the same reason, we chose to use only unlexicalized patterns.

We use two types of syntactic patterns: extractive patterns and distributive patterns. Extractive patterns identify head words of arguments of interest (agent, target, time, and location) and assign them to event anchors. A list of extractive patterns is provided in Table 1. Extractive patterns identify only the head words of event arguments. Given that arguments are often multiword expressions, we identify the full extent of the argument as the chunk containing the argument head word. For this task we use the chunker from the OpenNLP library.Footnote ³

Table 1.

Extractive patterns for event argument extraction (anchors are underlined, arguments are shown in bold)

Distributive patterns consider the cases in which one argument is shared between two or more events (argument sharing), or when one event has two or more arguments of the same type (argument conjunction).

Argument sharing considers cases in which two event anchors occur in a certain syntactic relation that indicates that an argument is being shared, although it is syntactically linked to only one of the anchors. For example, in ‘Low Speed Chase hit and ‘damaged the Stallion’, ‘Low Speed Chase’ is the agent for both event mentions ‘hit’ and ‘damaged’. However, the Nominal subject pattern (cf. Table 1) would identify ‘Low Speed Chase’ as the agent of only the anchor ‘hit’. Argument sharing is indicated with conjunction (e.g., conj(hit, damaged) in ‘Low Speed Chase hit and ‘damaged the Stallion’) and open clausal complement (e.g., xcomp(managed, avoid) in ‘Stallion managed to avoid the disaster’). An argument can only be shared with events that have not already been assigned an argument of the same type. For instance, in ‘The Stallion provoked and the Low Speed Chase answered’, we would not distribute an agent ‘The Stallion’ to the event ‘answered’ nor an agent ‘Low Speed ‘Chase’ to the event ‘provoked’, even though the anchors are conjoined.

The argument conjunction rule handles the cases in which one event has multiple arguments of the same type that are mutually syntactically conjoined. For instance, in ‘Stallion and Low Speed Chase were seriously damaged in the crash’ both ‘Stallion’ and ‘Low Speed Chase’ are the targets of ‘damaged’. However, the Passive nominal subject pattern (cf. Table 1) would identify only ‘Low Speed Chase’ as a target of ‘damaged’. By distributing the anchor ‘damaged’, we also assign ‘Stallion’ as its target.

It should be noted that the proposed extraction rules are designed specifically for the English language, and thus adapting the pipeline to a new language would require rewriting or at least adapting these rules. The development of the rules for English was a three-person-week effort.

3.3.2 Argument type disambiguation

Some extraction rules require an additional disambiguation step because the argument types cannot be uniquely determined from their syntactic roles. For example, variously typed arguments (e.g., time, location, agent, target, manner, cause) may be syntactically realized as a prepositional object of an event anchor. Our goal is to identify arguments denoting time, location, agent, or target. We define a set of disambiguation rules for each ambiguous extraction pattern. Similar to the extraction rules, we designed the disambiguation rules based on the analysis of the annotated development corpus (cf. Section 5.2), preferring more general rules over more specific rules.

We present the rules for determining the semantic type of a prepositional object, given that the pattern can be satisfied by arguments of all four semantic types. The rules apply in the order in which they are listed.

1. (1) If an argument is a named entity (NE) of type Location, it is assigned the location type; if it is a temporal expression, it is considered to be a temporal argument of the event.

2. (2) If an argument is a named entity of type Person or Organization and the preposition between the anchor and the argument is ‘by’, then the argument is an agent. If the preposition is ‘against’, then the argument is a target.

3. (3) If the prepositional object is introduced by a temporal preposition (‘before’, ‘after’, ‘during’, ‘since’, and ‘until’), the argument is considered to be of temporal type. If it is introduced by a locative preposition (‘above’, ‘below’, ‘behind’, ‘inside’, ‘around’, ‘along’, ‘across’, ‘near’, and ‘through’), the argument is a location of the event. As noted by Derczynski and Gaizauskas (Reference Derczynski and Gaizauskas2013), some temporal signals – some of which we use here, such as ‘before’ and ‘near’ – are ambiguous and can have both temporal and spatial reading. However, our choice of temporal and locational signals is rather pragmatic – we choose the prepositions based on the number of times they occur in the corpus (cf. Section 5.2) as locative and temporal signals. For example, in our development set, ‘before’ has never occurred as a locational signal, nor has ‘near’ occurred as a temporal signal. We leave a more systematic treatment of this issue for the future work.

If none of the previous rules apply, we compare the semantic similarity of the argument’s head word against three sets of concepts from WordNet. The sets T, L, and G represent temporal, locative, and general (non-locative and non-temporal) concepts respectively:

\begin{eqnarray*} L &=& \lbrace {\it location}, {\it geographical\_area}, {\it vehicle}, {\it building}, {\it transport}, {\it facility}\rbrace \\ T &=& \lbrace {\it time}, {\it duration}, {\it time\_unit}, {\it clock\_time}, {\it time\_period}\rbrace \\ G &=& \lbrace {\it object}, {\it concept}, {\it person}, {\it event}\rbrace \end{eqnarray*}

We defined the sets of locative and temporal concepts by identifying the most common WordNet hypernyms of the temporal and locative argument heads, manually annotated on the development set. For example, we observed that vehicles and facilities often make locative event arguments (e.g., ‘in the car’, ‘at the discotheque’). We defined the concept set G by selecting the most general non-temporal and non-locative concepts from WordNet.

The semantic similarity between a word w and a concept set S is the maximum of the Wu and Palmer (Reference Wu and Palmer1994) similarity between w and the concepts from S:

\begin{equation*} {\it ssim}(w,S) = \max _{s \in S}({\it wupalmer}(w,s)) \end{equation*}

The semantic type of the argument head word w is then determined based on its semantic similarity to the concept sets L, T, and G. The argument is considered locative if ssim(w, L) > ssim(w, T) and ssim(w, L) > ssim(w, G). Similarly, the argument is considered temporal if ssim(w, T) > ssim(w, L) and ssim(w, T) > ssim(w, G).

3.4 Relation extraction

In this work, we restrict our extraction to two main types of relations between events: temporal relations and event coreference relations. We use supervised machine learning models for both tasks, which we describe below.

3.4.1 Temporal relations

Most of the related approaches to temporal relation extraction are restricted in one of two ways: they either consider a subset of the possible event pairs (e.g., salient events of adjacent sentences or events from the same sentence in which one syntactically dominates the other) (Bethard Reference Bethard2008; Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010) or they assume a partial temporal ordering between events and represent the temporal structure as trees or directed acyclic graphs (Bramsen et al. Reference Bramsen, Deshpande, Lee and Barzilay2006; Kolomiyets et al. Reference Kolomiyets, Bethard and Moens2012). Both restrictions may easily lead to some temporal information remaining uncovered. Syntactic and discourse constraints narrow down the set of pairs for which temporal relations can be determined. Furthermore, imposing partial ordering implies that every temporal relation is reflexive, transitive, and antisymmetric. Since this assumption does not hold for every temporal relation of interest (specifically, overlap is not transitive, whereas equal is not antisymmetric), it can lead to spurious relations being extracted (e.g., spurious overlap relation due to incorrectly assumed transitivity).

In contrast, there is a rationale for restricting the pairs of events to be considered: determining the temporal relations between very distant pairs of event mentions in a document is impractical due to the large span of text between them (the information that needs to be ‘understood’ to determine the relation). Consequently, the effort required to annotate the temporal relations between all pairs of event mentions is simply too large. Moreover, due to discourse coherence (Wolf and Gibson Reference Wolf and Gibson2005) of narratives, temporally related event mentions tend to occur close to each other. Thus, we chose to consider all pairs of events from the same sentence and adjacent sentences. However, we make no further restrictions by assuming certain syntactic relations between events or by assuming partial temporal ordering.

Our temporal relation extraction procedure consists of two steps. In the first step we identify pairs of event mentions between which temporal relation can be established. In the second step we classify the type of relations identified as establishable in the first step. Aiming for robustness rather than fine-grained temporal semantics, we differentiate between four coarse-grained types of temporal relations – before, after, overlap, and equal. It has been observed that distinguishing between semantically fine-grained temporal relations is hard and that is why previous evaluations have also employed coarse-grained temporal relation types (Verhagen et al. Reference Verhagen, Gaizauskas, Schilder, Hepple, Katz and Pustejovsky2007Reference Verhagen, Sauri, Caselli and Pustejovsky2010). The mapping between the TimeML relation types and our coarse-grained types is shown in Table 2. We use the same rich set of features to both identify and classify temporal relations.

• • Positional features measure the distance between and relative position of two event anchors. They include a feature indicating whether the event anchors are in the same sentence, a feature indicating whether the anchors are in adjacent sentences, a feature indicating whether the events are adjacent (no other anchor is in between), and the distance between anchors (in number of tokens).

• • Lexical features include the word, lemma, stem, and POS of both event anchors. In addition, we use a feature indicating whether the word forms of event anchors are the same, a numerical feature indicating the semantic similarity of the anchors (measured on WordNet using the Wu and Palmer’s algorithm (Reference Wu and Palmer1994)), and the bag-of-words (BoW) between the two anchors (Bethard Reference Bethard2008). Similar basic lexical and POS features have been proposed in previous works (Bethard Reference Bethard2008; Llorens et al. Reference Llorens, Saquete and Navarro2010; UzZaman and Allen Reference UzZaman and Allen2010).

• • Syntactic features are computed only for event pairs from the same sentence and include the syntactic path between the anchors (concatenation of the dependency relation labels on the path between the anchors in the dependency tree) and binary features indicating whether the first event anchor syntactically dominates the second and vice versa. Similar features for capturing the syntactic path between the events have been proposed previously by Llorens et al. (Reference Llorens, Saquete and Navarro2010). In additionally, we use the adverbial complement marks of both anchors as features. An adverbial complement mark is the adverbial modifier of an event anchor that is in an adverbial complement relation to another event anchor (i.e., one of the events is a predicate of an adverbial clause governed by the other event), which often relates to a heading preposition feature proposed by Llorens et al. (Reference Llorens, Saquete and Navarro2010). For example, in ‘The storm began before Stallion rounded South Farallon Island’, the preposition ‘before’ is the adverbial complement mark of the anchor ‘rounded’.

• • Modifier features include all features that describe the modal, auxiliary, negation, and determination modifiers for both event anchors. These are the same features used for anchor extraction models (cf. Section 3.2).

Due to the large number of features (approximately 200K), for temporal relation extraction, we also chose to use linear discriminative models (logistic regression and linear SVM) using the LibLinear implementation (Fan et al. Reference Fan, Chang, Hsieh, Wang and Lin2008).

Table 2.

Temporal relation types mapping

4.1 Tensor product for labeled graphs

We use the tensor product of graphs as a mechanism to capture exact overlaps between the constructed event graphs and reference event graphs. A tensor product for the labeled graphs (Borgwardt Reference Borgwardt2007; Hammack et al. Reference Hammack, Imrich and Klavžar2011) is a binary operation that takes two labeled graphs G and G’ as input and results in a product graph, denoted as G × G’, with vertices from the set V(G) × V(G’).

Let $\mathcal {L}(v)$ and $\mathcal {L}(v_1, v_2)$ denote the labels of vertex v and edge (v ₁, v ₂) respectively. The vertices V(G × G’) and edges E(G × G’) of the product graph are defined as follows. The set of vertices comprises all pairs of vertices from G and G’ whose labels match:

\begin{equation*} V(G\times G^{\prime }) = \lbrace (v, v^{\prime }) \mid v \in V(G), v^{\prime } \in V(G^{\prime }), \mathcal {L}(v) = \mathcal {L}(v^{\prime }) \rbrace \end{equation*}

Analogously, the set of edges comprises all edges between pairs of matching vertices that are equally labeled in both input graphs:

\begin{align*} E(G\times G^{\prime }) = \big \lbrace & \big ((v_1, v^{\prime }_1), (v_2, v^{\prime }_2)\big ) \in V(G\times G^{\prime }) \times V(G\times G^{\prime }) \mid \\ & (v_1, v_2) \in E(G), (v^{\prime }_1, v^{\prime }_2) \in E(G^{\prime }), \mathcal {L}(v_1, v_2) = \mathcal {L}(v^{\prime }_1, v^{\prime }_2)\big \rbrace \end{align*}

We apply the above definition of the tensor product of graphs to the automatically extracted graph G_C = (V, E, A, m, r) and reference graph G_R = (V’, E’, A’, m’, r’). Since we wish to identify the exact overlap between G_C and G_R, we require a strict match between vertex labels and edge labels of the two graphs. The matching function is defined as follows:

1. (1) The labels of vertices v ∈ V(G_C) and v’ ∈ V(G_R) match, denoted $\mathcal {L}(v) = \mathcal {L}(v^{\prime })$, iff the anchors and all arguments of event mentions m(v) and m’(v’) match. The anchors of m(v) and m’(v’) match iff they point to the same token from text and have the same event type (e.g., Occurrence). Sets of arguments of m(v) and m’(v’) match iff they are identical, i.e., have the same head word and the same type.

2. (2) The labels of edges (v ₁, v ₂) ∈ E(G_C)∪A(G_C) and (v’₁, v’₂) ∈ E(G_R)∪A(G_R) match iff the corresponding relations are identical, i.e., $\mathcal {L}(v_1, v_2) = \mathcal {L}(v^{\prime }_1, v^{\prime }_2) \Leftrightarrow r(v_1, v_2) = r^{\prime }(v^{\prime }_1, v^{\prime }_2)$.

4.2 Evaluation metrics

The tensor product of graphs measures the exact overlaps between G_C and G_R; intuitively, the product graph will be larger as G_C becomes more similar to G_R. Thus, we define two measures for the overall quality of an event graph: relative overlap size (ROS) metric and largest connected component (LCC) metric. The ROS metric measures the size of the overlap between the constructed graph and the reference graph regardless of the structure of that overlap, whereas the LCC metric accounts for the structure of the overlap by considering the largest connected components of event graphs. In the context of news stories, ROS measures the ability of the system to capture all event-oriented information regardless of whether it belongs to the main narrative or to the side narratives of the story, whereas LCC measures the ability of the system to capture the structure of the main narrative (assuming that the largest connected component of the event graph corresponds to the main narrative of the news story). Both metrics abide by the following requirements:

1. (1) The scores assigned by the metric reflect the overall quality of the automatically constructed event graph well.

2. (2) The metric is easily interpretable and can be related to the standard precision and recall metrics.

To satisfy the latter, the metrics were designed to penalize for information from G_R missing in G_C and spurious information in G_C not existing in G_R.

Relative overlap size. This metric measures the relative size of the product graph G_C × G_R compared with both G_C and G_R. The size of a graph is computed as the sum of number of vertices and number of edges, i.e., |V(G)| + |E(G)|. We define the relative overlap size in terms of ROS-based precision and recall defined as follows:

(1) \begin{align} {\it ROS}_P(G_C, G_R) &= \frac{|V(G_C\times G_R)| + |E(G_C\times G_R)|}{|V(G_C)| + |E(G_C)|}, \end{align}

(2) \begin{align} {\it ROS}_R(G_C, G_R) &= \frac{|V(G_C\times G_R)| + |E(G_C\times G_R)|}{|V(G_R)| + |E(G_R)|} \end{align}

(3) \begin{equation} {\it ROS}(G_C, G_R) = 2 \cdot \frac{{\it ROS}_{P}(G_C, G_R) \cdot {\it ROS}_{R}(G_C, G_R)}{{\it ROS}_{P}(G_C, G_R) + {\it ROS}_{R}(G_C, G_R)} \end{equation}

Largest connected component. This metric considers the structure of the overlap between the graphs. It first computes the tensor product between the first graph and the largest weakly connected component of the second graph, then compares the size of the largest weakly connected component of that product with the size of the largest weakly connected component of the second graph. Let lcc(G) denote the largest weakly connected component of graph G. We define the metric in terms of LCC-based precision and recall defined as follows:

(4) \begin{align} {\it LCC}_{P}(G_C, G_R) &= \frac{\big |V\big ({\it lcc}(G_R \times lcc(G_C))\big )\big |}{|V({\it lcc}(G_C))|} & \end{align}

(5) \begin{align} {\it LCC}_{R}(G_C, G_R) &= \frac{\big |V\big ({\it lcc}(G_C \times lcc(G_R))\big )\big |}{|V({\it lcc}(G_R))|} \end{align}

(6) \begin{equation} LCC(G_C, G_R) = 2 \cdot \frac{{\it LCC}_{{\it P}}(G_C, G_R) \cdot {\it LCC}_{{\it R}}(G_C, G_R)}{{\it LCC}_{{\it P}}(G_C, G_R) + {\it LCC}_{{\it R}}(G_C, G_R)} \end{equation}

The intuition behind the LCC metric is that a narrative in a story (e.g., a news story) will be informationally coherent and will result in a connected event graph. If the main story is accompanied by additional stories (sidebar stories), it is likely that the event graphs of the sidebar stories will not be connected to the event graph of the main story (i.e., the temporal relations between the anchors of different narratives will be unidentifiable). This phenomenon is also observed in most gold graphs and automatically constructed event graphs in our corpus (cf. Section 5.3), where the ratio between the size (in the number of nodes) of the largest connected component and the whole graph is, on average, 0.83. Accordingly, we assume that the largest connected component of the event graph corresponds to the main narrative of the story. By comparing only the largest connected components, we emphasize the overlap between the main narratives and diminish the influence of the side narratives.

As an example of how event graph evaluation metrics are computed, consider the construction of an event graph from the story snippet given in Figure 2 (cf. Section 3). Let the graph given in Figure 2 be the reference graph G_R and the graph in Figure 3 be the automatically constructed graph G_C. The graph construction process omitted one reference event mention (‘competing’) and also falsely extracted one event mention (‘members’). Figure 4 presents the tensor product G_C × G_R of graphs G_C and G_R. The product graph contains six event mentions from G_R for which the anchor and all arguments have been correctly extracted in G_C and three correctly extracted relations from G_R. The largest weakly connected component in the product graph contains the vertices ‘pushed’, ‘swept’, and ‘rounded’. Note that both G_C and G_R in this example are weakly connected graphs (i.e., lcc(G_C) = G_C and lcc(G_R) = G_R), and hence the product is the same for both recall and precision components of LCC (i.e., lcc(G_C) × G_R = lcc(G_R) × G_C = G_C × G_R). The number of vertices and edges in the three graphs and their largest connected components are as follows:

\begin{align*} |V(G_R)|&=7 & |V(G_C)|&=7 & |V(G_C\times G_R)|&=6 \\ |E(G_R)|&=8 & |E(G_C)|&=6 & |E(G_C\times G_R)|&=3 \\ |V({\it lcc(G_R))}|&=7 & |V({\it lcc(G_C))}|&=7 & |V({\it lcc(G_C\times G_R))}|&=3 \end{align*}

Substituting these values into (1)–(6) gives the ROS and LCC scores:

\begin{align*} {\it ROS}_{{\it P}}(G_C, G_R) &\approx 0.69 & {\it LCC}_{{\it P}}(G_C, G_R) &\approx 0.43 \\ {\it ROS}_{{\it R}}(G_C, G_R) &\approx 0.60 & {\it LCC}_{{\it R}}(G_C, G_R) &\approx 0.43 \\ {\it ROS}(G_C, G_R) &\approx 0.64 & {\it LCC}(G_C, G_R) &\approx 0.43 \end{align*}

Fig. 3.

Example of an automatically constructed event graph.

Fig. 4.

Example of the tensor graph product between the reference event graph (Figure 2) and automatically constructed event graph (Figure 3).

4.3 Validation of evaluation metrics

For ROS and LCC to be useful, they must have some desirable properties. We demonstrate that both ROS and LCC have properties of standard metrics. Both measures are harmonic means of components that relate to standard precision and recall metrics used in numerous NLP tasks. We demonstrate that the precision component of both ROS and LCC decreases with an increase of spurious information in automatically extracted event graphs. Analogously, the recall component decreases with an increase in missing information. We also demonstrate that decrease in performance is caused by errors in the extraction of any event graph component (anchors, arguments, and relations).

Following the idea of UzZaman and Allen (Reference UzZaman and Allen2011), we simulate the construction of an event graph G_C by randomly adding and removing information from the reference graph G_R, and then computing the ROS and LCC scores on the product graph G_P of the modified graph G_C and the original reference graph G_R. The scores are averaged over 105 reference graphs that are manually created from the set of 105 news stories (cf. Section 6.4). On average, reference graphs have 42 vertices, 62 edges, and two weakly connected components. The largest weakly connected component has an average of 39 vertices. The results are shown in Figure 5.

Fig. 5.

Validation of ROS and LCC evaluation metrics: examination of metrics behavior when omitting true information ((a) anchors, (c) relations, and (e) arguments) or adding spurious information ((b) anchors, (d) relations, and (f) arguments).

We first remove the vertices (with their corresponding edges) from the reference graphs to simulate the case in which some event anchors are not extracted. We observe (Figure 5(a)) that for both ROS and LCC, the recall is inversely proportional to the number of removed vertices. Next, we add vertices to the reference graphs to simulate the case in which some false event anchors are extracted. Since the average degree of vertices in reference graphs equals approximately one, for each added vertex we also randomly add one relation. We observe (Figure 5(b)) that for both ROS and LCC, the precision is inversely proportional to the number of added vertices. Similarly, we observe that precision and recall decrease with the addition (Figure 5(c)) and removal (Figure 5(d)) of relations respectively. However, the precision drop for the LCC score becomes negligible after adding a small number of relations because many of the reference graphs G_R in our experiments are close to being connected. Thus, when deriving G_C from G_R by adding a small number of spurious relations, we quickly obtain a connected graph. Consequently, adding relations does not increase the size of the largest connected component of G_C. However, in a realistic setting, extracted graphs will be more fragmented and some spurious relations may significantly increase the size of the largest connected component and thus decrease the LCC precision score. Finally, we demonstrate that the removal and addition of event arguments (Figures 5(e) and (f)) also decrease ROS and LCC scores. Given that the vertex in the tensor product G_P is created only for vertices from G_C and G_R that match in all arguments, both removal of existing arguments and addition of spurious arguments affect both precision and recall. In other words, the vertex of G_C to which a false argument has been added or from which a true argument has been removed will not match any vertices in the reference graph G_R and thus will not appear in the graph product G_P.

5.1 Anchors annotation

Anchor annotation on the EvExtra corpus was conducted by six annotators. All annotators initially annotated a calibration set consisting of 10K tokens, after which the disagreements were analyzed and resolved by consensus. Out of 330K tokens, a common subset of 30K tokens was annotated independently by each of the six annotators. On this subset, we measured the inter-annotator agreement (IAA) as an pairwise-averaged F-score (Table 4). The IAA was computed after calibration and before resolving the disagreements. The remainder of the corpus (300K tokens) was split into six portions; each portion was independently annotated by one pair of annotators, and the third annotator resolved the disagreements. A comparison between the TimeBank corpus and the EvExtra corpus reveals that EvExtra is approximately three times larger than TimeBank (Table 3) in terms of annotated event anchors.Footnote ⁵

Table 3.

Comparison of event-annotated datasets

Table 4.

Inter-annotator agreement for factual event anchors

While annotators achieved high agreement on event anchor identification, they achieved modest agreement on event type classification (except for the Reporting class) even after two rounds of calibration. This level of agreement indicates that determining event type is a challenging task even for humans. The agreement was particularly low for Perception and State change because annotators frequently confused these two classes with the more frequent Occurrence class. For instance, in ‘The child fell and scratched its knee’, the anchor ‘fell’ is within the Occurrence class, whereas in ‘The stocks fell 3% in early trading’ it is within the State change class. Nonetheless, we believe that the resolving of disagreements substantially improves the quality of annotations, which is also indicated by the performance of the supervised model (cf. Section 6.1).

The IAA we obtained for anchor identification on the EvExtra corpus (0.87 F-score) is much higher than the agreement on the TimeBank corpus (0.78 P&R avg.),Footnote ⁶ especially given that on TimeBank, IAA is measured as an average of precision and recall, which is a less conservative estimate than the harmonic mean computed by the F-score. Moreover, we believe that our estimate of the human agreement is more reliable as it has been averaged over 15 pairs (all combinations of two out of six annotators), whereas the TimeBank IAA is measured from a single pair of annotators. This result seems to suggest that annotating non-factual events is more demanding for humans than annotating factual events.

5.2 Arguments annotation

EvExtra corpus contains 105 articles manually annotated with event arguments to enable the development of argument extraction model. Two annotators first independently annotated 25 articles, after which the disagreements were resolved by consensus. Each annotator then separately annotated another 40 news articles. We used the resolved set of 25 documents as the development set for our rule-based argument extraction, and the remaining 80 documents as the test set. The annotators were instructed to annotate four types of event arguments: agents, targets, locations, and temporal arguments. In line with the guidelines for annotating mentions in entity coreference resolution (Pradhan et al. Reference Pradhan, Ramshaw, Marcus, Palmer, Weischedel and Xue2011), we instructed the annotators to annotate the largest possible extent for each event argument. For example, in ‘The vessel Low Speed Chase, a 12 m racing sailboat with a crew of eight, competed on Saturday...’, the phrase ‘The vessel Low Speed Chase, a 12 m racing sailboat with a crew of eight’ would be annotated as the agent of the mention ‘competed’. Table 5 provides the statistics of the dataset, including the average length of argument expressions, broken down by argument type. The total average length of argument expressions is three tokens.

Table 5.

Statistics of the argument extraction dataset

Table 6 presents the IAA measured in terms of the F-score on the 25 documents annotated by both annotators. We measured the agreement on the argument head words (allows for extent mismatches as long as the head words match) and the strict agreement on the full argument (exact match of full argument extent). We believe that the average head word agreement of 86% is a fair agreement. As expected, mismatches in the extent annotation occur somewhat more frequently.

Table 6.

Inter-annotator agreement for argument annotation

5.3 Annotation of event graphs

In order to evaluate automatically extracted event graph using the proposed ROS and LCC evaluation metrics, we first need to create a set of reference graphs. To this end, we had two annotators annotate reference graphs for 105 EvExtra news articles.Footnote ⁷ Both annotators independently annotated 25 documents, and then each annotator annotated a separate set of 40 documents. For final evaluation, we use reference graphs from 80 documents; graphs from the remaining 25 documents annotated by both annotators were not used for evaluation because they were already used as a development set for the argument extraction model. The manual annotation of reference graphs mimicked the stages of the construction pipeline – annotation of anchors, arguments, temporal relations, and event coreference. We measured the inter-annotator agreement on the set of 25 articles annotated by both annotators using the LCC and ROS metrics. The observed agreement is 0.47 ± 0.13 of LCC and 0.69 ± 0.08 of ROS, which we consider to be in reasonable agreement.

6.1 Anchor extraction

We evaluate the performance of our supervised models for both anchor extraction tasks: anchor identification and event type classification. We select 70% of articles from the EvExtra corpus as the training set and leave the remaining 30% for testing. Here and henceforth (for all supervised models), we use grid search and 10-fold cross-validation to optimize the model hyperparameters and report the performance of the optimal model on the test set. All results were obtained with the logistic regression model, which we found to slightly outperform the linear SVM in all experiments. As a competitive baseline, we use the memorize model proposed by Bethard (Reference Bethard2008). The baseline model predicts for each word the most frequent class assigned to that word on the training set. If the word was never observed in the training set, it is assigned the most frequent class in the training set: class Not event and class Occurrence for the anchor identification and event type classification task respectively.

Table 7 shows the performance of anchor extraction on the EvExtra corpus. The upper part of the table shows the performance of the model (EvGraph-Anchors) on the anchor identification task. Our model substantially outperforms the memorize baseline. The overall anchor identification performance is close to the IAA scores, suggesting that the performance cannot be substantially improved with the current annotation quality.

Table 7.

Anchor extraction and classification performance on the EvExtra corpus

Occ.: Occurrence; IAct.: I_Action; Per.: Perception; Rep.: Reporting; SCh.: State change.

The lower part of Table 7 shows the performance on the event type classification task. The logistic regression model again outperforms the competitive baseline, this time by a wider margin. The performance gain is most discernible for the classes with the least number of instances in the training set – Perception and State change. We obtain the best results for the Reporting class, which is expected because reporting events exhibit little lexical variation in news stories (i.e., only a few verbs such as ‘say’, ‘report’, ‘tell’, and ‘add’ account for the majority of reporting events). The performance of the model is above the IAA scores for all classes, indicating that resolving annotator disagreements (which took place after measuring the IAA score and before model training) considerably improved the corpus quality. This improvement is especially notable for small classes (such as the Perception class), where the adjudication of only a handful of examples leads to large improvements in the annotation consistency and, consequently, improvements in the classifier performance.

In contrast to previous approaches based on the TimeBank corpus, we do not extract non-factual event mentions or mentions describing states. However, for the sake of comparison with the state-of-the-art models, we evaluate our model on the TimeBank corpus. We perform two experiments: in the first experiment, we both train and test our model on the TimeBank corpus; we refer to this model as the EvGraph-Anchors-TimeBank model. In the second experiment, we train the model on the EvExtra corpus but test it on the TimeBank corpus (EvGraph-Anchors model). The results are shown in Table 8, along with the state-of-the-art results from TempEval-2 (Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010). Our anchor identification and classification models achieve a performance comparable with state of the art, even though the features that aim to distinguish between factual and non-factual events have little impact when used on the TimeBank corpus. On anchor identification task, the EvGraph-Anchors-TimeBank model has a significantly better precision than the best performing model at TempEval-2 (Llorens et al. Reference Llorens, Saquete and Navarro2010). However, our model has a lower recall, presumably because it lacks the features specifically designed to recognize non-factual events and states. The EvGraph-Anchors model (trained on the EvExtra corpus) also has good precision (as all factual events are events) but significantly lower recall (as the model fails to recognize states and non-factual event anchors because such anchors were not present in the training set).

Table 8.

Performance of the anchor identification and classification models on the TimeBank corpus Occ.: Occurrence; ISt.: I_State; St: State; Rep.: Reporting; Per.: Perception; IAc.: I_Action; As.: Aspectual

Given that the EvExtra corpus is approximately three times larger than TimeBank, it is interesting to see how the size of the training set affects the performance on both tasks. To this end, we trained models on 25%, 50%, 75%, and 100% of the training set. Table 9 presents the performance of the four models on the test set. The performance improved considerably when the size of the training set was doubled from 25% to 50% of the full size. However, with a further increase in the training set size, the performance gains diminish. Thus, we therefore doubt that substantial performance gains can be obtained from annotating more data; improving the annotation quality would result in larger gains.

Table 9.

Influence of the training set size on performance

A closer inspection of the anchor identification misclassifications reveals that most errors relate to one of the following causes:

1. (1) Noun event anchors that are most often mentioned generically (e.g., violence, war) are not extracted even when referring to concrete events (e.g., ‘at least 41 people, mostly civilians, have been killed in violence that prompted UN...’). Such cases result in false negatives.

2. (2) Verbs in the infinitive form may refer to both something that actually occurred (e.g., ‘Violence forced the Arab League to end its own Syrian monitoring mission...’) and something that did not occur in the real world (e.g., ‘She underlined that the United States want to see continuing progress’). The classifier is often unable to distinguish between the two cases, producing both false positives and false negatives.

3. (3) Non-factual mentions occasionally have distant and implicit indicators of factuality (e.g., ‘That raises the possibility that its fighters are infiltrating across the border’). In such cases the model does not recognize that the anchors are not factual, which results in false positives.

6.2 Argument extraction

For the argument extraction task, we compared our rule-based system with the baseline method that extracts all nominal subjects of an event anchor as agents, all direct objects as targets, all locational named entities in the vicinity of the anchor as locative arguments, and all temporal expressions in the vicinity of the anchors as temporal arguments. In Table 10, we present the performance of the argument extraction method (EvGraph-Args) and the baseline on the test set of 80 manually annotated documents, broken down by argument type. The results are given separately for the argument heads (columns 3–5) and argument extent recognition (columns 6–8). In the latter case, only strict matches of the full argument extent are considered to be correct.

Table 10.

Argument extraction performance

The results indicate that our extraction rules are suitable for the recognition of argument heads, whereas the chunk-based detection of argument extent is moderately successful in identifying the full extent of arguments. This moderate performance is mostly due to the fact that arguments often extends beyond the boundaries of the chunk containing its head word, as extracted by the employed OpenNLP chunker. For example, in ‘The vessel Low Speed Chase, a 12 m racing sailboat with a crew of eight, competed on Saturday...’, the chunk of the argument head ‘Chase’ is ‘The vessel Low Speed Chase’, whereas the full extent of the argument is ‘The vessel Low Speed Chase, a 12 m racing sailboat with a crew of eight’. We could have resolved this mismatch by instructing the annotators to annotate argument extents as chunks provided by the employed chunker. However, this step would introduce a bias toward using chunking to identify argument extent, as well as a bias toward the specific chunker used. The fact that we use strict matching also lowers the results.

Overall, the EvGraph-Args model outperforms the baseline by a wide margin, justifying the use of a rich set of syntactic patterns and argument type disambiguation rules. The baseline model is better in terms of precision but is much worse in terms of recall and, consequently, F-score. This is expected considering that we optimized the argument extraction rules for the F-score. Performance on the locative arguments is substantially worse than on temporal arguments because while most temporal arguments are temporal expressions, recognized as such by the temporal expression tagger, the majority of locative arguments are not locational named entities. Thus, to identify locative arguments in such cases, we rely on measuring the WordNet-based semantic similarity with the locative concepts (cf. Section 3.3.2), which does not appear to be particularly robust.

As our argument extraction approach is primarily based on syntactic patterns, most extraction errors are caused by parsing errors. A single incorrect sentence parse may trigger several argument extraction errors.

6.3 Relation extraction

In this section, we describe the experiments on the extraction of temporal relations and within-document event coreference relations, i.e., relations forming the edges of an event graph.

6.3.1 Temporal relation extraction

We conducted all experiments on the combined TimeBank/AQUAINT corpus used for TempEval-2 (Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010). The dataset consists of 256 documents and 3,721 temporal relations annotated between events, 3,190 of which are between events from the same or adjacent sentences. We selected 180 documents for training and the remaining 76 documents for testing. To avoid including post hoc temporal knowledge in training, we ensured that all documents from the training set were created before the documents from the test set. Furthermore, some of the documents from the AQUAINT portion of the dataset contain passages that are verbatim (or almost verbatim) copies of passages from earlier created documents. To avoid having the same passages in both training and test sets, we eliminated all duplicate passages from the AQUAINT portion of the corpus.

In the TimeBank/AQUAINT corpus only the temporal relations inferable from text were annotated. Since we perform temporal relation extraction in two steps (relation identification followed by relation classification), we first needed to generate negative training instances for the relation identification task. We used all pairs of events from adjacent sentences for which no temporal relation was annotated in the dataset as negative instances.

The first part of Table 11 presents the performance of the model (EvGraph-TRels-Ident) on the temporal relation identification task. As the baseline, we use a model that connects adjacent event mentions (i.e., predicts that temporal relations are identifiable for all pairs of adjacent event mentions). Our model outperforms the baseline by a very wide margin, with balanced precision and recall.

Table 11.

Results for temporal relation identification and classification

In the second part of Table 11 we present the performance of the temporal relation classification model (EvGraph-TRels-Class). Here we use two baselines: a majority class baseline and a simple tense-based baseline. The majority class baseline predicts the most frequent class on the training set (Before class). The tense-based baseline predicts the temporal relation based on the tense and relative position of event anchors as follows: (1) An event anchor in the past tense is considered to occur before an event anchor in the present tense; (2) two event anchors in the same tense are considered to be temporally equal if they occur in the same sentence; (3) in all other cases, anchors are considered to be in temporal overlap. Our model outperforms both baselines by a very wide margin. The model performs worst on the Overlap class. This performance can be attributed to three possible causes: (1) Since most misclassifications occur between Overlap and Before instances as well as Overlap and After instances, we hypothesize that temporal overlap is less discernible from a text than other classes of temporal relations; (2) Overlap instances occur rarely in the training set; and (3) Overlap class as defined here includes several of Allen’s (Reference Allen1983) basic temporal relations (X overlaps Y, X starts Y, X finishes Y, and X during Y), and these basic relations may manifest themselves differently in text. While the first two issues could be remedied by compiling a balanced training set with the most discernible examples for the Overlap class, the third issue would require a more detailed investigation, which we leave for the future work.

Our results cannot be directly compared with the results from TempEval-2 (Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010) because we use a different set of labels (adding the Equal class) and consider different pairs of events (all pairs of events from adjacent sentences). To enable a direct comparison with the state-of-the-art approaches for the extraction of temporal relations, we merge temporal equality and temporal overlap classes and consider only pairs of events where (1) both event mentions represent main events from two consecutive sentences (as in TempEval-2 Task E), or (2) one event mention syntactically dominates the other (as in TempEval-2 Task F). The performance of our model in this setting (EvGraph-TRels-TimeBank), compared with the best performing models from TempEval-2 (Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010), is shown in Table 12. The results confirm that the EvGraph-TRels-TimeBank model performs comparably to the state of the art.

Table 12.

Temporal relation classification in the TempEval-2 setting

6.3.2 Event coreference

We perform the event coreference resolution experiments on the EventCorefBank dataset from Bejan and Harabagiu (Reference Bejan and Harabagiu2008). EventCorefBank is annotated for both within- and cross-document event coreference. Since we build event graphs from the individual documents, here we only consider within-document event coreference resolution. We split the dataset into training and testing set (80%–20% split ratio). We employ the commonly used event coreference baseline (Ahn Reference Ahn2006; Bejan and Harabagiu Reference Bejan and Harabagiu2010), which declares that two event mentions corefer if their anchor lemmas match or if one lemma is the nominalization of the other. This baseline is simple but competitive.

Table 13 presents the performance of our supervised model (EvGraph-Coref). Our model outperforms both baseline and unsupervised non-parametric Bayesian model (Bejan and Harabagiu Reference Bejan and Harabagiu2010) by a wide margin. Although direct comparison with other supervised event coreference resolution systems is not feasible due to differences in the datasets, the reported results (Chen et al. Reference Chen, Su, Pan and Tan2011; Lee et al. Reference Lee, Recasens, Chang, Surdeanu and Jurafsky2012) suggest that our mention comparison features achieve performance comparable with the state of the art. For all models, the recall is much lower than the precision. Closer inspection of the false negatives reveals two main causes for this result. First, in many cases the arguments of at least one of the mentions are implicit. For example, in ‘Jackson was arrested on DUI charge early yesterday...General Manager A.J. Smith issued the following statement on the arrest...’ the arguments of the second mention (arrest’) are implicit. In such cases the usefulness of features measuring the similarity between event arguments is diminished. Second, in some cases the entity coreference resolution from Lee et al. (Reference Lee, Peirsman, Chang, Chambers, Surdeanu and Jurafsky2011) fails to recognize matches between mention arguments. For example, in ‘Matt Smith has been cast as the next incarnation of the Doctor. Even the skeptics agreed that the right guy was chosen...’, entities ‘Matt Smith’ and ‘the right guy’ are not resolved as coreferent. Consequently, the match between the targets of mentions ‘cast’ and ‘chosen’ is not recognized.

Table 13.

Within-document event coreference resolution performance

6.4 Event graph evaluation

Finally, we evaluate the overall quality of the automatically extracted event graphs using the evaluation metrics proposed in Section 4.

6.4.1 Results

To compare our three-stage pipeline against a reasonable baseline, we use as a baseline system a combination of baselines from each pipeline stage. More precisely, the baseline pipeline consists of (1) the competitive memorize baseline for anchor extraction (cf. Section 6.1), (2) the rule-based baseline for argument extraction (cf. Section 6.2), (3) the tense-based baseline for temporal relations (cf. Section 6.3.1), and (4) the lemma-matching baseline for event coreference resolution (cf. Section 6.3.2). Table 14 presents the overall performance of the graph construction pipeline (EvGraph) along with the performance of the baseline system. We report the mean and standard deviation of ROS and LCC over the set of 80 documents. The results in Table 14 were obtained without applying transitive closure on the event graph. Applying transitive closure yielded consistently poorer results than when no transitive closure was applied. This result is not surprising because the temporal classification performance – although comparable to the state of the art – is only at about 60% of F1-score, and because transitive closure may induce several spurious relations due to a single incorrectly classified relation.

Table 14.

Overall quality of the automatically constructed event graphs

The full graph construction pipeline significantly outperforms the baseline. The differences are significant at p < 0.05 for LCC and p < 0.01 for ROS (Student’s two-tailed t-test). The obtained results indicate that the LCC metric is stricter than the ROS metric.

6.4.2 Headroom analysis

Our graph construction pipeline consists of multiple interdependent components. Thus, it is interesting to see how the performance of the individual components affects the performance of the entire system. To this end, we perform a headroom analysis of each component, which demonstrates the potential impact on overall system performance if the component in question would exhibit perfect performance. Although the so-obtained improvement estimates are overly optimistic, they are useful for comparing the potential benefits of enhancing different components in a resource-bounded setting. We perform the headroom analysis for the following components: (1) anchor extraction, (2) argument extraction, (3) temporal relation extraction, and (4) coreference resolution. In headroom analysis, the construction of an event graph for the component under test uses the gold standard annotations for that component and the automated output for all other components. For example, to construct an event graph for headroom analysis of the temporal relation extraction component, we automatically extract the anchors and arguments, copy temporal relations from the reference graph, and then automatically resolve event mention coreference.

In Table 15 we present the results of the headroom analysis. The results demonstrate that perfect temporal relation extraction would yield the largest improvements with respect to both evaluation metrics. At first, this result appears to contradict our earlier results, which indicated that in absolute terms, coreference resolution performs worse than temporal relation extraction (cf. Section 6.3). However, the number of temporal relations in the reference graphs is much larger than the number of coreferent event mention pairs, and thus the performance on temporal relation extraction has a greater effect on the overall event graph quality. Furthermore, the performance improvement is larger for the LCC metric (0.477 versus 0.225) than for the ROS metric (0.437 versus 0.287): A single correctly extracted relation can considerably increase the size of the largest connected component, and thus the LCC score.

Table 15.

Results of the headroom analysis (gold standard annotations for one of the pipeline steps)