1 Introduction
In the last 25 years, several systems performing event extraction have been presented within the NLP community. Diverse approaches aimed at building timelines from large document collections have been implemented, and technologies to support automatic storytelling have become a relevant research topic in the AI community (Ashish et al. Reference Ashish, Appelt, Freitag and Zelenko2006). Event processing has been addressed from a variety of perspectives, from data visualization to knowledge representation and modelling. However, the notion of event has been revised several times and often tailored to the task of interest, so that a number of different definitions of event has been introduced since the first MUC evaluation campaign. Furthermore, the notion of event has been studied also in other disciplines, such as philosophy, cognitive science and history, which the NLP community has hardly taken into account.
A further distinction concerns two different research areas within NLP: In the field of Topic Detection and Tracking, the identification of events is assimilated to the identification of topics within a stream of texts and the clustering of documents by topic.Footnote 1 Instead, in the field of Information Extraction (IE), the aim is to extract events expressed by words or phrases in a text. In this paper, we focus mainly on the latter perspective, since it has led to several standardization proposals and evaluation campaigns, and to the creation of a wide community of researchers working at Temporal Information Processing tasks. However, we are aware that Topic Detection and Tracking is going to attract more and more attention, because it is particularly suitable to perform coarse-grained event detection on large streams of documents, for instance on social media data.
Temporal Information Processing is a task that aims at automatically detecting and interpreting events (e.g., to live/the war), temporal expressions (e.g., 20/05/2015/this summer) and temporal relations within texts (e.g., in Waters recede before a tsunami the event recede happens BEFORE the event tsunami). Although event identification and processing may appear an easier task than the classification of temporal relations and expressions, which are often vague or implicit in natural language, this is still very challenging due to the ambiguous nature of the concept of event. The term ‘event’ itself has many readings: some authors use it to refer only to dynamic actions, others to refer also to static situations (Sasse Reference Sasse2002). This terminological confusion mirrors the inherent complexity of the concept of event: In fact, an event may designate both an ontological and a linguistic category. However, between the ontological level and the linguistic one, there is no one-to-one mapping because the same event may be expressed using various types of linguistic elements. As a matter of fact, even if verbs prototypically denote events whereas nominals denote objects, this distinction is not clear-cut in natural language (Hagège Reference Hagège1996). In particular, nominals exhibit a strong semantic ambiguity due to polysemy, showing alternations between eventive and non-eventive readings (Pustejovsky Reference Pustejovsky2005): for example, administration denotes an event in spending grew during his administration and a group in this administration is doing well. A variety of terminology is in use also in event classification. The best-known classification of events is the one proposed by Vendler (Reference Vendler1967), who distinguishes between states (non-dynamic situations persisting over a period of time and without an endpoint, e.g., believe), activities (open-ended dynamic processes, e.g., walk), accomplishments (processes with a natural endpoint and an intrinsic duration, e.g., build a house) and achievements (almost instantaneous events with an endpoint, e.g., find). The Generative Lexicon theory revisits Vendler’s classification introducing a three-way taxonomy of event types including states, processes and transitions. In the latter category, accomplishments and achievements are collapsed (Pustejovsky Reference Pustejovsky1991). Moreover, in the linguistic literature, all types of actions, states and processes often fall under the cover term ‘eventualities’, coined by Bach (Reference Bach1986) in his work on the algebra of events.
With this survey, we aim at providing an overview of the way events have been defined in IE (Section 2), with a focus on the different evaluation campaigns organized over the years (Section 2.2). We also account for multilingual event processing, presenting tasks and corpora that cover languages other than English, and for new domains involved in recent event definition efforts (Section 2.3). Finally, we present a case study in Section 3, taking the perspective of history scholars, i.e., researchers from another area that typically deal with events in their daily activity. We try to address the following questions: Was all the work devoted to event processing with IE techniques useful to serve real historical investigation? Were the various definitions of events provided over the years compatible with research practices adopted in other communities? How should events be defined to be processable with NLP tools but also to comply with historical research? We shed light on such questions by means of an online questionnaire, in which historians were involved in an ‘event definition’ exercise. The outcome of this study highlights the difficulties in shifting from a linguistic-driven perspective to the historical one, where a more abstract conception of events is prevalent.
2 The IE perspective on events
Starting in 1991, several evaluation campaigns and workshops devoted to various aspects of temporal information processing and in particular to the analysis of the notion of event have been organized and have fostered the creation of a research community around event detection and processing. The timeline in Figure 1, built by collecting information from websites and proceedings, summarizes the history of workshops, in the lower part, and evaluation campaigns, in the upper part, related to temporal processing and organized starting from MUC-3.Footnote 2
Timeline of evaluation campaigns (above) and workshops (below) in the field of event detection and processing. The Time & Space Track @Semeval 2015 includes the TimeLine, QA TempEval and Clinical TempEval tasks.
We describe them in detail in the following subsections.
2.1 First studies on events in the NLP community
In 2001, during the Workshop ‘Temporal and Spatial Information Processing’, three relevant works dealing with event annotation and processing were presented, each of them relying on a different notion of event. Filatova and Hovy (Reference Filatova and Hovy2001), whose system assigns a position on a timeline to events in newspaper articles, define events as propositions that contain a subject and a predicate. The system achieves a precision of 0.55 and a recall of 0.60. Schilder and Habel (Reference Schilder and Habel2001) present a tool for the automatic annotation of temporal expressions and events in news. The authors define events as expressions that have an implicit time dimension and are either verbs or noun phrases. The list of markable nouns is limited to those directly connected to a temporal expression or a temporal preposition (e.g., after the election in May) and belonging to the domain of interest (i.e., finance, opening of the stock exchange). In a further extension of the system, the authors perform event recognition through an ontology containing event-denoting nouns in the financial domain and information on event types (Schilder and Habel Reference Schilder and Habel2003). Finally, Katz and Arosio (Reference Katz and Arosio2001) propose a method to annotate temporal relations at sentence level, limiting events to verbs. The three works highlight the need to achieve a consensus on a definition of event, aimed also at making automatic approaches comparable.
In that same year, Setzer (Reference Setzer2001) presents Sheffield Temporal Annotation Guidelines, the first annotation scheme that takes into account all temporal information elements (i.e., events, temporal expressions, temporal relations and event identity). The author defines an event as something that happens, must be anchorable in time, can be instantaneous or may last for a period of time. States are therefore not taken into consideration and, from the linguistic point of view, candidate events include nominalizations, finite and non-finite verbs. Each event is associated with attributes giving grammatical and semantic information, e.g., aspect.
Built upon Sheffield Temporal Annotation Guidelines, TimeML (Pustejovsky et al. Reference Pustejovsky, Castaño, Ingria, Saurí, Setzer and Katz2003a) is a scheme for the annotation of events, temporal expressions and relations between events and/or temporal expressions (i.e. temporal, aspectual and subordination relations). Following Bach’s broad notion of event, TimeML identifies a wide range of linguistic expressions realizing events, i.e. tensed and untensed verbs (e.g., was captured, to thank), adjectives (e.g., sick), nominals (e.g., strike) and prepositional phrases (e.g., on board). The consolidation of TimeML as an international standard called ISO-TimeML (ISO 2008) has facilitated its adaptation to different languages, such as Spanish (Saurí Reference Saurı2010) and Korean (Im et al. Reference Im, You, Jang, Nam and Shin2009) and the release of annotated data, such as the English TimeBank (Pustejovsky et al. Reference Pustejovsky, Hanks, Saurí, See, Gaizauskas, Setzer, Radev, Sundheim, Day, Ferro and Lazo2003b).
2.2 Evaluation campaigns
Parallel to the works reported in the previous Subsection, several evaluation campaigns on temporal IE and processing have been carried out. As shown in Figure 1, such campaigns have become very frequent in the last decade, with some years characterized by multiple evaluations.
The first campaign was the Message Understanding Conference (MUC-3) in 1991. It hosted the ‘Scenario Template’ (ST) task, in which systems were required to identify information about a given event (e.g., an air vehicle launch) and relate such information to the entities involved in it. Thus, an event was considered as a set of relationships between participants, time and space: From a practical point of view, it was seen as a template with slots to be automatically filled. The ST task was proposed in five MUC editions, from 1991 to 1998. Throughout the years, teams participating in ST presented systems with a modular pipeline architecture based mainly on pattern-matching techniques in particular after the success of the FASTUS system in MUC-4 that used such approach (Appelt et al. Reference Appelt, Hobbs, Bear, Israel and Tyson1993). Results registered in the ST task are quite low if compared with the ones achieved in other MUC tasks such as Named Entity Recognition and Coreference (CO) resolution (Chinchor Reference Chinchor1998). For example, in MUC-7, the best system in the ST task obtained 0.51 F-score (Aone et al. Reference Aone, Halverson, Hampton and Ramos-Santacruz1998), whilst the best systems in the Named Entity Recognition and CO tasks achieved an F-score of 0.93 and 0.62, respectively (Mikheev, Grover and Moens Reference Mikheev, Grover and Moens1998; Hovy et al. Reference Humphreys, Gaizauskas, Azzam, Huyck, Mitchell, Cunningham and Wilks2013). The main difficulties of systems participating in the ST task were the complexity of texts to be processed, the high number of slots to be filled and the need of world knowledge for some of these slots.
In the ‘Event Detection and Recognition’ task, run for three years in the context of the ACE (Automatic Content Extraction) programme, an event is a specific occurrence involving participants, something that happens and can often be described as a change of state (Linguistic Data Consortium 2005). According to the ACE approach, extracting an event means marking up both the verb, noun, pronoun or adjective that most clearly expresses its occurrence (i.e., the event trigger) and the entire sentence containing that word (i.e., the event mention). However, only events belonging to a list of predefined types are taken into account, each with a number of subtypes (e.g., the event type Conflict has two subtypes: Attack and Demonstrate). Each event is associated with the entities playing a role in it (e.g., the location target of an Attack event) and a set of attributes such as genericity and tense.
It is not possible to make a precise comparison between ACE and MUC results because the former adopted a different evaluation measure called Value Score (Doddington et al. Reference Doddington, Mitchell, Przybocki, Ramshaw, Strassel and Weischedel2004). However, the two initiatives share the same limitation: They were both designed around specific domains and very limited types of events (Grishman Reference Grishman2010). Therefore, the proposed systems could hardly be adapted to different domains and applications. Another issue is that the corpora used for training and evaluation were artificially built by choosing the newspaper articles containing more events of interest: For example, forty-eight per cent of the events in the training corpus of ACE2005 belonged to the Attack subtype (Grishman Reference Grishman2010). This led to the creation of data sets that are not representative of journalistic language. Moreover, the complexity of ACE annotation makes the creation of consistent labeled data very challenging.
In order to address this last shortcoming, the ERE (Entities, Relations, Events) scheme has been developed within the DARPA DEFT programme (Aguilar et al. Reference Aguilar, Beller, McNamee, Van Durme, Strassel, Song and Ellis2014), with the goal to propose a lighter weight version of ACE. ACE and ERE share the same definition of events and the same event ontology (thus, event annotation is limited to the ACE types and subtypes). However, ERE simplifies the annotation by collapsing tags, accepting a looser event extent and reducing the set of attributes and values. Recently, a transition between this simple scheme (also known as Light ERE) towards a more sophisticated representation of events has been proposed under the name of Rich ERE (Song et al. Reference Song, Bies, Strassel, Riese, Mott, Ellis, Wright, Kulick, Ryant and Ma2015). In Rich ERE, the event ontology includes a new type and several new event subtypes. Moreover, the number of attributes is expanded and more attention is devoted to event CO. These DEFT ERE standards are the basis of the novel Event Nugget annotation scheme (Mitamura et al. Reference Mitamura, Yamakawa, Holm, Song, Bies, Kulick and Strassel2015). An event nugget is a semantically meaningful unit referring to an event and linguistically represented not only by a single word but also by a continuous or discontinuous multi-token expression. The Knowledge Base Population evaluation track of the Text Analysis Conference (TAC KBP) conducted a task on event argument extraction and a pilot task on event nugget detection (Song et al. Reference Song, Biesa, Strassel, Ellis, Mitamura, Dang, Yamakawa and Holm2016) in 2014,Footnote 3 and these same tasks are included also in the Event Track of Knowledge Base Population evaluation track of the Text Analysis Conference 2015 and 2016.Footnote 4
Although the Knowledge Base Population evaluation track of the Text Analysis Conference campaigns have been successful, their impact at large has been limited because the annotated datasets were distributed only to tasks participants. A different approach was adopted instead by TempEval organizers, who greatly contributed to improving state-of-the-art technologies in the field of Temporal Processing by making the data freely available after the campaigns. This consolidated also the success of TimeML annotation standard.
TempEval-1 (Verhagen et al. Reference Verhagen, Gaizauskas, Schilder, Hepple, Katz and Pustejovsky2007) was the first open and international evaluation competition that used TimeBank as a benchmark. TempEval-1 avoids the complexity of complete temporal annotation by focussing only on the identification of temporal relations between given pairs of temporal expressions and events. TempEval-2 (Verhagen et al. Reference Verhagen, Sauri, Caselli and Pustejovsky2010) was a more complex campaign than the previous one: It was multilingual and consisted of six subtasks including event extent identification and classification of event attributes. This subtask was proposed also in TempEval-3 (UzZaman et al. Reference UzZaman, Llorens, Derczynski, Allen, Verhagen and Pustejovsky2013). Only one out of seven participants in the event extraction and classification subtask uses a rule-based approach (Zavarella and Tanev Reference Zavarella and Tanev2013). The best performing systems rely on a supervised approach both for event extraction and event type classification: TIPSem (Llorens, Saquete and Navarro Reference Llorens, Saquete and Navarro2010), ATT1 (Jung and Stent Reference Jung and Stent2013) and KUL (Kolomiyets and Moens Reference Kolomiyets and Moens2013) are based on Conditional Random Fields, MaxEnt classification and Logistic Regression, respectively. They all take advantage of morphosyntactic information (e.g., POS) and semantic features at both the lexical and the sentence level, e.g., WordNet synsets (Fellbaum Reference Fellbaum1998) and semantic roles. Best results in event extraction are around 0.80 F1-score. However, when dealing with the classification of event types, system performances drop by almost ten points, with F1-scores all below 0.72.
SemEval-2015 hosted three tasks related to temporal processing in the ‘Time and Space track’ with a focus on new challenges, new evaluation approaches and new domains.Footnote 5 The TimeLine task addressed CO resolution of events and temporal relation extraction at a cross document level with the aim of building timelines (Minard et al. Reference Minard, Speranza, Agirre, Aldabe, van Erp, Magnini, Rigau and Urizar2015). QA TempEval introduced an extrinsic evaluation that took into consideration a specific end-user application, i.e., question answering (Llorens et al. Reference Llorens, Chambers, UzZaman, Mostafazadeh, Allen and Pustejovsky2015). Clinical TempEval moved past TempEval efforts from the news to the clinical domain (Bethard et al. Reference Bethard, Derczynski, Savova, Pustejovsky and Verhagen2015).
As a wrap-up of the different annotation schemes described in this section, we present in Figure 2 the same sentence annotated according to ACE, Light ERE, Event Nugget and TimeML guidelines. Differences in event types amongst ACE, Light ERE and Event Nugget are minimal (in this example are even null), whilst there is more variation concerning extent. ACE, Light ERE and TimeML annotate only events as single tokens, whilst Event Nugget annotation annotated multi-token and discontinuous expressions (charges. . .dropped in the third example). Moreover, in Light ERE, only actual events are eligible to be annotated (this is why dropped is not annotated in the second example). All the other schemes, instead, include the annotation of probable, possible and negated events. In ACE, Light ERE and Event Nugget events are connected to their arguments, i.e., entities such as him and witness. In TimeML, instead, the focus is on temporal links between two events (e.g., dropped and died) or between an event and a temporal expression (e.g., died and yesterday). In general, ACE, Light ERE and Event Nugget combine information on events with their argument structure, whilst in TimeML, the temporal dimension acquires more relevance, having its roots in Allen’s interval algebra (Allen Reference Allen1984).
Comparison of different event annotations. Red squares highlight event triggers whilst blue underlinings identify other annotated elements that in ACE, Light ERE and Event Nugget constitute event arguments. Connections between events and arguments are displays in dotted lines. For TimeML, temporal links are in green.
2.3 Adaptation of event processing to new domains
Most evaluation exercises presented so far were concerned with event processing in the news domain. Only recently, NLP researchers have started to look at different domains and develop domain-specific annotation guidelines and systems. For instance, following an increased interest in the temporal processing of clinical records, ISO-TimeML has been adapted to the clinical domain developing, as a result, the THYME annotation guidelines.Footnote 6 Following such guidelines, an event is ‘anything relevant to the clinical timeline’ (Styler et al. Reference Styler IV, Bethard, Finan, Palmer, Pradhan, de Groen, Erickson, Miller, Lin, Savova and Pustejovsky2014), for example diseases, medical treatments and all actions and states related to the patient’s clinical timeline. THYME guidelines formed the basis of both the i2b2 shared task in 2012 (Sun, Rumshisky and Uzuner 2013) and of the Clinical TempEval evaluation, organized within SemEval 2015 and aimed at assessing the performance of temporal IE systems on clinical notes and pathology reports.Footnote 7 The University of Colorado at Boulder proposed an extension of the THYME guidelines integrating ISO-TimeML, the Stanford Event CO (Lee et al. Reference Lee, Recasens, Chang, Surdeanu and Jurafsky2012) and the CMU Event CO guidelines (Hovy et al. Reference Hovy, Mitamura, Verdejo, Araki and Philpot2013) under the name of Richer Event Description.Footnote 8 Richer Event Description adopts the TimeML wide definition of events and annotates events, temporal expressions and entities, as well as temporal, CO and casual relations (Hovy et al. Reference Ikuta, Styler, Hamang, O’Gorman and Palmer2013).
Three editions of the BioNLP shared task in 2009, 2011 and 2013 evaluated systems for extracting events from biomedical data. In this field, the definition of event is strongly domain-dependent and expert biologists annotate the datasets. More specifically, a biological event is a temporal occurrence involving one or more genes or proteins (Kim et al. Reference Kim, Ohta, Tateisi and Tsujii2006): An event ontology that defines a set of processes and functions supports the annotation. During the 2013 evaluation campaign, different tasks were proposed: In the Genia Event Extraction, task systems were required to detect trigger words expressing molecular and sub-cellular events (e.g., mutation), assign a type to each event (e.g., anatomical or pathological), link events to their arguments (e.g., a molecule) and identify speculated and negated events (e.g., the failure of a mutation) (Nédellec et al. Reference Nédellec, Bossy, Kim, Kim, Ohta, Pyysalo and Zweigenbaum2013). EVEX, TEES-2.1, and BioSEM were the best performing systems in the extraction of events and of their primary arguments during BioNLP-ST 2013, with an F-score of 0.51. The first two systems combine Support Vector Machines and linguistic features, whilst the third one is rule-based (Hakala et al. Reference Hakala, Van Landeghem, Salakoski, Van de Peer and Ginter2013; Bjrne and Salakoski Reference Bjrne and Salakoski2013; Bjrne et al. Reference Bui, Campos, van Mulligen and Kors2013).
Event extraction from social media is another emerging area of research (Atefeh et al. Reference Farzindar and Khreich2015). Most of the works in this field address the task as a clustering problem following the Topic Detection and Tracking approach mentioned in Section 1, for example, using an unsupervised method and focussing on the detection of unspecified new events (Petrović, Osborne and Lavrenko Reference Petrović, Osborne and Lavrenko2010). Other works deal with the retrieval of retrospective events in microblogs, such as Twitter: Amongst others, Metzler, Cai and Hovy Reference Metzler, Cai and Hovy2012 propose a temporal query expansion technique to retrieve a ranked list of event summaries, having the events classified in different categories and types. Ritter, Mausam and Clark (Reference Ritter, Mausam and Clark2012) test a different approach appling IE techniques to identify events in a stream of tweet. The authors annotated manually event-referring phrases in a corpus of 1,000 tweets following the TimeML event definition and developed an automatic tagger that deals with the complexity of Twitter language (i.e., informal and ungrammatical style) achieving an F-score of 0.64.
The literature reports a number of works that try to tackle the semantics of historical texts using a combination of Semantic Web technologies and NLP approaches (Meroño-Peñuela et al. Reference Meroño-Peñuela, Ashkpour, van Erp, Mandemakers, Breure, Scharnhorst, Schlobach and van Harmelen2015). However, unlike what happened in the clinical domain, no attempt was made to find a domain-specific definition of event combining the historical perspective and ongoing research in the NLP field. Moreover, NLP techniques specifically developed for event processing have not been fully exploited and the current standardization efforts have received little attention in this domain. For example, in the Agora project, Van Den Akker et al. (Reference Van Den Akker, Aroyo, Cybulska, Van Erp, Gorgels, Hollink, Jager, Legene, van der Meij, Oomen, van Ossenbruggen, Schreiber, Segers, Vossen and Wielinga2010) aimed at enriching museums metadata through the extraction of historical event names from unstructured texts, event extraction is assimilated to the recognition of named entities. Therefore, only named events, such as French Revolution, are taken into account. Another choice usually made in projects dealing with historical documents is limiting the extraction of events to a set of specialized types. For example, in the FDR/Pearl Harbor project (Ide and Woolner Reference Ide and Woolner2004) research focussed only on communication events. This choice was driven by the goal of the project, which was to help historians of WWII to search and retrieve information from documents (e.g., government correspondence and memoranda) written before the Pearl Harbor attack in 1941. This specific set of events was categorized based on FrameNet (Baker, Fillmore and Lowe Reference Baker, Fillmore and Lowe1998) by assigning verbs in the corpus of reference to the ‘Communication’ frame and its sub-frames. Another project, the Semantics of History, focusses only on conflict-related and motion actions (Cybulska and Vossen Reference Cybulska and Vossen2011).
Another weak point of current NLP research for historical texts is the scarcity of corpora fully annotated with temporal information. For example, files tagged within the projects described above have not been publicly released. Two notable exceptions are the ModeS TimeBank (Nieto Saurí and Bernabé Reference Nieto, Saurí and Bernabé Poveda2011), containing Spanish texts from the eighteenth century, and the De Gasperi corpus, a collection of documents written by the Italian statesman Alcide de Gasperi and dating back to the beginning of the twentieth century. Both were manually annotated following a language-specific adaptation of TimeML. ModeS TimeBank was employed for theoretical studies on the evolution of the Spanish language, whilst the De Gasperi corpus was used to measure the performance of event extraction systems on historical texts within the EVENTI evaluation exercise.Footnote 9
In order to measure how an event extraction system trained on news performs on historical data, we evaluated the performance of two supervised systems (one for Italian and one for Spanish) trained on news and tested on in-domain and out-of-domain data, i.e. historical texts. For Italian, we used the state-of-the-art FBK-HLT-Time system (Mirza and Minard Reference Mirza and Minard2014) evaluated both on contemporary news and on the De Gasperi corpus in the framework of the EVENTI evaluation campaign for event extraction. The system achieves F1 0.87 (P 0.88, R 0.85) on news and F1 0.83 (P 0.89, R. 0.78) on historical documents, showing that the different performance in the two domains is limited to a drop of 0.07 points in recall. This is mainly due to the fact that the language in the De Gasperi corpus is very similar to contemporary Italian, and tokens corresponding to events are generally easy to recognize. The evaluation on Spanish data, instead, is remarkably different. We evaluated the TIPSem system (Llorens et al. Reference Llorens, Saquete and Navarro2010) on the Modes TimeBank, and we compared it with the TempEval 2013 results of the same system (UzZaman et al. Reference UzZaman, Llorens, Derczynski, Allen, Verhagen and Pustejovsky2013). In this case, the performance in the two domains is very different: On news data, the system achieves F1 0.89 (P 0.92, R 0.86), whilst its performance drops to F1 0.39 (P 0.27, R 0.72) on the ModeS TimeBank, since the corpus shows many diachronical language variations affecting precision.
This comparison shows that event recognition systems trained on news in some cases would be suitable for investigations in new domains, given that the event definition framework and the language of the documents to be processed are similar to the ones used for training. For instance, historians analysing corpora dating back to the previous century may still achieve satisfactory system performance.
In order to account for all corpora annotated so far with event information in different domains and languages, we report a summary in Table 1. The information presented in the table was gathered through the direct analysis of the resources downloaded from the Web and merging data from scientific papers. Resources listed in the table have been annotated following different schemes and cover five domains, with a prevalence of the news domain. The number of corpora in the list shows the interest of the NLP community in event processing. The most recent corpora confirm the trend towards new domains, new languages and more complex tasks integrating event extraction.
Corpora including event annotation in different domains. For each corpus the language, number of tokens, number of files and number of annotated events are provided. The symbol ‘-’ is used in case of missing information. Resources in boldface are available online at the moment of writing
# Light ERE, Rich ERE and Event Nugget corpora include both news and discussion forum data
** Number of characters instead of the number of tokens
3 What is an event in history?
As shown in the previous Section, past projects trying to apply NLP techniques to historical investigation have adopted heterogeneous approaches, and there has been no real effort amongst history scholars to standardize event definition taking into account past proposals made in the NLP community. However, researchers in history face daily issues related to the observation, analysis and interpretation of events. This gap between the two research communities may depend on a lack of communication and cross-fertilization, but also on the fact that events as defined in IE do not fully satisfy requirements from other disciplines. In order to clarify the reasons of this gap, we ran an investigation involving historians based on an online questionnaire. We circulated the survey in English and Italian, to facilitate the inclusion of different communities in this study. Both versions had the same set of questions. Only the examples taken from historical documents were different but with the same range of linguistic phenomena to investigate. The questionnaire was distributed via social media (i.e., Twitter and LinkedIn), mailing-lists (e.g., the Humanist Discussion Group) and targeted emails to individual historians, professional associations (e.g., the Australian Historical Association) and research centres (e.g., Institute of Historical Research at the University of London). After two months from its launch, seventy-four historians participated in the survey.Footnote 10 The expertise of the survey respondents covered many research fields such as cultural history, history of science, political science and biography thus providing insight into current research practices in the domain on interest.
The general goal of this analysis, that can likely be applied also to other domains, was to leverage knowledge about the way events are defined in historical research and to compare it with ongoing standardization efforts in the NLP community. To the best of our knowledge, this is the first questionnaire on this topic, whose outcome can potentially enrich the current theoretical discussion on the nature of events. Besides, it can be seen as a preliminary step towards the definition of annotation guidelines for developing NLP tools in this domain.
3.1 Questionnaire description and results
Two parts composed the questionnaire. In the first one, we collected participants’ demographic information and assessed their general interest in NLP. The second part aimed at shedding light on the notion of ‘event’ for historians based on three questions.
In the first question, participants were asked to list all the single words or expressions encoding events (if any) in three given sentences, without providing any definition of what an event is. The aim was to indirectly leverage an operational definition of events based on historians’ knowledge.
The sentences were different in the English and in the Italian questionnaire but they contained the same linguistic phenomena: negated verbs (e.g., has not been prepared), nominalizations (e. disarmament), aspectual nominals (e.g., end), cognitive verbs (e.g., think), named events (e.g., Korean war), nominals expressing states (e.g., independence) and multi-token expressions like phrasal verbs (e.g., taken place). We report in Table 2 the English sentences, taken from J. F. Kennedy’s public speeches.Footnote 11 Questionnaire participants could annotate single words or expressions conveying events, but also provide no annotation. In the first and the third sentence, a high percentage of respondents (thirty-seven per cent and sixty-eight per cent, respectively) did not detect any event, probably because these sentences contain a state (i.e., independence) and an opinion (i.e., I think. . .). The second sentence includes a named event (Korean war) and only the three per cent of respondents did not annotate any event.
English sentences annotated by the questionnaire participants. For each sentence, we report the absolute percentage of annotated events in terms of single tokens (ST), multi-token expressions (MT), verbal expressions (V) and non verbal expressions (NV). The three most common extents for each sentence are also reported. For Italian, we registered comparable results
The percentages listed in Table 2 are calculated by taking into consideration the total number of annotations per sentence. To decide if an expression was verbal or non-verbal, we looked at the part of speech of the words contained in the expression. In particular, all expressions containing at least one verb were considered verbal (e.g., menaced, work that out, think). Anyway the majority of the identified events are non-verbal (e.g., today, independence, conference, end of the Korean war, advice): seventy-five per cent in the first sentence, eighty-three per cent in the second and fifty-two per cent in the third. This contrasts with the outcome of the experiment reported by Hatzivassiloglou and Filatova (Reference Hatzivassiloglou and Filatova2003), in which nouns such as war and earthquake were never identified as events by a group of students annotating news. Events consisting of more than one token are annotated very frequently in all the sentences: thirty-two per cent of events annotated in the first sentence are multi-token, sixty-eight per cent in the second and forty-four per cent in the third. Some of these multi-token extents correspond to entire clauses, e.g., This country has not been prepared and I think we can work that out. This high number of multi-token events goes against the TimeML and Richer Event Description minimal chunk rule for tag extent, according to which only single tokens are to be annotated as events.Footnote 12 The distinction made in ACE and ERE between event trigger (the word expressing the event) and event mention (the sentence containing it) seems to better meet historians’ needs. Moreover, ACE, ERE and Event Nugget allow the annotation of multi-token event triggers (the latter also discontinuous cases).
Conclusion 1. The notion of event is seen as independent from its grammatical category, in line with TimeML. However, the minimal chunk annotation used in TimeML is not optimal. Amongst the considered standards, the multi-token annotation of continuous and discontinuous multi-token expressions proposed in Event Nugget addresses best historians’ view on events.
In the second question, we asked participants to rate the relevance of a list of properties to define when a word or expression can be labelled as an event. These properties included for instance impact, cause and frequency, and were inspired by the essay ‘What is an Event?’ written by the history scholar Robert Bedrosian.Footnote 13 The ratings included four possible values, i.e., ‘very important’, ‘somewhat important’, ‘not important’ and ‘don’t know’. Figure 3 presents the value distribution across the properties merging results from English and Italian questionnaires. Public Perception and Impact, i.e., the degree to which an event affects society or nature, are properties not related to the linguistic analysis of texts but to the historians’ interpretative work. Both were considered quite relevant, especially the latter. Predictability is the only property in which the value ‘not important’ prevails. On the contrary, Type has the highest positive consensus. In TimeML, event type information is conveyed by seven possible values of the class attribute, where both semantic (e.g., STATE) and syntactic criteria (e.g., I_STATE) are taken into account. A classification based on syntactic criteria would not be optimal for historians, for whom syntax does not have a primary importance in the interpretation of the text. On the other hand, the event ontology of ACE, ERE and Event Nugget is made of a list of types and subtypes which limits the annotation to a specific set of categories strongly connected to the news domain (e.g., type: JUSTICE, subtype ACQUIT in Figure 2). Other categories should be added to this ontology to make it more apt for the history domain so to include, for example, events of cognition, emotion and communication. The USAS (Rayson et al. Reference Rayson, Archer, Piao and McEnery2004) and the Historical Thesaurus of English tagsets (Kay et al. Reference Kay, Roberts, Samuels and Wotherspoon2009) contain twenty-one and thirty-seven main semantic categories, respectively and they have been already used to analyse historical texts (Archer Reference Archer, Taavitsainen, Jucker and Tuominen2014; Rayson et al. Reference Rayson, Baron, Piao and Wattam2015): For this reason, they can provide an interesting fine-grain classification of events for the history domain. Factuality, i.e., the distinction between actual real facts and imaginary, future, avoided and prevented events, has a limited interest for historians, whilst it is more relevant from a linguistic perspective. In fact, TimeML encodes this property through subordination links whereas other annotation schemes encode it as an attribute attached to the event (Saurí and Pustejovsky Reference Saurí and Pustejovsky2009; van Son et al. Reference van Son, van Erp, Fokkens and Vossen2014).
What are the most important properties for a historian in order to understand if a word (or a set of words) expresses a relevant event.
Preceding and consequent events appear to be very important for historians, and this is in line with the ongoing effort in NLP to encode intra- and cross-document event ordering. TimeML conveys this information by using temporal links, corresponding to thirteen types of binary temporal relations, inherited from Allen’s interval algebra. Besides, the challenge of cross-document event ordering has been recently addressed by the TimeLine task at SemEval-2015.Footnote 14 In TimeML, the temporal link tag is also employed to link events to points in time (e.g., 25/12/2014), durations (e.g., 3 month) and temporal expressions denoting recurring times (e.g., every month): This corresponds to the Temporal Grounding property, that is the degree to which an event can be pinpointed to a particular time or period, and the Frequency property. In the MUC ST as well as in ACE, ERE and Event Nugget, temporal relations between events or between an event and a temporal expression are not explicitly addressed. The link between an event and a temporal expression is encoded in the form of a temporal slot in case of MUC or of a temporal argument in case of ACE, ERE and Event Nugget (e.g., the Time-Arg argument ‘yesterday’ of the event trigger ‘died’ in Figure 2). The property of an event being the cause or the effect of another event (i.e., Cause) is strictly connected to the Agency property, i.e., who/what caused such event. TimeML does not include a specific relation for causative constructions but causes and effects denoted by events are temporally ordered using a temporal link (a cause always precedes the effect). However, attempts have been made to explicitly annotate causal relations as an extension of TimeML (Mirza and Tonelli Reference Mirza and Tonelli2014). In ACE, ERE and Event Nugget, Agency is annotated as event argument for several event types. For example, in the sentence ‘his father-in-law killed him’, father-in-law is the Agent argument of the trigger event killed of type LIFE. Event–event causality relations are planned as future development of the Rich ERE annotation, but they are currently not included in the guidelines. On the contrary, causal relations play an important role in the Richer Event Description guidelines (Hovy et al. Reference Ikuta, Styler, Hamang, O’Gorman and Palmer2013). As for Participants, TimeML does not foresee the annotation of the entities involved in an event, even if historians’ responses suggest that this information is quite relevant. Attempts have been made to add participants’ information to events (Pustejovsky Littman and Saurí Reference Pustejovsky, Littman, Saurì, Schilder, Katz and Pustejovsky2007), but this has not led to the extension of TimeML specifications. On the contrary, participants annotation is crucial in MUC, ACE, ERE and Event Nugget, in which several arguments have to be identified (e.g., Victim-Arg in Figure 2). Research on semantic roles can provide much guidance in this respect, for example, by taking inspiration from PropBank (Palmer, Gildea and Kingsbury Reference Palmer, Gildea and Kingsbury2005) or FrameNet frameworks. This was already proposed within the NewsReader project (Vossen et al. Reference Vossen, Rigau, Serafini, Stouten, Irving and Van Hage2014), where event extraction from news is performed by leveraging information related to events and participants from different sources and modelling them as knowledge graphs. An analysis of the application of NewsReader pipeline to the history domain may provide useful insights into the importance of semantic roles from historians’ perspective.
Conclusion 2. An event is a complex information object characterized by many properties. A new framework for the annotation of events in historical texts should take advantage of the temporal dimension as defined in TimeML but also look at other annotation efforts (e.g., semantic roles in FrameNet, participants’ information in Event Nugget) to cover all important properties of events.
Finally, in the third question, participants were asked to choose between two linguistic annotations of short text snippets containing the same specific phenomena in both English and Italian, i.e., states and multi-token expressions. This question had the aim of confirming or disproving the previous conclusions. The English questionnaire presented the following passage taken from a speech uttered by J.F. Kennedy:
After the key African state of Guinea, now [voting]1 − 2 with the Soviet Union in Communist foreign policy, after it [gained]1 − 2 its [independence]2, a Russian Ambassador [[showed]1 up]2 the next day. Our Ambassador did not [[show]1 up]2 for 9 months.
In the annotation marked with [. . .]1, only single tokens are annotated as events following the TimeML specifications. Moreover the state independence is not annotated. The option marked with [. . .]2 proposes looser criteria, annotating both multi-token event expressions and states. Only five per cent of participants preferred the first annotation, sixty-one per cent chose the second option and the rest did not give preference to either of the two annotations. We asked for the motivations behind this choice: Respondents said that a broad context is needed to represent events (An event is not one word, it’s syntactical, inter-relation between agent and object/patient). Besides, answers highlighted the importance of states and conditions (I feel that the state/condition is important.). In ACE, ERE and Event Nugget, states that result from actions, such as being dead, married or retired, are included in the annotation, but disagreement is an open issue for human annotators (Mitamura et al., 2015). On the other hand, in TimeML, only states that are temporally relevant (e.g., that are bound to a specific point or period of time) have to be annotated. Defining what states have to be annotated using a predefined set of annotation rules, as in TimeML, would be extremely critical because such rules could not cover all the information needs of historians.
Conclusion 3. Conclusion 1 about multi-token annotation is confirmed, showing that TimeML could not be applied to a new domain as is. Moreover, states/conditions are important and should be considered in the annotation of historical documents.
4 Conclusions
This paper presents a survey of the state of the art in event definition and processing in NLP, adopting an inter-disciplinary perspective. In the last twenty-five years, thanks to many workshops and evaluation exercises dedicated to the semantic and linguistic analysis of events, research has moved forward. However, a careful adaptation of existing annotation schemes is necessary to apply the outcome of these research activities to new domains. On the basis of the analysis of the state of the art and of historians’ replies to our questionnaire, we can now answer the questions posed in Section 1:
-
(i) Was all the work devoted to event processing with IE techniques useful to serve real historical investigation? NLP methods and technologies have not been fully exploited yet in the domain of history. Existing annotation schemes and systems constitute an important starting point but a careful adaptation is necessary to meet the requirements of domain experts. In the near future, we plan to propose an annotation paradigm for events that takes into account both historians’ suggestions collected through the survey and past experience in NLP.
-
(ii) Were the various definitions of events provided over the years compatible with research practices adopted in other communities? Several event definitions have been proposed over the years, each showing specific strengths and weaknesses. TimeML event definition relies on the broad notion of eventuality: The fact that it includes states as well as processes and actions is compatible with historians’ needs. On the other hand, states should be taken into consideration even if not bound to a specific point or period of time. Allowing only single token events does not meet research practices adopted in other domains. The multi-token choice proposed in the Event Nugget initiative addresses better this need.
-
(iii) How should events be defined to be processable with NLP tools but also to comply with historical research? Events can be defined as complex information objects characterized by many properties. These can be cast by combining different NLP analyses providing rich semantic information, such as semantic role labelling, causality detection and temporal relation processing. The role played by this information in historical research, however, can vary a lot according to the historiographical approach used. For example, the so-called evenemential approach defines history as a chronological accumulation of events in a coherent timeline (Simiand Reference Simiand1960). From this perspective, events are objective entities, atomic facts that do not need deep interpretation. In sharp contrast with this approach, more recent theories propose looking at events in a long-term perspective (Guldi and Armitage Reference Guldi and Armitage2014), in order to study them in their connection with other events taking into consideration recurring analogies and structures. Following these last assumptions, historians have the duty to pose problems and formulate hypotheses, not only to observe events emerged from the analysis of historical documents but also to interpret them (Febvre Reference Febvre1953). The distinction between an important event and one with no historical value is thus never definitive because the research question changes constantly according to the documents that historians are analysing (Marrou Reference Marrou1954). The challenge is to develop an annotation scheme and subsequently a system to support historical investigation and guide the interpretation of historians without replacing them or influencing their understanding of documents.
