1. Introduction
Extracting meaning from text is crucial for natural language understanding. Inferring locations of entities captures meaning from text. Most entities participate in events and as a result their locations change over time. For example, whereas buildings typically have one location during their existence, people and objects such as books do not. It is important to extract this kind of spatial knowledge that changes over a period of time.
Existing approaches to extracting spatial knowledge can broadly be divided into two categories: those that extract locations of events and those that extract locations of someone or something. Semantic role labelling (Palmer, Gildea, and Kingsbury Reference Palmer, Gildea and Kingsbury2005) falls into the former category, for example, Thelma Gutierrez
$\textit{[}went\textit{]}_{{verb}}$
[inside the forensic laboratory where scientists are trying to solve this mystery]
$_{{\textsc{arg}}_4}$
, where
${\textsc{arg}}_4$
indicates the end point of went. Research in the latter category includes geolocating Twitter users (Liu and Inkpen Reference Liu and Inkpen2015) and pairing companies with the location of their headquarters (Mintz et al.
Reference Mintz, Bills, Snow and Jurafsky2009), for example, IBM’s headquarters in New York.
Determining the temporal span when spatial knowledge is true has not been extensively researched. From the sentence John parked Jamie’s car at the Highland Garage, we can infer that John and the car were located at the Highland Garage minutes before and during parking. Similarly, we can infer that John will leave the Highland Garage shortly after parking, whereas the car will be at the garage for up to a few days but not months. In this article, we present a two-step methodology to extract temporally anchored spatial knowledge from text. First, we automatically generate potential spatial knowledge and second, we validate it. More specifically, we automatically generate potential spatial knowledge leveraging semantic roles or syntactic dependencies. Validation is done (a) manually to create two corpora and (b) using machine-learning models to assess whether the validation can be automated.
The major contribution of this article is a detailed comparison of the temporally anchored spatial knowledge obtained by manipulating semantic roles or syntactic dependencies. We compare the amount of potential knowledge generated, the percentages of potential knowledge that are valid, and experimental results with both gold and predicted linguistic information in a realistic scenario. We conclude that leveraging syntactic dependencies allows us to extract more temporally anchored spatial knowledge for entities and yields better results.
The work presented here extends our previous work on extracting temporally anchored spatial knowledge, which primarily manipulated semantic roles as detailed in Section 3.
Definitions and notation. We use the term temporally anchored spatial knowledge to refer to information regarding whether a given entity x is or is not located at some location y, and for how long with respect to an event. We use the notation location(x, y) to indicate the spatial relation between x and y, if any. We use the term potential spatial knowledge to refer to spatial relations location(x, y) that are yet to be validated.
There are two types of location(x, y) relations. Relations of type 1 are those whose arguments x and y are semantic roles of some verb, and relations of type 2 are those whose arguments x and y are not semantic roles of any verb. Type 1 can be further divided into type 1a if x and y are roles of the same verb, and type 1b if x and y are roles of different verbs. Consider the sentence Cambridge professor John Doe presented his paper in London. Relation location(John Doe, London) is of type 1 – more specifically, type 1a – and location(John Doe, Cambridge) is of type 2. Let us now consider the sentence Officer Jack found the missing diamond at a warehouse owned by Mr. Walker. The relation location(Jack, warehouse) is of type 1a (Jack is
$\textsc{arg}_1$
and warehouse is argm-loc of the same verb found) and the potential relation location(Mr. Walker, warehouse) is of the type 1b (Mr. Walker is
$\textsc{arg}_1$
of the verb owned and warehouse is argm-loc of found).
In Sections 4 and 5, we present and analyse two strategies to annotate temporally anchored spatial knowledge, and in Section 6, we present experimental results. Specifically, we leverage semantic roles to target temporally anchored spatial knowledge of type 1 and dependencies to target temporally anchored spatial knowledge of type 1 and type 2.
2. Background: existing corpora and tools
We add temporally anchored spatial knowledge on top of OntoNotes (Weischedel et al. Reference Weischedel, Hovy, Marcus, Palmer, Belvin, Pradhan, Ramshaw and Xue2011). OntoNotes contains over 64,000 sentences from several domains, for example, news, broadcast and telephone conversations, weblogs. It annotates, among other linguistic information, part-of-speech tags, named entities, parse trees, PropBank-style semantic roles (Palmer et al. Reference Palmer, Gildea and Kingsbury2005) and coreference. We use the CoNLL-2011 Shared Task distribution (Pradhan et al. Reference Pradhan, Ramshaw, Marcus, Palmer, Weischedel and Xue2011).
In this article, we work with the following named entity types: person, org (organisation), work_of_art, norp (nationalities, or religious or political groups) product, event, fac (facility), gpe (geopolitical entity) and loc (location). Weischedel and Brunstein (Reference Weischedel and Brunstein2005) detail the full set of named entity types. Regarding syntactic information, we work with the Stanford dependencies (De Marneffe and Manning Reference De Marneffe and Manning2008) and obtained them by transforming the gold parse trees into dependencies using Stanford CoreNLP (Manning et al. Reference Manning, Surdeanu, Bauer, Finkel, Bethard and McClosky2014).
The semantic role annotations in OntoNotes follow PropBank, which added semantic role annotations on top of the parse trees in the Penn Treebank (Marcus, Marcinkiewicz, and Santorini Reference Marcus, Marcinkiewicz and Santorini1993). PropBank uses numbered arguments (
$\textsc{arg}_0$
,
$\textsc{arg}_1$
, etc.) whose meanings are verb-dependent and argument modifiers (argm-tmp, argm-loc, etc.) which share a common meaning across verbs. The original PropBank corpus consists of (a) 3327 framesets, each frameset defines the numbered roles for a verb, and (b) actual semantic role annotations (numbered arguments and argument modifiers) for 112,917 verbs. On average, verbs in PropBank have 1.93 numbered arguments and 0.66 modifiers. Only 7410 (6.56%) verbs have an argm-loc annotated.
3. Previous work
Previous works in computational semantics that extract spatial information target events, or entities and objects. Semantic role labelling (Palmer et al. Reference Palmer, Gildea and Kingsbury2005) accounts for general locations indicating where something occurs. Role labellers to extract PropBank-style semantic roles have been studied for years (Màrquez et al. Reference Màrquez, Carreras, Litkowski and Stevenson2008), but these tools only extract spatial information between predicates and their arguments.
There have been several proposals to extract semantic links not annotated in well-known corpora such as PropBank (Palmer et al. Reference Palmer, Gildea and Kingsbury2005), FrameNet (Baker, Fillmore, and Lowe Reference Baker, Fillmore and Lowe1998) or Nombank (Meyers et al. Reference Meyers, Reeves, Macleod, Szekely, Zielinska, Young and Grishman2004). Gerber and Chai (Reference Gerber and Chai2010) augment NomBank annotations with additional numbered arguments appearing in the same or previous sentences. The SemEval-2010 Task 10: Linking Events and their Participants in Discourse (Ruppenhofer et al. Reference Ruppenhofer, Sporleder, Morante, Baker and Palmer2009) target cross-sentence missing numbered arguments in PropBank and FrameNet. Roberts, Skinner, and Harabagiu (Reference Roberts, Skinner and Harabagiu2013) target spatial containment relations between events, for example, in The bombing victim died immediately, died is spatially contained in bombing since the spatial bounds of bombing are larger. Unlike the work presented in this article, these previous efforts extract spatial information of predicates. None of them infer spatial knowledge between arguments of predicates or target temporally anchored spatial knowledge.
Previous work that targets entities includes extracting spatial relations between nominals (Davidov and Rappoport Reference Davidov and Rappoport2008), for example,
$[refugees]_{entity}$
from
$[Somalia]_{origin}$
, and possessives (Tratz and Hovy Reference Tratz and Hovy2013), for example,
$[Libya]_{location}$
’s
$[people]_{entity}$
. Relation extraction in general usually also includes spatial relations, for example, linking companies and the locations of their headquarters (Mintz et al.
Reference Mintz, Bills, Snow and Jurafsky2009). Spatial knowledge is intuitively associated with motion events, for example, drive, go, fly, walk, run. (Hwang and Palmer, Reference Hwang and Palmer2015) presented a classifier to detect caused motion constructions triggered by non-motion verbs, for example, The crowd laughed the clown off the stage (i.e., the crowd made the clown leave the stage). Our work targets both motion verbs and non-motion verbs (e.g., played, honored, taught, fighting) that allow us to infer temporally anchored spatial meaning. The task of spatial role labelling (Kordjamshidi, Van Otterlo, and Moens Reference Kordjamshidi, Van Otterlo and Moens2011, Reference Kolomiyets, Kordjamshidi, Moens and Bethard2013) aims at representing spatial information with so-called spatial roles, for example, trajector, landmark, spatial and motion indicators. Unlike us, spatial role labelling does not aim at extracting where entities are not located or when an entity is located (or not located) somewhere. But, doing so is intuitive to humans, as the examples and crowdsourced annotations described in this article show.
In the context of the TAC KBP temporal slot filling track (Garrido et al. Reference Garrido, Peñas, Cabaleiro and Rodrigo2012; Surdeanu Reference Surdeanu2013), relations common in information extraction (e.g., spouse, country_of_residency) are assigned a temporal interval indicating when they hold. Unlike them, the approach presented in this article builds on top of semantic roles and syntactic dependencies and also targets where entities are not located.
The work presented here extends our previous work on annotating and extracting temporally anchored spatial knowledge. We have anchored spatial knowledge on the day before, during or the day after an event (Vempala and Blanco Reference Vempala and Blanco2016c,a), and also at any point of time before, during or after an event (Blanco and Vempala Reference Blanco and Vempala2015; Vempala and Blanco Reference Vempala and Blanco2016b). In these previous works, we manipulated semantic roles to obtain temporally anchored spatial knowledge, thus we only targeted knowledge of type 1. In Vempala and Blanco (Vempala and Blanco, Reference Vempala and Blanco2018), we present preliminary experiments using syntactic dependencies to extract temporally anchored spatial knowledge of type 2. In this article, we expand our previous work as follows. First, we manipulate semantic roles and syntactic dependencies to obtain temporally anchored spatial knowledge and thus target knowledge of type 1 and type 2. Second, we provide (a) a detailed analysis of the spatial knowledge obtained using both strategies (roles and dependencies), including the amount of potential and validated knowledge, and (b) experimental results using both gold and predicted linguistic information. We conclude that syntactic dependencies allow us to infer more temporally anchored spatial knowledge for entities than semantic roles.
4. Corpora creation
We follow a two-step methodology to create corpora annotated with temporally anchored spatial knowledge. First, we generate potential temporally anchored spatial knowledge. Second, we crowdsource annotations to validate the potential spatial knowledge.
4.1 Generating potential spatial knowledge
In this section, we detail two approaches to automatically generating potential spatial relations: manipulating semantic roles and manipulating syntactic dependencies.
Manipulating semantic roles. We generate potential relations location(x, y) by combining all arg
$_i$
(
$x_{verb}$
, x) and argm-loc(
$y_{verb}$
, y) semantic roles within a sentence (
$x_{verb}$
and
$y_{verb}$
need not be the same). Then, we enforce the following restrictions:
-
1. x and y must not overlap;
-
2. the head of x must be a person, org, work_of_art, fac, norp, product or event named entity;
-
3. the head of y must be a noun subsumed by physical_entity.n.01 in WordNet, or a fac, org, gpe or loc named entity; and
-
4. the heads of x and y must be different than the heads of all previously generated pairs.
We define these restrictions to ease the annotation effort and minimise the amount of invalid potential relations generated. Restriction 1 discards potential relations location(x, y) when x and y overlap, which is only possible when x
$_{verb}$
and y
$_{verb}$
are different. Restrictions 2 and 3 are designed to improve the likelihood that the potential relations will be validated, for example, locations whose head is an adverb such as here and there (11% of all argm-loc roles) do not yield valid additional spatial relations. Restriction 4 simply discards potential relations that have already been generated.
OntoNotes annotates 9162 argm-loc roles, and the total number of location(x, y) relations generated prior to enforcing any restrictions is 44,997. The total number of location(x, y) relations generated after enforcing restriction 1 is 37,272, after restrictions 1 and 2 is 3697, and after restrictions 1–3 is 1863. A total of 1732 location(x, y) relations are generated after enforcing all the restrictions. Among these relations, 723 belong to type 1a (i.e., x
$_{verb}$
and y
$_{verb}$
are the same) and 1009 belong to type 1b (i.e., x
$_{verb}$
and y
$_{verb}$
are different).
Figure 1 presents a sample sentence and potential spatial relation generated using semantic roles.
Sample sentence and potential spatial knowledge generated using semantic roles.
Manipulating syntactic dependencies. We generate potential relations location(x, y) following the steps below:
-
1. y is a gpe or loc named entity;
-
2. x is a person, fac, product or work_of_art named entity belonging to the same sentence than y; and
-
3. x is reachable from y
$_{verb}$
, where y
$_{verb}$
is the closest verb going up the dependency tree from y; -
4. y
$_{verb}$
is not the verb to be or to have.
We defined restrictions 2 and 3 to reduce the amount of potential spatial relations to be annotated. Restriction 4, however, is designed after pilot annotations revealed that temporally anchored spatial knowledge could rarely be inferred from them.
OntoNotes annotates 19,478 gpe and 1858 loc named entities (potential ys); and 18,823 person, 1080 fac, 734 product, and 1253 work_of_art named entities (potential xs). Pairing all potential xs and ys within a sentence results in 10,136 pairs (restrictions 2 and 3). Enforcing restriction 1 reduces the number of pairs to 9351, and enforcing restriction 4 further reduces the number to 8775. Out of these 8775 pairs, 7029 have a person named entity as x, 951 a fac, 411 a product and 384 a work_of_art. We annotated 25% of potential location(x, y) relations belonging to each y
$_{verb}$
, which total to 1689 potential spatial relations. Among these relations, 478 belong to type 1 (a: 227, b: 251) and 1211 belong to type 2.
Figure 2 presents a sample sentence and potential spatial relation generated using dependencies.
Sample sentence and potential spatial knowledge generated using syntactic dependencies.
Manipulating roles versus dependencies: overlap. We note that the two approaches are not directly comparable due to the restrictions we impose in each case. Only 5% of the total gpe (geopolitical entity), loc (location) named entities are the second argument of an argm-loc semantic role. Hence, there is minimal overlap between the potential spatial knowledge generated using semantic roles and syntactic dependencies.
4.2 Annotating temporally anchored spatial knowledge
We crowdsourced annotations in order to validate the potential spatial relations automatically generated. For each potential location(x, y) relation, we ask the question ‘Do you think x is located at y?’ before, during and after the event y
$_{verb}$
took place. The annotators first choose a coarse-grained label:
-
• yes or no if x is (or is not) located at y before/during/after y
$_{verb}$
; -
• unk if the question is intelligible but the answer is unknown, that is, neither yes nor no;
-
• inv if the question is invalid.
If they choose yes, they also select one of the following fine-grained labels:
-
• secs, mins, hours, days, weeks, months, years or inf (for infinity) for before and after;
-
• entire if x is located at y for the entire duration of y
$_{verb}$
, some otherwise.
From the label distribution obtained (Section 5), we observe that the fine-grained labels secs, mins, hours, days and weeks are annotated only for a small portion of the potential spatial relations. Hence, we consider as fine-grained labels (a) < years and
$\geq$
years for before and after, and (b) entire and some for during. We refer to these labels as clustered fine-grained labels. The final label was decided using majority vote.
We used Amazon Mechanical TurkFootnote a to validate the potential spatial relations generated from semantic roles. We created one Human Intelligence Task (HIT) per potential location(x, y) relation and recruited annotators with previous approval rate
$\geq$
95% and 5000 or more previous approved HITs. We rejected submissions that took unusually short time compared to other submissions, and those from annotators who always chose the same label. We collected seven annotations per HIT and paid $0.05 per HIT.
We utilised CrowdflowerFootnote b platform to validate location(x, y) relations generated from syntactic dependencies. We asked one question per potential location(x, y) relation and collected annotations from contributors with high trust scores. We collected three annotations per question and paid $0.05 per annotation.
4.3 Annotation quality
To assess the quality of the annotations, we calculate Pearson correlations between each annotator and the majority label, and the percentages of annotators that agree (Tables 1 and 2). Pearson correlations are calculated with the following mappings:
-
• Coarse-grained labels: yes: 1, no:
$-1$
, unk: 0, and inv: 0. -
• Clustered fine-grained labels: < years: 1,
$\geq$
years: 2, some: 1, entire: 2, no:
$-1$
, unk: 0, and inv: 0.
Annotation quality for temporally anchored knowledge generated with semantic roles
We show (a) weighted averages of Pearson correlations (r) between each annotator and the majority label for coarse-grained and clustered fine-grained labels (column 2) (b) percentages of instances for which all (100%), and at least 6 (85%), 5 (71%), 4 (57%) and 3 (43%) out of 7 agree for coarse-grained labels and (c) percentages of instances for which 100% agree, and at least 75%, 66% and 50% agree for clustered fine-grained labels.
Annotation quality for temporally anchored knowledge generated with syntactic dependencies
We show (a) weighted averages of Pearson correlations (r) between each annotator and the majority label for coarse-grained and clustered fine-grained labels (column 2) (b) percentages of instances for which all (100%), at least 2 (66%) and only 1 (33%) out of 3 agree for coarse-grained labels and (c) percentages of instances for which 100% agree, and at least 66% and 50% agree for clustered fine-grained labels.
Pearson correlations range from 0.66 to 0.81 for temporally anchored knowledge generated with semantic roles, and from 0.59 to 0.62 for knowledge generated with syntactic dependencies. For pairs generated from semantic roles (Table 1), we observe that the correlations for both coarse-grained and clustered fine-grained labels are the highest for during (0.81, 0.79) followed by before (0.73, 0.68) and after (0.66, 0.65). For pairs generated from dependencies (Table 2), the differences in correlations are relatively minor with both coarse-grained and clustered fine-grained labels: before: 0.62, 0.62; during: 0.63, 0.60; and after: 0.59, 0.58. Thus, it is harder to determine whether an entity is or is not in a location after an event than before or during the event. The lower correlations for knowledge generated with dependencies are most likely due to the fact that neither the entity (x) or location (y) are necessarily a direct dependent of a verb.
The percentages of annotators that agree show that a majority label exists most of the time. For pairs generated from semantic roles (Table 1), four or more annotators (out of 7) agree in 70.0% of coarse-grained labels and
$\geq$
50% of annotators agree in 80.09% of fine-grained labels. Regarding temporal anchors, we observe the same trend as with Pearson correlations: the percentage is the largest with during (coarse: 81.4, fine: 93.3) followed by before (65.8, 74.5) and after (62.9, 72.5).
In the case of pairs generated from dependencies, we present the percentage of instances for which three, at least two and none of the annotators agree on a label in Table 2. We observe that (a) in 87.7% of instances, two out three annotators agree on the coarse-grained and (b) a majority of annotators agree on the clustered fine-grained label in 97.3% of instances. Note that Pearson correlation measures whether annotators choose similar labels, but the percentage of instances for which annotators agree considers all disagreements as equal.
4.4 Annotation examples
Table 3 presents annotation examples for the corpus generated from semantic roles. In Table 3 sentence 1, x is the
$\textsc{arg}_0$
of explains (Mei), y is the argm-loc of preserved (Macau) and the
$y_{verb}$
is preserved. Annotators chose unk label for the three temporal anchors as we cannot say anything about the location of x. In Table 3 sentence 2, annotators determine that Abdul Rahman (x) is located in Germany (y) during living (
$y_{verb}$
), but not before or after. In Table 3 sentence 3, annotators indicated that Balqes (x) was located at the apartment (y) for years
before and after shown (
$y_{verb}$
), and during the entire duration of shown. In Table 3 sentence 4, the annotations indicate that The state owned Petro Basque Tanker (x) is located in the concrete pier (y) for a few minutes before and during hit (
$y_{verb}$
). Additionally, annotators chose unk label for after, although one could argue that the tanker had to be around the pier after hit, even if for a very short period of time. Note that the temporally anchored spatial knowledge in Table 3 sentence 4 is of type 1a, and in Table 3, sentences 1–3 is of type type 1b.
Annotation examples for pairs generated from semantic roles
We provide annotations (C: coarse-grained, F: fine-grained) for selected potential additional spatial knowledge (is x located at y with respect to
$y_{verb}$
?)
Table 4 presents annotation examples from the corpus generated from syntactic dependencies. In Table 3 sentence 1, annotators indicated that both Hillary Rodham Clinton (x
$_1$
) and President Clinton (x
$_2$
) were in the US (y) for years
before and after shown (
$y_{verb}$
) and also for the entire durations of shown. In Table 3 sentence 2, annotators determined that Fidel Castro (x) was located in Caracas for a few days
before and after awarded (
$y_{verb}$
), and for the entire duration of awarded. In other words, they understood that Fidel Castro made a trip to Caracas to receive an award, and he stayed for a few days (but less than a week) before and after getting the award. In Table 3 sentence 3, the annotators inferred that the USS Cole was not located in Yemen and Aden before and during sent (
$y_{verb}$
) took place. They also inferred that the USS Cole was located in Yemen and Aden for a few weeks
after sent. Additionally, annotators indicated that the ship was not in Capitol Hill. In Table 3 sentence 4, annotators indicated that it is unk whether Bin Laden was or was not in Afghanistan, as the sentence does not provide enough information about his location.
Annotation examples for pairs generated from syntactic dependencies
We provide annotations (C: coarse-grained, F: fine-grained) for selected potential additional spatial knowledge (is x located at y with respect to
$y_{verb}$
?).
5. Corpora analysis
In this section, we analyse the annotated corpora with respect to their coarse- and fine-grained labels. Specifically, we discuss label distributions per temporal anchor depending on the argument x and the verb
$y_{verb}$
.
5.1 Coarse-grained label distributions
Figure 3 presents distributions for temporally anchored spatial knowledge generated from semantic roles depending on the numbered role of x. From the overall distribution (bottom right), we observe that 63.5% of the potential spatial relations are valid (yes and no labels). We also observe that x is likely to be located at y during
$y_{verb}$
(top right, 62.07%) much more likely than before (top left, 13.05%) and after (bottom left, 15.4%). Also, when the argument is arg
$_1$
semantic role, almost 70% of the potential spatial relations are validated (yes: 34.00, no: 35.74). On the other hand, when the argument x is arg
$_3$
semantic role, 75% of the potential spatial relations are invalid with respect to all the three temporal anchors. We note, however that
$\textsc{arg}_3$
is very infrequent (0.23%).
Coarse-grained label (yes, no, unknown and invalid) distribution for pairs generated from semantic roles. We show distributions based on the numbered role (x) per temporal anchor and overall. The percentages of numbered roles are as follows: arg
$_0$
: 60.85%, arg
$_1$
: 32.02%, arg
$_2$
: 6.90% and arg
$_3$
: 0.23%.
Figure 4 presents distributions for temporally anchored spatial knowledge generated from syntactic dependencies depending on the named entity of x. The overall distribution (bottom right), shows that 80% of the potential spatial relations automatically generated are valid (yes: 56.23, no: 24.45). For all three temporal anchors, over 75% of the potential spatial relations are valid. We also observe that when the argument x is a product named entity, it is the most likely to be located at y for all three temporal anchors. Finally, we note that the ratios of yes and no with temporal anchors before and after are much higher for the potential relations generated with syntactic dependencies (Figure 4) than semantic roles (Figure 3).
Coarse-grained label (yes, no, unknown and invalid) distribution for pairs generated from syntactic dependencies. We show distributions based on the named entity type of argument (x) per temporal anchor and overall. The percentage of named entity types are as follows: person: 84.25%, fac: 9.24%, work_of_art: 3.73%, and product: 2.78%.
5.2 Fine-grained label distributions
Figures 5 and 6 present the fine-grained label distributions per temporal anchor. Generally speaking, years and inf are by far the most frequent labels for before and after temporal anchors, and entire for during temporal anchor. We also note that the percentages of secs, mins, hours, days, weeks and months are much larger with semantic roles (Figure 5) than dependencies (Figure 6), but the percentage of the minority label some is much larger with dependencies than roles. Based on this distribution, we carry out experiments (Section 6) with clustered fine-grained labels: < years and
$\geq$
years.
Fine-grained label distribution for pairs generated from semantic roles.
Fine-grained label distribution for pairs generated from syntactic dependencies.
5.3 Verbs and coarse-grained label distributions
We now discuss observations from the coarse-grained label distributions depending on the verb tenses and verb classes of
$y_{verb}$
(Figures 7 and 8). We obtained verb tenses following deterministic rules (Loaiciga, Meyer, and Popescu-Belis Reference Loaiciga, Meyer and Popescu-Belis2014) and used the WordNet lexical filenames to obtain verb classes. The discussion in this section is limited to verb tenses and verb classes that account at least for 5% of all potential spatial knowledge automatically generated.
Coarse-grained label (yes, no, unknown and invalid) distribution for pairs generated from (a) semantic roles and (b) syntactic dependencies. We show distributions based on the tense of
$y_{verb}$
per temporal anchor and overall.
Coarse-grained label (yes, no, unknown and invalid) distribution for pairs generated from (a) semantic roles and (b) syntactic dependencies. We show distributions based on the lexical file name in WordNet of
$y_{verb}$
per temporal anchor and overall.
Verb tenses. The distribution of verb tenses of the potential spatial knowledge generated from semantic roles (Figure 7(a), percentage to the right of each tense) shows that tenses are clearly not uniformly distributed. Indeed, most are simple past (33.1%), followed by present (14.8%), present participle (10.5%) and past participle (8.9%). The distributions of coarse-grained labels show similar trends per temporal anchor regardless of verb tense. This may appear surprising, as one may intuitively expect the verb tense to be a good predictor of temporal anchors. Finally, as one would expect based on the overall distributions (bottom right graphs in Figures 3 and 4), the percentage of yes is larger for during than before and after across all verb tenses.
Figure 7(b) shows the distribution of verb tenses of the potential spatial knowledge generated from syntactic dependencies. The distribution of verb tenses is similar but more biased: simple past accounts for 45.9%, present for 19.6% and past perfect for 12.1%. Similar to the observation regarding potential knowledge generated with semantic roles, the label distributions follow similar trends regardless of temporal anchor. We also note that the ratios of validated knowledge (yes and no labels) are almost uniform across verb tenses for before (
${\approx}80$
%) and after temporal anchors (
${\approx}70$
%), but the distributions of yes and no labels are not: no label is much more frequent than yes label with past perfect, a trend that is reversed with other tenses. Also, note that the percentage of pairs whose
$y_{verb}$
is in present participle goes down to less than 5% for syntactic dependencies from 10.5% for roles.
Verb classes. Figure 8 shows the distributions of coarse-grained labels per temporal anchor depending on the verb class of
$y_{verb}$
. We note that regardless of whether the potential spatial knowledge is generated with roles or dependencies, most of the potential knowledge validated does not involve a motion verb (e.g., walk, fly). Indeed, motion verbs only account for 11.2% and 14.4%, respectively.
The distributions of verb classes of potential knowledge generated from semantic roles are depicted in Figure 8(a). We observe that overall (bottom right) the ratio of validated knowledge (yes and no labels) ranges from
${\approx} 60$
% to
${\approx} 80$
%, and the distributions per verb class follow the same trends regardless of temporal anchor. We also observe the most inv labels (
${\approx} 25$
%) with communication verbs (e.g., say). Finally, potential spatial knowledge involving a motion verb is more frequently annotated yes (
${\approx} 50$
%), and potential knowledge involving stative verbs is the most frequently annotated with no. Stative verbs usually express a state rather than an event; some examples include feel, love, sit and stand.
Figure 8(b) presents the coarse-grained label distributions of verb classes of potential knowledge generated with syntactic dependencies. Communication verbs are the most common (33.6%) followed by motion (14.4%), social (13.4%) and stative (10.2%) verbs. Overall (bottom right), there are more yes labels than no labels for all verb classes, but that is not the case if we look at specific temporal anchors. For example, for before and after, the ratios of yes and no labels are virtually the same with social (e.g., take, act, revolt) and possession verbs (e.g., have, own, occupy). Unlike with semantic roles, we observe that stative verbs are more likely to be annotated yes than no.
6. Experiments
In this section, we explain the experiments done with the annotations from both corpora. Each location(x, y) relation has three labels corresponding to temporal anchors before, during and after, that is, three instances per relation. We perform two classification tasks:
-
1. Coarse-grained classification to predict yes, no or unk for each instance.
-
2. Clustered fine-grained classification to predict < years,
$\geq$
years (entire and some in case of during), no or unk for each instance. This classification is inspired by previous work on determining event durations (Pan, Mulkar, and Hobbs Reference Pan, Mulkar and Hobbs2006).
Features extracted from pairs generated from semantic roles
$x_{verb}$
and
$y_{verb}$
denote the verbs to which x and y attach
We discard all the instances with inv label. We found it advantageous to train one classifier per temporal tag. For each corpus, we divide the instances into 80% train and 20% test by ensuring all the location(x, y) relations generated from a particular sentence either belong to train set or test set. We use scikit-learn (Pedregosa et al. Reference Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg, Vanderplas, Passos, Cournapeau, Brucher, Perrot and Duchesnay2011) GridSearch implementation to train one SVM per temporal anchor and tune the C and gamma parameters using 10-fold cross-validation over the train set. All the results are reported on the corresponding test set.
Features. Tables 5 and 6 list all the features used across the experiments presented in this work. We tune the feature sets using cross-validation with the training sets, and the best features selected for pairs generated from roles and dependencies have little overlap. We consider 26 features for pairs generated with semantic roles (Table 5). Basic features capture predicate information (words and part-of-speech tags). Lexical, heads and semantic features are standard in semantic role labelling (Gildea and Jurafsky Reference Gildea and Jurafsky2002) and we do not elaborate on them. Situation entity features are adapted from a system for assigning situation entity type labels to clauses (Friedrich, Palmer, and Pinkal Reference Friedrich, Palmer and Pinkal2016). Situation entity types are a linguistic categorisation of semantic clause type, whereby each clause of text is labeled according to the type of situation it introduces to a discourse. The main types are state, event, generic and generalizing sentence (also known as habituals). Diagnostic criteria for situation entity types include lexical aspect (stative vs. dynamic) of the main verb, genericity of the clause’s subject, and whether the clause is episodic, habitual or static. It is primarily these criteria which features 25–36 aim to capture.
Features extracted from pairs generated from syntactic dependencies
$y_{verb}$
is the closest verb going up the dependency tree from y.
Regarding pairs generated with syntactic dependencies, we consider 40 features (Table 6). Basic features capture elemental information regarding y
$\_{verb}$
, argument x and location y. Lexical features capture context around x and y. Heads features categorise x and b based on their heads and named entity types. Syntax features capture dependency structure of x and y. Finally, semantic features add information regarding location and temporal semantic roles in the sentence.
7. Results
We performed experiments on both corpora using gold linguistic information as well as predicted linguistic information. Gold linguistic information (part-of-speech tags, named entities, syntactic dependencies and semantic roles) is taken from the CoNLL-2011 Shared Task release, and predicted linguistic information is obtained using SyntaxNet (Chen and Manning Reference Chen and Manning2014) (syntactic dependencies) and the CoNLL-2011 Shared Task auto files (semantic roles).
The baseline system for both corpora predicts the most frequent tag per temporal anchor. For pairs generated with semantic roles, baseline results per temporal anchor are 0.31 F1 for before, 0.64 F1 for during and 0.22 F1 for after. For pairs generated with syntactic dependencies, baseline results per temporal anchor are 0.29 F1 for before, 0.49 F1 for during and 0.29 F1 for after.
7.1 Results with gold linguistic information
Table 7 presents results with pairs generated from semantic roles and coarse-grained labels. Basic features and situation entities features yield virtually the same results individually (overall, 0.51 and 0.50 F1, respectively). Situation entities features, however, are more beneficial than basic features when combined with lexical, semantic and heads features (overall, 0.59 vs. 0.63 F1). This leads to the conclusion that situation entities successfully capture information about verbs that is not readily available in their part-of-speech tags. Regarding temporal anchors, we observe that results with during are consistently higher than with before and after, following a similar trend as with the agreements observed (Section 5). The difference in results obtained with basic features and the best system (situation entities + lexical + semantic + heads) is statistically significant (two-tailed Z test, p
$<$
0.05).
Experimental results (Precision, Recall and F1) with pairs generated from semantic roles and coarse-grained labels
We provide results using several feature combinations and extracting features from gold linguistic information.
Table 8 presents results with pairs generated from syntactic dependencies and coarse-grained labels. In general, using different feature combinations has a smaller impact on results than when experimenting with pairs generated from semantic roles (0.50–0.54 F1 vs. 0.51–0.63 F1). We note, however, that using all features obtains the best results with respect to all temporal anchors and that results are more uniform across temporal anchors (overall: 0.54 F1; before: 0.55, during: 0.56, after: 0.50). F1 for before and during are higher than after, similar to the agreements observed (Section 5). The differences in results obtained with basic features and all features (basic + lexical + heads + syntax + semantic) are statistically significant (0.50 F1 vs. 0.54 F1 2-tailed Z test p
$<$
0.05).
Experimental results (Precision, Recall and F1) with pairs generated from syntactic dependencies and coarse-grained labels
Regarding clustered fine-grained labels, we present results with the best combination of features (all features with both roles and dependencies, Tables 7 and 8) in Table 9. We observe similar trends as with coarse-grained labels: (a) results are better for during than before and after with pairs generated from both semantic roles and syntactic dependencies and (b) results are less uniform with pairs generated from roles (0.53–0.71 F1) than dependencies (0.50–0.52 F1). Regarding results per label,
$\geq$
years obtains better results than
$<$
years (before and after), and some is never predicted (for the temporal tag during, recall that there are very few instances annotated some, Section 5).
Top five features ranked based on mutual information score for pairs generated from semantic roles and dependencies
We also perform univariate feature selection using mutual information score (Kraskov, Stögbauer, and Grassberger Reference Kraskov, Stögbauer and Grassberger2004) to select the most useful features incrementally. Mutual information measures how much information the presence or absence of a feature contributes in making the correct classification decision. We present the percentage contribution (y-axis) of top N features (x-axis) for pairs generated from semantic roles and syntactic dependencies in Figure 9. Percentage contribution is measured as the ratio with respect to the best results obtained (Tables 7 and 8). Recall that the best results with roles are 0.56 (before), 0.75 (during), and 0.59 (after) and with dependencies 0.54 (before), 0.56 (during), and 0.50 (after). There are two major conclusions from these graphs. First, few features (roles: 5, dependencies: 1) are required to obtain the best results for during temporal anchor. Second, all features contribute to the learning for before and after temporal anchors, especially for after with semantic roles and before with syntactic dependencies.
Percentage contribution of top N features for pairs generated from semantic roles and syntactic dependencies. The top N features are ranked by their mutual information score. Experiments are performed using coarse-grained labels and gold standard linguistic annotations.
7.2 Results with predicted linguistic information
Experimenting with gold linguistic information is worthwhile, but results are not necessarily representative of what one would obtain in a realistic scenario, that is, using predicted linguistic information. First, generating potential spatial knowledge heavily relies on semantic roles and syntactic dependencies. Thus, missing and spurious roles and dependencies will lead to potentially wrong temporally anchored spatial knowledge. Second, features should be extracted from predicted linguistic information to estimate results in a realistic scenario. In this section, we experiment with predicted linguistic information to provide insights into extracting temporally anchored spatial knowledge in a realistic scenario. Evaluation metrics (Precision, Recall and F1, Tables 11 and 12) are calculated using the pairs generated with predicted linguistic information that overlap with the ones generated from gold linguistic information (286 out of 842 instances with semantic roles, and 655 out of 1076 instances with syntactic dependencies).
Experimental results (Precision, Recall and F1) with pairs generated from semantic roles and predicted linguistic information
We provide results using the best features combinations from Table 7.
Experimental results (Precision, Recall and F1) with pairs generated from syntactic dependencies and predicted linguistic information
We provide results using the best features combinations from Table 8.
Table 11 presents results using pairs generated from semantic roles, predicted linguistic information and the best feature combination (all features, Table 7). Regarding coarse-grained labels, results are virtually the same as the one obtained with gold linguistic information (overall, 0.62 vs. 0.63), and we observe the same trends per temporal anchor. When experimenting with clustered fine-grained labels, we observe a decline (overall, 0.57 vs. 0.61 F1). The decline is minimal with before (0.53 vs. 0.55 F1) but substantial with after (0.43 vs. 0.53 F1).
Table 12 presents results using pairs generated from syntactic dependencies, predicted linguistic information and the best feature combination (all features, Table 8). Regarding coarse-grained labels, F1 are lower across all temporal anchors except during: before: 0.49 versus 0.55, during: 0.56 versus 0.56, after: 0.44 versus 0.50. The differences are much smaller with clustered fine-grained labels: 0.49 versus 0.51 F1 with before, same results with during (0.52 F1), and 0.50 versus 0.51 with after.
The results obtained with pairs generated from semantic roles are higher than the ones obtained with syntactic dependencies when working with both gold and predicted linguistic information. These results, however, are not directly comparable with each other because of the different test sets. Out of the 1689 pairs generated using dependencies, y is a semantic role of some verb in only 205 (12%) pairs and y is an argm-loc(y
$_{verb}$
, y) in only 60 (3.5%) pairs. Hence, there is very little overlap between the potential spatial knowledge generated from the two approaches.
8. Semantic roles versus syntactic dependencies
As discussed earlier, PropBank semantic roles are focused on the annotation of events and their participants, without singling out modifiers of the entities that might be those participants. As such, spatial knowledge about entities must be inferred from spatial knowledge of the related events. Syntactic dependencies, on the other hand, provide a thorough, comprehensive treatment of every element of a sentence, and this includes complete coverage of any entities and their modifiers. Therefore, as would be expected, syntactic dependencies represent a much richer set of relations that can be mined for entity specific spatial knowledge. Working with syntactic dependencies for these purposes is especially beneficial for three main reasons:
-
1. Dependencies allow us to target temporally anchored spatial knowledge of type 1 and type 2. Because PropBank-style semantic roles annotate semantic arguments of verbs, they disregard temporally anchored spatial knowledge when either an entity x or a location y is not the argument of a verb. In our experiments, 72% of temporally anchored spatial knowledge generated from dependencies is of type 2 (1211 out of 1689 generated pairs), that is, either x or y is not a semantic role of a verb.
-
2. The potential temporally anchored spatial knowledge generated with dependencies is validated (yes and no labels) more frequently. Specifically, yes is chosen 56.2% and no 24.5% with dependencies (total: 80.7% Figure 4), and 30.2% and 33.3% with roles (total 63.5% Figure 3).
-
3. Dependencies are more reliable than roles when working with predicted linguistic information. Specifically, only 34% of potential temporally anchored spatial knowledge about entities can be generated when working with predicted semantic roles (286 out of 842 pairs), compared to 61% with predicted syntactic dependencies (655 out of 1076 pairs). Thus, despite the apparent better results obtained with roles (Tables 11 and 12), syntactic dependencies would obtain much higher recall values in a realistic scenario.
9. Conclusions
We have presented a two-step methodology to extract temporally anchored spatial knowledge from text, that is, determine whether an entity x is located or not located at a location y, and specify when with respect to an event. Specifically, we automatically generate potential knowledge manipulating semantic roles or syntactic dependencies and then validate the potential knowledge. Crowdsourcing experiments show that temporally anchored spatial knowledge is ubiquitous and the validation step can be done by non-experts. Additionally, most of the potential knowledge automatically generated is valid (coarse-grained labels yes and no).
We have experimented with support vector machine classifiers and several feature combinations using both gold and predicted linguistic information. Results show that the task of extracting temporally anchored knowledge about entities can be automated, including both coarse-grained and clustered fine-grained labels. When working with predicted linguistic information (i.e., in a realistic scenario), syntactic dependencies are better suited than semantic roles, as dependencies allow us to extract temporally anchored knowledge involving entities and locations that are not the argument of a verb. Additionally, dependencies allow us to generate more of the potential knowledge about entities that one would generate with gold linguistic information (dependencies: 61%, roles: 34%).
