2.1 Refactoring and normalising annotated data

Previous efforts have standardised corpora for ease of reuse (Johnson et al. Reference Johnson, Baumgartner, Krallinger, Cohen and Hunter2007; Heimonen et al. Reference Heimonen, Pyysalo, Ginter and Salakoski2008) and for comparative evaluation (Pyysalo et al. Reference Pyysalo, Airola, Heimonen, Björne, Ginter and Salakoski2008). However, they have not investigated corpora as diverse as ACE and BioInfer. Furthermore, they have not required that the data sets have multiple relation types that are not determined by the types of the entities forming a relation.

Johnson et al. (Reference Johnson, Baumgartner, Krallinger, Cohen and Hunter2007) report a feasibility study on corpus refactoring. The effort is motivated by a previous study suggesting that format is a primary factor in corpus uptake (Cohen et al. Reference Cohen, Fox, Ogren and Hunter2005). The authors take the Protein Design Group corpus as an example, converting the document-level annotation to character-offset annotation in two popular formats. Our standardisation is also motivated by accessibility. However, normalisation of diverse data sets is an additional key goal, allowing comparison of relation extraction across domains.

Pyysalo et al. (Reference Pyysalo, Airola, Heimonen, Björne, Ginter and Salakoski2008) describe an effort to normalise four protein–protein interaction corpora to a ‘a shared level of information, consisting of undirected, untyped binary interactions.’ Like Pyysalo et al., our reannotation of BioInfer flattens nested entities, maps n-ary to binary relation mentions and ignores directionality of relations. Unlike Pyysalo et al., our reannotation additionally maps part-whole to part-part relations. Crucially, where Pyysalo et al. focus exclusively on protein-protein interactions as their only relation type, we have the explicit goal of multi-type relation extraction across domains. Therefore, we maintain all physical entity types and all BioInfer relations over physical entity types. In addition, we consider data from the news and biomedical domains.

Heimonen et al. (Reference Heimonen, Pyysalo, Ginter and Salakoski2008) describe a process for converting the complex, structured annotation of the BioInfer corpus to binary relations. They argue that this is a necessary simplification towards extraction of detailed knowledge about protein–protein interactions and demonstrate that the binarisation is largely valid with limited loss of information. Heimonen et al. do not perform other types of normalisations described here. In particular, they do not map part-whole to part-part relations (see Section 6.2). Furthermore, they do not explore compatibility across domains.

2.2 Generic relation extraction (GRE)

Relation extraction can be addressed using supervised, bootstrapping or generic approaches. One way to characterise them is in terms of adaptation cost, i.e. the amount of work necessary to adapt them to a new domain or task. In these terms, supervised approaches (e.g. Aone et al. Reference Aone, Halverson, Hampton and Ramos-Santacruz1998; Bunescu et al. Reference Bunescu, Ge, Kate, Marcotte, Mooney, Ramani and Wong2004) incur the highest cost as systems need to be built largely from scratch for each new domain. Bootstrapping approaches (e.g. Brin Reference Brin, Atzeni, Mendelzon and Mecca1999; Agichtein and Gravano Reference Agichtein and Gravano2000) incur less cost as they require only a small amount of seed data. Finally, generic approaches (e.g. Conrad and Utt Reference Conrad and Utt1994; Hasegawa, Sekine and Grishman Reference Hasegawa, Sekine and Grishman2004) provide domain adaptation for free, as parameters do not need to be modified for new domains or tasks.

Another way to characterise these approaches is in terms of the ontology creation problems they address, i.e. whether they address the instantiation task where instances are added to an ontology in a new domain given a relation schema (the taxonomy of relation types to be identified) or whether they also address the task of learning the relation schema for the new domain. In these terms, the supervised and bootstrapping approaches address only the ontology instantiation problem, while the generic approaches also address the problem of learning relation schemas from data. The tradeoff is in terms of accuracy, where generic approaches suffer when compared to supervised and bootstrapping approaches. However, generic approaches have high utility in terms of developing cheap components for applications like paraphrase acquisition (Hasegawa, Sekine and Grishman Reference Hasegawa, Sekine and Grishman2005), on-demand information extraction (Sekine Reference Sekine2006) and automatic summarisation (Hachey Reference Hachey2009a).

In Section 8, we present summary experiments for GRE across news and biomedical domains. These experiments illustrate the utility of the standardised corpora described in this paper.

2.3 Data sets

The data for the news domain is derived from the IE corpora that was prepared for the 2004 and 2005 ACE-shared tasks (Mitchell et al. Reference Mitchell, Strassel, Huang and Zakhary2005; Walker et al. Reference Walker, Strassel, Medero and Maeda2006).Footnote ⁵ The data discussed here is drawn from the newswire and broadcast news materials prepared for the 2004 and 2005 RE tasks (LDC 2004b, 2005b). Table 2 summarises the documents in these corpora. The first column (Source) corresponds to the name of the organisation from which the data was obtained. The second column (Type) corresponds to the media type of the data source. Newswire indicates that the data was obtained from a printed news feed. Broadcast news indicates that the data is obtained from the transcript of a spoken news programme. The data from broadcast news sources is generally well edited, though does not contain capitalisation. The third and fourth columns correspond to the range of months during which the sources were published (Epoch) and the number and distribution of documents from each source organisation (Num docs). The total number of documents is 348 and 298 for ACE 2004 and 2005, respectively. In addition, the overall newswire/broadcast news splits are approximately 36.8%/63.2% and 27.2%/72.8%, respectively.

Table 2.

Sources for ACE 2004 and ACE 2005 news data

The data for the biomedical domain is derived from the IE corpora, which have been prepared and freely distributed as the BioInfer corpus (Pyysalo et al. Reference Pyysalo, Ginter, Heimonen, Björne, Boberg, Järvinen and Salakoski2007).Footnote ⁶ This consists of 1,100 sentences that were selected from the PubMed database of biomedical literature.Footnote ⁷ The corpus data was collected by entering known pairs of interacting proteins from the Database of Interacting Proteins (DIP)Footnote ⁸ as PubMed search terms. Resulting abstracts (including titles) were searched for sentences containing mentions of two proteins that are known to interact. The epoch of the resulting corpus includes publication dates up to December 2001, which is when the sentence selection process was carried out.

3.1 Refactoring ACE

The ACE data is encoded in SGML, does not include sentence or word token markup and uses character standoff annotation for entities and relations. Therefore, conversion to RE XML requires SGML-to-XML conversion, tokenisation and mapping from character offset to token offset. First, character entity references are converted to standard XML and spaces are added, where necessary, to maintain correct character offsets. ACE does not have token markup. Therefore, we add sentence boundary and word token markup using LT-TTT2,Footnote ⁹ a general purpose text tokenisation tool based on the generic XML text processing tools in LT-XML2 (Grover et al. Reference Grover, Matthews and Tobin2006).Footnote ¹⁰

Next, the conversion from character to token standoff is performed using LT-XML2 tools. This is achieved by first wrapping each character with an element and then mapping the standoff from the character elements to the word token elements. Occasionally the original ACE markup spans substrings of words, for example, in the possessive ‘its’, only the substring ‘it’ is marked up as an entity mention. In cases such as this, the converted markup must point at the tokens rather than token substrings so the converted entity mention is ‘its’ rather than ‘it’. Although we do not utilise it in the experiments here, information about the original character offsets is preserved using the end offset (@eo) and start offset (@so) attributes on ne elements. Thus, the ne element for ‘its’ is marked up with @eo ‘-1’.

After this, the data is well-formed XML using token offsets and the conversion to basic RE XML is a simple XML-to-XML transformation. Figure 4 contains an example document with the basic RE XML markup. This is drawn from the ACE data and contains markup for the following sentence:

The markup in the figure specifies five entities (ne). The first ne element (with @id ‘e60’), for example, contains the markup for the Person entity ‘American saxophonist David Murray’, which starts (@fr) and ends (@to), respecitively, with word tokens ‘w292’ and ‘w295’ and has ‘David Murray’ as its head. The markup in the figure also specifies three relations: a Citizen-or-Resident relation between ‘saxophonist’ and ‘American’, a Business relation between ‘David Murray’ and ‘Amidu Berry’ and a Business relation between ‘David Murray’ and ‘Awadi’. In Section 6 we argue that the Citizen-or-Resident relation should actually be between ‘David Murray’ and ‘American’ and we describe rules for automatically mapping this and other similar cases where the ACE annotation marks relations with a nominal rather than a named entity mention.

Fig. 4.

Example of refactored ACE document with basic RE XML markup.

3.2 Refactoring BioInfer

The BioInfer data, already encoded in XML, includes sentence and word token markup, and uses token standoff annotation for entities and relations. Therefore, conversion to the RE XML document type is a matter of simple XML-to-XML transformation. In addition, while the information is not used for the experiments here, Not relations (specifying negation) are mapped to a negation attribute (@neg) on relation elements (rel). And, Equal and Corefer relations are converted to coreference information in the form of a grounded entity identifier attribute (@gid) on entity elements (ne).

Figure 5 contains an example document with the basic RE XML markup. This is drawn from the BioInfer data and contains markup for the following sentence:

The markup in the figure specifies two entities (ne). The first ne element (with @id ‘e75’) contains the markup for the Substance entity ‘Beta-catenin’, which starts (@fr) and ends (@to) with the word token (w) with @id ‘w211’. The markup in the figure also specifies a relation (with @id ‘r32’) of type Causal between entity ‘e75’ (‘Beta-catenin’) and entity ‘e77’ (‘structures’). Note that not all BioInfer entities consist of continuous token sequences. The phrase ‘alpha 5 and beta 1 integrins’, for example, is annotated with two entity mentions ‘alpha 5 integrins’ and ‘beta 1 integrins’. For the first mention, the @fr, @to markup here simply indexes into ‘alpha’ and ‘integrins’ and these boundaries are used for the experiments. However, the non-continuous annotation is saved in exattr elements.

Fig. 5.

Example of refactored BioInfer document with basic RE XML markup.

4.1 Adding TTT linguistic information to ACE

Figure 7 contains the RE XML markup corresonding to the linguistic information from TTT for the ACE example sentence. The markup in the figure specifies one verb phrase ‘recruited’ indicated by the ‘B-VP’ value of the @phr attribute on the corresponding w element. The markup for ‘recruited’ also indicates the following: the part-of-speech is past tense verb (@p=‘VBD’), the lemma is ‘recruit’ (@l=‘recruit’) and the verb phrase is in active voice (@voice=‘act’). In addition, the markup in the figure specifies three noun phrases ‘American saxophonist’, ‘David Murray’ and ‘Amidu Berry and DJ Awadi’.

Fig. 7.

RE XML markup for TTT linguistic information (ACE example).

4.2 Adding TTT linguistic information to BioInfer

Figure 8 contains the RE XML markup corresponding to the linguistic information from TTT for the BioInfer example sentence. The markup in the figure specifies one verb phrase ‘is also found’ with main verb ‘found’ indicated by the @headv attribute on the corresponding w element. The markup for ‘found’ also indicates the following: the part-of-speech is past participle verb (@p=‘VBN’), the lemma is ‘find’ (@l=‘find’) and the verb phrase is passive (@voice=‘pass’). In addition, the markup in the figure specifies two noun phrases, ‘Beta-catenin’ and ‘these structures’.

Fig. 8.

RE XML markup for TTT linguistic information (BioInfer example).

5.1 Adding dependency parse information to ACE

Figure 10 contains the Minipar dependency parse of the example ACE sentence. Word tokens constitute nodes in the dependency graph, arks specify relations where the word token at the end of the arrow is the dependent token and the annotations (e.g. s+subj) between arrow heads and word tokens specify the relation types. Dependency relations include, e.g. a modifier (mod) relation from governor noun ‘Murray’ to dependent adjective ‘American’, and an object (obj) relation from ‘recruited’ to dependent noun ‘Berry’. Note that the subject (s+subj) relation from governor verb ‘recruited’ to dependent noun ‘Murray’ is not actually a single relation in the Minipar output. It orignially consists of a surface subject (s) relation from governor verb ‘recruited’ to dependent noun ‘Murray’ and a subject (subj) relation from governor verb ‘recruited’ to an introduced empty node marked as being coreferent with the noun ‘Murray’. These are collapsed to a single s+subj relation as a postprocessing step. This serves to connect verbs directly to arguments with deep grammatical relation types. Figure 11 contains the RE XML markup corresponding to the dependency parse. The first dpg element, for example, encodes the modifier relation between ‘American’ (@cr=‘mod:w295’) and ‘Murray’ (@d ‘w292’).

Fig. 10.

Dependency parse for an example of ACE sentence.

Fig. 11.

RE XML markup example of ACE dependency parse.

5.2 Adding dependency parse information to BioInfer

Figure 12 illustrates the Minipar dependency parse of the BioInfer example sentence.Footnote ¹⁴ Again, word tokens constitute nodes in the dependency graph, arks specify relations where the word token at the end of the arrow is the dependent token and the annotations (e.g. s+obj) between arrow heads and word tokens specify the relation types. Dependency relations include, e.g. a surface subject/object (s+obj) relation from ‘found’ to ‘Beta-catenin’, and a nominal in modifier (mod*in*n) from ‘found’ to ‘structures’. Note that the nominal in modifier (mod*in*n) relation from governor verb ‘found’ to dependent noun ‘structures’ is not actually a single grammatical relation in the Minipar output. It originally consists of two relations: a modifier (mod) relation from governor ‘found’ to dependent ‘in’ and a nominal preposition complement (pcomp-n) relation from governor ‘in’ to dependent ‘structures’. These are collapsed to a single relation as a post-processing step following Lin and Pantel (2000). This serves to connect the prepositional complement directly to the words modified by the preposition. Figure 13 contains the RE XML markup corresponding to the dependency parse. The first dpg element, for example, encodes a relation of type object (s+obj) between the governor token with identifier ‘w214’ (@cr=‘s+obj:w214’) and the dependent token (@d) with identifier ‘w211’ (i.e. ‘Beta-catenin’ is the object of ‘found’).

Fig. 12.

Dependency parse for example of BioInfer sentence.

Fig. 13.

RE XML markup example of BioInfer dependency parse.

6.1 Reannotating ACE

Reannotation of ACE is motivated by the prevalence of nominal entity mentions in ACE, where entities can be referenced by their name (i.e. named mention), by a common noun or noun phrase (i.e. nominal mention) or by a pronoun (i.e. pronominal mention). The mapping is facilitated by the presence of detailed linguistic annotation in the ACE corpus, which makes it possible to automatically map many nominal entity mentions to named entity mentions. Several aspects of the ACE annotation are used in the mapping rules: entity extent, entity type and entity mention type. This information is represented by typesetting conventions illustrated in the following text snippet:

This contains one nominal and two named entity mentions:

1. (1) ‘Amidu Barry’ with type Person (PER) and mention type named (NAM).

2. (2) ‘one half of PBS’ with type Person and mention type nominal (NOM).

3. (3) ‘PBS’ with type Organisation (ORG) and mention type named.

We will also talk about embedding and embedded entity mentions. In the above example, ‘one half of PBS’ is embedding and ‘PBS’ is embedded.

Table 3 contains the full list of possible entity mention types with the number and proportion of occurrences in the ACE 2004 and the ACE 2005 data sets. The rules used here only consider nominal and prenominal entity mentions for possible mapping. Unquantified nominal mentions are generally not coreferent with named entity mentions and other mention types are too rare. The ACE 2004 entity types include Person (PER), Organisation (ORG), Facility (FAC), Location (LOC), Geographical/Political (GPE), Vehicle (VEH) (LDC 2004c). The ACE 2005 entity types include Person (PER), Organisation (ORG), Geographical/Social/Political (GPE), Location (LOC), Facility (FAC), Vehicle (VEH) and Weapon (WEA) (LDC 2005a).

Table 3.

Entity mention types in the ACE source data. Columns contain the mention-type label (label), a description (description), an example (example) and the count and percentage of occurrences for ACE 2004 and ACE 2005

Entity phrase heads and coreference are also annotated in ACE. Head markup is based on the notion of headedness in syntactic grammars, where a head is the word or category that gets propagated up a phrase structure tree. Another way of describing this notion is that a head is the main word associated with the root of a phrase or sentence and as such is the word that is described or specified by the non-head branches in a parse tree. Heads are indicated here by an underscore, e.g. ‘one’ in ‘one half of PBS’.Footnote ¹⁵ Coreference markup indicates the entity mentions that refer to the same underlying entity (e.g. ‘one half of PBS’ and ‘Amidu Barry’ above). In the following description, coreference will either be noted or obvious from the context. Finally, some of the mapping rules described below make use of aspects of the linguistic preprocessing introduced in Section 4.

Figure 14 contains three example mappings from different rules. Mapping rule 1 is possible because the entity mention ‘Michael Martin’ is coreferent with and embedded within the entity mention ‘Commons speaker Michael Martin’ and because the latter is annotated as having a prenominal mention type, indicating that it occurs in a modifying position before another noun. Thus, the embedded relation mention Executive (‘Commons’, ‘Commons speaker Michael Martin’) is converted to Executive (‘Commons’, ‘Michael Martin’).

Fig. 14.

Example of ACE mappings from nominal to named entity mentions.

Mapping rule 5 is possible because the entity mention ‘Amidu Barry’ is coreferent with and immediately to the left of the entity mention ‘one half of PBS’ and because the latter is annotated as having a nominal mention type. Thus, the embedded relation mention Membership(‘one half of PBS’,‘PBS’) with nominal entity mention ‘one half of PBS’ is converted to the non-embedded, fully named relation mention Membership(‘Amidu Barry’,‘PBS’). Mapping rule 6 is analogous except that it maps to a named entity mention to the right, converting the embedded relation mention Part-Whole(‘Ecuador’, ‘Ecuador's capital’) with nominal entity mention ‘Ecuador's capital’ to the entity mention Part-Whole(‘Ecuador’,‘Quito’).

The full list of mapping rules is found in Table 4. The first column (#) lists the rule number. The second column (Description of mapping rule) contains a brief textual description of the mapping rule, where the mention type of the original entity mention is on the left, followed by the ↦ symbol, followed by a specification of how the target entity mention for the mapping rule is identified. Finally, the third (ACE 2004) and fourth (ACE 2005) columns contain a count of how many times each rule fired and the percentage of total firings accounted for by each rule for the respective data sets. Rules are ordered from those that are the most constrained to those that are the least constrained.

Table 4.

Full list of rules for mapping from nominal to named entity mentions in ACE. Columns contain the rule identifier (#), the rule description (description) and the number and percentage of firings for ACE 2004 and ACE 2005

Rule 2 maps to a coreferent and embedded entity mention occurring immediately to the left of the head of the original entity mention, for example:

Rule 3 maps to any coreferent and embedded entity mention, for example:

Rule 4 maps to any coreferent and embedding entity mention, for example:

Note that this relation can be further simplified to Part-Whole(‘Senegal’, ‘African’) by taking entity heads only.

Rules 5 and 6 map to immediately adjacent coreferent entity mentions. These rules, as described above, are illustrated in Figure 14(b) and (c). Rules 7 and 8 map to coreferent entity mentions found respectively to the left or the right, and have only a copular verb phrase (i.e. a verb phrase where the lemma of the main verb is ‘be’) and any number of adverbs intervening, for example:

Rules 9 and 10 again map to coreferent entity mentions found respectively to the left or the right. However, they allow two intervening verb phrases and any number of adverbs as long as the second verb phrase consists of an infinitival copula (i.e. ‘to be’), for example:

Rules 11 and 12 map once again to coreferent entity mentions found respectively to the left or the right. In this instance, however, they allow any number of coreferent entity mentions to intervene between the original nominal entity mention and the target named or pronominal entity mention, for example:

where both nominal entity mentions (i.e., ‘a Washington insider’ and ‘a lobbyist for General Motors’) are coreferent with the named entity mention ‘Card’.

Finally, Rules 13 and 14 are general rules that map to any coreferent entity mentions found respectively to the left or the right, for example:

Rules 13 and 14 fired a total of ninety-two times for the ACE 2004 data and eighty-two times for the ACE 2005 data (Table 4). In order to assess the accuracy of Rules 13 and 14, a random sample of twenty firings was inspected for ACE 2004. Six (30%) were found to be noisy (details below), which correspond to 27.6% of the full ninety-two firings of Rules 13 and 14. Since Rules 1–12 do not introduce noise, the overall error rate for the ACE 2004 rule firings listed in Table 4 is 3.8% (27.6/719), and the overall error rate for the full ACE 2004 data set used in the experiments here (see Table 7) is less than 2% (27.6/1,400). We consider here a small amount of noise to be a reasonable tradeoff for a larger data set.

Among the sample, four firings create relation mentions that are questionable or could arguably have been mapped to a more suitable target entity mention, e.g. Rule 13 triggers the following mapping:

Here the pronominal entity mention ‘it’ could arguably be mapped to the named entity mention ‘Labor’. However, mapping away from pronominal mention here is inconsistent with the overall goal of a corpus of relations over named and pronominal entity mentions.

Another noisy mapping is encountered when a nominal entity mention is mapped out of an embedded entity mention, but the other entity mention is a possessive pronoun that is not mapped, for example:

This questionable mapping occurred twice in the sample. In the current work, the mapping is allowed to fire and the resulting relation mentions are kept in the final data.

6.2 Reannotating BioInfer

The reannotation of BioInfer is motivated by the fact that the BioInfer annotation sometimes marks part-whole and part-part relation mentions differently depending on their syntactic context. It is also motivated by the fact that the BioInfer annotation sometimes marks relations with more than two arguments. Table 5 contains details of the mapping rules. The first column (#) lists the rule number. The second column contains a brief rule description on the line where the rule number is given (e.g. Part-Whole ↦ Part-Part). Below this, the second column contains a list of relation types that are affected in the original BioInfer source data. The third column (Source) contains a count of how many relation mentions are mapped and the corresponding percentage (of the total number of mapped relation mentions). The fourth column (Mapped) contains a count of how many new relation mentions are created by the mapping rules and the corresponding percentage of total new relation mentions.

Table 5.

List of rules for reannotation of in BioInfer. Columns contain the rule identifier (#), the rule description and the affected relation types (description/relation type) and the number and percentage of relation mentions of the given type in the source (source) and the mapped (mapped) data

Rule 1 in Table 5 addresses relation mentions that are marked differently depending on their syntactic context, by mapping part-whole relation mentions to part-part. Consider the following two sentences:

The first sentence is annotated with one Bind(‘Smooth muscle talin’, ‘skeletal muscle actin’) relation mention. The second sentence, however, is annotated with two Contain relation mentions, where the entity mention ‘complex of birch profilin and skeletal muscle actin’ is the whole and the entity mentions ‘birch profilin’ and ‘skeletal muscle actin’ are the respective parts. In the BioInfer relation-type schema (Pyysalo et al. Reference Pyysalo, Ginter, Heimonen, Björne, Boberg, Järvinen and Salakoski2007), Bind is defined as a non-covalent binding (i.e. formation of a complex, association) between the arguments and Contain is defined as a component being part of a complex. For the annotation to be consistent across the two sentences, the second sentence should also have a relation mention between ‘birch profilin’ and ‘skeletal muscle actin’. Therefore, a Co-X relation mention is added between each entity mention that is annotated as being part of the same whole, e.g. Co-Contain(‘birch profilin’, ‘skeletal muscle actin’). Co-X relations do not have a clear semantics; however, they do exist and are expressed in the text. Therefore, they are used here for the relation identification experiments (Section 8.1) but ignored for the relation characterisation experiments (Section 8.2).

Rule 2 in Table 5 addresses relations over more than two entity mentions in BioInfer, by mapping n-ary relation mentions to binary relation mentions over all possible entity mention pairs. Consider the following example:

Here, the top relation mention with three arguments is replaced by the three distinct binary relation mentions that follow it. Note that it has been argued that biomedical relations need to be n-ary (Rzhetsky et al. Reference Rzhetsky, Iossifov, Koike, Krauthammer, Kra, Morris, Yu, Dubou, Weng, Wilbur, Hatzivassiloglou and Friedman2004; Wattarujeekrit, Shah and Collier Reference Wattarujeekrit, Shah and Collier2004; McDonald et al. Reference McDonald, Pereira, Kulick, Winters, Jin and White2005; Cohen and Hunter Reference Cohen and Hunter2006). However, while the mapped binary relation mentions here do not necessarily capture the simultaneous interaction expressed in the original annotation, they are appropriate for many applications (e.g. large-scale automatic search and knowledge discovery tasks like GRE) and are compatible with the domain-general notion of relation adopted here. Furthermore, of 2,424 relations that result from the refactoring described in Section 3.2, only thirty-four are n-ary relations (Table 5). In addition, the original relation identifiers are retained, allowing n-ary relations to be reconstructed.

7.1 ACE data for GRE

Filtering entity mentions. First, all entity mentions that do not have a mention type of named (NAM), pronominal (PRO) or prenominal (PRE) are filtered. (The full list of entity mention types can be viewed by referring back to Table 3.) This serves to remove all nominal mentions, which are not reliably recognised by most named entity recognition systems. Prenominal mentions are kept because they are often names (e.g. ‘Labour’ in ‘[_nom^per [_pre^org Labour] nominee]’), though not always (e.g. ‘British prime minister’ in ‘[_nam^per [_pre^per [_pre^gpl British] prime minister] Tony Blair]’). The ACE 2005 data actually distinguishes between named and non-named prenominal mentions. However, the ACE 2004 (development) data does not, so the distinction is ignored for the evaluation here. Entity mentions are also filtered based on mention class. The filtering here removes all entity mentions that are underspecified (e.g. ‘many people’), generic (e.g. ‘extremist groups’) or negatively quantified (e.g. ‘no one’). This leaves only entity mentions that are specific referential, i.e. mentions that refer to a particular, unique object or set of objects in the real world.

Filtering relation mentions. Next, relation mentions are removed where one of the entity mentions is no longer part of the annotation because of the entity filtering rules. Finally, relation mentions in ACE 2004 with relation-type Discourse are removed. According to the ACE 2004 Annotation Guidelines for Relation Detection and Characterization (LDC 2004b), ‘a Discourse relation is one where a semantic part-whole or membership relation is established only for the purposes of the discourse’. Examples include ‘Many of these people’ and ‘each of whom’. In ACE 2004, 279 discourse relation mentions were filtered. In ACE 2005, discourse relation mentions were discontinued. After filtering, the ACE 2004 data has 13,358 entity mentions and 1,511 relation mentions (down from 22,736 and 4,374, respectively, in the original source). And the ACE 2005 data has 10,345 entity mentions and 975 relation mentions (down from 18,086 and 3,658, respectively).

Converting to the final schema. Finally, entity and relation types are changed to the final schema. This is a simple automatic mapping from the original schema, which serves to simplify the schemas and make them more similar across the development and test sets. Table 6 lists the mapping rules, with the first column (#) containing the numeric rule identifier, the second column (Source) containing the types as they are found in the original source data, the third column (Target) containing the types after mapping and the last four columns containing the number and proportion of occurrences in the ACE 2004 and ACE 2005 data sets, respectively. In the Source and Target columns, entity and relation-type labels prefixed with ‘T:’ are types and labels prefixed with ‘S:’ are subtypes. Rows 1 through 2 of Table 6(a), for example specify that all entity mentions with type GPE (geo-political) or LOC (location) are changed to have a single common type GPL. And, Rows 1 through 4 of Table 6(b) specify that all relation mentions with subtype Located, Near, Based-In or Org-Location (located, based, headquartered, operates etc.) are changed to have type GEN-AFF (affiliation) and subtype Located.

Table 6.

Changes in ACE entity and relation-type schemas. Columns contain the rule identifier (#), the source types (source), the target types (target) and counts and percentages for ACE 2004 and ACE 2005. ‘T:’ and ‘S:’ indicate relation types and subtypes, respectively

The GRE data set. Tables 7 and 8 contain the generic relation identification (GRI)- and GRC-type distributions for the final ACE 2004 and ACE 2005 data sets as used for the experiments here. The first column lists the gold standard type. For GRI, this is a binary distinction between a pair of entity mentions being in a relation or not being in a relation. Possible entity pairs are all those occurring within the same sentence. For GRC, the first column lists the relation type (with supertypes typeset in small capital letters). The next seven columns list the entity pair subdomains. These data subsets are constructed based on four entity types: Facility/Vehicle/Weapon (fvw or f), Geographical/Political/Location (gpl or g), Organisation (org or o) and Person (per or p). Note that for ACE, the the number of GRI Y instances is less than the total number of GRC instances because relation mentions are removed where one of the entity mentions are prenominal (e.g. ‘Scottish’ in ‘Scottish National Health Service’). This is to make the GRI task consistent with the named entity recognisers used for a related extrinsic evaluation (Hachey Reference Hachey2009a), which do not mark prenominal entity mentions. These instances are not filtered for the GRC data in order to maximise the number of data points for GRC evaluation.

Table 7.

Relation distributions for GRE news development data (ACE 2004). The first column specifies the relation type and the following columns specify the entity pair subdomains

Table 8.

Relation distributions for GRE news test data (ACE 2005). The first column specifies the relation type and the following columns specify the entity pair subdomains

7.2 BioInfer data for GRE

Filtering entity mentions. First, we only consider relations between Physical entities, defined as a physical, biochemical object[s] (Ginter et al. Reference Ginter, Pyysalo, Björne, Heimonen and Salakoski2007). While BioInfer annotates relations between Physical entities, properties and processes, the experiments here focus on the subtask of relations between Physical entities. Therefore, BioInfer entity mentions with Property and Process supertypes are ignored. We also ignore Textbinding entity mentions, which are used to mark spans of text that express a relation (example in Figure 5). In addition, we filter redundant annotations of the same entity mention. This can happen, for example, when a plural pronoun refers to more than one specific entity mention in the same sentence:

where ‘which’ refers back to ‘4a’ and ‘4b’. Here, mentions that do not take part in a relation are removed until only one is left.

Filtering relation mentions. Next, relation mentions are removed where one of the entity mentions is no longer part of the annotation because of the entity filtering rules. Relation mentions are also filtered based on type. In particular, relation mentions with type Rel-Ent are removed. These are BioInfer relations where an unnamed entity mention refers to a named entity mention, for example:

where the Rel-Ent(‘dose’, ‘IL-6’) relation mention indicates that dose refers to dose of IL-6. The original BioInfer data contains fifty Rel-Ent relation mentions. After filtering, the BioInfer data has 5,800 entity mentions and 2,116 relation mentions (down from 7,818 and 3,020, respectively, in the original source).

Converting to the final schema. Finally, entity and relation types are changed to the final schema. For BioInfer, this is a matter of choosing a level in the full relation-type schema from the source data that gives several entity pair subdomains with a sufficient number of relation types and instances for evaluation of the GRC task. Figure 15 contains a simplified version of the entity-type schema. The entity pair subset for each relation mention is determined by choosing the lowest level in this schema where the types of the entity mentions are siblings. For example, the subdomain for a relation-forming pair consisting of an Individual-Protein (P) entity mention and a Protein-Complex (C) entity mention would be P-C. For a pair consisting of a Source entity mention and an Individual-Protein entity mention, with parent-type Substance (B), however, the subdomain would be R-B. The relation type for the GRC task is simply the second-level type from the full-relation schema (Pyysalo et al. Reference Pyysalo, Ginter, Heimonen, Björne, Boberg, Järvinen and Salakoski2007), i.e. one of Causal, Part-Of, Observation or Is-A.

Fig. 15.

Simplified entity-type schema for BioInfer.

The GRE data set. Table 9 contains the GRI- and GRC-type distributions for the final BioInfer data set as used for the experiments here. The first column lists the gold standard type. For GRI, this is a binary distinction between an entity mention pair being in a relation or not being in a relation. For GRC, the first column lists the relation type (with supertypes typeset in small capital letters). The next seven columns list the entity pair subdomains. Note that the number of instances in the GRC data subsets is less than the corresponding number of GRI Y instances because the number of relation mentions that have vague or undetermined types are ignored for the relation characterisation experiments (but not the relation identification experiments). These consist of Corelate (a general, unspecified relation between the arguments), Humanmade (a relationship that is forced or caused by human intervention), Relate (a general, unspecified, non-directional relationship used when no details of the relationship are known) and Co-* (the relations created by Rule 1 for mapping BioInfer entity mentions from Section 6.2).

Table 9.

Relation distributions for GRE biomedical test data (BioInfer). The first column specifies the relation type and the following columns specify the entity pair subdomains

8.1 Experiment I: portability of generic relation identification (GRI)

The first step in GRE is GRI, where the goal is to identify relation-forming entity mention pairs using methods that port across domains without modification of model parameters. The input to the GRI task consists of sentences from source documents with entity mention markup. For the purpose of the intrinsic evaluation here, gold standard entity mention annotation is used, serving to isolate the errors that are due to the GRI module. All pairs of entity mentions that occur in the same sentence are considered to be the candidate relation mentions. Only considering intrasentential relation mentions is a simplifying assumption. However, in the three data sets used for the current work (which all contain at least 900 gold standard relation mentions), there is only one instance of a gold standard relation mention where the entity mentions are in different sentences. The GRI task, therefore, is to consider each pair of entity mentions within a sentence and determine whether the pair constitutes a relation mention or not.

Accuracy is measured in terms of precision (P) and recall (R):

and, f-score (F) is calculated in the standard way: F = 2PR/(P + R). Paired Wilcoxon signed ranks tests across entity pair subdomains are used to check for significant differences between systems. The paired Wilcoxon signed ranks test is a non-parametric analogue of the paired t test. The null hypothesis is that the two populations from which the scores are sampled are identical. Following convention, the null hypothesis is rejected for values of p less than or equal 0.05. Subdomains are formed by taking just those relations between two entities of given types (details given in Tables 7, 8 and 9).

8.1.2 Results

Table 10 contains P, R and F results. The best score for each measure is in bold and scores that are statistically distinguishable from the best (p ≤ 0.05) are underlined. The first row corresponds to a baseline system that accepts all pairs of entity mentions occurring in the same sentence. The final row corresponds to the upper bound calculated in terms of mean human agreement with respect to the adjudicated gold standard. Results suggest that the GRI accuracy is comparable when applying the newswire-optimised models directly to the biomedical domain. In both domains the best recall is achieved by the Comb model and the f-score is at least as good as the next best model (in the biomedical domain, the Comb f-score is actually significantly better than the Deps f-score).

Table 10.

Comparison of P, R and F on news and biomedical test sets. The best score in each column is in bold and those that are statistically distinguishable from the best are underlined

The highest f-score on the news test data is obtained using the dependency path model, though this is not statistically distinguishable from the Toks or Comb models. In terms of recall, the Comb model obtains the highest score (0.538), which is significantly better than the Toks and Deps models. The Deps model, however, obtains a precision score that is significantly better than the Comb model. For the current work, the combined model is considered to be the best as it achieves the highest recall, while the f-score is statistically indistinguishable from the other models. The prioritisation of recall is motivated by the fact that weighting is generally applied to co-occurring entity pairs for applications of GRI. For example, relation mining systems (Conrad and Utt Reference Conrad and Utt1994; Smith Reference Smith2002) use statistical measures of association such as pointwise mutual information, φ² and log likelihood ratio to estimate association strengths. Thus, a certain amount of noise in GRI should be acceptable if the subsequent weighting scheme is assumed to give higher weight to true relation-forming entity pairs.

In the biomedical domain, the Comb model performs best in terms of f-score with a score of 0.453, though it is statistically indistinguishable from the Toks model. This is a stronger result than in the news domain where there was no significant differences among the f-scores of the Toks, Deps and Comb models. Consistent with the news domain, there are no significant differences among the precision scores of the Toks, Deps and Comb models and, importantly, the Comb model is significantly better than the Toks and Deps models in terms of recall in both domains. Interestingly, the f-score of the Baseline model is statistically indistinguishable from the Comb model on the biomedical data. Since Baseline recall is the same for both domains (1.000), this is due to higher precision (0.268 as opposed to 0.110), which is due to the higher proportion of true relation-forming pairs (27% for BioInfer compared to approximately 10% for the ACE data sets). This is artificially high, however, since the BioInfer creators selectively sampled sentences that include mentions of proteins that are known to interact. Comb precision is significantly better than the Baseline precision on both domains.

8.2 Experiment II: portability of generic relation characterisation (GRC)

The second step in GRE is GRC, where the goal is to automatically annotate each relation mention with a label that describes the relation type using methods that port across domains without modification of model parameters. The input to the GRC task is the output from the GRI task and consists of sentences from the source document with entity mentions and relation-forming pairs identified. For the purpose of the intrinsic evaluation here, gold standard entity and relation-forming entity pair annotations are used, serving to isolate the errors that are due to the GRC module. The primary modelling task of GRC is to induce a partition (or clustering) over the relation-forming pairs, where the goal of the clustering is to group them by relation type. For the current work, each relation mention (i.e. pair of co-occurring entity mentions) from the output of the GRI task is an instance for clustering. In other words, the clustering instance level is entity pair tokens instead of entity pair types.

Accuracy is measured in terms of two different approaches to calculating precision (P), recall (R) and f-score (F) of clustering output with respect to a gold standard. F _1:1 is based on a one-to-one mapping between clusters and gold standard classes. If there are more classes than clusters, then some classes are left unaligned (likewise if there are more clusters). Calculating the one-to-one mapping can be expensive; however, a simple greedy search through possible alignments with a beam of width five has linear time and space complexity and provides a reasonable approximation. F_pw is based on the number of pairs of data points that are in the same cluster and in the same class (tp), the number of pairs in the same cluster but in different classes (fp) and the number of pairs in different clusters but in the same class (fn). As above, the paired Wilcoxon signed ranks tests across entity pair subdomains are used to check for significant differences between systems.

8.2.1 Systems

Unreduced. In the second approach, no dimensionality reduction is performed. Here feature vectors are extracted for each relation mention and weighted using tf*idf, which is calculated as follows:

(1)

where tf_i,j is the number of times feature i occurs in the context of relation-forming entity mention pair j and df_i is the number of relation-forming pair contexts in which feature i occurs. Cosine is used to measure the similarity between feature vectors in the unreduced feature space.

SVD-reduced. In the third approach, dimensionality reduction is performed using singular value decomposition (SVD), a linear algebraic least squares method (Eckart and Young Reference Eckart and Young1936). In case where X_r×f is a tf*idf-weighted relation-by-feature (R×F) matrix, SVD performs a decomposition of X into the product of three matrices with n latent semantic dimensions:

In the resulting decomposition, the R and F matrices represent relation mentions and features in the new space and S is a diagonal matrix of singular values in decreasing order. These are generally sorted by decreasing magnitude of the singular values. Based on tuning experiments on the ACE 2004 data, n is set to 5 for the experiments here.

LDA-reduced. While SVD has proved successful, its representation of words and documents (or relations) as points in a Euclidean space is not easy to interpret. In the fourth approach, dimensionality reduction is performed using Latent Dirichlet allocation (LDA) (Blei et al. Reference Blei, Ng and Jordan2003), a generative probabilistic version of latent semantic analysis (Berry et al. Reference Berry, Dumais and O'Brien1995; Landauer, Foltz and Laham Reference Landauer, Foltz and Laham1998). Here LDA is used to model the contribution of different topics to a relation mention by treating each topic as a probability distribution over features, where a relation mention is a probabilistic mixture of topics. In case where T is the number of topics, the probability of the ith feature is written as follows:

(2)

where z_i is a latent variable indicating the topic from which feature f_i is drawn, P(f_i|z_i = j) is the probability of drawing feature f_i under topic j and P(z_i = j) is the probability of topic j for the current relation mention. Intuitively, P(f|z) indicates the features that are important to a topic and P(z) is the prevalence of those topics for a given relation mention (Griffiths and Steyvers Reference Griffiths and Steyvers2004). The resulting similarity model contains four free parameters: the number of topics T, two hyperparameters (β and α, which determine the nature of the Dirichlet priors on P(f_i|z_i = j) and P(z_i = j), respectively) and the constant C for divergence-to-similarity conversion. Based on tuning experiments on the ACE 2004 data, these are set to T = 0.97|TotalFeatures|, β = 0.0001, α = 50/T and C = 8 for the experiments here.

8.2.2 Results

Table 11 contains P, R and F results. The best score for each measure is in bold and scores that are statistically distinguishable from the best (p ≤ 0.05) are underlined. The first row corresponds to a baseline system that randomly partitions the data into n clusters. The final row corresponds to the upper bound calculated in terms of mean human agreement with respect to the adjudicated gold standard. All clustering approaches here use a feature set consisting of (1) words occurring between the two entities, (2) words occurring within the entity phrases and (3) words and grammatical relations occurring on the path connecting the two entities in the dependency parse (Hachey Reference Hachey2009b). The best score for each evaluation measure is in bold and systems that are statistically distinguishable from the best (i.e. p ≤ 0.05) are underlined. Table 11(a) contains results for the news domain development set (ACE 2004); Table 11(b) contains results for the news domain test set (ACE 2005); and Table 11(c) contains results for the biomedical domain test set (BioInfer).

Table 11.

Comparison of precision, recall and f-score results on all data sets

In terms of the f-score results of the clustering systems, the LDA-reduced similarity model achieves the highest scores in most combinations of data sets and evaluation measures. Moreover, it is significantly better than the baseline across all combinations. The LDA-reduced model is significantly better than the unreduced and SVD-reduced models in terms of F_pw on both the news development and test sets, though not on the biomedical test set. In terms of recall, however, the LDA-reduced model is significantly better than the unreduced model for all combinations except in terms of R _1:1 on the news test set. The effect of the hyperparameters can be observed in the relatively high recall for the LDA-reduced model. Here the small values of α (means across subdomains of 0.63, 0.67 and 0.65, respectively, for the ACE 2004, ACE 2005 and BioInfer) can be expected to result in skewed topic distributions, which subsequently lead to skewed distributions over clusters. This effect can be observed in terms of the very strong negative correlation between values of α and pairwise recall (Pearson's r of −0.686, −0.733 and −0.865, respectively, for the ACE 2004, ACE 2005 and BioInfer).