1 Introduction
Relation extraction (RE) is a subtask of information extraction whose aim is to automatically extract instances of semantically related entities from corpora.
Biographical relation extraction (BRE) attempts to obtain pairs belonging to a biographical relation, such as (Person)hasBirthDate(Date), (Person)hasDeathPlace (Location), or (Person)hasProfession(Occupation).
For example, from a sentence like ‘Kimberley Deal (born June 10, 1961) is an American singer, songwriter and musician,’ the goal of a BRE system is to extract the following structured data:
— Kimberley Deal hasBirthDate10 of June of 1961
— Kimberley Deal hasNationalityAmerican
— Kimberley Deal hasProfessionsinger
— Kimberley Deal hasProfessionsongwriter
— Kimberley Deal hasProfessionmusician
These data can be arranged, for instance, into proper noun ontologies, which are useful for many applications such as question answering systems (Mann Reference Mann2002; Ravichandran and Hovy Reference Ravichandran and Hovy2002) or information retrieval engines (Wan et al. Reference Wan, Gao, Li and Ding2005).
In recent years, several strategies have been applied for RE, like semi-supervised classifiers or more unsupervised approaches. These methods are focused on minimizing the effort of manually annotating corpora. However, one of the main approaches for extracting high-quality instances of semantic relations from unstructured text is still the use of supervised classifiers, trained with annotated sentences containing pairs of the desired relations. These classifiers may contain different kinds of features: from orthographic and lexical data to more complex features such as part-of-speech (PoS) tags, semantic knowledge, or syntactic information.
In order to build accurate feature-based classifiers, some works adjust the feature space to a specific relation and/or pattern, creating high-quality extractors (Fleischman, Hovy and Echihabi Reference Fleischman, Hovy and Echihabi2003). Other works showed that several features (and combinations of them) behave well with some relations, while their performance decrease in other scenarios (Agichtein Reference Agichtein2005). However, there has been little research in feature engineering for RE, and it has been usually performed only for English (Kambhatla Reference Kambhatla2004; Zhou et al. Reference Zhou, Su, Zhang and Zhang2005).
In this respect, evaluating the effectiveness of different linguistic knowledge for RE is important not only for this task, but also for theoretical and computational linguistics in general, in order to know the best way for representing linguistic structures containing semantic relations.
The main objective of this paper is to perform an evaluation of different features for BRE for Portuguese (pt) and Spanish (sp). Additionally, two freely available corpora for this task were also created and analyzed.
Taking this into account, a systematic evaluation of several linguistic features is performed. Extraction is carried out between entities appearing in the same sentence, by means of generic features that have been used for RE, extracted from different levels of linguistic knowledge.
Concerning the corpora, two collections of sentences containing biographical relations (one for Portuguese and another one for Spanish) were created and annotated with the following semantic relations: hasBirthPlace, hasBirthDate, hasDeathPlace, hasDeathDate and hasProfession.Footnote 1 Furthermore, a careful analysis of the relation patterns was carried out, aimed at knowing how the target relations are expressed in Portuguese and Spanish. The manual effort of building annotated corpora was reduced by applying a method based on distant-supervision (Mintz et al. Reference Mintz, Bills, Snow and Jurafsky2009). Then, machine learning algorithms were applied in order to build and to evaluate several classifiers, differing only in the types of linguistic features they have.
The results of several experiments show that using lemmatization—as well as basic semantic knowledge provided by named entity recognizers—improves the accuracy of bag-of-words classifiers. Furthermore, pseudo-syntactic information (mainly represented by bigrams of lemmas) can be used to avoid computationally expensive processes such as syntactic parsing, which, in turn, seems not to provide great benefits on the system.
The remainder of this paper is organized as follows. Section 2 introduces some related work. Then, Section 3 provides a general view of how the corpora were constructed as well as an analysis of their linguistic structures. Next, Section 4 is focused on the description of the features, particularly on the types of linguistic knowledge used by the classifiers. Then, Section 5 reports the diverse experiments performed over the Portuguese and Spanish texts. Finally, some conclusions are put forward in Section 6.
2 Related work
There has been a large body of work on extraction of semantic knowledge from unstructured text in recent years. In this section, some seminal papers on RE are introduced, followed by several state-of-the-art approaches for this task. Then, various works focused on BRE are presented, paying special attention to the linguistic knowledge they use. After that, some papers that evaluate different features in RE are shown. Finally, several specific works of RE for Portuguese and Spanish are also introduced.
Hearst (Reference Hearst1992) was the first one to experiment with a pattern-matching approach for the identification of semantic relations. She made use of a small set of initial patterns to get hypernymy relations, thus increasing the set of patterns with a bootstrapping technique. Brin (Reference Brin1998) applies a similar method, but performs an evaluation of the new patterns, in order to promote only those that show a good precision.
Other work that iteratively learns new extraction patterns is Agichtein and Gravano (Reference Agichtein and Gravano2000), which present Snowball. This tool extracts and evaluates the quality of patterns in order to obtain new semantically related pairs without human intervention. Ravichandran and Hovy (Reference Ravichandran and Hovy2002) make use of question answering pair examples to automatically extract patterns. KnowItAll (Etzioni et al. Reference Etzioni, Cafarella, Downey, Kok, Popescu, Shaked, Soderland, Weld and Yates2004) also extracts instances of semantic relations in an unsupervised fashion, by learning new extraction rules in a bootstrapping process. Espresso is another tool for RE from the web. It starts by using patterns with high recall and lower precision. Then, Espresso uses the web in order to filter those patterns with low precision (Pantel and Pennacchiotti Reference Pantel and Pennacchiotti2006). Bunescu and Mooney (Reference Bunescu and Mooney2007) also use small sets of positive and negative seed pairs for obtaining sentences containing the desired relations.
Recently, new approaches for RE have been proposed. Distant-supervision takes advantage of large repositories of related pairs to automatically classify sentences, then used for training classifiers (Mintz et al. Reference Mintz, Bills, Snow and Jurafsky2009).
Another paradigm, Open Information Extraction, attempts to extract large sets of related pairs without manually specifying semantic relations (Banko et al. Reference Banko, Cafarella, Soderl, Broadhead and Etzioni2007). Similarly, woe performs open information extraction taking advantage of the semi-structured resources of Wikipedia (Wu and Weld Reference Wu and Weld2010). Woe uses syntactic information extracted from parsing Wikipedia, but does not outperform new approaches of open information extraction (Etzioni et al. Reference Etzioni, Fader, Christensen, Soderland and Center2011; Gamallo, Garcia and Fernández-Lanza Reference Gamallo, Garcia and Fernández-Lanza2012).
2.1 Biographical relation extraction
More related to our work, several papers and competitions focus on the extraction of pairs belonging to biographical relations.
Mann (Reference Mann2002) describes a method for creating ontologies of proper nouns for question answering systems, by selecting instances of a common noun followed immediately by a proper noun (the composer (common noun)Wolfang Amadeus Mozart (proper noun)). Inspired by this work, Fleischman et al. (Reference Fleischman, Hovy and Echihabi2003) apply machine learning algorithms in order to classify instances of similar ‘common noun–proper noun’ examples.
Jijkoun, De Rijke and Mur (Reference Jijkoun, De Rijke and Mur2004) evaluate different surface and syntactic patterns for obtaining pairs for question answering. Their results show that syntactic parsing improves the performance of the question answering systems, despite the fact that the extraction is less precise than that obtained through surface patterns.
Garera and Yarowsky (Reference Garera and Yarowsky2009) present several approaches for extracting biographical facts: their objective is to include structural information of biographical texts (position of the facts in the document, extraction of implicit knowledge, etc.) into the extractors, thus improving the performance of surface patterns for RE.
Liu et al. (Reference Liu, Nie, Yu and Wen2010) describe BioSnowball, a framework to collect personal information from the web in order to generate Wikipedia-style pages.
Other works explore the extraction of biographical knowledge, although they are not focused only on this kind of relations. LEILA (Suchanek, Ifrim and Weikum Reference Suchanek, Ifrim and Weikum2006) is a system that can extract instances of semantic relations from natural language documents, by using syntactic parsing. Finally, Pasca et al. (Reference Pasca, Lin, Bigham, Lifchits and Jain2006) also extract a large set of hasBirthDate pairs by generalizing patterns found with a small set of initial seed pairs.
In recent years, two competitions have included tasks about biographical fact extraction: the Knowledge Base Population (KBP) Slot Filling task (in the Text Analysis Conference, TAC)Footnote 2 and the Person Attribute Extraction (in the Web People Search Evaluation Campaign, WePS),Footnote 3 both of them since 2009. The objective of these competitions is to extract attribute values from a predefined set of biographical relations. In some scenarios, the systems should perform clustering of web pages, coreference resolution, as well as cross-sentence and cross-document fact extraction, so the tasks are not directly comparable to the approach followed in this paper.
2.2 Linguistic knowledge
Different RE systems, from pattern-matching approaches to more complex statistical methods, use linguistic knowledge in order to improve the extraction of related pairs.
First, strategies for RE (Hearst Reference Hearst1992) simply applied lexical patterns (‘entity_1such as entity_2’) to obtain the related pairs. Then, in order to increase the recall of their systems, other works performed different types of pattern generalization, by means of the longest common substring (Finkelstein-Landau and Morin Reference Finkelstein-Landau and Morin1999), applying a suffix tree (Ravichandran and Hovy Reference Ravichandran and Hovy2002), or by computing the edit distance between the different patterns (Ruiz-Casado, Alfonseca and Castells Reference Ruiz-Casado, Alfonseca and Castells2005).
The above mentioned Mann (Reference Mann2002) included PoS-tags in the patterns for extracting specific relations such as hasProfession. In a similar way, BioSnowball (Liu et al. Reference Liu, Nie, Yu and Wen2010) also uses a combination of lexical and PoS-tag sequences for representing the patterns.
A novelty in the work of Suchanek et al. (Reference Suchanek, Ifrim and Weikum2006) was the use of syntactic analysis (provided by a link grammar) in the system. Similarly, Akbik and Broß (Reference Akbik and Broß2009) apply linguistic patterns (using dependency grammars) for populating a semantic wiki from the English Wikipedia.
Some other works also use syntactic parsing for RE. Bunescu and Mooney (Reference Bunescu and Mooney2005), Snow, Jurafsky and Ng (Reference Snow, Jurafsky and Ng2005), and Nguyen, Matsuo and Ishizuka (Reference Nguyen, Matsuo and Ishizuka2007) created different statistical models using the results of syntactic parsing. More recently, Yan et al. (Reference Yan, Okazaki, Matsuo, Yang and Ishizuka2009) perform RE by combining surface patterns extracted from the Web with dependency patterns obtained from the parsing of Wikipedia.
RGAI (Nagy and Farkas Reference Nagy and Farkas2010) was the system with the best results in the Attribute Extraction subtask of WePS-3. Its method consists of two parts: first, candidate paragraphs are extracted if they contain relevant lexical information for each relation (e.g., born, birth or birthday for hasBirthDate); then, a validator classifies the candidate attribute as positive or negative based on regular expressions, string-matching, and other relation-dependent heuristics.
Sun et al. (Reference Sun, Grishman, Xu and Min2011) achieved the top results at the 2011 KBP Slot Filling task, by training distant-supervision classifiers. Concerning the linguistic knowledge they use, this system obtains features from the token sequence (words after and before the candidate entities, intermediate pattern of tokens), from the semantic classification of the entities (Person, Organization, etc.), and from the dependency tree (the shortest dependency path between the candidate pair, and the head and dependent word of each candidate entity).
Concerning the effectiveness of various linguistic features, some works evaluated the impact of different knowledge over the ACE corpus (for generic RE). Kambhatla (Reference Kambhatla2004) and Zhou et al. (Reference Zhou, Su, Zhang and Zhang2005) show that the effective incorporation of different features based on linguistic knowledge improves the performance of the classifiers. However, Jiang and Zhai (Reference Jiang and Zhai2007) conclude that adding more complex features may hurt the performance of the extractors.
Other strategies using kernel methods (as well as several kernel combinations) were also explored in order to improve the RE using the ACE corpus (Zhao and Grishman Reference Zhao and Grishman2005; Zhang et al. Reference Zhang, Zhang, Su and Zhou2006; Zhou et al. Reference Zhou, Zhang, Ji and Zhu2007; Nguyen, Moschitti and Riccardi Reference Nguyen, Moschitti and Riccardi2009).
2.3 Relation extraction for Portuguese and Spanish
As it was shown in the previous sections, most of the work on RE is performed for English. However, there are some papers on RE for Portuguese and Spanish that need to be mentioned.
Specifically for Portuguese, several works were focused on different types of RE. Oliveira et al. (Reference Oliveira, Santos, Gomes and Seco2008) deal with the extraction of relations between concepts—such as hypernym, hyponym, part_of, inclusion, etc.—from a Portuguese dictionary. Using this strategy as well as other extractions based on pattern-matching, the Onto.PT project aims to automatically construct a lexical ontology for Portuguese (Oliveira and Gomes Reference Oliveira and Gomes2010). Other works such as (Costa and Branco Reference Costa and Branco2012) identify events and temporal expressions in unstructured text.
The second HAREM evaluation included a shared task on generic RE between named entities (ReRelEM) (Mota and Santos Reference Mota and Santos2008). Three different systems were presented: REMBRANDT uses knowledge extracted from Wikipedia for Named Entity Recognition (NER), and a set of heuristics and grammatical rules for identifying the relations (Cardoso Reference Cardoso, Mota and Santos2008). SEI-Geo applies Hearst patterns (and other strategies such as using trigger words and verbs) for establishing relations between locations (Chaves Reference Chaves, Mota and Santos2008). Finally, SeRELeP also attempts to identify relations by applying simple rules on previously identified named entities (Bruckschen et al. Reference Bruckschen, de Souza, Vieira, Rigo, Mota and Santos2008). Concerning biographical relations, Soares, Martins and Calado (Reference Soares, Martins and Calado2011) evaluate various algorithms and features for classifying sentences (but not pairs of entities) containing this kind of relations.
For Spanish, Sánchez-Cuadrado et al. (Reference Sánchez-Cuadrado, Lloréns, Morato and Hurtado2003) take advantage of syntactico-semantic patterns for building a thesaurus of zoology. Sierra et al. (Reference Sierra, Alarcón, Aguilar and Bach2008) propose the use of verbal patterns for extracting (i) hypernym and hyponym relations, (ii) synonym, as well as (iii) individuation (quantity/mass and member/group) instances. Other works such as Soler-Alcina (Reference Soler and Alcina2008) also apply lexical patterns for obtaining instances of the part-whole relation (in the domain of ceramics).
Aguado de Cea et al. (Reference Aguado de Cea, Gómez-Pérez, Montiel-Ponsoda and Suárez-Figueroa2008) present a method (and a tool) for re-using ontology design patterns (by associating them to lexico-syntactic patterns) for ontology population (for English, German, and Spanish).
Garcia and Gamallo (Reference Garcia, Gamallo, Saint-Dizier and Mehta-Melkar2011a) compare the impact of several features for distant-supervision extraction in Spanish, focusing on the hasProfession relation.
Finally, Garcia and Gamallo (Reference Garcia, Gamallo, Nakov, Kozareva, Ganchev and Hobbs2011b) apply a sentence compression method in order to increase the coverage of a RE system in Spanish and Portuguese.
In this paper, the effectiveness of several types of linguistic features for BRE is explored, by training different feature-based classifiers: first, by using simple lexical features, and then by evaluating the incremental addition of more complex features based on morphosyntactic analysis, pseudo-syntactic representation, as well as on syntactic knowledge.
3 The corpora
In this section, the method for obtaining the corpora used during the experiments as well as their main characteristics are presented. Two corpora were built, one for Portuguese and another one for Spanish.
3.1 Building the corpora
Manually creating a labeled corpus is an expensive process, but it is crucial for many applications relying on high-quality data for building statistical models. So, in order to speed up this process, a strategy inspired by distant-supervision was applied.
Distant-supervision assumes that if two entities participate in a relation (e.g., hasBirthPlace: ‘Woody Allen’, ‘The Bronx’), any sentence containing these two entities might express the same relation (hasBirthPlace) (Mintz et al. Reference Mintz, Bills, Snow and Jurafsky2009). Thus, obtaining large sets of pairs belonging to specific relations (from databases such as Freebase, DBpedia, etc.) permits to automatically classify extracted sentences containing different pairs as positive or negative for the desired relations.
Although this strategy gave good results for some relations and datasets (Hoffmann, Zhang and Weld Reference Hoffmann, Zhang and Weld2010), Riedel, Yao and McCallum (Reference Riedel, Yao and McCallum2010) observed that it might produce a large amount of noisy patterns if the knowledge base and the text do not belong to the same—or similar—domain. Taking this into account, the corpora used in this paper were created by manually correcting the output of a distant-supervision extraction.
It is worth mentioning that applying this method for languages other than English is not always feasible, since the attributes of many relations are language-specific, and the most common (and largest) knowledge bases are mainly focused on English.Footnote 4
So, language-independent pairs (containing dates) were obtained from Freebase and the English DBpedia. The other sets of pairs (language-dependent) were extracted from the Portuguese and Spanish Wikipedia infoboxes (and also from the incipient versions of DBpedia for these languages).
The following strategy was applied:
(1) First, large sets of pairs for each relation and language were automatically extracted from the mentioned sources. For hasBirthDate and hasDeathDate, 460, 703 pairs were used. For hasBirthPlace, 45, 588 and 8, 952 for Portuguese and Spanish, respectively. The amount of hasDeathPlace pairs was 11, 664 (pt) and 1, 319 (sp). For hasProfession, only the most common occupation nouns were used, by selecting those with more than 20 occurrences in the seed list, in order to avoid noisy sentences. Therefore, 68 and 96 different nouns instantiating this relation were used for Portuguese and Spanish, respectively.
(2) Then, these pairs were used for extracting sentences containing known entities from the Portuguese and Spanish versions of Wikipedia (with 700 MB and 1.6 GB of data, respectively). Furthermore, 1 GB and 225 MB of journalistic corpora were also used for Portuguese (Público) and for Spanish (El País). The final extraction had Wikipedia/newspaper ratios of 90%/10% (pt) and 93%/7% (sp).
(3) Both Portuguese and Spanish corpora were first analyzed by some modules (sentence chunking, tokenizing, lemmatizing, PoS-tagging, and dates and named entity recognition) of FreeLing (Padró et al. Reference Padró, Collado, Reese, Lloberes and Castellón2010). For avoiding noisy sentences, they were selected only when the entities they contained were exactly the same as in the seed pairs. Moreover, these entities had been classified by FreeLing with the expected class (e.g., Person, Location), except for dates, which were selected if at least the year was the same as in the seed examples.
(4) The extracted sentences were automatically labeled as positive for a specific relation if they contain both entities of a pair. Otherwise, they were marked as negative.
(5) Sentences were enriched with syntactic knowledge in order to build a linguistic structure (explained below), useful for extracting different types of features that will be evaluated later. DepPattern, a multilingual suite for syntactic analysis (Gamallo and González Reference Gamallo and González2011) was applied for generating dependency graphs from the sentences.
(6) Then, the label (positive or negative) of each candidate pair as well as the boundary and the semantic classification (Person, Location, Date, etc.) of the two entities were manually corrected.
(7) Finally, the entities of each related pair were substituted by X (Person) and Y (attribute: Y_Loc, Y_Dat or Y_Occ).
Both corpora for Portuguese and Spanish contain five biographical relations: hasBirthPlace, hasBirthDate, hasDeathPlace, hasDeathDate, and hasProfession. Moreover, the corpora have 268, 469, and 152, 817 tokens for Portuguese and Spanish, respectively.
Table 1 shows the number of labeled pairs for each relation and language. Positive pairs are used as negative examples for different relations with the same entity classes. Thus, a sentence such as ‘Person was born in Location’ (positive for hasBirthPlace) will be used as a negative example for hasDeathPlace.Footnote 5
Number of candidate pairs for each relation and language of the annotated corpora. Pos. stands for positive pairs for the semantic relation and Neg. for the negative ones
3.2 Linguistic structure
Each labeled, tagged, and parsed sentence represents a linguistic structure containing all the relevant information required by the RE system. A linguistic structure or parsed sentence can be conceived as a knowledge-rich space incorporating several levels of linguistic information.
Figure 1 contains an example of the hasProfession relation, where the original sentence, the related terms with a polarity label, and the linguistic structure are shown. Note that the same sentence will be also labeled as positive for hasBirthDate, with the attribute Y_Dat being the entity in pos 3.Footnote 6
Example of a sentence (in English) labeled for hasProfession with the related entities (X and Y_Occ), the polarity, and its linguistic structure.
In the example (Figure 1), columns 1, 2, 3, and 4 respectively stand for position, token, lemma, and PoS-tag. Last two columns contain syntactic information: 5 identifies the head position of the current token, while the label of the dependency is shown by column 6. This structure was inspired by the output parsing format defined in Lin (Reference Lin2003), adopted by the evaluation of the CoNLL Shared Tasks.
The syntactic analysis was obtained by a dependency parser. Dependency parsing establishes binary links between a head and a dependent, generating a dependency graph from a sentence. Dependency links encode information about the head and the dependent of a relation, as well as the syntactic function of this dependency. Figure 2 exemplifies a dependency graph of the previous example sentence, where the arrows point from the dependent to the head.
Example of a dependency graph. First row contains the PoS-tag, enriched with semantic classes, while the syntactic function is shown in the third row.
3.3 Linguistic properties of the corpora
Before extracting the features for building the classifiers, an analysis of the patterns containing the desired relations was carefully performed in the corpora: first, by creating a histogram of the number of tokens that occur between the two candidate entities, and second, by finding the most common patterns for each relation and language.
Figure 3 shows a histogram that summarizes the intermediate number of tokens (from 0 to 18) between the candidate entities in the Portuguese and Spanish corpora.
Histogram of positive/negative pairs versus intermediate context in number of tokens. Values are micro-average results of the five analyzed relations.
Even though the intermediate number of tokens differs for each relation (as the most common patterns will show), the histogram of Figure 3 clearly proves that semantic relations occur more frequently the closer the entities are (with the best ratio with less than eight intermediate tokens, depending on the language and on the relation). As the size of this window increases, more negative pairs appear, and little positive ones are found.
Concerning the patterns that express biographical relations, Table 2 includes the two best patterns per relation and language, taking into account their percentage in the sets of positive sentences as well as their precision. Note that these patterns are greatly simplified by using disjunctions as well as by omitting some optional elements.
Best patterns (taking into account their precision and their percentage in the sets of positive sentences) per language and relation. Elements in curly brackets are—each one—optional, vertical bars represent disjunction, and NP PoS-tags refer to a proper noun. For space reasons, patterns are highly simplified by omitting some verbal inflections and other optional elements. BD means hasBirthDate; BP,hasBirthPlace; DD,hasDeathDate; DP,hasDeathPlace, and Pr. refers to hasProfession. % is the percentage of each pattern in the positive sentences of each relation. Prec is its precision in the corpus
The taxonomy presented in Table 2 shows that target relations often occur in biographical-specific patterns with high precision, which contain in parenthesis biographical facts about a previously mentioned person. This table also indicates that some relations heavily depend on these patterns (which represent between 36% and 49% of the positive pairs), while others occur in a wide range of structures.
However, it is worth mentioning that, except for these frequent and high-precision patterns, other structures are very ambiguous and their use to identify a specific relation may depend on lexical data outside the intermediate window.
In order to represent the patterns, these analyses point out the importance of choosing the best intermediate window as well as the right features. Note that the patterns underlying biographical sentences not only are based on lexical and syntactic structures, but also contain a great variety of punctuation marks designing relevant linguistic information.
4 Features
In order to evaluate the effectiveness of the linguistic knowledge for RE, features were organized from shallow to deep analysis, according to the linguistic complexity they contain. Therefore, this section presents and discusses the different features used for training the classifiers.
4.1 Basic features
Before introducing the basic features, it is worth noting that the selection of candidate entities is restricted to specific semantic classes for each relation. Thus, X will always be a Person proper noun, and Y will be Date in hasBirthDate and hasDeathDate and Location in hasBirthPlace and hasDeathPlace. Finally, Y will be an occupation noun in hasProfession. The effectiveness of this restriction will be further analyzed by using a baseline that does not take advantage of this semantic classification.
Apart from the use of NER tools, the first category of features is not purely linguistic. It includes information about the position of both X and Y in the sentence. This category contains four different features (their values come from the example shown in Figure 1):
• Absolute position of X in the sentence: 0
• Absolute position of Y in the sentence: 8
• Direction of the relation: X_Y (1) or Y_X (2): 1
• Distance (in number of tokens) between both entities: 7
Although these features do not characterize the semantic relation at all, they are widely used in the literature, and preliminary experiments showed they are helpful for combining with some linguistic features (explained below) that do not include implicit information about the position of the entities. Thus, these features will be combined with the subsequent categories including lexical, morphosyntactic, pseudo-syntactic, and syntactic knowledge.
4.2 Lexical
These features concern the use of the lexical units present in the linguistic structure. Two main types of features were taken into account:
• Tok: including the tokens between the two related entities as well as some surrounding elements (defined below)
• Lem: using lemmas, the normalized form of each token
Tokens are common features used to characterize linguistic structures containing semantic relations. Using them as bag-of-words adds important lexical information. An example of the Tok features extracted from the example can be seen below:
Sentence: X (born June 10, 1961) is an American Y_Occ
Tok: < (>, < born >, < Date >, <) >, < is >, < an >, < American >
Note that Tok and Lem are not simple bag-of-words features. They include the boundaries of complex entities such as compound proper nouns (e.g., Kimberly_Deal) and temporal expressions (e.g., June 10, 1961), by means of different modules included in FreeLing.
In highly inflected languages such as Portuguese and Spanish, using tokens may involve data sparseness. For example, each verb has about 50 different forms in these languages. Therefore, using lemmas as features allows the classifier to generalize this information. The previous example sentence will be encoded as follows:
Lem: < (>, < bear >, < Date >, <) >, < be >, < a >, < american >
Note that these features will also be extracted from similar sentences such as ‘X (born June 10, 1961) was an American Y’. So, Lem features reduce the data sparseness even if they lose some morphosyntactic information included in the inflection of nominal and verbal forms.
Using bag-of-words strategies (with tokens, lemmas, and other features) for representing patterns involves the definition of two thresholds. First, the surrounding context of X and Y, in order to characterize relations defined outside the window between the entities: ‘The city of birth of X is Y’. If the system only takes into account the elements between the candidate pair, this kind of instances will not be extracted. However, including large contexts before the first entity and after the second one, may add noise and irrelevant elements to the classifier.
Second, the size of the intermediate window (between X and Y) also needs to be defined: If the window is small (5, 6, 7 tokens), the number of features and the complexity of the patterns will be reduced, but the classifier will have low coverage (since many instances will be excluded, as shown in Section 3.3). Increasing the window size will also increase the coverage of the system at the expense of growing the number of features. These windows will be empirically defined in Section 5.1.
4.3 Morphosyntactic
The feature space may be enriched by including morphosyntactic information, such as PoS-tags. PoS-tags can be added to the classifiers in two different ways: (i) by enriching the features based in lemmas (Lem_PoS) and by adding a new level of linguistic abstraction (PoS), representing a linguistic structure by means of morphosyntactic categories and not by lexical units. Thus, adding these features to a classifier might help to improve the recognition of linguistic patterns. The PoS-tag features extracted from the example sentence are the following:
Sentence: X (born June 10, 1961) is an American Y
Lem_PoS: < FA >, < bear_VB >, < Date >, < FC >, < be_VB >, < a_DI >, < american_ADJ >
PoS: < FA >, < VB >, < Date >, < FC >, < VB >, < DI >, < ADJ >
4.4 Pseudo-syntactic
Syntactic analysis provides information about the function of different items in a linguistic structure. However, syntactic parsing may present some problems, since it is computationally expensive and usually its performance is still not very high, especially for languages other than English. Moreover, languages other than English often suffer lack of free tools for syntactic parsing.
Taking these assumptions into account, it is interesting to evaluate some features that, in some way, represent pseudo-syntactic information by means of encoding the position of the different elements in a sentence. Thus, the following features were evaluated during the experiments:
• Bigrams: bigrams of lemmas
• Trigrams: trigrams of lemmas
These features were encoded by using sequences of two (Bigrams) and three (Trigrams) adjacent lemmas:
Sentence: X (born June 10, 1961) is an American Y
Bigrams: < X_(>, < (_bear >, < bear_Date >, < Date_) >, <)_be >, < be_a >, < a_american >, < american_Y >
Trigrams: < X_(_bear >, < (_bear_Date >, < bear_Date_) >, < Date_)_be >, <)_be_a >, < be_a_american >, < a_american_Y >
Other pseudo-syntactic features were also tested, such as intermediate and large lexico-syntactic patterns. However, they turned to be not helpful for any relation and language.
4.5 Syntactic
Finally, the last set of features are related to (pure) syntactic analysis. In recent years, many works have taken advantage of parsing in order to syntactically represent linguistic structures containing semantic relations.
Syntax encodes, in a deep linguistic level, the function of each element in a sentence. This way, the same syntactic functions could be extracted from sentences with very different surface structure. The following examples share the main syntactic structure, despite the fact that their surface form notoriously differ:
Example 1: X (born June 10, 1961) is an American Y(singer)
Example 2: Many music journalists said that X was probably the best Y(singer) in America
Note that in the above sentences, X is the Subject of the verb to be, while Y (singer) is the attribute of the same verb.
In order to represent syntactic knowledge as features, three different ways were explored and evaluated:
• Deps: single syntactic dependencies
• SDPs: shortest dependency path (between X and Y)
• Pths: full dependency path
Single dependencies are triples that represent links between two related items (a head and a dependent) with a syntactic label. The first element of the dependency defines the syntactic function, while the head and the dependent are the second and third units, respectively.Footnote 7
Sentence: X (born June 10, 1961) is an American Y
Deps: < Subj;be;X >, < Pt;bear;FA >, < Ad;X;bear >, < Obj;bear;Date >, < Pt;bear;FC >, < Sp;Y;a >, < Ad;Y;american >, < Attrib;be;Y >
From this point of view, dependencies are similar to bigrams of lemmas, but built from (and providing) syntactic information. Note that in the previous example, the first and the last triples are crucial for defining the relation hasProfession, while the others add extra information. This observation inspired the next feature, called the shortest dependency path (SDP).
The SDP represents the shortest link between two entities in a dependency tree. Figure 4 shows (with normal line), the SDP between X and Y in the Example 1 sentence. This way, the SDP of this sentence can be represented as the feature: < Subj;be;X//Attrib;be;Y > .
Example of a shortest dependency path (with normal line) between X and Y.
Although the SDP is often considered as one of the best ways for representing a relation between two entities in a sentence, some works have warned that important information (such as negation) might be lost by using SDP (Wu and Weld Reference Wu and Weld2010). Figure 5 shows a sentence with the same SDP as the previous one, but not having the same semantic relation.
Example of a shortest dependency path between X and Y that does not represent a semantic relation. In this example, crucial information (the negation of the verb) is lost.
Therefore, a different path between the entities, using the full dependency path (Pths, which includes the dependents of the head elements in the SDPs), was evaluated. An example of this third syntactic feature can be seen below:
Sentence: X (born June 10, 1961) is an American Y
Pths: < Subj;be;X//Pt;bear;FA//Ad;X;bear//Obj;bear;Date//Pt;bear;FC//Sp;Y;a//Ad;Y;american//Attrib;be;Y >
This last feature will probably include irrelevant information in many cases, but it could also provide crucial elements ignored by the SDPs.
As in the previous categories, these syntactic features were evaluated alone and in several combinations. It is worth noting that the dependency parsers used to analyze the corpora (DepPattern, see Section 3.1) do not provide full coverage in some sentences. Since they are rule-based, some complex sentences were not totally parsed, so in these cases the SDP and/or the FDP were not extracted. However, these parsers performed better than MaltparserFootnote 8 trained with the BosqueFootnote 9 and AnCoraFootnote 10 corpora for Portuguese and Spanish, respectively. Several evaluations show that the parsers used in this paper achieve about 88%/79%/83% (pt) and 85%/74%/79% (sp) of precision/recall/f-score (Gamallo and González Reference Gamallo and González2013).
5 Experiments
This section shows the results of several experiments performed in order to know the effectiveness of each of the linguistic features previously defined.
First, some experiments concerning the size of the window between the entities are shown. Then, machine learning classifiers were trained using the features extracted from the linguistic space. The linguistic space is exploited using two strategies: first, by evaluating the performance of each type of feature individually, and second, in a bottom-up way, that is, by starting with the most basic features and, then, systematically including more complex features.
Experiments were carried out using the libsvm implementation of support vector machines (SVM) (Chang et al. Reference Chang and Lin2011), since in preliminary tests it performed better than other algorithms such as Naive Bayes, Decision Trees, and Logistic Regression. In order to evaluate the ten relations (five for each language), a one versus all approach was applied.
For each classifier, 80% of the data were used for training and 20% for testing (all of them randomly selected). For setting the SVM parameters, ten-fold cross-validations were performed on the test sets.
In order to evaluate the performance of each classifier, the following measures were used: Precision is the number of correct positive decisions divided by the number of total positive decisions (true positives and false positives) made by each classifier. Recall refers to the number of correct positive decisions divided by the total number of positive examples in the test set. F-score is the harmonic mean of precision and recall.
Average results are presented using both micro-average and macro-average values. Micro-average is calculated by constructing a global contingency table with the results for every relation, and then calculating precision, recall, and f-score using these sums. Macro-average values are the average of the individual results for each relation.
5.1 Windows sizes
Our first experiment consists in evaluating different windows sizes for the extraction task. Specifically, this experiment attempts to answer the following questions: What is the best intermediate window between X and Y? And what are the best windows for the sequences before the first and after the second entities?
This evaluation was performed with classifiers built with Basic and Tok features (Section 4) for all the relations and languages.
First, classifiers were trained using an intermediate window of two tokens, and incrementally adding two more tokens in every iteration until achieving a window of 20 elements. In these experiments, four tokens before the first entity and four after the second one were used.
Figure 6 contains the micro-average precision and recall of the five Portuguese classifiers as well as for the five Spanish systems. As this figure shows, using more than 16 tokens in the intermediate window does not improve the precision of the system. However, a large window allows the classifier to analyze more sentences, so the recall increases. Note that in Section 3.3, Figure 3 showed that most positive patterns occur with a small (< 8) intermediate window. For the rest of the experiments, an intermediate window of 16 was selected.
Micro-average precision (P) and recall (R) of five classifiers for Portuguese and Spanish (one per each relation) versus intermediate window size (2–20, in number of tokens).
In the following, more tests were carried out in order to find the best size of the surrounding windows. The context of two entities X and Y consists of: the intermediate (between X and Y, already defined in 16 tokens) and the surrounding windows: before the first target entity (previous) and after the second one (post).
Several experiments were performed using different sizes of these windows (from 0(previous)/16(intermediate)/0(post) to 8/16/8).
However, the results of these tests show that changing the number of features in the previous and post contexts does not affect very much the performance of the system. In this respect, modifying from zero to eight the surrounding windows gave maximum differences of about 1% of f-score, and did not follow similar tendencies for each classifier.
Concerning the patterns, it is worth mentioning that the new structures obtained with larger surrounding windows are harder to be represented than those with shorter previous and post contexts. Therefore, the average results indicate that using windows of three tokens before and after the entities is enough for trading-off performance and efficiency (in number of features).
5.2 Feature effectiveness
In order to know the effectiveness of the different features the following strategy was applied: First, classifiers built with an individual type of feature (Tok, Lem, etc.) were used. The results of these experiments suggest what type of feature better represents the linguistic level they belong to (lexical, pseudo-syntactic, or syntactic). Then, several feature combinations were evaluated by training models with the best features of each level, trying to avoid information overlapping. Note that Basic features (described in Section 4.1) were incorporated in all the classifiers.
5.2.1 Semantic restriction and contextual subspaces
Before this main set of experiments, two baseline models were developed in order (i) to assess the importance of the semantic restriction of the candidate entities, and (ii) to determine the best way of representing the elements of the previous and post windows in the feature vector.
As Sections 3 and 4.1 noted, the approach followed in this work uses semantic knowledge in order to restrict the candidate pairs to be analyzed (classifying only Person–Date pairs for hasBirthPlace and hasDeathPlace, and not for hasBirthDate or hasProfession, for instance). Therefore, two baseline models were used to evaluate how named entity recognition affects learning.
Baseline 1 (B1) classifiers use more candidate pairs than the other models, since they do not take advantage of semantic classification of proper nouns. Thus, negative pairs (which cannot belong to the testing relation: e.g., Location–Number for hasBirthDate) were automatically added to the training and testing sets. The number of candidates of each set is 5.2 (pt) and 4.9 (sp) times larger than the standard ones.
Baseline 2 (B2) systems differ from B1 in the semantic restriction of the candidates, since B2 applies semantic filtering on the candidate pairs.
Both B1 and B2 classifiers were built with Tok features extracted from the following windows: 3/16/3.
The first two columns of Tables 3 and 4 show the results of the two baseline models for Portuguese and Spanish, respectively. Results are shown per relation as well as in micro- and macro-average values.Footnote 11
Precision, Recall, and F-score per relation and model for Portuguese. Abbreviations are expanded in footnote Footnote 11
Precision, Recall, and F-score per relation and model for Spanish. Abbreviations are expanded in footnote Footnote 11
As expected, the comparison of the B1 to the B2 results proves the effectiveness of the applied semantic filtering, which increases in more than 25% the micro-average f-score of the classifiers. Semantic restriction allows the models to not classify pairs that cannot belong to the target relation. Therefore, both precision and recall of all the classifiers and languages improve, except for the hasDeathDate relation, whose precision values decrease (≈ 16%), but with much higher recall values (≈ 60%).
B2 and Lx1 models were trained with the same set of features, but the latter differs from the baselines on the way it represents the surrounding elements in the feature vector. Unlike baseline models, which mix all the features in the feature space, the following classifiers divide these features into three subspaces (previous, intermediate, and post). Therefore, the differences between B2 and Lx1 results (Tables 3 and 4) are due to the distinct representation of the surrounding elements in the feature vector.
Again, the performance of every classifier improves with the proposed feature representation. Contextual subspaces allow classifiers to better represent the sentences where the relation occurs, by identifying the relevance of the relative word order in a pattern.
It is worth noting that the differences between B2 and Lx1 results follow similar tendencies: in general, f-score results increase between 2% and 4% within the new scenario. However, those relations whose attribute (Y) is a Date have a higher improvement (between 9% and 10% in Portuguese and 5% and 9% in Spanish). As it was shown in the pattern analysis (Section 3.3), these relations occur frequently in biographical structures containing both birth and death dates. So, the use of different subspaces helps these classifiers to disambiguate between many hasBirthDate/hasDeathDate candidate pairs. This strategy was applied in the training of the subsequent models.
5.2.2 Lexical and morphosyntactic level
The next experiment aims at knowing what is the best way for representing the lexical units in the sentences (Section 4.2). Three models were evaluated: Lx1 (built with Tok features, as already mentioned), Lx2 (built with Lem), and Ms2 (based on PoS features), which contains morphosyntactic information.Footnote 12
The results of these models (Tables 3 and 4) show, as hypothesized in Section 4.2, that the use of lemmas instead of tokens (Lx2 versus Lx1) generalizes the patterns, by reducing the data sparseness of the classifiers.
This generalization involves little improvements in all the relations and languages (≈ 0.5%), except hasDeathDate in Spanish. Note, however, that the improvements against Lx1 models were not statistically significant.Footnote 13 A deep analysis of these results showed that models with lemmas as bag-of-words correctly classify some candidates with less frequent verbal forms (e.g., ‘nacidas’Footnote 14).
Ms2 models are constructed with a further step of generalization, by substituting the lexical information of the sentences by their morphosyntactic representation.
Again, as the pattern analysis shows (Table 2), some relations heavily depend on biographical sentences, where punctuation takes an important role in the definition of the linguistic structures. These patterns are relatively well covered by PoS features, but removing the lexical information of the models involves a decrease in their general performance (≈ 3%), except for hasProfession relation, where representing adjectives (and other word classes less relevant for defining a relation) by their PoS-tag simplifies the feature space of the classifiers.
This set of tests confirms the validity of features based on lemmas (Lem) for representing the lexical units. The following experiments perform a similar analysis by evaluating the performance of models based on pseudo-syntactic features.
5.2.3 Pseudo-syntactic level
Two different models were developed using pseudo-syntactic information: BiG, built with bigrams of lemmas, and TrG, containing trigrams of lemmas.
Despite the results showing variation between relations and languages, in general bigrams of lemmas work better than models built using trigrams. In Portuguese, trigrams work better for hasBirthPlace, due to the great amount of patterns where the two target entities are covered by the same trigram feature < X_ (_Y_Loc > . Similarly, hasProfession results show less variation, since their patterns are usually well represented by both bigrams and trigrams of lemmas.
Best pseudo-syntactic models, however, do not clearly overcome Lx2 classifiers, which perform slightly better in almost every scenario in Portuguese. Nevertheless, the BiG micro-average performance in Spanish is higher than Lx2, because BiG models only work noticeably better in hasProfession and hasDeathDate. In this case, only the improvements of BiG models for hasProfession were statistically significant.
5.2.4 Syntactic level
Finally, the effectiveness of syntactic parsing for the RE task was evaluated by comparing three different strategies: Dep models, which contain syntactic dependencies as the main feature; Pth, built with dependency paths; and classifiers using the shortest dependency path (SDP).
The three last columns in Tables 3 and 4 contain the results of the models that use syntactic information. Note that for some patterns, the dependency path and the SDP could not be extracted. This fact was caused for two main reasons: (i) lack of punctuation of some sentences, which involved parsing errors, and (ii) parser coverage, which cannot establish some dependencies in the entities path. Thus, 40% (pt) and 54% (sp) of the data do not have Pths features, while the SDPs were not extracted from 54% (pt) and 63% (sp) of the corpus.
Therefore, recall values of these classifiers are noticeably low (except for hasProfession and for hasBirthDate in Portuguese, whose patterns were better analyzed by the syntactic parsers).
In general, single dependencies do not have high precision values, unless they contain unambiguous knowledge (e.g., < Subj;nascer;X > and < em;nascer;Y >, where the verb ‘nascer’ means to be born in Portuguese), so the average results are worse than B2 and BiG models.
Pth and SDP classifiers clearly depend on the coverage as well as on the patterns typology. The precision values of these models surpass Lx2 and BiG in some relations (with values of 100% precision in several experiments). However, other patterns that contain ambiguous Pths and SDPs (or even that do not have these features) caused a lower performance.
Between these two models, SDP seems to have better precision than Pth, although both of them have recall problems, even in sentences with Pths and SDPs features successfully extracted.
So, this test indicates that using only features based on syntactic analysis (and even more depending on the parser accuracy and coverage) is not suitable for building RE systems. However, the high precision of some of the Pth and SDP models suggests that they could be good features to combine with other information based on lexical and/or pseudo-syntactic analysis.
5.3 Feature combinations
To optimize the combination of features in terms of performance and efficiency, different features from each linguistic level (lexical, pseudo-syntactic, and syntactic) were combined in order to find the best classifiers as well as to identify redundant features with information overlapping.
First, best lexical (Lem) and pseudo-syntactic (Bigrams) features were combined for building L/B models. Then, Lem were also used by training classifiers with Pths features (L/P).Footnote 15 Finally, three different levels of linguistic analysis were combined for building classifiers (L/B/S), by using the best possible combination of them: Lem, Bigrams, and SDPs features.
Table 5 contains the results of these three combinations for each relation and language, together with their macro- and micro-average values.
Precision, Recall, and F-score per relation, language, and model. Classifiers combine features from different levels of linguistic analysis: L, lexical (Lem); B, pseudo-syntactic (Bigrams); P and S, syntactic (Pths and SDPs, respectively)
The first model (L/B), which combines features based on lexical and pseudo-syntactic analysis, performed statistically better than these features taken individually. Classifiers for Portuguese improve between 0.6% and 0.8% when compared to Lx2 and BiG, respectively. In Spanish, the increase was of 1.27% and 0.9%. Furthermore, this combination involves improvements in both precision and recall of all the models (with the only exception of hasDeathPlace recall in Portuguese).
So, lexical and pseudo-syntactic features, even built with the same base information (lemmas), are suitable combination for developing classifiers without computationally expensive resources such as syntactic parsers.
The next feature combination (L/P) was carried out by using lexical (Lem) and syntactic information (Pths). In Portuguese, the full dependency path also helps to surpass the performance of Lx2 models in all the relations (with an increase of ≈ 0.4% of micro-average f-score). These models also worked better (although just 0.2%) than the previous combinations (L/B).
In Spanish, the use of syntactic information together with features based on lemmas involves little improvements (0.2%) in the classifiers when compared to the use of lemmas as the unique type of feature. However, these combinations do not overcome models built with bigrams of lemmas (BiG and L/B).
Nonetheless, note that the Portuguese parser extracted more Pths and SDPs features than the Spanish one, as referred in Section 5.2.4.
Finally, the third combination (L/B/S) explores the use of Tok, Bigrams, and SDPs features in the same classifiers.
In Portuguese, this last combination works better than L/B (≈ 0.3%) and L/P (0.07%) models, especially in those relations where SDP systems had achieved good results (> 75% f-score): hasBirthPlace and hasProfession. However, L/B/S classifiers do not improve the performance of previous models in hasDeathDate and hasDeathPlace, in fact producing lower macro-average values than L/B and L/P systems.
In Spanish, the addition of syntactic features does not present noticeable improvements in the classifiers, although the average results are significantly better than those achieved by models built with just a single type of feature.
In sum, this last set of experiments demonstrates that combining different features that do not involve significant information overlap helps to improve the general performance of the classifiers. This means that complementary features permit better classifiers to be built. Taking into account that languages such as Portuguese and Spanish suffer lack of free or state-of-the-art tools for syntactic parsing, it could be interesting to evaluate these features with more mature systems.
5.4 Learning curve
Concerning the importance of the amount of training data used by the classifiers, Figure 7 shows the micro-average learning curve of the L/B/S classifiers for Portuguese and Spanish.
Micro-average f-score of five classifiers for Portuguese and Spanish versus training data (25–637 pairs). Testing was performed with 200 random candidate pairs.
These tests were carried out by using random sets of 25, 50, 75, 100, 200, 300, 400, 500, 600, and 637Footnote 16 training sentences. The curves indicate that the classifiers achieve their peaks with about 500 sentences. More training data gives little improvement in performance. However, other tests with relations that have more data (hasBirthPlace and hasProfession) suggest that the classifiers start overfitting the training data with about 1, 400 sentences, since their f-score results do not improve and sometimes decrease.
Finally, Figure 7 also shows that the learning curves have very similar tendencies for each language.
5.5 Error analysis
In order to know the main sources of errors made by the classifiers, a careful analysis of the false positives and false negatives of the L/B/S results were carried out, including every misclassification of all relations and languages. The errors are grouped together by their typology. Moreover, the micro-average percentage (between pt and sp results) of each type of error was also calculated.
5.5.1 Ambiguity and parsing errors (≈ 43%)
This class of errors occurs when lexical and/or pseudo-syntactic features are not enough for representing the relation between two entities, and the SDP (which could disambiguate the pattern) is not extracted or not correctly built. The following Portuguese sentence (false positive for hasBirthPlace) exemplifies this kind of error:
Sentence: [. . .] filho de Per e de X nasceu no Y_Loc [. . .]
Trans.: [. . .] son of Per and of X was born in Y_Loc [. . .]
Pattern: [NP and of]previousX nascer em o Y_Loc [. . .]post
5.5.2 Patterns without explicit information (≈ 25%)
Other misclassifications arise when the pattern (or even the whole sentence) does not contain explicit information concerning the target relation, thus producing a false negative. The following example shows a pattern where the hasBirthDate relation (in Portuguese) between X and Y_Dat is not explicit, and the main lexical data is outside the scope of the extracted features:
Sent.: [. . .] teve sete filhos: Per em Dat, Per em Dat, X em Y_Dat [. . .]
Trans.: [. . .] had seven children: Per in Dat, Per in Dat, X in Y_Dat [. . .]
Pattern: [em Dat,]previousX em Y_Dat [. . .]post
5.5.3 Less frequent patterns (≈ 15%)
Some other positive pairs are misclassified (false negatives) when they appear in very infrequent patterns (with just one or two—or even zero—examples in the training corpus). Some of them might be correctly classified if they contain relevant features (such as a dependency path), but usually they are false negative instances:
Sent.: [. . .] X, que era apenas um homem da zona rural de Y_Loc [. . .]
Trans.: [. . .] X, who was just a man from the rural zone of Y_Loc [. . .]
Pattern: X, que ser apenas um homem de a zona rural de Y_Loc
5.5.4 Useless shortest dependency path (≈ 14%)
As shown in the previous section, SDPs are high-precision features. However, they can hurt the performance of a classifier when they are ambiguous for both true positive and true negative examples. The following pair of sentences are, respectively, a true negative and a true positive example of hasProfession in Spanish, represented by the same SDPs:
Sent. 1: X y un Y_Occ fueron dos de los homenajeados [. . .]
Trans.: X and a Y_Occ were two of the honorees [. . .]
Sent. 2: X, poeta y Y_Occ, recibió el patrocinio de [. . .]
Trans.: X, poet and Y_Occ, was sponsored by [. . .]
SDPs: < Coord;y;X//Coord;y;Y_Occ >
In this type of scenarios, the L/B/S classifiers labeled as positive (for hasProfession) both of the sentences.
Besides that, SDPs extracted from biographical sentences—that often contain non-lexical dependencies between entities and surrounding punctuation—do not have high precision, since they appear indistinctly in positive and negative patterns.
5.5.5 Other errors (≈ 3%)
Finally, there are other less frequent sources of misclassifications, such as (i) lemmatization problems, and (ii) named entity recognition errors.
One of the few examples of errors produced by lemmatization includes the Portuguese participle ‘morto’ (dead), which was incorrectly lemmatized as ‘matar’ (to kill) instead of ‘morrer’ (to die). So, a hasDeathPlace pair such as the following was classified as negative for this relation:
Sentence: X, morto em Y_Loc [. . .]
Trans.: X, dead in Y_Loc [. . .]
Pattern: X, matar em Y_Loc
Other less frequent sources of errors derive from the NER tool, which might incorrectly classify as a common noun a Person name, or vice-versa. In the following example, ‘Belén’ (Location) was misrecognized as a common noun, so it was not a candidate entity for hasDeathPlace: ‘X (Location, Date - Belén, Date).’
5.6 Discussion
This section briefly discusses the results of the performed experiments, taking into account the hypotheses referred to during the presentation of the features (Section 4) as well as some of the previous works analyzed in Section 2.
First, it is worth noting that filtering the candidate pairs by their semantic class allows the systems to increase their micro-average f-score by more than 25%, by improving both the recall and the precision. However, unlike other approaches (Kambhatla Reference Kambhatla2004; Zhou et al. Reference Zhou, Su, Zhang and Zhang2005), the semantic classification of the other proper nouns in the patterns does not contribute to a better performance. Furthermore, the use of different subspaces for representing the surrounding elements of a pattern also improves the behavior of every classifier.
Then, it is also important to note that, once the candidate entities have been annotated with a NER, simple classifiers (those with only Basic and lexical features) are enough to achieve good precision and recall values (about 91% and 92% for Portuguese and Spanish, respectively). Thus, it can be inferred that (i) it is feasible to build high-quality proper noun ontologies with little manual effort, and that (ii) approaches based on supervised classifiers are useful for the BRE task.
Concerning the effectiveness of the features, first experiments show that generalizing the lexical units by means of lemmatization (Lem) improves the classifier performance. This fact agrees with tests carried out by Agichtein (Reference Agichtein2005), when he decides to use words (tokens) and stems. The use of PoS-tags, however, did not produce any improvement on the performed experiments (except for some patterns that depend little on lexical knowledge, such as ‘Y_OccX,’ as Mann (Reference Mann2002) noticed).
With regard to pseudo-syntactic features, the experiments performed in this paper showed that bigrams of lemmas were the best way to represent this level of analysis. Trigrams of lemmas also work well with patterns where the related entities are close together, but they still depend on other features for better representing the relation. Besides that, lexico-syntactic patterns had little impact on the classifiers.
Even though syntactic parsing might be computationally expensive and less accurate than other processing steps (lemmatization, NER), it provides useful information. Though syntactic features did not work well individually, their combination with lexical and pseudo-syntactic information allowed the systems to improve their classification. However, it is worth noting that some SDPs might be a source of errors if they represent entities linked by punctuation and/or coordination instead by lexical data.
Feature combination tests indicated that using complementary information obtained from different linguistic levels helps to develop better classifiers with less feature overlapping. In this respect, the best combinations for both Portuguese and Spanish were those based on lemmas, bigrams of lemmas, and SDPs. Apart from that, the performed experiments also show that using just lexical (Lem) and pseudo-syntactic data (Bigrams) permits high-precision classifiers to be built without using syntactic parsing.
In this regard, it is worth remembering that the parsers used in these experiments—but also other state-of-the-art parsers trained with free corpora—did not provide dependency paths for almost a half of the corpora, so the results should be considered as preliminary. Thus, the knowledge provided by syntactic features do not always add new information to the classifiers, since some dependency paths are similar to some bigrams of lemmas.
Therefore, some of the results of this work agree with Jijkoun et al. (Reference Jijkoun, De Rijke and Mur2004) and Jiang and Zhai (Reference Jiang and Zhai2007), who had concluded that more complex features might hurt the performance of the classifiers (or increase the recall with a cost in precision). Nevertheless, syntactic information has also been reported as useful in previous works (Kambhatla Reference Kambhatla2004; Zhou et al. Reference Zhou, Su, Zhang and Zhang2005; Mintz et al. Reference Mintz, Bills, Snow and Jurafsky2009). Hence, it would be interesting to analyze the performance of syntactic features using different corpora and/or other parsers.
From the above, several conclusions can be made, which might be useful for improving the extraction of semantically related entities: First, a previous analysis of the patterns where the target(s) relation(s) occur can be useful for adapting the feature space of the classifiers. In this respect, works such as Garera and Yarowsky (Reference Garera and Yarowsky2009) introduced several strategies designed ad hoc for biographical fact extraction.
Second, errors in lemmatization can be avoided by analyzing those lexical units that are crucial for a desired relation (e.g., ‘to die’ or ‘to be born’ verb for relations related to death and birth).
Third, if the patterns heavily depend on punctuation and/or coordination, syntax might not be very helpful, since parsers better represent syntactic relations established through lexical elements.
Finally, though the performance of the tested classifiers is good enough for accurately extracting instances of the evaluated relations, further work is needed in order to assess whether the methods presented in this paper are suitable for extractions with other relations and texts (Grishman Reference Grishman2010).
6 Conclusions and further work
Our work is an attempt to shed some light on the following questions concerning extraction of biographical semantic relations in Portuguese and Spanish: which are the more appropriate linguistic features to improve the performance of supervised machine learning approaches? Which are those allowing better combinations without feature overlapping? And what is the best way to obtain features for extracting different semantic relations from languages with less linguistic resources?
The experiments described in this paper can help us to outline some ideas to answer these questions. First, results showed that it is preferable to use lemmas instead of words, and that semantic knowledge is crucial for filtering the candidates of a classifier. Furthermore, combining lexical features with pseudo-syntactic information such as bigrams of lemmas improves the performance of the classifiers. Finally, syntactic parsing also provides useful knowledge to be combined with features from different levels of analysis, even if it does not involve great benefits in the systems.
Taking this into account, the results also pointed out that it is possible to build high-quality classifiers even for languages with fewer resources, since combinations of features obtained by lemmatization are often enough to train reliable RE classifiers.
Notice that this work dealt with several linguistic levels: lemmatization, PoS-tagging, named entity recognition, as well as pseudo-syntax and syntactic analysis.
Moreover, this paper showed that semi-structured knowledge available in Wikipedia is reliable enough to build a large amount of training data, without requiring bootstrapping techniques. This way, distant-supervision strategies can be adapted to rapidly obtain corpora for supervised classifiers, by reducing the time consumption of manually annotating corpora from scratch.
Semi-structured data such as infoboxes or category trees can be used as big repositories of seed examples to learn further semantic information on named entities, in the same way as lexical resources (e.g., WordNet) are often used as initial meaning repositories to deal with complex semantic phenomena related to lexical words: word sense disambiguation, anaphora resolution, etc.
Finally, let us mention the contribution of this work to the study of RE from languages other than English. Even if one of the main characteristic of the Web is its multilingual scope, most work on extraction of relations between named entities has been done only in the English version. This paper provides freely available annotated corpora with five biographical relations for both Portuguese and Spanish, as well as a careful analysis of the patterns where biographical relations occur. We hope there will be very soon more pieces of work on multilingual semantic extraction.
