1 Introduction
Information extraction (IE) systems acquire concepts and information about them from collections of text. Yet, extracted relations are typically between plain terms. Such representation is usually not enough to unambiguously refer to a concept, because the same word might have different meanings, while different words might have the same meaning. On the one hand, this problem is not severe in the extraction of domain knowledge, where, based on the ‘one sense per discourse assumption’ (Gale, Church and Yarowsky Reference Gale, Church and Yarowsky1992), ambiguity is low. On the other hand, when dealing with broad-coverage knowledge, if ambiguities are not handled properly, it becomes impractical to formalise the extracted information and perform tasks such as knowledge inference (Moldovan and Novischi Reference Moldovan and Novischi2002) or coherent summarisation (Barzilay and Elhadad Reference Barzilay and Elhadad1997).
To make IE systems more useful, term ontologising (Pantel Reference Pantel2005), which is a kind of word-sense disambiguation (WSD) (Navigli Reference Navigli2009), is needed. This task aims at moving towards an ontological structure by associating the extracted terms to representations of their meanings.
However, whereas most WSD techniques rely on the context where the words occur to find their most suitable sense, the local context is sometimes very limited or unavailable, as in dictionary definitions, word lists or text snippets. Moreover, when dealing with heterogeneous knowledge sources, it is not always easy to set the context of a word (e.g. window, sentence, document), which may have to be defined manually, according to the specific source or entry. Therefore, in order to provide greater flexibility, we aim to ontologise without accessing the local context. Our view enables the creation of an IE system with two independent modules: (i) one for extracting relations between terms (hereafter, tb-triples); and (ii) another for ontologising the arguments of the latter.Footnote 1 This modular approach is a useful way of coping with information sparsity, since it allows for the extraction of knowledge from different heterogeneous sources, sometimes out of context, and provides a way to harmoniously integrate it in a common knowledge base.
The present work is focused on the second module, ontologising. Our goal is to map each term argument, in a relation, to a concept, represented, for instance, as a synset in a wordnet-like knowledge base. Wordnets are a popular way of organising lexical-semantic knowledge on a language, inspired by PWN (Fellbaum Reference Fellbaum1998), but adapted to many other languages (see Bond and Paik (Reference Bond and Paik2012)). They are structured in synsets—groups of synonymous words/word senses that contain the possible lexicalisations of a natural language concept – and semantic relations, including hypernymy and part-of, established between the synsets, according to the meaning they transmit.
In Open Information Extraction systems (Etzioni et al. Reference Etzioni, Fader, Christensen and Soderland2011), where many different predicates are extracted for describing a relation between two entities (hereafter, relation phrases), it is common to deal with relation ontologising as well. Yet, the previous task is out of the scope of our work, which targets only IE systems that acquire a set of normalised relations of pre-defined types.
In the previous work, we proposed alternative algorithms to map the arguments of tb-triples into the synsets of a thesaurus, and we compared them in that task (Gonçalo Oliveira and Gomes Reference Gonçalo Oliveira and Gomes2012). They were shown to depend on the quantity of extracted knowledge because, in order to find adequate synset matches, they exploit the network established by the extracted tb-triples. Moreover, the presented comparison had a few limitations because: (i) it used a set of tb-triples extracted automatically, not hand-checked for correction; (ii) it was based on a gold standard (GS) created manually by just one person; (iii) it was made only for Portuguese.
In this work, we revisited those algorithms and compared their performance when mapping tb-triples into existing wordnets for both English and Portuguese, and ended up concluding that their utilisation is, in fact, much broader. They can be used not only in the creation of new wordnets, but also for enriching existing wordnets, by exploiting their own structure. The last point is of great importance because wordnets, typically handcrafted, have frequently coverage issues, especially when it comes to world knowledge, which is confirmed by the amount of works aiming both at enriching wordnets or aligning them with other knowledge bases.
In the remaining of this article, we enumerate some related works and introduce the task of ontologising tb-triples, in the scope of our work. After that, we describe the revisited algorithms and complement their description with an example for each. We then report on the performed experiments and discuss the obtained results. In the end, we draw some conclusions on the broader utilisation of these algorithms.
2 Related work
Word Sense Disambiguation (WSD) (Navigli Reference Navigli2009) is the task of selecting the most adequate sense of a word in a particular context, from a list of possible senses, as in a dictionary or lexical ontology. For English, Princeton WordNet (PWN) (Fellbaum Reference Fellbaum1998) is commonly used as a sense inventory. The goal of WSD thus becomes the assignment of a word in context to one of its possible PWN senses. At the same time, in the so called knowledge-based WSD, PWN can be used as an external source of knowledge. Approaches to knowledge-based WSD are typically unsupervised and explore the structure of PWN for setting a context for the word senses, which are then compared with the context of words in text. The word sense context can be obtained, for instance, from the taxonomic tree of PWN (Resnik Reference Resnik1995), related words and words in the synset glosses (Banerjee and Pedersen Reference Banerjee and Pedersen2002), or by exploiting the graph structure established by relations and word senses (Agirre and Soroa Reference Agirre and Soroa2009).
Our work is more specific and does not require a full disambiguation of text. It aims to move from knowledge structures based on terms, identified by their orthographical form, to ontological structures, which handle ambiguities and are thus aware of word meanings. This task was originally baptised as ontologising, when a method was presented to link terms, extracted from text, to PWN synsets, after inducing several features of their senses (Pantel Reference Pantel2005). Besides enabling the formalisation of knowledge according to words and their meanings, and providing tasks such as inference, these methods can be used to enrich ontologies with information extracted from text.
Whether aiming for the automatic acquisition of knowledge on specific domains (e.g. Pustejovsky et al. (Reference Pustejovsky, Zhang, Kotecki and Cochran2002); Wiegand, Roth and Klakow (Reference Wiegand, Roth and Klakow2012)) or the extraction of open-domain facts (e.g. Agichtein and Gravano (Reference Agichtein and Gravano2000); Etzioni et al. (Reference Etzioni, Fader, Christensen and Soderland2011)), IE systems typically represent concepts as terms, which are lexical items identified by their written form or by their lemma. In those systems, semantic relations are commonly denoted by tb-triples tt = (a R b), where the arguments (a and b) are terms whose meaning is connected by a relation described by R. Even when targeting the acquisition of lexical-semantic relations that would eventually enrich PWN (e.g. Hearst (Reference Hearst1992); Pantel and Pennacchiotti (Reference Pantel and Pennacchiotti2006)), most IE systems output a triple between terms, not synsets, which is usually not enough to unambiguously refer to a concept and/or mapping it to an existing synset.
Although, PWN is a broad-coverage resource, it is limited and incomplete in many domains (Hearst Reference Hearst1992; Caraballo Reference Caraballo1999), and therefore not enough for several NLP tasks, especially when world knowledge is needed. In addition to the automatic acquisition of tb-triples, researchers working on IE from text had to go further, in order to apply some of the ontologising ideas to enrich PWN. To name a few works, PWN was augmented with additional relations obtained from its own glosses (Harabagiu and Moldovan Reference Harabagiu and Moldovan2000), topic signature relations (Cuadros and Rigau Reference Cuadros and Rigau2008), domain concepts (Navigli et al. Reference Navigli, Velardi, Cucchiarelli and Neri2004), named entities (Toral, Muñoz and Monachini Reference Toral, Muñoz and Monachini2008) and all kinds of information from Wikipedia (Suchanek, Kasneci and Weikum Reference Suchanek, Kasneci and Weikum2007; Navigli and Ponzetto Reference Navigli and Ponzetto2012), among others. Moreover, PWN has been linked to the upper ontology SUMO (Pease and Fellbaum Reference Pease and Fellbaum2010) and to the descriptive ontology DOLCE (Gangemi et al. Reference Gangemi, Guarino, Masolo and Oltramari2010).
In Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006), two methods are presented for taking advantage of the structure of PWN to ontologise tb-triples, extracted from text. The anchor approach assumes that terms related in the same way to a fixed term are more plausible to describe the same sense. To select a suitable synset for attaching a term x in tt = (x R y), it computes an overall sense score (r(sx ), see (1)) by exploiting: the possible senses of x (sx ∈ Sx ); senses of terms related to y the same way as x (s x′ ∈ S x′); the number of times each of the latter senses (synsets) occurs in a relation, of the same type, with y (f(s x′)); and the distances of the previous synsets in the hypernymy hierarchy (d(sx, s x′)).
\begin{equation}
r(s_x) = \sum \limits _{s_{x^{\prime }} \in S_{x^{\prime }}} \bigg (\frac{1}{d(s_x, s_{x^{\prime }})+1} \times f(s_{x^{\prime }})\bigg )
\end{equation}
All synset pairs with r(sx ) higher than a threshold are selected as attachment points for the tb-triple. Alternatively to the anchor approach, they describe a clustering approach, which selects suitable synsets using generalisation through hypernymy links in PWN. First, it sets semantic clusters containing the tb-triples with a common generalisation. Then, to find the attaching points for the arguments of each tb-triple, it selects the generalisation of the best cluster, regarding that this selection is supported by the remaining tb-triples and, at the same time, is not too general nor too specific. It should be stressed that this approach may result in the attachment of a term to its hypernym and not to its actual representation in the ontology.
This kind of approach was also applied to the Open Information Extraction paradigm (OIE, (Etzioni et al. Reference Etzioni, Fader, Christensen and Soderland2011)), where large sets of facts, represented as tb-triples, are extracted from text. In Soderland and Mandhani (Reference Soderland and Mandhani2007), an earlier attempt to ontologise the output of an Open Information Extraction system in PWN is proposed. To this end, they exploit both the extraction context, with words in the sentences where each tb-triple was extracted from, and the synsets context, which contains words in the synset, in sibling synsets and in direct hyponyms. The term arguments are mapped to synsets that include one of their senses and share the most similarities with the triple context. After this, the relation phrase is normalised, the logical semantics is formalised, the meta-properties of each relation are learned, a correctness probability is given to each relation and, in the end, an inference engine combines the derived relations with the relations in PWN.
In fact, when it comes to Open Information Extraction, ontologising the relation phrases is as critical as ontologising their arguments. Although meaningful, relation phrases often do not match a known relation type directly and there are generally different textual variations to denote the same relation. In the recent years, there has been work on ontologising relation phrases, which might involve not just the normalisation of the phrase, but also clustering of synonymous phrases, phrase disambiguation, and relation formalisation, by identifying typical semantic classes of its arguments (see e.g. Min et al. (Reference Min, Shi, Grishman and Lin2012); Nakashole, Weikum and Suchanek (Reference Nakashole, Weikum and Suchanek2012); Flati and Navigli (Reference Flati and Navigli2013); Moro and Navigli (Reference Moro and Navigli2013)). Although this feature could enlarge our work, it is currently out of the scope of this article. We are only concerned with ontologising relations of a pre-defined and unambiguous type.
Back to the main topic, in the context of the automatic construction of wordnets, alternative algorithms for ontologising tb-triples were presented (Gonçalo Oliveira and Gomes Reference Gonçalo Oliveira and Gomes2012). Similar to Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006) approaches, they do not consider the extraction context as cues for disambiguation. But they do not use the wordnet structure directly, either. Instead, they exploit all the extracted information for computing similarities between the term arguments and the terms in candidate synsets. This option, which provides greater flexibility, relied especially on two facts: (i) these algorithms were originally used to add relations to a synset-oriented thesaurus without any previously assigned relations nor synset glosses; (ii) the information they use can be acquired from different heterogeneous sources (e.g. dictionaries, corpora, ontologies), where it is not always straightforward to set an extraction context. Moreover, as the wordnet structure is not available/used, each term argument is always attached to a synset containing it, in contrast to Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006) clustering approach where it can be attached to one of its hypernyms instead.
The algorithms presented in Gonçalo Oliveira and Gomes (Reference Gonçalo Oliveira and Gomes2012) were compared when ontologising tb-triples extracted from a Portuguese dictionary into a handcrafted thesaurus of Portuguese. In this article, we show that, besides the extracted information, this kind of approach can also take advantage of the wordnet structure, after its ‘deconstruction’ into tb-triples, thus broadening its utility. We also confirm that it is a multilanguage approach, as we used it not only for Portuguese, but also for English.
3 Moving from term-oriented to synset-oriented structures
Our goal is the integration of term-oriented structures (tb-triples) into knowledge bases where concepts are represented as synsets. In other words, we aim to map the term arguments of the tb-triples to suitable synsets in a wordnet.
For each tb-triple, all proposed approaches for this task start by selecting the candidate synsets for ontologising each of its argumentsFootnote 2 . These synsets will be those that contain a word of the tb-triple arguments. Then, different strategies, including those described in Section 4, can be followed for selecting the most suitable candidates, according to the meaning transmitted by the tb-triple. For two sample tb-triples, (plot hypernym-of intrigue) and (waist part-of trunk), Table 1 illustrates how they are ontologised in PWN 3.0, given the candidate synsets. There, a ✓ marks the appropriate choice of synsets and a × is used for incorrect alternatives.
Ontologising examples
This task has some similarities to knowledge-based WSD because it exploits a knowledge source, in this case built up of tb-triples, to perform disambiguation. But here, the available context is very limited. In fact, our context consists of nothing but the two arguments of the tb-triple. This is still often enough information for a human to disambiguate the argument terms, as shown in the examples of Table 1 – for instance, waist and trunk would be clearly related in the body parts domain, but not in the elephants (elephant trunk), cars (luggage compartment), trees (bole) nor shoes (shank) domain.
On the one hand, the limited context makes our task more challenging. On the other hand, it makes our approach more flexible, as it enables to ontologise tb-triples extracted from different heterogeneous sources (e.g. corpora, dictionaries, encyclopedias, search engine logs. . .), including resources that provide none or limited context (e.g. dictionaries, word lists, text snippets), into wordnet-like resources that might be very incomplete in terms of glosses or relations.
4 Algorithms
In this section, we present five different strategies for mapping terms a and b, in a tb-triple tt = (a R b), to suitable synsets Ai and Bj of a wordnet W, such that Ai ∈ W, Bj ∈ W, Ai = {a i0, a i1, . . ., ain }, Bj = {b j0, b j1, . . ., bjm }. This results in a relational triple with synset arguments (sb-triple), st = (Ai R Bj ).
The proposed algorithms are especially well-suited for the following broader tasks:
-
• ‘Upgrading’ a synset-oriented and relationless thesaurus to a wordnet, by the addition of relation instances that connect available synsets (see Gonçalo Oliveira and Gomes (Reference Gonçalo Oliveira and Gomes2012)).
-
• Augmentation of an existing wordnet with the attachment of new relation instances.
For ontologising terms in tb-triples, these algorithms exploit a larger set of tb-triples, which is provided to them and constitutes a lexical network N. In the scope of this article, a lexical network is a graph-like organisation of a set of tb-triples (see Figure 1). In fact, the typical result of a IE system can be seen as a lexical network. Its main difference towards a wordnet is that word senses are not handled. Therefore, in a lexical network, each word defines one and only one node, even if it has more than one sense. Synonymy is just one more relation.
Relevant part of the lexical network exploited in the illustration of the ontologising algorithms.
In the case of thesaurus upgrading, the lexical network N may be established by the extracted tb-triples. Otherwise, if W contains relations, exploiting them is also an option. For this purpose, W should be converted into a lexical network by transforming each sb-triple into one or several tb-triples, as Table 2 illustrates for the sb-triple {bicycle_seat, saddle} part-of {bicycle, bike, wheel}.
Synset-triple and respective term-triples
All algorithms, presented in this section, are performed after collecting the candidate synsets from W, which are those containing term a, A⊂W : ∀(Ai ∈ A) → a ∈ Ai , and all with term b, B⊂W : ∀(Bj ∈ B) → b ∈ Bj .
Besides their description, an example is presented for illustrating each algorithm. All examples are made towards ontologising the tb-triple
$tt = ({\it ticket\_office}$
part_of theatre) to the synsets of PWN, in the following scenario:
-
• Candidate synsets:
-
– Only one for the argument ticket_office
-
– A 1 = {box office, ticket office, ticket booth} (the office where tickets of admission are sold).
-
-
– Three for the argument theatre:
-
– B 1 = {theater, theatre, house} (a building where theatrical performances or motion-picture shows can be presented).
-
– B 2 = {dramaturgy, dramatic art, dramatics, theater, theatre} (the art of writing and producing plays).
-
– B 3 = {field, field of operations, theater, theater of operations, theatre, theatre of operations} (a region in which active military operations are in progress).
-
-
-
• Hypothetic lexical network N to be exploited, with the following relevant tb-triples (also in Figure 1), among others:
-
– t 1 = (box_office part-of theater)
-
– t 2 = (ticket_office part-of theater)
-
– t 3 = (ticket_office part-of house)
-
– t 4 = (theater synonym-of theatre)
-
– t 5 = (box_office hyponym-of office)
-
– t 6 = (office part-of house)
-
We note that, for illustrative purposes, PWN’s lexical network as well as the implicit synonymy relations in its synsets are not used. Therefore, the hypothetic N does not contain additional relevant tb-triples, nor does it contain the word ticket booth.
4.1 Related proportion (RP)
Based on a similar assumption to Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006) anchor approach, to map term a, this algorithm fixes term b. For each synset Ai ∈ A, ni is the number of terms aik ∈ Ai such that the tb-triple (aik R b) holds in N. The related proportion rp is computed according to (2).
\begin{equation}
rp(A_i, [a, R, b]) = \frac{n_i}{1 + log_2(|A_i|)}
\end{equation}
This equation considers the size of the synsets, but minimises the negative impact of very large synsets by applying a logarithm to their size. If rp < θ, a predefined threshold, a is not mapped. Otherwise, a is mapped to the candidate synset(s) with the highest ni . Term b is mapped using the same procedure, but fixing a instead.
4.1.0.1 Example:
In the illustrative example, since there is only one candidate with ticket_office (A 1), this term would be mapped to synset A 1. Next, to map theatre using RP, ticket_office is fixed and ni is computed for each synset containing theatre:
-
• N has three part-of tb-triples between ticket_office and terms in B 1 (tt, t 2, t 3) → n B 1 = 3.
-
• N has two part-of tb-triples between ticket_office and terms in B 2 and B 3 (tt, t 2) → n B 2 = n B 3 = 2.
The resulting ni is used to compute RP:
-
•
$rp(B_1) = \frac{3}{1 + log_2(3)} \approx 1.16$
-
•
$rp(B_2) = \frac{2}{1 + log_2(5)} \approx 0.60$
-
•
$rp(B_3) = \frac{2}{1 + log_2(6)} \approx 0.58$
Given that B 1 maximises RP, the resulting sb-triple would be (A 1 part-of B 1).
4.2 Average cosine (AC)
Assuming that related concepts are described by words related to the same concepts, AC exploits all the relations in N, and not just those of type R, to select the most similar pair of candidate synsets. A term adjacency matrix M(|V| × |V|) is first obtained from N, where |V| is the total number of nodes (distinct terms). If the terms in indexes i and j are connected (related), Mij = 1, otherwise, Mij = 0.
In order to ontologise a and b, the most similar pair of synsets, Ai
∈ A and Bj
∈ B, is selected according to the adjacencies of the terms they include. The similarity between Ai
and Bj
, represented by the adjacency vectors of their terms,
$\vec{A}_i = \lbrace \vec{a}_{i0}, \ldots , \vec{a}_{in}\rbrace$
and
$\vec{B}_j = \lbrace \vec{b}_{j0}, \ldots , \vec{b}_{jm}\rbrace$
, is given by the average similarity of each term aik
∈ Ai
with each term bjl
∈ Bj
(see (3)).
\begin{equation}
sim(A_i, B_j) = \frac{\sum \limits _{k=1}^{|A_i|}\sum \limits _{l=1}^{|B_j|} \cos (\vec{a}_{ik}, \vec{b}_{jl})}{|A_i||B_j|}
\end{equation}
While this equation has been used to find similar nouns in a corpus Caraballo (Reference Caraballo1999), we adapted it to measure the similarity of two synsets, represented as the adjacency vectors of their terms in N.
4.2.0.2 Example:
To use AC in the illustrative example, N would have to be represented as a large matrix where, for each term, there would be a vector of ones (adjacencies) and zeros. For the sake of simplicity, only part of the calculation of sim(A 1, B 1) is presented. As in the cosine calculation, the numerator is an inner product, only vector positions that are higher than 0 in both vectors matter. These are the positions of nodes adjacent to both terms represented by the vectors. The other positions would result in zero. Therefore, supposing that, although there are additional tb-triples in N, none of the remaining involve any term in A 1 or B 1, the adjacency vectors can be represented by the following setsFootnote 3 :
-
• boxoffice = {boxoffice, theater, office}
-
• ticketoffice = {ticketoffice, house, theatre}
-
• ticketbooth = {ticketbooth}
-
• theater = {theater, theatre, boxoffice, ticketoffice}
-
• theatre = {theatre, theater}
-
• house = {house, ticketoffice, office}
From these sets, the following cosines are computed:
-
•
$\cos ({\it box office}, {\it theater}) = \frac{2}{\sqrt{3} \times \sqrt{4}} = \frac{\sqrt{12}}{6}$
-
•
$\cos ({\it box office}, {\it theatre}) = \frac{1}{\sqrt{3} \times \sqrt{2}} = \frac{\sqrt{6}}{6}$
-
•
$\cos ({\it box office}, {\it house}) = \frac{1}{\sqrt{3} \times \sqrt{3}} = \frac{1}{3}$
-
•
$\cos ({\it ticket office}, {\it theater}) = \frac{2}{\sqrt{3} \times \sqrt{4}} = \frac{\sqrt{12}}{6}$
-
•
$\cos ({\it ticket office}, {\it theatre}) = \frac{1}{\sqrt{3} \times \sqrt{2}} = \frac{\sqrt{6}}{6}$
-
•
$\cos ({\it ticket office}, {\it house}) = \frac{2}{\sqrt{3} \times \sqrt{3}} = \frac{2}{3}$
-
•
$\cos ({\it ticket booth}, {\it theater}) = \cos ({\it ticket booth}, {\it theatre}) = \cos ({\it ticket booth}, {\it house}) = 0$
which results in
\begin{equation*}
sim(A_1, B_1) = \frac{\frac{\sqrt{12}}{6} + \frac{\sqrt{6}}{6} + \frac{1}{3} + \frac{\sqrt{12}}{6} + \frac{\sqrt{6}}{6} + \frac{2}{3}}{3 \times 3} \approx 0.33
\end{equation*}
Supposing that sim(A 1, B 2) and sim(A 1, B 3) have a lower value than 0.33, the resulting sb-triple would be (A 1 part-of B 1).
4.3 Number of triples (NT)
Pairs of candidate synsets (Ai, Bj ) are scored according to the number of tb-triples of type R, in N, between any of their terms. That is to say, the pair that maximises nt(Ai, Bj ) is selected (see (4), where contains(t, N) returns 1 if the network N contains the tb-triple t and 0 otherwise).
\begin{equation}
nt(A_i, B_j) = \frac{\sum \nolimits _{k=1}^{|A_i|}\sum \nolimits _{l=1}^{|B_j|} contains([a_{ik},R,b_{jl}], N)}{|A_i||B_j|}
\end{equation}
4.3.0.3 Example:
Using NT in the illustrative example would comprise the following steps:
-
(1) First, for each candidate pair, count how many tb-triples are held in N between their terms:
-
• (A 1, B 1) → 4 (tt, t 1, t 2, t 3)
-
• (A 1, B 2) → 3 (tt, t 1, t 2)
-
• (A 1, B 3) → 3 (tt, t 1, t 2)
-
-
(2) Second, divide the previous number by the number of possible term pairs between one term from one and another from the other synset:
-
•
$nt(A_1, B_1) = \frac{4}{9}$
-
•
$nt(A_1, B_2) = \frac{3}{15}$
-
•
$nt(A_1, B_3) = \frac{3}{18}$
-
As the best scoring synset pair is (A 1, B 1), the resulting sb-triple is (A 1 part-of B 1).
4.4 Minimum distance (MD)
MD assumes that related synsets contain terms that are close in N, regardless the type of relations connecting them. Therefore, it selects the closest pair of synsets, given the average distance between their terms (see (5)).
\begin{equation}
\overline{dist}(A_i, B_j) = \frac{\sum \nolimits _{k=1}^{|A_i|}\sum \nolimits _{l=1}^{|B_j|} dist(a_{ik}, b_{jl})}{|A_i||B_j|}
\end{equation}
The distance between two nodes of N (dist(a, b)) is the number of edges (relations) in the shortest path between them, computed using the well-known Dijkstra’s algorithm (Dijkstra Reference Dijkstra1959). If a term in a synset (aik or bjl ) is not in N, they are removed from Ai and Bj before this calculation.
4.4.0.4 Example:
To apply MD to the illustrative example, the distance between A 1 and each synset in B would have to be computed. For the sake of simplicity, only the calculation of the distance between A 1 and B 1 is presented. For this purpose, the distance between each term in A 1 and in B 1 is first computed, as follows:
-
•
$dist({\it box\_office}, {\it theater}) = 1$
(box_office part-of theater) -
•
$dist({\it box\_office}, {\it theatre}) = 2$
(box_office part-of theater synonym-of theatre) -
•
$dist({\it box\_office}, {\it house}) = 2$
(box_office hyponym-of office part-of house) -
•
$dist({\it ticket\_office}, {\it theater}) = 1$
(ticket_office part-of theater) -
•
$dist({\it ticket\_office}, {\it theatre}) = 2$
(ticket_office part-of theater synonym-of theatre) -
•
$dist({\it ticket\_office}, {\it house}) = 1$
(ticket_office part-of house) -
• The word ticket_office is removed from A 1 because it is not in N, thus |A 1| = 2, while |B 1| = 3
After this,
$\overline{dist}(A_1, B_1)$
can be computed:
\begin{equation*}
\overline{dist}(A_1, B_1) = \frac{1 + 2 + 2 + 1 + 2 +1}{2 \times 3} = \frac{9}{6}
\end{equation*}
For
$\overline{dist}(A_1, B_2)$
and
$\overline{dist}(A_1, B_3)$
, more information would be needed about N, which has more than just the relevant tb-triples shown. If those distances were higher than
$\overline{dist}(A_1, B_1)$
, the resulting sb-triple would be (A
1 part-of B
1).
4.5 PageRank (PR)
The PageRank algorithm (Brin and Page Reference Brin and Page1998) ranks the nodes of a graph according to their structural importance. At each iteration, PageRank is computed according to (6), where t represents the iteration number, α is the so called dampening factor (typically 0.85), In(vk ) is the number of edges to node vk and Out(vi ) is the number of edges from vi .
\begin{equation}
PR(v_k; t + 1) = (1 - \alpha ) + \alpha \sum \limits _{v_l \in In(v_k)} \frac{PR(v_l; t)}{Out(v_l)}
\end{equation}
Traditionally, the initial weights are uniformly distributed across all the nodes in the graph. But these weights can be biased according to the desired purpose. For instance, in the personalised PageRank algorithm, also used for WSD (Agirre and Soroa Reference Agirre and Soroa2009), PageRank is run in a graph representation of PWN and, in the first iteration of the algorithm, only the synsets that include context words have an assigned weight. After several iterations, synsets are ranked according to their relevance to the context.
As ontologising is a kind of WSD, the algorithm we propose also personalises PageRank for selecting the most adequate synsets for the arguments a and b. There are two main differences though: (i) our context consists of a and b only; (ii) instead of a network of synsets, we have N, a network of terms. Therefore, for ontologising a tb-triple, we PageRank the terms in N, with initial weights of 0.5 in the nodes of a and b (our ‘context’) and 0 in the rest. This is how the final rank is biased towards a and b. Each synset in W is then scored with the average PageRank (
$\overline{PR}$
) of the terms it includes
\begin{equation}
\overline{PR}(A_i) = \frac{\sum \nolimits _{k=1}^{|A_i|} PR(a_{ik})}{1 + log_2(|A_i|)}
\end{equation}
Given the initial weights, PR(aik
) will be the relevance of the term aik
when in the presence of a context consisting precisely of a and b and nothing else. Finally, a pair of synsets (Ai, Bj
) is selected, such that Ai
and Bj
maximise
$\overline{PR}(A_i)$
and
$\overline{PR}(B_j)$
respectively.
4.5.0.5 Example:
For using this algorithm in the illustrative example, first we should once again recall that, besides those presented, N contains more tb-triples of different types. So, PageRank would be run for 30 iterations in the full N, with weights of 0.5 in the two ‘context’nodes, which are those corresponding to the words ticket_office and theatre, and of 0 in the remaining nodes. Suppose the following PR values:
-
•
$PR({\it box\_office}) = 4.6 \times 10^{-5}$
-
•
$PR({\it ticket\_office}) = 6.1 \times 10^{-6}$
-
•
$PR({\it ticket\_booth}) = 0$
-
• PR(theater) = 7.3 × 10− 5
-
• PR(theatre) = 1.3 × 10− 4
-
• PR(house) = 1.2 × 10− 4
After that,
$\overline{PR}$
can be computed for each of the synsets in A and B. For A
1 and B
1, it would be:
\begin{eqnarray*}
\overline{PR}(A_1) &=& \frac{4.6 \times 10^{-5} + 6.1 \times 10^{-6}}{1 + log_2(3)} = 2.084 \times 10^{-5}\\
\overline{PR}(B_1) &=& \frac{7.3 \times 10^{-5} + 1.3 \times 10^{-4} + 1.2 \times 10^{-4}}{1 + log_2(3)} = 1.292 \times 10^{-4}
\end{eqnarray*}
If, for the remaining synsets we have the following
$\overline{PR}$
:
-
•
$\overline{PR}(B_2) = 9.9 \times 10^{-5}$
-
•
$\overline{PR}(B_3) = 1.06 \times 10^{-4}$
This would result in the sb-triple (A
1 part-of B
1) because, for each target argument, A
1 and B
1 are the candidate synsets that maximise
$\overline{PR}$
.
5 Experimentation and results
In this section, we report on experiments with two different resources, where the algorithms presented before were used to ontologise tb-triples into available wordnets for English and for Portuguese. As we used tb-triples obtained from the same wordnets, the evaluation of the algorithms is straightforward, as it may rely on confirming if the resulting sb-triples originally existed in those wordnets. Correct triples are those with both arguments mapped to the same synsets as in the original wordnet. For Portuguese, we made an additional experiment where, instead of using the tb-triples of the wordnet, we exploited a lexical network extracted automatically from text. In contrast to the other situations, which are closer to a scenario of enriching an existing wordnet with new relations, this latter situation is closer to the scenario of adding connections between synsets, in the automatic creation of a wordnet from a set of synsets and tb-triples.
In addition, to the algorithms described in Section 4, we combine RP and AC (RP+AC) in a new algorithm where, when RP’s θ < 0.5, ontologising is performed with AC. This was made because, as the lexical network N becomes smaller, RP’s score drops drastically and it is often below the set threshold, θ = 0.5. Moreover, in order to enable a rough comparison with an alternative approach for this task, we implemented an adapted version of Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006) anchor approach. Still, for reasons further discussed regarding its different nature, the latter results are shown separately.
5.1 Experimental setup
All the experiments exploited a term-based lexical network N, obtained from the target wordnet, except for the second and third parts of experiment 2, where an external network, extracted automatically from text, was used. Regarding evaluation, we created an ontologising GS and defined the evaluation measures. In the following, we explain the previous points.
5.1.1 Wordnet as a lexical network
Given a wordnet W, N is established by all the tb-triples obtained from each of its sb-triples, done in a similar fashion to what Table 2 illustrates.
5.1.2 Gold standard for relation R
All the tb-triples of type R, in N, are first collected and shuffled. For each tb-triple tt = (a R b), the candidate synsets for its arguments are selected—we recall that the candidates for term a, A⊂W, include all the synsets with a sense of a, Ai ∈ A → a ∈ Ai and the same for b, B⊂W, Bj ∈ B → b ∈ Bj . Each pair of candidates (Ai, Bj ) is then marked as correct or incorrect, according to the existence of st = (Ai R Bj ) in W. This procedure is illustrated in Table 1 for two tb-triples.
The GS consists of all tb-triples with at least two candidates for one of the arguments a or b and their correct attachment points (synsets) in the wordnet. In other words, the GS only contains tb-triples where, at least, one of the words had more than one sense in the wordnet. Without this restriction, results would be better, but the comparison of the algorithms would also be noisier. Only in the third experiment for Portuguese the GS contained also tb-triples with just one attachment point for each argument.
5.1.3 Evaluation measures
In each experiment, the arguments of the reference tb-triples were ontologised in the candidate synsets, resulting in a sb-triple. Given the original wordnet, we can have hints on the performance of the algorithms by comparing the set of resulting sb-triples (Tr ) with the original ones (To ) and computing precision (P), recall (R) and F1-score (F1):
\begin{equation*}
P = \frac{|Correct(T_r)|}{|T_r|} \quad R = \frac{|Correct(T_r)|}{|T_o|} \quad F_1 = 2 \times \frac{P \times R}{P + R}
\end{equation*}
5.2 Experiment 1: ontologising in WordNet
In this experiment, tb-triples were ontologised into the synsets of PWN 3.0. For this purpose, we followed the procedure in Section 5.1 to create GSs for hypernymy (just between nouns) and part-of (part-meronymy) relations, each with the first 400 tb-triples entries. On average, there were 2.03 and 1.775 candidate synsets for each tb-triple argument, respectively in the hypernymy and in the part-of GS. For illustrative purposes, Table 3 shows 10 tb-triples of each GS.
First 10 tb-triples of the hypernymy and part-of gold standards for PWN 3.0
First, in order to validate the algorithms, we used the full lexical network N, which contains about 1.2 M tb-triples, including about 257 k of hypernymy triples between nouns and 38 k part-meronymy tb-triples, as well as all other relation types covered by PWN. Then, to simulate more realistic scenarios, we gradually decreased the size of N, by removing tb-triples randomly. For instance, when a wordnet is in the process of creation and it is not as complete as PWN, it is expected to provide a smaller N. Or when the wordnet contains just synsets and no relations, N can be established by a set of tb-triples, extracted automatically from text, possibly the same that are being ontologised to the available synsets.
Tables 4 and 5 show the results of ontologising the hypernymy and the part-of tb-triples respectively. Besides the full N, they have results when using parts of N. For instance, a half of N (1/2) is a set of 600 k tb-triples, randomly selected from the original 1.2 M. Out of curiosity, when using 1/32 of N, respectively 64.5% and 65.6% of the term arguments of the tb-triples in the hypernymy and part-of GS were still present. The proportion of GS tb-triples with both arguments in N was lower, more precisely 40.3% and 45.5% also for the hypernymy and part-of. The first lines of the Tables 4 and 5 are the results of three baselines, respectively:
-
• Random: for each tb-triple, a random pair of candidate synsets, respectively from A and B, is selected for mapping the arguments a and b. The presented numbers are averages of 10 runs.
-
• All-to-all: maps the arguments of each tb-triple to all of its candidate synsets, which results is all possible (Ai, Bj ) combinations.
-
• Sense #1: for each tb-triple, a pair of candidate synsets,
$(A_1, B_1) : A_1~\in ~A~\wedge ~B_1 \in ~B$
, is selected, where A
1 and B
1 contain, respectively, the most frequent senses of a and b, according to PWN. If the tb-triples had been extracted from raw text, this would probably be a higher-bound baseline. However, in this scenario, it is not very relevant, because the tb-triples were obtained randomly, from PWN itself, thus making the probability of selecting the most frequent sense the same as selecting any other sense.
Ontologising 400 hypernymy tb-triples in WordNet 3.0
Ontologising 400 part-of tb-triples in WordNet 3.0
With the full N, the high results of RP, AC, RP+AC, NT and MD confirm that, in theory, these algorithms make sense and suggest that they can be used for wordnet creation or enrichment. As expected, when N becomes smaller, performance decreases. But it decreases more for some of the algorithms than for others. This is also observed in Figure 2, which shows the evolution of F1 with all the algorithms for hypernymy and part-of, as N is reduced. For comparison, it also shows the baselines random (RAND), all-to-all (A2A) and most frequent sense (S#1).
F1 when ontologising hypernymy (top) and part-of (bottom), as the lexical network N is reduced.
In all situations, all algorithms perform better than the random baseline, except for RP, when using 1/32 of the network (37.5 k tb-triples). In fact, the performance of RP decreases drastically for smaller networks, which is caused by the threshold θ = 0.5. As RP exploits only relations of the target type with a fixed argument, the size reduction leads to lower scores. On the performance of PR, when using the full N, the results are far from perfect but, as N becomes smaller, the loss of performance is less pronounced (especially for part-of) than for the other algorithms. This happens in such a way that, when using 1/32 of the original N, PR performs close to the best. This is related with the structural noise of the great amount of connections, together with the ambiguity in the full N. As PR exploits random walks, it considers the structure of the whole graph and it is more tolerant to contextual gaps.
RP+AC and NT perform especially well. As the network becomes smaller, they strengthen their position as the best algorithms, at least until 1/16 of the network. With 1/32 it is less clear, due to the aforementioned behaviour of PR.
A brief remark on the time cost of each algorithmFootnote 4 : RP and NT are the fastest algorithms and, using the full N, concluded both the experiments consecutively in less than 40 s each. RP+AC took less than 1.5 minutes and AC about 6 minutes. When using a cache for distances, MD took about 8.5 minutes. PR is the slowest algorithm and took about 6 h to conclude these experiments, one after each other. When using just 1/32 of N, RP and NT took less than 10 s to conclude both experiments consecutively. All others took in less than 40 s for the same task, except PR, which took 17 min.
The performance of our algorithms cannot be directly compared to those reported in Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006) for part-of and causation, using either the anchor and the clustering approach (e.g. best F1=53% and 36.5% respectively) because those were obtained in a different scenario, using different datasets and their evaluation was made against a handcrafted GS. Also, those algorithms exploit the hypernymy relations already in wordnet. They are thus suitable for enriching wordnets, but less flexible than the algorithms we propose. In fact, since they do not exploit an external lexical network exclusively, they are not applicable when there are no pre-existing hypernymy relations between synsets. Furthermore, the clustering approach may result in the attachment of term arguments to their generalisations (hypernyms) which, though acceptable, could be too general to be useful. This behaviour is not our goal and simply cannot be evaluated with our GS.
On the other hand, we could simulate the anchor approach with a few adaptations, namely: (i) for each tb-triple, the attachment points are only those with the highest overall sense score (see equation 1)—multiple attachment points for a term are only selected in the case of ties; (ii) when a term is not covered, a new synset is created with only that term (does not apply for the PWN experiences); (iii) modifiers in multiword expressions are not removed—looking at Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006) example: expert analysis and Sandag study instead of just analysis and writing.
Table 6 shows the results obtained with our implementation of the anchor approach (hereafter AN′). They were separated from the full results because, even with the adaptations, the comparison would be unfair to all. And we should point out an additional difference, this time in the evaluation scenario: since AN′ does not exploit just a lexical network, in order to simulate the reduction of the resource, we used not only the same reduced lexical network as in the other algorithms, but we also reduced the hypernymy relations in PWN, in the same proportion.
Ontologising 400 hypernymy and part-of tb-triples in WordNet 3.0, using the adapted anchor approach
Apart from the aforementioned differences, the results of the anchor approach with the full PWN are of the same order of magnitude as the best results of the proposed algorithms. On the other hand, when the resource is reduced, it performs a few points below the best proposed algorithms. A remark should be given on the recall of AN′, which clearly outperforms the proposed algorithms. This is the result of ties, which occur more frequently for smaller lexical networks and fewer hypernymy relations, thus leading to multiple attachment points and lower precision.
5.3 Experiment 2: ontologising in OpenWordNet-PT
In the second experiment, instead of PWN, we used OpenWordNet-PT (OpenWN-PT) (de Paiva, Rademaker and de Melo Reference de Paiva, Rademaker and Melo2012), a Portuguese wordnet with synsets projected from the Universal WordNet (de Melo and Weikum Reference de Melo and Weikum2009) for Portuguese, and relations inherited from PWN. OpenWN-PT is much smaller than PWN 3.0 and also less complete. It contains about 48 k lexical items, organised in 39 k synsets, connected by 83 k sb-triples. For this experiment, we also created a GS for hypernymy and another for the part-of relations of OpenWN-PT, each with 400 tb-triples, also produced as described in Section 5.1. On average, there were 2.28 and 1.85 candidate synsets for each tb-triple argument, respectively in the hypernymy and in the part-of GS. For illustrative purposes, Table 7 shows 10 tb-triples of each GS.
First 10 tb-triples of the hypernymy and part-of gold standards for OpenWN-PT
This experiment is divided in three parts. First, similarly to what we did for PWN, the algorithms were validated by ontologising the tb-triples of the GS in the OpenWN-PT synsets, using the full lexical network extracted from the same resource. Second, towards a more realistic scenario, we did the same but using an external lexical network instead, automatically extracted from text. In the third part of this experiment, we created a new GS, where the constraint of having more than one candidate synset for the tb-triple arguments was removed. The last conditions are those expected in a scenario of relationless thesaurus upgrading to a wordnet—synsets are available, but sb-triples are not; tb-triples, some between polysemous and others between monosemous words are extracted from text and then ontologised.
We should also add that, this time, we only used the random candidate and the all-to-all baselines, because OpenWN-PT does not contain information on sense numbers, specifically for Portuguese.
5.3.1 Part 1: validation with its own lexical network
The lexical network N obtained from OpenWN-PT contains about 160 k tb-triples, in contrast to the 1.2 M in PWN’s lexical network. Similarly to what we did to PWN, we validated the ontologising algorithms for this resource, using the full N to ontologise the tb-triples from the GS, into OpenWN-PT’s synsets. Table 8 shows the obtained results.
Ontologising 400 hypernymy and 400 part-of tb-triples in OpenWN-PT, using its own lexical network
Though not as expressive as those for PWN, these results also show that the algorithms clearly improve the baselines. Performance is lower mostly due to the lack of exhaustiveness of OpenWN-PT. Also, although the relations of this resource are inherited from PWN, they were not hand-checked for correctness and there are some inconsistencies. In this experiment, the MD algorithm was clearly the best for hypernymy (≈ 81% F1), while RP and RP+AC had the best performance for part-of (≈ 91% F1).
Similarly to what we have done with PWN, we also used AN′ for this task. Its results, reported in Table 9 confirm that this approach leads to good results. For hypernymy, it outperforms MD’s F1 by 4% points, while for part-of it is 2.5% points below RP+AC and RP’s F1. We recall that AN′ exploits not only the extracted tb-triples, but also the hypernymy hierarchy between the wordnet synsets, which makes it applicable for enriching the wordnet but less suitable to populate it from scratch. Therefore, since the remaining scenarios using OpenWN-PT aimed at testing the performance when upgrading a relationless thesaurus to a wordnet, it would not make sense to apply AN′, because it uses the hypernymy sb-triples already in the wordnet. In those scenarios, only the wordnet synsets and an existing external lexical network for Portuguese, not obtained from the wordnet, were exploited.
Ontologising 400 hypernymy and 400 part-of tb-triples of OpenWN-PT with the adapted anchor approach, using OpenWN-PT’s lexical network and its hypernymy hierarchy
5.3.2 Part 2: using an external lexical network
After validation, instead of gradually reducing the size of N, we used PAPEL (Gonçalo Oliveira et al. Reference Gonçalo Oliveira, Gomes, Santos and Seco2008), a lexical network for Portuguese. PAPEL was created automatically, based on the regularities in the definitions of a Portuguese proprietary dictionary. This lead to the discovery of systematic patterns, manually compiled in several grammars, and used for the extraction of a broad range of semantic relations between words in the definitions and their definienda. The current version of PAPEL, 3.5Footnote 5 , contains about 102 k lexical items, connected by 191 k tb-triples that cover a broad range of relation types, whose correction rate is dependent on the type, including synonymy (98%–100%), hypernymy (91–94% correct), causation (90–95%), purpose (69%), property-of (72–78%) and several types of meronymy, such as member-of (93–96%) and part-of (not evaluated).
This situation becomes very close to a real scenario—imagine that a large amount of knowledge is automatically extracted from some independent source (e.g. dictionary) and then needs to be ontologised in available synsets. If the synsets are part of a thesaurus that does not contain sb-triples, and if the synsets do not contain glosses, the whole lexical network is an important source of additional context. After this procedure, the thesaurus becomes a wordnet. Table 10 shows the results of ontologising OpenWN-PT tb-triples in OpenWN-PT synsets, but using PAPEL as the lexical network N Footnote 6 . This means that the sb-triples of OpenWN-PT are ignored while ontologising, and used just for evaluation purposes.
Ontologising 400 hypernymy and 400 part-of tb-triples in OpenWN-PT, using PAPEL as an external lexical network
In this situation, while the best algorithms (RP+AC and NT) outperform the random baseline by 9% and 8% points, others perform below it. Still, even for the best algorithms, F1 scores are lower than 50%. The poorer performance shown here confirms that this is a challenging task, especially in this scenario where, instead of a uniform removal of tb-triples, some parts of N might be quite complete while others are possibly inexistent. The different coverages of PAPEL and OpenWN-PT might also contribute to this performance. PAPEL only covers 48% of the lexical items in OpenWN-PT. But while this should contribute to the lower performance, it does not mean that the selection of the attachment points is completely random for the uncovered terms because, we recall, most algorithms also use available tb-triples involving other words in the candidate synsets and may thus tolerate some lexical gaps. Some of the terms in OpenWN-PT and not in PAPEL are named entities, commonly not found in a wordnet. Another aspect is that OpenWN-PT is developed by a Brazilian team while, in its creation, PAPEL exploited an European Portuguese dictionary.
5.3.3 Part 3: including single candidate tb-triples
We recall that our previous GSs had only tb-triples with more than one candidate sb-triple in the wordnet. This makes comparison more clear, but penalises performance. In this part of the experiment, we used the same settings as previously, using PAPEL, but different GSs, created in a similar fashion, but this time from completely random tb-triple entries, as the constraint of having more than one candidate synset was removed. Therefore, the new GSs also contained 400 entries with hypernymy tb-triples and 400 entries with part-of tb-triples, but respectively only 265 and 191 had more than one candidate sb-triple. For part-of, this number is slightly less than a half of the tb-triples in the GS. On average, also in the hypernymy and part-of GSs, there were 1.92 and 1.60 candidates respectively for each term argument, which is clearly less than in the GS used in the previous experiments. The results of this experiment are shown in Table 11.
Ontologising 400 hypernymy and 400 part-of tb-triples in OpenWN-PT, using PAPEL as an external lexical network, against a GS that includes single candidate tb-triples
Under these conditions, as expected, all the obtained results are considerably higher than in the previous experiments. Furthermore, differences in the performance of each algorithm are also less pronounced and comparison is more difficult. This applies also for the baselines, which are now harder to beat because, for tb-triples with a single candidate, the correct answer is always selected. In fact, some of the proposed algorithms performed below the baselines, namely RP and PR for hypernymy and MD for part-of. But these numbers are the closest we can get to a real scenario, using OpenWN-PT, as it is.
For hypernymy, the best algorithms in terms of F1 were RP+AC and MD (≈ 57%), which improve the random baseline by about 3.5% points. RP+AC has the highest precision (≈ 58%).
For part-of, the global performance is higher than for hypernymy, because its GS contains even more single candidate tb-triples. There, all algorithms but MD outperform the random baseline, and NT has the best F1 (≈ 72.5%). RP has the best precision (≈ 73%).
These results confirm that the best algorithm depends on the relation type we are ontologising. Moreover, they suggest that, especially for hypernymy, we are still far from obtaining a highly accurate resource. Yet, some of the problems are caused either by lack of information in PAPEL, content issues of OpenWN-PT, or both. As far as we know, the creators of OpenWN-PT are currently dealing with the following issues:
-
• Synsets with terms in need of some kind of normalisation/correction, such as:
-
– lemmatisation of plurals, occasionally in the same synsets as their singulars, as in canções, mapas or navios (songs, maps, ships)
-
– lack of spaces/underscores for separating some multiword expressions, as in estaremembulição, migalhadepão or carnedeporco (should be estar_em_ebulição, migalha_de_pão and carne_de_porco)
-
– typographic errors, as in fèbre or fehadura (should be febre and fechadura).
-
-
• Erroneous synsets, possibly due to translation issues, such as:
-
– {Inglaterra, Esportistas da Inglaterra} (England, English sportsmen), aligned to the ‘England’ PWN synset, as a division of the UK
-
– {Estados Unidos da América, América, EUA, Continente Americano?. . .} (United States of America, America, USA, American Continent?, . . .)
-
– {Líbia, Grande Jamahiriya Socialista Popular Árabe da Líbia, línguas da Líbia?} (Libya, Socialist People’s Libyan Arab Jamahiriya, languages of Libya?)
-
– {perna, braço, mão} (leg, arm, hand), aligned to the PWN synset {mitt, hand, manus, paw}
-
We are thus waiting for newer versions of this or similar resources to generate a better GS and make stronger conclusions on the real performance of these algorithms for Portuguese. This does not invalidate the results of our previous experiments regarding the comparison of the ontologising algorithms and the confirmation that they can be used both for enriching and for creating wordnets. To have a better insight on the current issues, in the next section, we report on the errors found and their main causes.
5.3.4 Error analysis
While it is not possible to have a more reliable and freely available wordnet than OpenWN-PT, we made a quick error analysis for the last experiment. For each relation type, we selected a random sample with 15% of the incorrectly produced sb-triples, using RP+AC, one of the best algorithms according to our experiments with PWN. We analysed the cause of these errors and divided them into types.
With this exercise, we noticed that not all errors were caused by limitations of the algorithm. Some were due to a problematic translation between the PWN and its OpenWN-PT equivalent—for instance, the translation of the English word font (as a synonym of baptistery) to etiqueta_de_tipo_de_letra (font_type_tag), or the inclusion of the Portuguese word alguém (someone) in the synset representing the first letter of the alphabet {A, a}. The other sb-triples were, in fact, acceptable, but were not in the GS. Since our GSs were created directly from the wordnet contents, without any kind of inference, they did not contain sb-triples that would be correct if transitivity rules were applied—for instance, {artifact, artefact} was selected as a hypernym of {adornment, design, figure} but, in PWN, it is a hypernym of {decoration, ornament, ornamentation} which is the hypernym of {adornment, design, figure}. In a similar fashion, {Idaho} was selected as a part of {America} (the continent), but in PWN it is first a part of {USA}, which is part of {North America}, a part of {America}.
Tables 12 and 13 show the classes of error types, their proportion and an example for each class, respectively for hypernymy and part-of. For hypernymy, about 73% of the errors were caused by limitations of the algorithm, which was not powerful enough to tolerate the lack of knowledge in PAPEL (≈ 20%), the lack of knowledge both in PAPEL and OpenWN-PT (≈ 35%), or the presence of the same term in hypernym-hyponym pairs (≈ 18%). As an example of the first, PAPEL does not cover the terms nova_avaliação nor reavaliação (revaluation) and so does not contain a connection between these terms and any other term in the candidate synsets. As an example of the second, there is not enough information in OpenWN-PT nor PAPEL to make a distinction between the candidate synsets {assinatura} (signature) and {subscrição, assinatura} (subscription). The former ended up connected with {aliança, anel} (wedding_ring), but it is the second, {consenso, aliança, acordo . . .} (agreement), of which it is the hyponym. The third situation happens when hypernym-hyponym pairs share at least one term, which makes it difficult to select the correct position of the synsets in the hierarchy, especially for those algorithms that do not consider the relation type (e.g. AC).
Error analysis for ontologising hypernymy tb-triples of OpenWN-PT in the raw gold standard with its synsets, using PAPEL as an external lexical network
Error analysis for ontologising part-of tb-triples of OpenWN-PT in the raw gold standard with its synsets, using PAPEL as an external lexical network
For part-of, about 71% of the errors were caused by limitations of the algorithm, namely the lack of knowledge in PAPEL (≈ 22%), lack of knowledge in both resources (≈ 32%) and the presence of the same term in two candidate synsets (≈ 17%). As an example of the first, PAPEL does not contain enough tb-triples to support the selection of the sb-triple {parada, etapa} (stage, leg) part-of {jornada , viagem} (journey, journeying) and selected {jornada, dia} (day, daylight, daytime) instead of the second synset. As an example of the second error class, PAPEL does not have enough information on the ‘electrical device’ sense of the word ‘antenna’, nor has OpenWN-PT got a synonym for its ‘head appendage’ sense. This resulted in the sb-triple {antena} (feeler, antenna) part-of {receptor, radioreceptor, recetor} (receiver, receiving system).
6 Concluding remarks
We have presented several experiments on ontologising tb-triples in wordnets of English and Portuguese. Those have confirmed that, in theory, ontologising can be performed by exploiting an external lexical network, structured in terms and not word senses. This has some advantages over approaches that need existing sb-triples in the wordnet (e.g. hypernymy relations in Pennacchiotti and Pantel’s (Reference Pennacchiotti and Pantel2006)), because the lexical network may be obtained not only from the target wordnet, but also from information extracted automatically or in any other external resource. With this flexibility, this approach may be followed not only in the enrichment, but also in the creation of new wordnets.
In more realistic scenarios, performance dropped but it still outperformed our baselines. Experimental results reveal that the revisited algorithms might not be enough for creating a highly accurate resource, especially when there is some noise in the wordnet. But they should be considered, especially when there is not enough data and/or context. In fact, the AC and RP+AC algorithms were used in the automatic creation of Onto.PT (Gonçalo Oliveira and Gomes Reference Gonçalo Oliveira and Gomes2014), a large Portuguese wordnet. More precisely, they were used to attach tb-triples of different relation types, extracted from text, to the synsets of Portuguese thesauri and wordnets. For English, we also concluded that, when the lexical network is very incomplete, and possibly noisy, PageRank is the most tolerant algorithm.
In addition to the discussed limitation regarding the lack of knowledge in the lexical network, we are aware of other limitations in our work. The first assumed limitation is that synsets are identified by their words. It is thus impossible to handle two senses of the same word separately in the minority of situations where the corresponding synsets contain exactly the same words. Another limitation occurs when a polysemous word is in a single-term synset. As there is not enough context on the single-term synset, it is difficult to differentiate it from a larger one with the same word. We should also comment on situations when a word has more than one suitable sense, either due to the fine granularity of the sense inventory or, for instance, because it is assumedly vague. The proposed algorithms account for the latter situations because, when they lead to ties, the arguments can be mapped to all the tied synsets. Although some algorithms are more prone to lead to ties (e.g. RP) than others (e.g. PR), it should still be better to select only one of the suitable meanings than none.
We believe that the proposed algorithms have potential and can be used in the automatic creation or enrichment of other wordnets, besides Onto.PT. Yet, more experiments should be done, in order to ontologise other relation types besides hypernymy and part-of. Especially for English, it would also be interesting to analyse the behaviour of the algorithms using external lexical networks, possibly with more relation types than the limited set one can find in PWN. In the algorithms that exploit all available relations and not just the type that is being ontologised (e.g. AC, MD, PR), it would be important to analyse the impact of using different relation types or considering the relation direction. The more available information (relations), the better these algorithms should perform and this is also what we have seen in the previous experiments for Portuguese. Still, if these relations are extracted automatically or if they are not very well semantically-defined, they can be an additional source of noise and result in lower performance. A final line of future work would involve using the scores of the algorithms as confidence measures for each sb-triple. This would provide useful hints on reliability, which would enable the selection of sb-triples to use in other tasks, according to their confidence.
