2 WRPA method outline

The WRPA method is divided into three blocks as shown in Figure 1: learning WRPA components, validating the WRPA method and components, and applying the method in order to obtain the WRPA corpus. The first block, in turn, is divided into three phases: acquiring the candidate paraphrase patterns, and learning the relation and paraphrase classifiers. In what follows, we present an overview of the whole process, organized according to the steps in Figure 1.

Fig. 1.

WRPA method outline.

1A in Figure 1. We first learn a set of candidate paraphrase patterns expressing a concrete relation. Acquiring paraphrases implies ensuring sameness of meaning. In this sense, the distributional hypothesis (Harris Reference Harris1954), which states that words occurring in the same contexts tend to have similar meanings, makes up the basis of the methodology that many authors apply (1 in Figure 2). This methodology allows for the acquisition of a special type of paraphrases: those where a formal mapping is not necessarily observed.

Fig. 2.

Relational paraphrasing.

We apply this hypothesis to relational paraphrasing (2 in Figure 2). Relational paraphrases are those paraphrases in which the same type of relation is expressed in both members of the paraphrase pair. The relations we deal with are directional and binary between two entities: the source and the target (i). An example of such a relation is authorship, where the source corresponds to an author and the target corresponds to a work (ii). source and target are classes standing for sets of instances.

1. (i) Relation ⊂ source × target

   source${\xrightarrow {Relation}}$target

2. (ii) Authorship ⊂ author × work

   author$\xrightarrow {Authorship}$work

Pairs of source and target entities in a concrete relation stand for the ‘context’ (using distributional-hypothesis terms) of our paraphrase candidates. In concrete, our approach is based on the hypotheses that the meaning of the text around the source and target entities will be similar throughout their different occurrences, and that this meaning will hold the relation addressed in some way.

One of the main problems facing paraphrase acquisition is having an appropriate data source. Collecting paraphrases requires the availability of a corpus where different wordings for the same meaning coexist. The web is the biggest source of redundancy in existence, but it is also unstructured and unrestricted. We have therefore chosen Wikipedia. Its structure allows for an effective automatization of the task and its encyclopaedic nature allows for the acquisition of reliable and consistent examples. Imagine, for example, that we look for the Wikipedia page for ‘Paul Auster’. Once the page corresponding to the novelist is located (sometimes after a disambiguation procedure), we are reasonably sure that this page contains information about the author, and that mentions of ‘Paul’ or ‘Auster’ are likely to refer to him. The adequacy of mentions is notably less reliable when accessing web pages.

WRPA obtains pairs of source and target entities taking advantage of Wikipedia-structured information: source entities are extracted from the titles of Wikipedia pages, and target entities from infoboxes and itemized information embedded in these pages. Then, WRPA uses these entities as X and Y anchor points, respectively, for the extraction of paraphrase candidate fragments in the free text of the body of Wikipedia articles (3 in Figure 2).

In more detail, the extraction of the source and target entities is performed within the distant learning paradigm for relation extraction, initially proposed by Mintz et al. (Reference Mintz, Bills, Snow and Jurafsky2009), which uses supervised learning but with supervision not provided by manual annotation but obtained from the occurrence of positive training instances in a knowledge source or reference corpus. As we do not bootstrap the process, the distributional hypothesis is only used for obtaining snippets of text expressing a relation between two entities and is not applied to obtain new entities. It is important to point out that, although WRPA applies relation extraction techniques, our objectives are different: while relation extraction systems are geared towards obtaining the semantic relation held by pairs of entities in a corpus, relational paraphrasing focuses on the wording used to express those relations. Our objective is to build paraphrase corpora for specific relations, not to extract as many as possible relations occurring in a text.

Finding a large number of good ‘contexts’ is essential in our approach, since the quality and number of the obtained paraphrases depends in great measure on these contexts. The use of various Wikipedia structured sources allows for the obtention of a large number of source and target entities. Furthermore, we guarantee the correctness of these entities by extracting them directly from structured and semantically labelled data.

Once the set of source and target entities has been obtained, they are used as X and Y anchor points in the body of Wikipedia articles, the sentences where they appear are extracted and the snippets within X and Y are further processed. As reducing paraphrase expressions to sequences of words has a rather limited expressive power, our method generalizes these word sequences into regular expression patterns including words, lemmas and PoS tags using an A* approach (Nilsson Reference Nilsson1982). Finally, a set of candidate paraphrase patterns for a concrete relation is obtained.

As reflected in Figure 2, our pairs of entities are seen from multiple perspectives in this paper. Thus, several terms are used to allude to them: when we refer to the distributional hypothesis underlying our method, we talk about context; when we refer to relational paraphrasing, we say source and target entities; when we refer to the technique applied within the Wikipedia framework, we use the term anchor point. 1B and 1C in Figure 1. Two classifiers are built: the relation classifier checks whether a paraphrase candidate is an instance of the relation addressed, while the paraphrase classifier establishes whether two candidates are actual paraphrases. 2 in Figure 1. We carried out a series of experiments in order to validate the performance of the WRPA method and components. They were applied to several relations in two languages. More specifically, our experiments are divided into two blocks: person experiments, which include the person–date_of_birth, –date_of_death and –place_of_birth relations in the English Wikipedia; and the authorship experiment, which focuses on the author–work relation in the Spanish Wikipedia. Moreover, we performed a further validation applying our method to 13 more person relations. Therefore, the method was applied to 17 relations in total.Footnote ⁵

Person relations are simple relations. Authorship is complex in the sense that it includes several professional or artistic activities like painter–painting, director–film or inventor–invention. It should be noted that these relations are not treated separately, but embedded in a single complex one: authorship.Footnote ⁶

The person relations and the authorship one were chosen because they are frequent in biographies, for its high presence is Wikipedia-structured information and for their diverse nature.

These experiments suggest that the WRPA method is independent of the language and the relation addressed, and that the corpus may be extended to any language and relation present in Wikipedia.

3 in Figure 1. Finally, the application of the WRPA method resulted in the obtention of the WRPA corpus, which consists of WRPA-authorship and WRPA-person.

3 Related work

The omnipresence and complexity of paraphrasing have given rise to a great variety of approaches to the automatic treatment of this phenomenon in NLP. Although this task is far from being solved, interesting and useful methods have been set out.Footnote ⁷ These methods can be grouped in three basic paraphrase tasks as follows:

• Paraphrase recognition. When faced with two snippets of text, deciding whether they are paraphrases or assigning a degree of paraphrasability to the pair.

• Paraphrase generation. When faced with a snippet of text, generating a (ranked) set of paraphrases of the snippet.

• Paraphrase extraction (or acquisition). When faced with a corpus, extracting from the corpus a (ranked) set of paraphrases.

It should be noted that these tasks are combined or embedded in larger systems on many occasions.

Moreover, some works are devoted to building paraphrase corpora, which are generally compiled using paraphrase extraction techniques. Existing paraphrase corpora are different in nature, which basically depends on the data source and method applied to acquire them.

In this section, we set out related work in paraphrase extraction (Section 3.1) and paraphrase corpora (Section 3.2), the fields in which the WRPA method and corpus are respectively embedded. Some works on corpus building in Section 3.2 can be linked to the corresponding paraphrase extraction systems in Section 3.1. As our final objective is building paraphrase corpora, more attention is paid to this field.

4 Learning WRPA components

WRPA performance is based on the learning and application of a set of patterns associated with each relation addressed. In this section, we set out the learning process (1 in Figure 1). It is performed in three steps, grouped in two sections: learning the set of candidate paraphrase patterns (Section 4.1), and learning the relation and paraphrase classifiers (Section 4.2).

4.1 Learning the set of candidate paraphrase patterns

Learning the set of candidate paraphrase patterns (1A in Figure 1) proceeds in four steps, reflected in Figure 3. First, the working corpus is set up (Section 4.1.1). Second, collections of anchor points are extracted (Section 4.1.2). Anchor points are then used for sentence selection in the text, and relevant snippets are subsequently extracted from these sentences (Section 4.1.3). Finally, candidate paraphrase patterns are obtained from these snippets applying additional processes (generalization and ngram modelling) when needed (Section 4.1).

Fig. 3.

The process of learning the set of candidate paraphrase patterns.

4.1.1 Working corpus set-up

Our corpus consists of the Spanish Wikipedia and the English Wikipedia. We downloaded the versions of February 2009 into an MySQL database from Wikipedia dumps. We used the JWPL software (Zesch, Müller, and Gurevych Reference Zesch, Müller and Gurevych2008)Footnote ³⁰ to access Wikipedia. Although the use of Wikipedia dumps prevents us from working on the latest version of Wikipedia, it allows us to perform the experiments on a static version, avoiding the problems arising from content instability.

WRPA extracts source–target pairs and, from them, our paraphrase candidates, taking advantage of the following Wikipedia components:

• • Source pages. Wikipedia pages describing source entities.

• • Redirection pages that point to source pages.

• • Infoboxes that are present in the source pages containing attributes corresponding to target entities.

• • Target sections. Itemized sections in the source pages containing target entities.

• • Target pages. Pages linked to source pages containing an itemized list of target entities.

• • Category-page links, which assign categories to articles, and the category graph.

For each relation addressed, the first step is to collect Wikipedia articles corresponding to the source (e.g. person and author). To do so, we first select Wikipedia categories that correspond to the source. This process can sometimes be done straightforwardly: when the source corresponds to an existing Wikipedia category (‘person’ in the English Wikipedia), this category, which we call top category, is selected. When such a correspondence does not exist (‘author’ in the Spanish Wikipedia), a manual inspection is needed in order to get top Wikipedia categories related to the source.Footnote ³¹ A set of top categories is selected in this case. From this top category or top category set, the category graph of Wikipedia is traversed top-down following the subcategory links. Once the set of relevant categories and subcategories for a source has been established, we collect the set of pages under them using the category-page links. These pages constitute the working corpus for each relation addressed. The working corpora are then cleaned and segmented into sentences.

The second step is to collect target-related infobox attributes (e.g. date_of_birth and work). Again, this task can sometimes be done straightforwardly and sometimes it requires further work. For the person experiments, we took advantage of the set of generic attributes and mappings to specific ones provided in the framework of the Slot-Filling Task in the TAC KBP contest.Footnote ³² Thus, we only had to select the generic attributes corresponding to our target entities. For the authorship experiment, we had to follow a less direct approach: we collected all the infoboxes with their attributes occurring in the working corpus and we manually selected the ones containing an attribute referring to works. Once the set of relevant infobox attributes was built, the pages in the working corpus containing any of these attributes were selected. They constituted what we call the learning corpus.

In our work, we talk about different partitions of the same corpus, namely Wikipedia: (1) the whole English Wikipedia and Spanish Wikipedia, our point of departure data sources; (2) the working corpus, which consists of the source pages; and (3) the learning corpus, which consists of the source pages containing a target entity in the structured information, i.e. infoboxes and/or target sections, and the target pages (see Section 4.1.2).

4.1.2 Anchor point extraction

In this step, the set of the source and target entities is extracted. These entities will be used as X and Y anchor points, respectively, in the candidate paraphrase extraction process.

As shown in Figure 4, we extract X anchor points from the titles of source pages; Y anchor points, in turn, are extracted from infoboxes, target sections and target pages. We sometimes extract a third anchor, Z, which consists of additional information complementing Y. The relation holds between X and Y; Z simply makes Y more reliable. Z, when present, is extracted together with Y. In our experiments, Z is only obtained for authorship and corresponds to the work date.

Fig. 4.

Anchor point extraction from Wikipedia.

X extraction is simpler than Y extraction. Once the source pages have been selected (Section 4.1.1), X extraction simply consists in looking for it in the title of each of these pages. Name variants for X, when they exist, are also extracted from redirection pages linked to the source page and/or from infobox attributes such as ‘alternate names’ or ‘nicknames’.

In the case of Y, a first set of anchor points is extracted from infoboxes. As information encoded in the attribute values shows variability (e.g. the variable format of the dates in Figure 5), we learned a set of grammars for extracting these values: date, date+place, place and work grammars in our experiments. Grammars were inferred using the ALERGIA system (Carrasco and Oncina Reference Carrasco, Oncina, Carrasco and Oncina1994), an efficient stochastic regular grammar inference engine that learns only with positive examples. Probabilities in ALERGIA-generated grammars were not taken into account.

Fig. 5.

Infobox attributes and their values.

In the case of person experiments, information extracted from infoboxes is enough for Y anchor point extraction, because, as can be seen in Table 1, (i) the English Wikipedia is extensive enough, (ii) there is a relatively high number of person pages with an infobox, (iii) many of the target attributes appear in person infoboxes and (iv) attributes are univalued, allowing for an easier and more accurate extraction. However, in the authorship experiment, the coverage of Ys in infoboxes is low, because (i) the Spanish Wikipedia is smaller, (ii) few author pages have an infobox, (iii) few infoboxes have a work attribute and (iv) this attribute is multivalued consisting of an unconstrained string with possible noise.

Table 1.

Counts for relevant Wikipedia pages for authorship and person experiments

In order to improve the recall of the target entities in the authorship experiment, we take advantage of, when they exist, the target sections and the links to target pages. Target sections present the difficulty involved in obtaining the section limits (heading and ending), a lack of homogeneity in the format and the presence of complementary material which results in noise. By way of illustration of the productivity of target sections and pages, Paul Auster's infobox lacks the work attribute. In contrast, 33 works (25 were correct) were extracted from the target section and 18 (all correct) were added from the target page. Table 2 contains some anchor points extracted by WRPA.

Table 2.

Anchor point examples

4.1.3 Sentence snippet selection

Sentence snippet selection consists of first gathering the sentences that contain the anchor points in the body of the articles, and second selecting relevant snippets within these sentences.

Sentences are extracted from the free text of the same pages where the corresponding pair of anchor points has been extracted from the structured information. Sentences extracted by our method present the following structure:

\begin{equation} string_1\; X\; string_2\; Y\; string_3\; Z\; string_4 \end{equation}

Here, X, Y and Z stand for our entities or anchor points. They may occur in any order (what is shown in the above formula is the canonical order). String_i stands for a snippet of text. Only the anchor Y is mandatory and it is looked for firstly. In case X occurs before or after Y, the string between these two anchors is string ₂. In this case, string ₁ is not used. If Z occurs, the string between Y and Z is string ₃. In this case, string ₄ is not used. In case X or Z are not present, the distinctions between string ₁ and string ₂, or string ₃ and string ₄, respectively, are no longer relevant. In these cases, the beginning and the end of the sentence are taken as limits. Table 3 shows frequency information for different orderings.

Table 3.

Sentence snippet counts for authorship. The underscore equals to a string of text. When no underscore appears, the text appears between the variables

In order to improve the recall, we look for all the variant forms in which X and Y can be expressed. In order to expand X, apart from using the variants obtained in the anchor point extraction step, we apply a person–name grammar (Arévalo, Civit, and Martí Reference Arévalo, Civit and Martí2004) to each entity. In order to expand Y, the grammars used for the extraction of infobox attribute values are also used for the generation of Y variants. One of WRPA's strengths for learning a large number of patterns is the use of anchor point variants instead of restricting the search to the original ones alone. By way of illustration, in the authorship experiment, the average number of variants for X is 5.2. Using only the initial anchor point instead of the whole set of variants obtains only 28% of the total of patterns.

Relevant sentence snippets are subsequently extracted. The criterion for selecting them depends on its frequency and the presence and order of X, Y and Z. In the following, we will concentrate on XY and XYZ snippets. We hypothesize that the most relevant information expressing the relation can be found between a source and a target in that order, as the former is likely to correspond to the subject of the sentence and the latter to the object, and the canonical sentence order is SVO. Also, this subset is common in our learning corpus, as can be seen in Table 3 for the authorship experiment. The set of snippets only containing Y is larger because subject (normally the X) elision is extremely frequent in Spanish. We decided not to work with this subset because the absence of X makes it much less reliable. Moreover, including YX patterns might result in a set of snippets different in nature from XY patterns; however, again, this would involve an increase of noise. We are leaving these issues for future work.

Table 4 shows examples of XY snippets extracted by our method. In the case of proper names, numbers and dates, we replaced specific words with their generic tags to reduce sparseness (e.g. < location > or < date >).

Table 4.

Sentence snippet examples

4.1.4 Pattern acquisition

Pattern acquisition consists of transforming sentence snippets into candidate paraphrase patterns. The process is more demanding in the case of complex relations such as authorship, due to the high variability of the sentence snippets and their low relative frequency. In simple relations such as date of birth, the sentence snippets are less variable and more frequent and, thus, the process is more straightforward.

For simple relations, the obtained snippets are simply transformed into regular expression patterns. For instance, the snippet in (f) in Table 4 is transformed into the regular expression X +was +born +in + < location > +on +(.+).*. The underlined group in the regular expression must be recognized by the grammar that corresponds to the target (in this example, the date grammar). This consistency check allows for the removal of many spurious candidates extracted by the pattern matching mechanism. For complex relations, the direct use of the snippets as patterns leads to severe overfitting and further work is therefore needed. We have developed two approaches: generalization and ngram modelling.

Generalization aims to extract patterns accounting for a larger number of texts from our initial sentence snippets. It implies going from a representation of the snippet as a sequence of words to its representation as a sequence of tokens with word forms, lemmas and PoS tags. Additionally, each token can be mandatory, skippable or omitted. The generalized pattern will be subsequently transformed into a regular expression formula.

In concrete, we first perform PoS tagging and named-entity recognition and classification on sentence snippets using the Freeling toolbox (Padró et al. Reference Padró, Collado, Reese, Lloberes and Castellón2010).Footnote ³³ Then, we represent each token in the snippet as a < word, lemma, PoS, matching condition > tuple as shown in Table 5. Conditions state which part of the token has to be matched (w for word, l for lemma, or p for PoS) and its mandatory, skippable (?) or omitted (-) nature. Before generalization, all tokens fulfil the w condition. Snippets longer than 10 tokens are removed.

Table 5.

Generalization pre-process corresponding to (a) in Table 4

Then, generalization is performed using an A* approach (Nilsson Reference Nilsson1982). The search is controlled by three parameters: N, minimum number of matches with other snippets; M, maximum number of states to be traversed; and B, maximum number of generalized patterns to be returned.Footnote ³⁴ The search proceeds until either of the following conditions is fulfilled: (i) a state matching at least N other snippets is reached or (ii) the number of traversed states reaches M. Once either of these stopping criteria is fulfilled, the B best scored patterns are returned. Note that the algorithm returns an optimum value only when the first halting criterion (N matches) holds, the second condition (M states) only prevents an expensive and unsuccessful search. In each state, snippets are represented as a sequence of <word, lemma, PoS, matching condition> tuples. The operators allow for moving from w to w?, from w? to l and so on. An example of the generalization process can be seen in Table 6.

Table 6.

Generalization process corresponding to Table 5

As the number of applicable operators in each state is huge, we perform a clustering of the candidates using an agglomerative approach with Levenstein distance as measure in order to reduce the search space. In this way, pattern generalization is carried out within each cluster and a set of generalized patterns is obtained from each cluster independently. Table 7 shows the contents of one of these clusters.

Table 7.

Cluster example

A final filtering sets the number of tokens fulfilling the w condition in a pattern from 1 to 4 (one of them being a common noun or a main verb) and the number of words fulfilling the p? condition from 0 to 4. The reason for establishing such restrictive constraints is that we are looking for patterns with lexical content. Not establishing these constraints would result in paraphrase candidates with no text between X and Y, or only functional words or punctuation marks between them, which are not relevant for our purpose of building paraphrase corpora. Examples of generalized patterns can be found in Table 8 (they are derived from the seventh example in Table 7).

Table 8.

Generalized pattern and ngram examples

The alternative approach to generalization is the use of ngram fragments between the involved anchor points. This can be considered to be an intermediate approach between generalization and a pure bag of words clustering, which is not appropriate for our purposes because word order is pertinent. In this approach, all word ngrams of any length are extracted from all the sentence snippets of the form XY[Z]. This set of ngrams is filtered to remove those not including any main verb or common noun. Moreover, a frequency threshold is used in order to discard infrequent ngrams. Finally, a regular expression is built for each remaining ngram, including the ngram within the anchor points and possibly a snippet of text (string_i) with a limited length (LL, in tokens) between both X and the ngram and between the ngram and Y (|string_i| ⩽ LL).

\begin{equation} X\; \{string_1\}\; ngram\; \{string_2\}\; Y \end{equation}

We also build patterns including a pair of ngrams between X and Y:

\begin{equation} X\; \{string_1\}\; ngram_1\; \{string_2\}\; ngram_2\; \{string_3\}\; Y \end{equation}

Examples of the so-called simple and double ngrams can be found in Table 8.

4.2 Learning the classifiers

The next steps in WRPA are learning a relation classifier, which classifies a specific pattern as an example or not of the relationship of interest (1B in Figure 1), and learning a paraphrase classifier, which decides, given a pair of patterns for a relation, whether or not they are paraphrases (1C in Figure 1). In both cases, we used a supervised approach for learning decision trees in the WEKA toolbox (Hall et al. Reference Hall, Frank, Holmes, Pfahringer, Reutemann and Witten2009).Footnote ³⁵ In the experiments reported in this paper, we built these two classifiers for the authorship relation only. The simpler nature of person experiments made this process unnecessary.

4.2.1 Learning the relation classifier

In order to build the classifier, we first created a learning data collection. With this aim, the set of candidate paraphrase patterns was applied to the body of the articles in the learning corpus and 977 instances that matched different patterns were randomly selected. They were then manually annotated using the CoCo annotation tool (España-Bonet et al. Reference España-Bonet, Vila, Rodríguez and Martí2009).Footnote ³⁶ The annotation consisted in deciding whether the pattern instances expressed an authorship relation or not. By way of illustration, the instance (a) in Table 9 is a positive example and (b) is a negative one.

Table 9.

Relationship annotation

The classifier was learned using a total of 5,231 features extracted from this set of instances. These features consisted of the most frequent nouns, verbs and their lemmas, the most frequent word ngrams (Section 4.1), and the most frequent subpatterns containing a verb or a noun. The relation classifier resulted in 32 rules corresponding to branches of the decision tree and 1,067 nodes. It has a depth of 87 and the average length of branches is 33.34. We finally applied this classifier to the whole set of candidate paraphrase patterns and selected the positive ones.

4.2.2 Learning the paraphrase classifier

In order to create the learning data set, we collected the pairwise cross product of all the patterns considered as positive by the relation classifier. We then applied them to the learning corpus and 8,746 instance pairs that matched different patterns were randomly selected for manual annotation using the CoCo interface. The manual annotation consisted of determining whether the pairs were paraphrases or not. Our point of departure is that while what we call general authorship expressions can be applied to any type of authorship (e.g. is the author of can be applied to painting, writing, inventing, etc.), concrete authorship expressions can only be applied to one or some types (e.g. wrote can only be applied to writing). In the light of these ideas, the basic criteria were to annotate as paraphrases those pairs where both members expressed general authorship ((a) in Table 10) and those pairs where one member expressed general authorship and the other expressed concrete authorship (b). In the latter case, the meaning of the general case includes the meaning of concrete one. The specific–specific authorship relations were annotated as paraphrases only when they belonged to the same type(s) (c). When they did not belong to the same type(s), they were mutually exclusive and thus annotated as non-paraphrases (d).

Table 10.

Paraphrase annotation. G(eneral), S(pecific). α and β are authorship types

The classifier was learned using a total of 234 features measuring different types of overlapping, including the most frequent words, word ngrams, lemma ngrams and tree edit distance information calculated with the EDITS suite (Kouylekov and Negri Reference Kouylekov and Negri2010).Footnote ³⁷ The paraphrase classifier resulted in 55 rules.

Although the classifiers are only built for authorship and for Spanish in this work, they can be built for other relations and languages as the features, i.e. the most frequent nouns, verbs, etc., are not fixed but extracted from the set of instances corresponding to the relation in question.

5 Validating the WRPA method and components

WRPA validation consists in evaluating the set of candidate paraphrase patterns and the two classifiers (2 in Figure 1). Candidate patterns are validated according to their ability to match a snippet of text ending in a correct Y. Following this approach, we applied the set of patterns to the body of the articles in the learning corpus: we looked for the occurrences of any variant of X in the sentences in the learning corpus (X is determined by the author in the title of the source page) and applied the pattern in order to obtain all possible Ys (corresponding to that X). We then computed their precision, recall and F1. The values for frequency, i.e. number of occurrences of each pattern, were also obtained.

In the case of precision, if the resulting Y is a variant or the original Y, the agreement is positive. For estimating recall, in the case of relations linked to univalued attributes, i.e. person experiment, we make the conservative assumption that all the pages contain at least a Y corresponding to each relation in the free text. Obviously, this is not always the case, because many pages lack these relations or some of them are simply not pertinent (e.g. a date of death is not adequate for living people). For each relation, recall is thus the ratio between the number of Ys recovered by the obtained patterns and the total number of pages in the learning corpus. In the case of multivalued attributes, i.e. the authorship experiment, it is not easy to hypothesize how many works occur in Wikipedia pages, so there are no clear ways to measure recall automatically. As an approximate alternative, we select the author pages containing links to their corresponding work pages and assume that all the works by an author can be obtained from these work pages. This resulted in 46 works per author on average. We then select from these works the ones appearing in the author page free text, which resulted in 13 works per author on average. We then compute the recall as the ratio between the number of Ys recovered by our patterns and the total number of Ys in the work pages occurring in the author pages. Thus, recall is only measured using author pages containing a link to a work page.

Table 11 presents the number of occurrences, precision, recall and F1 for the five most frequent patterns (individual and overall values) and the results for all the patterns in each person relation. As can be seen, the frequency is considerably higher in the first or first two patterns. Precision is very high in all cases, due to the relatively low variability of the patterns, and the fact that patterns have not undergone generalization or ngram processes. Date of birth resulted in a high recall, basically due to the high recall of the first and most frequent pattern. In the case of date of death, the recall is lower due to the conservative criteria used for defining it.

Table 11.

Person experiment results

The results for authorship are shown in Table 12. We set out the results of the generalization, ngram and double-ngram approaches individually (rows 1–7). After removing the patterns showing a precision under 0.5, we split the remaining patterns in each approach into three groups: those having a precision of 1, those having a precision of between 0.5 and 1, and those having a precision of 0.5, respectively. Results are shown in a cumulative way in Table 12. The third group is empty for some sources. The evaluation of several combinations between the top sets is also presented (rows 8–13).

Table 12.

Authorship experiment results. Pat.=number of patterns; Freq.=frequency; P=precision; R=recall

Patterns coming from generalization are more precise than those obtained from ngrams and double ngrams, though the latter show better recall. As patterns within each approach are sorted decreasingly by precision, increasing the number of patterns results in an improvement in recall at a cost of a drop in precision. The results of the different combinations, in turn, are significant since they demonstrate that the generalization and ngram approaches are complementary for pattern building, which allows for the acquisition of non-overlapping data. As an illustrative example, the overlapping between the works extracted by the pattern subsets shown in rows 2 and 4 is only 6%. The relatively low recall in all cases is due to the greater difficulty of the task and the constraints we have imposed, namely that they must contain at least a common noun or a main verb. Limiting ourselves to XY patterns also contributes to a drop in recall.

As our objective is to build paraphrase corpora, our main interest is to obtain precise and linguistically rich paraphrases rather than a high recall. Actually, we observed that some patterns with a low recall are, in some sense, more interesting from a linguistic point of view. Patterns such as X escribi Y (‘X wrote Y’) show a high recall and the recall is lower in cases such as X haba por fin publicado Y (‘X finally published Y’). However, the second pattern is linguistically richer and therefore of greater interest for the analysis of the linguistic mechanisms underlying paraphrasing.

Regarding the classifiers, they were validated using 10-fold cross-validation. The relation classifier achieved an accuracy of 85% and the paraphrase classifier an accuracy of 76%.

A direct comparison of the whole task of relational paraphrase corpus building to related approaches is not possible because no comparable corpora exist. However, we can perform a partial comparison of the subtask of learning the set of candidate paraphrase patterns to works on pattern acquisition.

One of the relations used in this paper to illustrate our approach, namely, date of birth, is also explored by Ravichandran and Hovy (Reference Ravichandran and Hovy2002). Ravichandran and Hovy (Reference Ravichandran and Hovy2002)’s patterns differ from ours in four main aspects:

• • They extract patterns from the web, while we extract them from Wikipedia.

• • They accept XY and YX patterns, while we limit ourselves to the former.

• • They extract patterns split into prefix, infix and suffix strings, while we limit ourselves to infixes.

• • We apply a consistency check (grammar application) of the obtained Ys to ensure grammaticality, while they apply a simple matching procedure.

In their paper, a list of the 10 most precise patterns for this relation is presented. Three of them are comparable to ours (they consist of XY infixes) and also appear in the top positions in our set: X was born on Y, X was born in Y, X was born Y. In Ravichandran and Hovy (Reference Ravichandran and Hovy2002), these patterns have a precision of 0.85, 0.6 and 0.59, respectively. In our work, these patterns have a precision of 1.0 in all three cases (Table 11 shows the first two cases). Our better results in precision are due to the structured and more limited nature of our corpus, and the consistency check performed. Although no recall information is provided by Ravichandran and Hovy (Reference Ravichandran and Hovy2002), we assume that our recall is lower, due to our less expressive patterns (only infixes and XY patterns are extracted) and our smaller corpus.

Regarding authorship, a comparison to the DIPRE system (Brin Reference Brin, Atzeni, Mendelzon and Mecca1999) can be made, although DIPRE and WRPA differ in two main aspects: first, DIPRE is applied to English and WRPA to Spanish; second, DIPRE is only applied to the author–title (books) relation and WRPA to authorship in general. DIPRE patterns are again more expressive (YX patterns are accepted, as well as prefixes and suffixes) than ours and come from a greater but less reliable source (24 million web pages). Brin (Reference Brin, Atzeni, Mendelzon and Mecca1999) does not provide values for precision and he performs a qualitative analysis of the results instead. DIPRE acquires 3,972 occurrences of books after convergence. Even though this result does not correspond to the whole authorship relation, it is comparable to our snippet counts in Table 3 (4,338 for XY and 1,866 for YX).

6 Applying the WRPA method to build the WRPA corpus

The process of the construction of the WRPA corpus (3 in Figure 1) consists of candidate paraphrase pattern set application and filtering with the two classifiers. Contrary to the validation process, here the application of the patterns was performed on the free text of the whole working corpus, rather than on the learning one.

In the building of the WRPA-person corpus, the whole set of patterns was applied to the working corpus to obtain the number of occurrences. The patterns were then organized in three groups according to the relation involved. All patterns within each group are considered to be paraphrases of each other in the sense that they all express the same type of relation. This massive grouping is possible due to the low variability of person patterns. The WRPA-person corpus consists of 355 patterns with 27,581 occurrences for date of birth, 433 patterns with 3,261 occurrences for date of death and 942 patterns with 15,208 occurrences for place of birth.

For the building of the WRPA-authorship corpus, we selected the group of patterns with a higher F1 (row 12 in Table 12) and, from them, we selected the patterns accepted by the authorship classifier. Then, we obtained all the occurrences of these patterns from the working corpus, cross-paired them, computed the Levenstein distance between all pairs, sorted them by distance and discarded the most distant half. We finally applied the paraphrase classifier and collected the instance pairs accepted as paraphrases. The WRPA-authorship corpus consists of 81,101 pairs. The size of the WRPA-person and WRPA-authorship corpora is not comparable. The reason is, as already stated, that the person corpus is not expanded to all possible pairs, but presented as a single massive grouping.

In order to check the quality of the final WRPA-person and WRPA-authorship corpora, subsets for each were manually revised. WRPA-person validation consisted in manually revising 25% of the cases in each subcorpus to determine whether they were an expression of the corresponding relation. Besides using this revision for evaluation, the rejected patterns were removed from the corpus. Table 13 shows the final volume of each WRPA-person subcorpus and its precision. Moreover, a further validation was performed: the WRPA method was applied to 13 more person relationsFootnote ³⁸ and the results were revised in the same way. The figures for these new relations can also be seen in Table 13. They constitute the WRPA-person-2 corpus.

Table 13.

WRPA-person and WRPA-person-2 validation. Pat.=number of patterns; P=precision

The precision for age shows a major decrease. Age, understood as a relation, is very unstable and it normally appears associated with concrete events in the life of a person in Wikipedia. This makes the value of the corresponding attribute in infoboxes highly variable. Due to its low precision, we decided not to include the age relation in the WRPA-person-2 corpus. The remaining relations show a high precision, only locations (place of birth, residence and origin) are slightly lower. A reason for that is that a same location may appear in a text expressing different relations. Also, locations sometimes do not appear in isolation, but combined in infoboxes (e.g. California, USA).

In the case of WRPA-authorship, the manual validation comprises two steps. The first step was previous to building the paraphrase pairs and consisted of a manual validation of 922 instances corresponding to different patterns. The objective was to reject those instances not expressing an authorship relation. The global precision obtained was 0.63. The second step was performed after building the paraphrase pairs and consisted in manually validating 1,000 of these pairs, stating whether they were paraphrases or not. The criterion used was the same as in Section 4.2.2. In this case, the precision obtained was 0.91.

7 WRPA corpus main features

The WRPA corpus consists of a collection of relational paraphrases extracted from Wikipedia.Footnote ³⁹ In what follows, WRPA corpus main features are set out. In concrete, the nature of paraphrases in WRPA is discussed and compared with those in other paraphrase corpora. Also, we analyse the quantity, variety and quality of WRPA paraphrases. Then, some actual uses and applications of WRPA are presented. This section finishes with an analysis of the most common errors found in the corpus.

Paraphrase nature. Paraphrase multifaceted nature prevents from the creation of general and comprehensive paraphrase corpora, that is, paraphrase corpora covering the phenomenon as a whole. Only corpora covering specific paraphrase facets, directly linked to the way paraphrases were obtained, may be created. In this sense, some paraphrase corpora rely on the formal similarity between the paraphrase members in order to establish paraphrasability. Part of the MSRP, for example, was built using string edit distance. Other corpora apply transformations to one member of the pair in order to obtain the other. The P4P corpus was built in this way. Despite following different routes, both string similarity and transformation techniques generally cover paraphrases with some kind of formal mapping. In contrast, other corpora do not necessarily show formal similarity between the pairs. This absence of formal mapping may be the result of different gathering techniques. In the case of the MSRVD corpus, paraphrases are the result of using videos as stimulus. WRPA, in turn, follows the line of works that rely on the distributional hypothesis. In this case, as paraphrasability is established through context, what may be found in between is not predictable. By way of illustration, together with paraphrases with a high level of formal similarity (6), our corpus also contains paraphrases that are completely different in form (7).

1. (6)

   1. a. author realizó su primer álbum en solitario, work

      author made his first solo album, work

   2. b. author publicó su primer álbum en solitario, work

      author released his first solo album, work

2. (7)

   1. a. author corresponden los títulos work

      the titles work correspond to author

   2. b. author es autor de los libros work

      author is author of the books work

These ideas have been demonstrated in Vila et al. (Reference Vila, Bertran, Martí and Rodríguez2013), which presents the results of annotating 1,000 pairs of the WRPA-authorship corpus, together with the 856 pairs in P4P and 3,900 pairs of MSRP, with our typology of paraphrases (Barrón-Cedeño et al. Reference Barrón-Cedeño, Vila, Martí and Rosso2013), comprising 24 paraphrase types.Footnote ⁴⁰ The same-polarity type, which includes cases of lexical-unit substitution, synonymy among them, is one of the most frequent types in the three corpora (25%–50% of the cases approximately). Addition–deletion is the other most frequent type in P4P and MSRP (12.91% and 25.94%, respectively); in contrast, the second most frequent type in WRPA is precisely semantic (16.22%), which includes those cases of semantic similarity where a formal mapping is not always possible.

The higher presence of semantic paraphrases in WRPA makes it a valuable resource in NLP, as this type of paraphrase presents considerable computational challenges and requires further study. Barrón-Cedeño et al. (Reference Barrón-Cedeño, Vila, Martí and Rosso2013) illustrate this in the field of automatic plagiarism detection: cases of plagiarism comprising semantic paraphrases are the most difficult to detect by plagiarism detection systems.

Next to the ideas that paraphrase multifaceted nature prevents from the creation of corpora covering the phenomenon as a whole and that only corpora covering specific paraphrase facets may be created, it is important to note that the presentation and organization required by these corpora can also be different: for example, the MSRP corpus is presented as a collection of sentence pairs; the WRPA corpus, in contrast, makes more adequate its presentation as a collection of patterns organized in sets/pairs for each relation.

Paraphrase quantity. The WRPA corpus currently covers 16 relations in two languages (English and Spanish). Moreover, it can be extended to any relation and language with sufficient coverage in Wikipedia by applying our acquisition method. Also, for each relation, a great number of paraphrases are provided. We consider that a total of 4,421 patterns for person and 81,101 pairs for authorship to be significant numbers. It is important to note that a direct numerical comparison between existing paraphrase corpora is not feasible due to their different nature.

Paraphrase variety. Through the variety of relations covered (simple and complex, and with univalued and multivalued targets), we observed that there is a spectrum between two extremes within relational paraphrasing. Very complex relations such as authorship show a high variability with respect to semantic content, that is why paraphrases cannot be combined in a single paraphrase grouping. Very simple relations such as date of birth show a low variability, so a single group is possible (Section 6). Between these two extremes there is a whole spectrum of possibilities. What all these relations share is the underlying paraphrase concept, which is wider than in corpora such as MSRP or P4P: in WRPA, paraphrases are those units expressing the same kind of relation, even though the semantic content is not always the same, as in example (8) for the alternate name relation.

1. (8)

   1. a. person also writes under the name person

   2. b. person changed his name to person

Paraphrase quality. Taking advantage of Wikipedia-structured information and semantically labelled data to obtain our anchor points, as well as extracting our paraphrases from the body of the articles of the same encyclopaedia, allowed for the acquisition of reliable and consistent examples. The precision of the paraphrase pattern sets is good and their quality is further tested through the use of the two classifiers, whose accuracy is also reasonably high (Section 5). The same can be stated for the results of the manual validation of the corpus (Section 6).

Corpus applicability. WRPA is not fully accurate because it was automatically created without a final and complete human revision. Therefore, this corpus cannot be used as gold standard for evaluating paraphrase systems. However, it is useful in other NLP tasks. By way of illustration, patterns in WRPA-person were used in the Slot Filling Task in the TAC-KBP-2012 contest (Gonzàlez et al. Reference Gonzàlez, Rodríguez, Turmo, Comas, Naderi, Ageno, Sapena, Vila and Martí2012). Also, paraphrase corpora annotated with paraphrase types show a great potential for the study of the phenomenon of paraphrasing and the development of NLP tools involving paraphrasing. As already explained, a subset of WRPA-authorship was annotated with our paraphrase typology (Vila et al. Reference Vila, Bertran, Martí and Rodríguez2013).

Due to the complexity of the task and the diversity of the paraphrase phenomenon, the method applied is not error free. In what follows, an analysis of the drawbacks present in the WRPA corpus is performed. According to Androutsopoulos and Malakasiotis (Reference Androutsopoulos and Malakasiotis2010) and Madnani and Dorr (Reference Madnani and Dorr2010), extraction methods based on the distributional hypothesis can produce pairs of templates that are not true paraphrase pairs, even though they share the same context. In fact, pairs involving antonyms are frequent. Lin and Pantel (Reference Lin and Pantel2001) find X solves Y to be very similar to X worsens Y, and the same problem has been reported by Bhagat and Ravichandran (Reference Bhagat and Ravichandran2008). WRPA is not different in this regard. It extracted cases of antonymy, such as (a) in Table 14 for place of birth; cases that do not express the relation in question, such as (b) for authorship; cases in which the relation does not necessarily hold, such as (c) for spouse; cases in which it is difficult to determine the adequacy of the pattern, such as (d) for age; and cases where there exists a confusion with other relations, such as (e) for origin, where the preposition on clearly introduces a date. These errors have been removed from the final corpus when detected in our manual revisions.Footnote ⁴¹

Table 14.

Patterns not expressing the expected relation

Other issues causing error in WRPA, specially regarding the authorship classifier, can be seen in Table 15. The example in (a) ends by a list of Ys instead of a single Y. The example in (b) introduces a series of alternatives to the name of the work (better known as or which can be translated as). In (c), a large amount of complementary information appears (revolutionized the field of theoretical linguistics and based on his doctoral thesis). These cases make the task more difficult.

Table 15.

Troubling instances