2.1 Heuristics to capture keyphrase relevancy

Medelyan (Reference Medelyan2005) points out that the main challenge of keyphrase extraction systems that use classification algorithms is to define informational heuristics that can be determined for each term and used to distinguish terms that are keyphrases and that are not keyphrases between the set of candidate terms. Table 1 shows a summary of the heuristics used in automatic keyphrase extraction systems to capture the relevance of a keyphrase term.

Table 1.

Heuristics to capture the relevance of terms

2.2 Systems for automatically extracting keyphrases

The literature in NLP is rich in proposals of systems that, based on corpora with texts having keyphrases labeled by people, try automatically extracting keyphrases. The systems for automatically extracting keyphrases from text are predominantly divided into two categories: supervised and unsupervised. Supervised methods uncover the mapping between features about the syntax, structure, and semantics of the text and certain keyphrases that best summarize the meaning of the text, whereas the unsupervised methods are mostly based on clustering and graph-based ranking.

A wide variety of keyphrase extraction methods have been presented in the literature. Some methods are based on supervised learning (Frank et al. Reference Frank, Paynter, Witten, Gutwin and Nevill-Manning1999), unsupervised learning (Mihalcea and Tarau Reference Mihalcea and Tarau2004; Bougouin, Boudin, and Beatrice Reference Bougouin, Boudin and Beatrice2013), and semi-supervised learning (Ammar et al. Reference Ammar, Peters, Bhagavatula and Power2017; Ye and Wang Reference Ye and Wang2018). In addition, word representation has been used as word embeddings which has also been used in keyphrase extraction (Wang, Liu, and McDonald Reference Wang, Liu and McDonald2015; Debanjan et al. Reference Debanjan, Kuriakose, Shan and Zimmermann2018ab). The main challenge of keyphrase extraction is that sometimes keyphrases are not present in the text, so some methods will fail to extract those keyphrases. The literature of keyphrase extraction has demonstrated some methods that extract keyphrases missing from the text (Silveira, Furtado, and Pinheiro Reference Silveira, Furtado and Pinheiro2015; Meng et al. Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017; Wang, Liu, and McDonald Reference Wang, Liu and McDonald2015).

Keyphrase extraction algorithm (KEA) (Frank et al. Reference Frank, Paynter, Witten, Gutwin and Nevill-Manning1999; Witten et al. Reference Witten, Paynter, Frank, Gutwin and Nevill-Manning1999) is one of the first systems that extracts keyphrases from documents. It applied a Naive Bayes (Domingos and Pazzani Reference Domingos and Pazzani1997) classifier for keyphrase extraction as a classification task in which the textual terms are represented by three features: (i) TFxIDF, (ii) position of the first occurrence, and (iii) keyphraseness. Medelyan and Witten (Reference Medelyan and Witten2006) improved the original KEA and proposed the KEA++, a method based on machine learning and which works in two main stages: (i) candidate identification, which identifies thesaurus terms that relate to the documents content and (ii) filtering, which uses a learned model to identify the most significant terms based on certain properties or features.

Wikify! Mihalcea and Andra (Reference Mihalcea and Andra2007) uses the property of Wikipedia’s keyphraseness, which quantifies the probability of the term appearing as a Wikipedia tag, and was inspired in reproducing the publishers’ editorial style in Wikipedia and select keyphrases.

The multi-purpose automatic topic indexing (MAUI) system (Medelyan, Frank, and Witten Reference Medelyan, Frank and Witten2009) enhances KEA’s framework by proposing features that capture a semantic knowledge from Wikipedia. MAUI divides the process of keyphrase extraction into two steps: (i) the selection of the candidates to be labeled a keyphrase and (ii) the application of the learning algorithm. At the candidate selection step, all n-grams up to a maximum length of three words that do not begin or end with a stop word and that appear more than once in the text are selected as candidate terms. Moreover, MAUI has extended the set of features that characterize the text by creating six new ones: (i) spread, (ii) semantic relatedness, (iii) Inverse Wikipedia Linkage, (iv) phrase length, (v) Wikipedias keyphraseness, and (vi) node degree. MAUI uses bagged decision trees.

Recently, Zhang et al. (Reference Zhang, Wang, Gong and Huang2016) tackle the problem of keyphrase extraction in a different context using an innovative approach. The context is that of extracting keyphrases from tweets. The approach is based on deep recurrent neural network (RNN) in order to rank the keyphrase candidates. Instead of looking for a pattern from features of the text, the approach identifies the potential keyphrases exploring the sequence of words in the text. Zhang et al. (Reference Zhang, Li, Song and Zhang2018) present a neural keyphrase extraction framework that exploits conversation context. They proposed four types of neural encoders (averaged embedding, RNN, attention, and memory networks) for capturing salient content to help in indicating keyphrases in target posts.

On many occasions in the keyphrase extraction task, candidate terms that are not assigned as a keyphrase by the authors are labeled as non-keyphrases and used as negative examples in training by supervised classifiers. Often these negative examples outnumber the positive examples, creating an unbalanced training set. The literature presents data balancing techniques (Chawla et al. Reference Chawla, Bowyer, Hall and Kegelmeyer2002) in order to reduce the negative effects that unbalanced data can cause in training. Sterckx et al. (Reference Sterckx, Caragea, Demeester and Develder2016) address this problem by creating a large test collection of articles with many different opinions per article, evaluating the effect on extraction performance, and presenting a procedure for supervised keyphrase extraction with noisy labels.

The literature review also contemplates approaches that use unsupervised learning for keyphrase extraction. Graph-based ranking methods follow an unsupervised approach for keyphrase extraction. Mihalcea and Tarau (Reference Mihalcea and Tarau2004) proposed a graph-based ranking model called TextRank, which builds a graph that represents the interconnection between the words or other text entities with meaningful relations. The principle of the model is the fact that when one vertex links to another one, it is casting a vote for the other vertex. The higher the number of votes that are cast for a vertex, the higher the importance of the vertex, and this information is also taken into account by the ranking model.

Grineva, Grinev, and Lizorkin (Reference Grineva, Grinev and Lizorkin2009) used a model to build a graph and analyze the interconnectivity of the vertices. However, instead of using statistics gathered from a training set, they use semantic information derived from Wikipedia. Bougouin et al. (Reference Bougouin, Boudin and Beatrice2013) improved TextRank and presented TopicRank, a keyphrase extraction method based on graphs that classify candidate sets of keyphrases in topics and use them as vertices in a complete graph. A ranking model based on a graph is applied to allocate the significance score for each topic. Keyphrases are generated selecting a candidate for each topic with the largest score value. Bougouin, Boudin, and Daille (Reference Bougouin, Boudin and Daille2016) propose TopicCoRank, a keyphrase annotation method built on the existing TopicRank method (Bougouin, Boudin, and Beatrice Reference Bougouin, Boudin and Beatrice2013). The authors used two graph representations, one for the document and one for the specialized domain. Then, they applied a co-ranking algorithm to perform both keyphrase extraction and assignment in a mutually reinforcing manner.

The literature review also has semi-supervised approaches for keyphrase extraction. Danesh, Sumner, and Martin (Reference Danesh, Sumner and Martin2015) developed the SGRank using an unsupervised keyphrase extraction approach by combining statistical heuristics with a graph-based algorithm in which candidate n-grams are ranked based on a modified version of TFxIDF as used in KP-Miner (El-Beltagy and Rafea Reference El-Beltagy and Rafea2010). Wang and Li (Reference Wang and Li2017) constructed a system that uses external knowledge to enrich heuristics that represent a candidate for the keyphrase, including Wikipedia, IEEE taxonomy, and pre-trained word embeddings, among others. The system involves three steps: (i) generation of candidates, (ii) ranking of keyphrases, and (iii) classification of the type of keyphrase. To assign candidates as keyphrases, two unsupervised algorithms TextRank (Mihalcea and Tarau Reference Mihalcea and Tarau2004) and SGRank (Danesh, Sumner, and Martin Reference Danesh, Sumner and Martin2015) are incorporated into a Random Forest, providing their output as complementary heuristics. Ammar et al. (Reference Ammar, Peters, Bhagavatula and Power2017) developed a system using semi-supervised learning based on the extraction of end-to-end relationships of Miwa and Bansal (Reference Miwa and Bansal2016) for keyphrase extraction and identification of types and semantic relationships between keyphrases of the same type.

Ye and Wang (Reference Ye and Wang2018) proposed semi-supervised methods for extracting keyphrases using labeled and unlabeled data for learning. Two strategies are proposed: (i) unlabeled documents are first tagged with synthetic keyphrases obtained from unsupervised keyphrase extraction methods or a self-learning algorithm, and then combined with labeled samples for training and (ii) a multitask learning framework is investigated to jointly learn to generate keyphrases as well as the titles of the articles.

All of the works presented here consider the universe of terms that appear in the text. We identified that there are keyphrases in the corpus that are not textual, that is, they are not articulated in the texts. Therefore, keyphrase extraction methods that do not consider not-in-text keyphrases are not comparable with those assigned by the best performing human taggers.

Recently, Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) proposed a feature called inference semantic relatedness (ISR) for terms that do not appear in the text, with the objective of capturing if the semantic relatedness of a not-in-text term (in relation to the other terms in the text) is higher than the semantic relatedness of the textual term from which it was inferred. Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) proposed to apply a generative model based in RNN for keyphrase prediction. This model can generate keyphrases based on an understanding of the text, regardless of the presence or absence of keyphrases in the text. The basic idea of the proposed keyphrase generation model is to use an RNN Encoder–Decoder model to convert the content of the original text with an RNN encoder and generate corresponding keyphrases by producing the probability of all words in the vocabulary with an RNN decoder. Experiments with five benchmark datasets were performed, and the results show that this proposed model significantly outperforms existing supervised and unsupervised extraction methods. Chen et al. (Reference Chen, Zhang, Wu, Yan and Li2018) proposed a new sequence-to-sequence architecture for keyphrase generation, CorrRNN, which captures correlation among multiple keyphrases. A coverage mechanism (Tu et al. Reference Tu, Lu, Liu, Liu and Li2016) is employed to indicate whether the word has been summarized by previous phrases. Furthermore, preceding phrases are taken into account to avoid the repetition of phrases and to improve result coherence.

Word embeddings have also been used in keyphrase extraction. Wang et al. (Reference Wang, Liu and McDonald2015) proposed an approach that uses word embeddings as an external knowledge base for both keyphrase extraction and generation. They proposed an unsupervised graph ranking approach using word embeddings as an external knowledge base for extracting keyphrases that appear in a document and then introduced a keyphrase generation approach built on top of the extraction model using the power of word embeddings to suggest abstractive keyphrases not appearing in a document. Debanjan et al. (Reference Debanjan, Kuriakose, Shan and Zimmermann2018a) and Debanjan et al. (Reference Debanjan, Kuriakose, Shan, Zimmermann and Talburt2018b) proposed a method that represents candidate keyphrases extracted from a scientific article by domain-specific phrase embeddings and ranking them using a theme-weighted PageRank algorithm (Langville and Meyer Reference Langville and Meyer2004), such that the thematic weight of a candidate keyphrase indicates how similar it is to the thematic representation or the main theme of the article, which is also constructed using the same embeddings. Papagiannopoulou and Tsoumakas (Reference Papagiannopoulou and Tsoumakas2018) presented an unsupervised method for keyphrase extraction that uses local word embeddings (more specifically, GloVe vectors (Pennington, Socher, and Manning Reference Pennington, Socher and Manning2014)) to detect the candidate keyphrases and to provide a complete set of representative keyphrases for a particular title and abstract.

3.1 Architecture of inference method

The task of identifying keyphrases can be modeled as the problem of selecting from a list of candidate terms $C = \{c_1, c_2, ..., c_n\}$ the subset $K = \{k_1, k_2, ..., k_z\}$ representing the terms that summarize the contents of the document $d \in D$, where D is a collection of documents. We treat keyphrase extraction as a task of binary classification. Each candidate term $c \in C$ of the training set is marked as a positive or negative example, depending on whether the authors associated the term as a keyphrase or not in the corresponding document. The classification algorithm learns the model based on the training set, analyzing the values of the heuristics of each candidate example.

The authors assign keyphrases to a document based on their semantic meaning, instead of following only the written content of the text. In this work, an inference methodFootnote ^a is proposed to generate part of the set of candidates, more specifically, a complementary form to add to the set of candidates terms that are absent from the text. Figure 1 illustrates the steps that our method follows.

Figure 1.

Inference of not-in-text candidates from the knowledge model for learning keyphrases of a document d.

3.1.2 Inference strategy

We propose to add to the set of candidates C, terms that do not appear in the text and/or in the corpus, obtained from an inference strategy that infers terms (which do not appear in the text) according to the knowledge model that has a semantic relationship with the terms of the text.

It is assumed that when people assign keyphrases that are not present in the text, they somehow express the existence of a semantic relationship between those keyphrases and the meaning of the text, either because it represents the main subject of the document, a generalization of specific terms present in the text or a synonym for terms that represent the content of the document. Thus, it is argued that there must be an association between the missing keyphrases of the text and the terms present in the text. In this sense, the inference process attempts to obtain not-in-text terms, which are related to the terms in the text.

The inference strategy generates a set $NT = \{nt_1, nt_2, ..., nt_j\}$ of not-in-text candidate terms $(NT \not \subset T)$, or in other words, candidate terms that are not in the text. For each textual term from $ti \in T$, one searches the semantic relations between two terms in a knowledge model by means of a triplet $\{t_i, \phi, nt_j\}$, where t _i is a textual term, nt _j is a term absent of the text, and $\phi$ expresses a semantic relation. We assume that it is possible to find for a textual term t _i in d a semantic relatedness with one or more not-in-text terms nt _j, thus Tnt _j represents the set of all the textual terms that have a semantic relation with nt _j. If $nt_j \not \in T$ and $|Tnt_j| \geq z$, where $|Tnt_j|$ represents the number of terms of the text that inferred the term not-in-text nt _j and z the minimum amount of inference by terms of the text, then nt _j is a not-in-text candidate term and is included in the set NT. The complete candidate set $C = T \cup NT$ contains the terms that appear in the document texts and those not-in-text terms $(nt_j \not \in T)$ that do not appear in the text and/or corpus, but that possess a semantic relatedness with d.

For example, Figure 2 displays a sports text with the corresponding keyphrases. The set of terms in the text T is formed by the terms in the text, such as terms referring to names of clubs and players (Barcos, Palmeiras, and Brazilian Championship). In this example, the keyphrase that is not explicitly expressed in the text refers to the type of sport (in this case, football). A semantic relationship between the name of the clubs and/or players can retrieve, from a knowledge model, this not-in-text term as a candidate for the keyphrase, composing the set of not-in-text NT candidate terms.

Figure 2.

Sports Domain News from corpus News corpus Silveira et al. Reference Silveira, Furtado and Pinheiro2015.

3.2 Inference of not-in-text candidate terms

As defined above, the set of candidate keyphrases C contains terms that appear in a document d, and not-in-text terms that have a semantic relationship in a knowledge model with terms that are in d. Then two traditional strategies can be used in the inference of not-in-text candidates terms: (1) distributional semantics and (2) symbolic models of knowledge bases.

The knowledge model of distributional semantics when executed in the corpus itself allows us to capture more specific terms in the corpus domain. A term that does not appear in the text of a particular document but which is present in the text of other documents of the same corpus of d _i and which has a semantic relation to the terms of the text of d _i possibly represents a specific term in the domain of the corpus. On the other hand, symbolic models of the knowledge base allow us to capture more general terms from external resources (knowledge bases) and, not necessarily, terms present in the other documents of the corpus. The inference process of not-in-text terms can be accomplished by combining knowledge models to increase the number of not-in-text candidate terms that are keyphrases, while not restricting the inference only to the terms present in the corpus.

3.2.1 Inference of not-in-text terms from distributional semantic models

Distributional semantics models, such as LSA (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990), can be used to describe the distributional vector of words from a large corpus according to their occurrence patterns, obtaining the similarity between words using these vectors.

Formally, M is an n × m matrix of document term from a collection of documents D. Each column of M corresponds to a document $(d \in D)$, and each line corresponds to a text term $(t_i \in T)$. The dimensions of M, n, and m correspond to the number of terms in the text |T| and documents |D|, respectively. Therefore, each candidate term of the text $t_i \in T$ and each document $d \in D$ correspond to a row and a column, respectively, of the matrix M. Each cell of matrix M corresponds to the number of times that the term t _i occurs in each document $d \in D$. If the term t _i occurs x times in document d, then M [t _i, d] = x. The occurrence of terms can be modified so that the terms are weighted according to the specificity for the documents, that is, rare terms are given more weight than common terms. The value of the TFxIDF property is used as the value of the occurrence of the term t _i in document d. Then, the matrix M is subjected to a singular value decomposition (SVD) process, generating M _k, in order to find a reduced dimensional representation of M that emphasizes the strongest relationships and removes the existing noises.

In this approach, a sorted list $TS = \{t_1, t_2, ..., t_n\}$ of top n similar words, where n is the quantity of terms in M, is computed for each textual term from $t_i \in T$, according to the cosine similarity of the distributional vector of words of the matrix M _k, creating a set P with n triplet, in the form, $\{t_i, \phi, nt_n\}$, that is, $P =\{\{t_i, \phi, t_1\}, \{t_i, \phi, t_2\}, ..., \{t_i, \phi, t_n\}\}$. The cosine similarity between the two-term vectors (t _i and t _j) is calculated as follows:

(1)\begin{equation}similarityCos(t_i, t_j) = \frac{\sum_{i=1}^{n} A_iB_i}{\sqrt{\sum_{i=1}^{n}A_i^2} \sqrt{\sum_{i=1}^{n}B_i^2}},\end{equation}

where A and B correspond to the distributional vector in M _k of the terms t _i and t _j, respectively. Finally, given a term of the text t _i, for each triple $\{t_i, \phi, t_n\} \in P$, where t _n represents the nth most similar term to t _i, if $t_n \not \in T$ and $|NT_{ti}| < k$, where NT _ti represents the set of not-in-text terms inferred from t _i and k the amount of not-in-text terms for each term of the text, then t _n is a not-in-text term and is included in the set NT.

3.2.2 Inference of not-in-text terms from symbolic models of knowledge bases

Traditionally, symbolic knowledge bases express the semantic relations between two terms by means of a triplet $\{t_i, \overbrace{\phi}^n, nt_j\}$, where t _i and nt _j are terms in a natural language and $\overbrace{\phi}^n$ represents the path of size n that links the two terms by means of relationships between them, for example, football, isA, sport. The relationships can be of several types, such as capableOf, propertyOf, effectOf, isA, and partOf.

We assume the existence of a knowledge base κ as input and for each term of the text of $t_i \in T$, we search the knowledge base κ by triples of size n = 1, with the form $\{t_i, \overbrace{\phi}^1, nt_j\}$ or $\{nt_j, \overbrace{\phi}^1, t_i\}$. If $nt_j \not \in T$, then nt _n is a not-in-text candidate term and is included in the NT set. Formally, this can be written logically through the following rules of inference:

(2)\begin{eqnarray}\frac{(nt_j, \overbrace{\phi}^1, t_i), (t_i \in T), (nt_j \not \in T)}{nt_j \in NT}\end{eqnarray}

(3)\begin{eqnarray}\frac{(t_i, \overbrace{\phi}^1, nt_j), (t_i \in T), (nt_j \not \in T)}{nt_j \in NT}\end{eqnarray}

For example, given a term of the text sport, it is possible to deduce from a knowledge base the following triples with n = 1: {football, IsA, sport}, {sport, isA, game}, forming NT = {football, game}. The use of n > 1 searches for matches related to football and game, obtain, for example, {football, RelatedTo, ball}, {attacker, HasContext, soccer}, {game, IsAn, activity}, and {domino, IsA, game}, obtaining NT = {football, sport, ball, striker, activity, domino}. The intuition is that not-in-text terms that are semantically related to text terms through a direct relationship (n = 1) are potential candidates for keyphrases.

3.3 Heuristic methods for selecting not-in-text candidates

The deduction process obtains all the existing terms that have a similarity relationship with the textual terms. For example, in a text about sport, it is possible to deduce the soccer team names and the players as well as other terms used in soccer, such as goal, referee, and stadium. However, not all the deduced not-in-text terms are considered good candidates for keyphrases. Therefore, we need to measure the relevance of the not-in-text terms, allowing the algorithms to learn and characterize a significant example of not-in-text keyphrases. Thus, we use the feature proposed by Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) and we define other two new heuristics that measure the relevance of a not-in-text term to be considered a good candidate of a keyphrase.

3.3.1 Inference semantic relatedness

The first heuristic considers that the semantic relationship of a term is a good indication of its relevance to be a keyphrase. To characterize the quality of not-in-text terms, Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) have created a Boolean heuristic called ISR, which captures whether the semantic relatedness of the not-in-text term nt _j (in respect to other terms of the text) is higher than the semantic relatedness of the term t _i which inferred it. It is argued that terms that have a greater semantic relationship with the terms of the document are more likely to represent the text. More specifically, not-in-text terms that are semantically more related to the text are potential candidates for keyphrases.

Formally, given $\{nt_i, \phi, t_i\}, \{nt_i, \phi, t_j\}, ..., \{nt_i, \phi, t_n\}$, where $nt_i \in NT$ is a not-in-text term, $T_t = \{t_i, t_j, ..., t_n\} \subset T$ are textual terms with semantic relationship to nt _i and $\phi$ represents the relationship in a knowledge base between nt _i and each term of T _t. Thus, the ISR(nt _i) to the not-in-text terms can be written as follows:

(4)\begin{equation}ISR(nt_i) = \left\{\begin{matrix} 1, & if (SemRelat(nt_i) \ge SemRelat(t_c))\\ 0, & otherwise \end{matrix},\right.\end{equation}

where $nt_i \in NT$ represents the not-in-text term, $t_c \in T_t$ corresponding to the term of higher value of TFxIDF in T _t and SemRelat(x) is the value of semantic relatedness (according to Section 0.1) of the term x, whose value quantifies the semantic similarity of the term x with the other candidates.

For example, considering a crime text that does not mention the type of crime, but contains the terms revolver and death in the text. These terms (revolver and death) can infer, through searching for the semantic relationship in a knowledge model, the not-in-text term murder. The basic ISR hypothesis is that if murder has a bigger semantic relationship measurement (SemRelat(x)) with the text than the terms revolver and death, then it is more relevant as a keyphrase.

Finally, it is important to note that a not-in-text nt _i term can be inferred from n terms of the text, which form the set $T_t = \{t_i, t_j, ..., t_n\}$. In the ISR property, the comparison of the semantic value of the not-in-text nt _i is performed with the text term $t_k \in T_t$ with the highest value of TFxIDF. It is considered that the term t _k that best represents the set T _t is the most significant for the document, that is, the term with the highest value of TFxIDF. For example, the terms revolver and death infer the term murder. Thus, considering that murder = revolver, death and that $TFxIDF(death) \ge TFxIDF (revolver)$, where TFxIDF(x) is the value of the property TFxIDF of the term x, the term death is used in the ISR in the comparison of the semantic similarity with the not-in-text murder, in other words ISR(murder) = 1, if $SemRelat(murder) \geq SemRelat(death)$ and 0, otherwise.

3.3.2 Semantic power inference

The second heuristic, called semantic power inference (SPI), captures the descriptive strength of a not-in-text candidate nt _i. This strength is based on the intuition that not-in-text keyphrases must have a strong semantic relationship with the text. The descriptive power is based on the intuition that not-in-text keyphrases are deduced from terms that have a high semantic value with the text, that is, not-in-text terms that are semantically related to text terms that have high semantic value with the document are potentially more suitable candidates for keyphrases.

For example in a text about football, terms such as the name of the club and the name of the players have high semantic relationship. Subsequently, all the not-in-text terms inferred from these probably have a strong semantic relationship with the text. Formally, given a not-in-text term nt _i deduced from a document d where a subset of terms that are in the text $T_t = \{t_i, t_j, ..., t_n\}$, SPI(nt _i) is computed as:

(5)\begin{equation}SPI(nt_i) = \frac{\sum_{t_i \in T_t} SemRelat(t_i)}{\sum_{t_k \in T} SemRelat(t_k)},\end{equation}

where SemRelat(x) is the value of the semantic relatedness feature (according to Section 0.1) of the term x in the document d, whose value quantifies the semantic similarity of the term x with the other candidates, T _t is the set of terms of the text that inferred the not-in-text term nt _i, and T the set of all terms of the text.

3.3.3 Inference frequency versus inverse document frequency

The third feature is based on the intuition that not-in-text terms that are good candidates of a keyphrase are those that are inferred from various terms belonging to a specific document and that, in turn, are not often inferred in other documents. Therefore, based on TFxIDF, we propose a heuristic called inference frequency x inverse document frequency (IFxIDF) for not-in-text terms. This heuristic captures the frequency of a not-in-text term to be inferred from a specific document multiplied by the inverse frequency of its occurrence in the corpus. Given the not-in-text term nt _i inferred in a document d, IF × IDF(nt _i) is calculated as:

(6)\begin{equation}IF\times IDF(nt_i) = \frac{freq(nt_i,d)}{\sum_{nt_j \in NT} freq(nt_j,d)} \times -log_2 \frac{d_{nt}}{|D|},\end{equation}

where freq(nt _i, d) is the number of times that nt _i was inferred from any textual term of the document d, NT is a not-in-text terms set of document d, d _nt is the quantity of documents of the corpus in which nt _i is inferred, and |D| is the number of documents of the corpus. The first component in this expression refers to the normalized frequency of nt _i in the document d. The second component is the negative logarithm of the inverse frequency in the documents, in which high values indicate rare terms.

It is assumed that not-in-text keyphrases are represented by terms that are significant for a particular document. Therefore, not-in-text terms inferred from multiple documents are not relevant to a particular document and thus are not good candidates for the keyphrase. The rationale behind this is that terms that are infrequent in the corpus and frequent in a document are good candidates of keyphrases because they discriminate the document.

4.1 Training and test sets

For evaluating the proposed approach, five widely adopted scientific publication datasets were used. The first test scenario was executed in CiteULike (Medelyan Reference Medelyan2009), a corpus of scientific articles that contains 180 academic documents from two journals: HighWire and Nature. In total, 332 taggers were associated with 4638 keyphrases. However, we are going to use only 935 keyphrases because they were identified by at least 2 taggers. As stated before, 21% of the keyphrases were not present in the texts of the documents.

The second is a corpus of paper abstracts, Inspec (Hulth Reference Hulth2003). This corpus contains 2000 journal abstracts in English from the areas of Computer and Control, and Information Technology. Each abstract has keyphrases assigned by a professional indexer from a set of controlled terms and from a set of uncontrolled terms that can mean any suitable term. In this corpus, nearly 27% of the keyphrases are absent from the texts.

The third dataset was built by Kim et al. (Reference Kim, Medelyan, Kan and Baldwin2010) for the keyphrase extraction task of the SemEval-2010. This corpus has 284 scientific articles from conferences and workshops of the ACM Digital Libraries. All collected papers contain author-assigned and reader-assigned keyphrases. In this corpus, on average 15% of the reader-assigned keyphrases and 19% of the author-assigned keyphrases do not appear in the text.

The fourth and fifth datasets contain scientific publication and were built by Nguyen and Kan (Reference Nguyen and Kan2007) (NUS corpus) and Krapivin et al. (Reference Krapivin, Autayeu and Marchese2008) (Krapivin corpus), respectively. The NUS corpus contains 211 computer science articles, and Krapivin corpus has 2304 documents in the computer science domain.

All analyzed corpora have keyphrases associated by authors or readers that are absent from the text of the documents and the corpus. Table 2 shows the statistics for the number of documents, the percentage of keyphrases that are not present in the text of the referred to document and the percentage of keyphrases that are not included in any document of the corpora.

Table 2.

Statistics of documents and not-in-text keyphrases in the corpora evaluated

^a Percentual of not-in-text keyphrases, considering the author-assigned keyphrases.

^b Percentual of absent keyphrases from the document abstract.

4.2 Models of knowledge

Different scenarios have been evaluated that extend the set of candidates by means of the inference process with the application of different knowledge models, such as knowledge base symbolic model, distributional semantics with the LSA, as well as a hybrid model that combines the symbolic model and distributional semantics.

• M1 – symbolic model of knowledge base using ConceptNet;

• M2 – symbolic model of knowledge base using WordNet;

• M3 – distributional semantics, with LSA executed in the corpus itself;

• M4 – distributional semantics, with LSA executed in TakeLabFootnote ^e;

• M5 – hybrid model that combines the symbolic model with WordNet and the distributional semantics executed in the corpus.

The expansion of the candidate set by adding the non-text terms using a symbolic model of knowledge base (M1, M2, and M5) was performed using the inference strategy described in Section 3.2.2 Thus, the not-in-text terms using distributional semantics (M3, M4, and M5) were obtained according to the inference strategy described in Section 3.2.1, showing that the M3 and M5 models used their own corpus to construct the LSA matrix, while in the M4 model the LSA matrix was constructed through training the New York Times Annotated Corpus and Wikipedia.

4.3 Heuristics to characterize the relevance of candidate terms

To evaluate the use of heuristics that capture the relevance of not-in-text candidate terms with the keyphrase, we have varied the set of properties that characterize an example. Six sets of properties were used:

• A1 – has the same properties as KEA: (i) TFXIDF, (ii) first occurrence, and (iii) keyphraseness;

• A2 – has the same properties as the MAUI, that is, the same properties as KEA, plus six other properties: (i) amplitude of occurrence, (ii) semantic relationship, (iii) inverse linking on Wikipedia, (iv) term, (v) Wikipedia’s keyphraseness, and (vi) degree of the node;

• A3 – has the same properties as MAUI, plus the ISR property.

• A4 – has the same properties as MAUI, plus the property of Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) – IFxIDF.

• A5 – has the same properties as MAUI, plus the SPI property.

• A6 – has the same properties as MAUI, plus the ISR, IFxIDF, and SPI properties.

4.4 Data balancing

As the number of positive examples (terms that are keyphrases) is significantly smaller than the number of negative examples (terms that are not keyphrases), different existing data balancing strategies were applied in the training set: (i) random oversampling and (ii) synthetic minority oversampling technique (SMOTE) (Chawla et al. Reference Chawla, Bowyer, Hall and Kegelmeyer2002), increasing by 200% examples in the minority class (in this case, positive examples).

In general, SMOTE and random oversampling have similar results; however, when the ISR heuristic is used, the random oversampling strategy outperforms the SMOTE. The interpretation is that the introduction of a Boolean property, such as ISR, can undermine the SMOTE process of generating synthetic examples. Thus, in the experiments with balanced data, the SMOTE balancing strategy was applied using the set of properties KEA (A1) and MAUI (A2) and the balancing random oversampling strategy was applied when using the other sets of properties (A3–A6).

4.5 Experimental setups

We configured the experiments with the following resources and tools:

• Encyclopedic knowledge base Wikipedia (snapshot on February 4th, 2014 with 12,853,780 articles);

• Knowledge base ConceptNet 5.0 (Liu and Singh Reference Liu and Singh2004) and WordNet 3.1 (Miller Reference Miller1995; Fellbaum Reference Fellbaum1998);

• Distributional Semantic Model TakeLab (Saric et al. Reference Saric, Glavas, Karan, Snajder and Basic2012) and LSA (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990);

• Lemmatizer Stanford CoreNLP Toolkit (Manning et al. Reference Manning, Surdeanum, Bauer, Finkel, Bethard and McClosky2014);

• Weka Framework (Hall et al. Reference Hall, Frank, Holmes, Pfahringer, Reutermann and Witten2009) and the random forest algorithm (Breiman Reference Breiman2001).

The experiments were run on a machine with two 6-core Intel (R) Xeon (R) CPUs E5-2630L v2 @ 2.40GHz, with 40GB of RAM. Each corpus goes through all three steps described in Figure 1. For example, if we consider the corpus CiteULike, step (1) preprocessing is performed in approximately 3 h, step (2) inference strategy, using the inference of terms by means of WordNet, is performed in approximately 36 h, while step (3) machine learning is accomplished in approximately 156 h.

4.6 Results and discussion

To evaluate the inference process of not-in-text terms from different knowledge models, experiments were carried out by extending the set of candidates, adding, in addition to the terms of the text, not-in-text terms obtained from the models M1, M2, M3, M4, and M5 (described in Section 4.2). In each experiment, the f-measure was calculated separately for each set of candidate terms: (i) only examples representing the terms of the text T, (ii) only examples representing the not-in-text terms NT, and (iii) all examples, that is, terms of the text and not-in-text, $C = T \cup NT$, performing an exact match of the term classified as a keyphrase by the learning algorithm in relation to the keyphrases assigned by the author to the corresponding document.

For each set of candidates (T, NT, and C), the keyphrases assigned by the author according to the corresponding set are considered as gold standard, that is, when calculating the f-measure (F1-measure) for the set of candidates in the text (T), the gold standard is regarded as only the keyphrases assigned by the author that are present in the text of the document, and so consider a set of keyphrases smaller than the universe of all the terms assigned as keyphrases by the author. Thus, the f-measure is calculated using the exact match evaluation framework between the extracted keyphrases by machine learning algorithm and the gold standard keyphrases of the document corresponding.

Tables 3–7.show the results (in respect to the f-measure, considering all the keyphrases classified by the learning algorithm) between the assessment scenarios for the CiteULike, Inspec, SemEval-2010, Krapivin, and NUS corpora, respectively, considering the balanced dataset with the SMOTE (Chawla et al. Reference Chawla, Bowyer, Hall and Kegelmeyer2002) (A1 and A2) and random oversampling (A3–A6) methods. In the tables, the column K represents the keyphrases, where the value of T represents the f-measure calculated only for the candidate terms of the text, NT where the f-measure is calculated only for the not-in-text candidate terms, and C is calculated for the complete candidate set $(T \cup NT)$. The column Only text shows the f-measure value, considering candidates only as the candidate terms of the text, whereas the columns With not-in-text candidate terms consider that the candidate set contains textual and not-in-text terms, according to the process of inference of not-in-text candidate terms, described in Section 3.2, from different knowledge models, symbolic model (M1 and M2), distributional semantics model (M3 and M4), and hybrid model (M5). The values highlighted in bold represent the best f-measure results in each experiment.

Table 3.

Results in terms of f-measure of the learning of textual and not-in-text keyphrases for the CiteULike corpus

Table 4.

Results in terms of f-measure of the learning of textual and not-in-text keyphrases for the Inspec corpus

Table 5.

Results in terms of f-measure of the learning of textual and not-in-text keyphrases for the SemEval-2010 corpus

Table 6.

Results in terms of f-measure of the learning of textual and not-in-text keyphrases for the Krapivin corpus

Table 7.

Results in terms of f-measure of the learning of textual and not-in-text keyphrases for the NUS corpus

The data shown in Tables 3–7 indicate an increase in the f-measure value of the experiments whenever the set of candidates is improved with the addition of not-in-text terms inferred from a knowledge model. For example, for the CiteULike corpus (Table 3), using the MAUI property set (A2), the f-measure value changes from 49.6% (with the set of candidates consisting only of terms from the text) to 53.1% (with the set of candidates formed according to M5), representing an increase of 7.1% in the f-measure value. The same pattern in the results is verified for the other four corpora, considering the set of properties of the MAUI (A2) and two scenarios of variation of the set of candidates: (i) set of candidates formed by only terms from the text and (ii) set of candidates conforming to the M5 model. In the Inspec corpus (Table 4), the f-measure increases from 32.4% to 33.2%, representing an increase of 2.5%; for the SemEval-2010 corpus (Table 5), the f-measure rose from 22.7% to 24.3%, representing an increase of 7.0%; for the Krapivin corpus (Table 6), the f-measure increases from 21.6% to 23.7%, representing an increase of 9.7%; and for the NUS corpus (Table 7), the f-measure increases from 24.7% to 26.5%, representing an increase of 7.3%.

In the CiteULike corpus (Table 3), for the experiment applying the M5 model and the A6 property set, we obtained a f-measure result of 57.9% for the set of text candidates (T), 8.6% for the candidate set not-in-text (NT) and 53.8% for the total set of candidates ($C = T \cup NT$). It should be noted that the gold standard differs in each set of candidates. When calculating the f-measure of the set T, the gold standard is only considering the keyphrases assigned by the author that are in the text, whereas in calculating the f-measure of the set C the gold standard is considering all the keyphrases attributed by the author (whether present or not in the text).

As mentioned, it is assumed that the keyphrases missing from the text have a semantic relationship with the terms present in the text, either by expressing a generalization of specific terms of the text (hyperonymy) or being the synonym of the terms that represent the subject of the text (synonymy). In an analysis carried out in the corpora, it can be seen that most of the missing keyphrases in the text have a relation of hyperonymy or synonymy with terms of the text. For example, in a document from the CiteULike corpus dealing with the maintenance of Borneo’s biodiversity, the threat of extinction of orangutans is explicitly described in the text. From WordNet, it is possible to infer the relation {great ape, hypernym, orangutan}, “great ape” being a non-text term assigned as a keyphrase by the authors, representing a generalization of orangutans. In another document of the same corpus that deals with the social interaction between people, the authors analyze in the text the use of the mobile phone in communication among people; however, they attribute the term cell as a keyphrase of the document. From WordNet, it is possible to infer the relation {cell, synonym, mobilephone}. This way cell, synonymous with mobile phone, is added to the set of not-in-text candidates and classified as not-in-text keyphrase. In the CiteULike corpus, in an analysis of the candidate set improved using WordNet, approximately 58% of the not-in-text keyphrases have a hyperonymic relationship with terms of the text, 37% have a synonymic relation, and 5% have a different type of semantic relation. In the corpus Inspec, approximately 52% of the not-in-text keyphrases have a hyperonymic relation with terms of the text, 33% have a synonymic relation, and 15% have another type of semantic relation. In the SemEval-2010 corpus, about 40% of the not-in-text keyphrases have a hyperonymic relation with terms of the text, 47% have a synonymic relationship, and 13% have another type of semantic relation.

The symbolic models used in the experiments make use of external resources, more specifically, use the knowledge base, to obtain not-in-text terms that have different semantic relations with the terms of the term. Considering that most of the missing keyphrases of the text captured by the knowledge bases, for example WordNet, are obtained through synonymic or hyperonymic relations, it is assumed that the knowledge base symbolic models extract more generic not-in-text terms.

The distributional semantics models used in the experiments construct an LSA matrix by exploiting the co-occurrence of words in a corpus and make use of a measurement of similarity to obtain the terms present in the LSA matrix that are more similar to the terms of the text. When using the corpus itself in the distributional semantics model, it is assumed that not-in-text terms specific to the corpus domain are obtained, since they are present in other corpus documents. Some of the not-in-text keyphrases are contained in the corpus documents, for example, in the CiteULike corpus, about 35% of the not-in-text keyphrases are present in other corpus documents.

The hybrid model (M5) obtains the best results. Our interpretation is that the combination of a symbolic model with a distributional semantic model enables the capturing from external resources the more generic not-in-text terms inferred from the symbolic model of the WordNet, as well as capturing the not-in-text terms more specifically related to the corpus domain, augmenting the number of not-in-text candidate terms that are in fact keyphrases with a semantic relationship with the text and is not restricted to deducing only terms presented in the corpus. More specifically, in a text about health care, the symbolic model of the WordNet enables the capturing of the not-in-text term disease, while the distributional semantic model LSA enables the capturing of the not-in-text terms that are more specific with the corpus domain, such as breast cancer.

The extension of the set of keyphrase candidates with the not-in-text terms obtained from a knowledge model enables an improvement in the accuracy of the learning algorithms and overcomes the limitation of the state-of-the-art systems that consider as candidates only the terms of the text and/or training corpus. However, it is assumed that it is not enough to only increase the pool of candidates with not-in-text terms. To increase the accuracy of learning algorithms, it is also necessary to have heuristics that characterize the relevance of not-in-text candidate terms to keyphrases.

We performed experiments using the set of candidates formed by text terms and not-in-text terms obtained by the hybrid model (M5), varying the properties of the candidate terms: A1, A2, A3, A4, A5, and A6 (as described in Section 4.3). It should be noted that in scenarios A1 to A6, the properties (i) TFxIDF, (ii) first occurrence, and (iii) range of occurrence are not applied to characterize not-in-text terms, since they make use of information in the text of the document, for example, the TFxIDF property uses the frequency of the term occurrence in the text of the document. Therefore, these properties have no value for not-in-text terms. In the scenarios A3 to A6, the properties (i) ISR, (ii) SPI, and (iii) IFxIDF are used to characterize the relevance of not-in-text terms, not having a value for the terms of the text.

The addition of heuristics to characterize the relevance of not-in-text terms improves the accuracy of learning algorithms. For the CiteULike corpus, the f-measure changes from 53.3% in scenario A2 (set of MAUI properties) to 53.8% in scenario A6 (set of properties of MAUI + ISR + IFxIDF + SPI), representing an increase of about 1%. The same pattern of results is verified for the other corpora. In the Inspec corpus, the f-measure increases from 33.2% (A2) to 34.7% (A6), representing an increase of approximately 5%; in the SemEval-2010 corpus, the f-measure increases from 24.3% (A2) to 26.4% (A6), representing an increase of approximately 9%; in the Krapivin corpus, the f-measure increases from 23.7% (A2) to 25.3% (A6), representing an increase of approximately 7%; and in the NUS corpus, the f-measure increases from 26.5% (A2) to 27.6% (A6), representing an increase of approximately 4%.

The results presented in Tables 3–7 show the importance of the new heuristics to aid the algorithm in dealing with the learning of not-in-text candidates obtained by the inference process. In addition to improving the set of candidates that includes not-in-text terms inferred from a knowledge model, it is necessary to add new heuristics to characterize the relevance of not-in-text terms to increase the accuracy of learning algorithms in the classification of not-in-text keyphrases.

Considering the results obtained in the experiments with the use of the ISR, IFxIDF, and SPI properties (Tables 3–7), it is noticed that by adding one of these properties to the MAUI properties set (A3–A5), the learning algorithms can distinguish the terms that are not-in-text candidates, improving the accuracy of the algorithms. By combining all these properties (A6), the accuracy of learning algorithms is even greater, improving the extraction quality of not-in-text keyphrases.

The heuristics defined in this work for the not-in-text terms capture the descriptive and discriminative capacity of a specific document. Considering the results of the experiments with the addition of the ISR, IFxIDF, and SPI properties, it is understood that the keyphrases missing from the text: (i) have a semantic relation with the terms of the text, allowing the inference of these terms from models of knowledge, (ii) have a greater semantic relationship with the text if compared with the terms of the text that inferred them (according to the ISR heuristic), (iii) have a descriptive power, that is, they are deduced from terms with high semantic value for the document (according to the SPI heuristic), and (iv) have a discriminatory power, that is, they are deduced from several terms of a specific document and are not often deduced from other documents.

Figure 3 shows a document of the Inspec corpus with the keyphrases extracted by the proposed method with the inference of the not-in-text terms from the model M5 and with the set of properties A6. In the figure, the underlined terms correspond to the keyphrases present in the text, the terms shaded in gray correspond to the candidate terms and/or keyphrases that are not in the text and that were obtained through the inference process, as well as the terms that are highlighted in bold corresponding to the keyphrases correctly extracted by the learning algorithm.

Figure 3.

An example of the keyphrases extracted with the application of the proposed method for a document from corpus Inspec.

The improvement of the set of candidates, in addition to the terms of the text, with the not-in-text terms obtained from a knowledge model, and the use of heuristics that characterize the relevance of the not-in-text terms to the keyphrases allow the algorithm to learn not-in-text terms, increasing accuracy and overcoming the limitation of traditional keyphrase extraction systems that extract only keyphrases present in the text and/or in the corpus. In the example shown in Figure 3, it is noted that the learning algorithm correctly extracted one of the absent keyphrases from the text attributed by people.

It is important to emphasize that, independently of the knowledge representation model used, our method also improves the learning of keyphrases derived from terms that are in the text, that is, besides improving the accuracy in the identification of not-in-text keyphrases, it is possible to verify an improvement in the f-measure for the terms of the text. For instance, comparing the results obtained in the CiteULike corpus with the terms of the text in experiment A2 with the training set only with the terms of the text and with the addition of not-in-text terms from the hybrid model (WordNet and LSA executed in the corpus) of the CiteULike corpus, we obtained 420 and 404 true positives for the examples that are keyphrases, respectively. Our hypothesis is that the improvement of the training set with the not-in-text terms ends up giving the classification algorithm greater power of discrimination and thus also impacting the learning of keyphrases that are in the texts of the corpus.

Table 8 shows the results (relative to f-measure) for the top 5, top 10, top 15, as well as for all terms classified as keyphrases by the learning algorithm for the corpora CiteULike, Inspec, SemEval-2010, Krapivin, and NUS Corpus. The scores given by each tree of the random forest algorithm to each candidate term are averaged and used to produce the final ranking of terms to keyphrases, obtaining the top 5, top 10, and top 15 keyphrases. For these experiments, we used (i) the inference of the not-in-text terms from the hybrid model that combines WordNet and LSA executed in the corpus itself (M5), and (ii) the set of properties A6 (according to Section 4.3). It is verified that for all corpora, the results are better when considering all the terms classified as keyphrases by the classification algorithm. For example, for the CiteULike corpus, when considering only the top 5 keyphrases, we have a result of f-measure for the set C of 50.5%, whereas when considering all keyphrases, we have a f-measure result of 53.8%, that is, there is an increase of 6.5%.

Table 8.

Results in terms of f-measure for keyphrase extraction using knowledge models to top 5, 10, and 15 keyphrases

Table 9 presents the results (in relation to f-measure) for the set of text candidates (T), set of not-in-text candidates absent from all corpus documents (NT _c), set of non-text candidates (NT), and the set of all candidate terms ($C = T \cup NT$) for the corpora CiteULike, Inspec, SemEval-2010, Krapivin, and NUS Corpus. For these experiments, we used (i) the inference of the not-in-text terms from the hybrid model that combines WordNet and LSA executed in the corpus itself (M5) and (ii) the set of properties A6 (according to Section 4.3). It can be seen that the use of the M5 model (which combines the symbolic model of knowledge base with the distributional semantic model) allows us to infer as candidates not-in-text terms absent from the text as well as not-in-text terms absent from the corpus.

Table 9.

Results in terms of f-measure for extraction of keyphrases absent from the text and the corpus using knowledge models

4.6.1 Evaluation of the proposed method with state-of-the-art systems

One limitation of traditional keyphrase extraction systems is that the set of keyphrase candidates is formed only by terms present in the text. An improvement of this limitation is made by Bougouin et al. (Reference Bougouin, Boudin and Daille2016) using TopicCoRank, a method for identifying keyphrases based on a unified graph that connects two graphs representing the document topics, the controlled keyphrases, and the relations between them. Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) propose a model to extract keyphrases missing from the text, but which are present in the training corpus and Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) define a heuristic for not-in-text candidate terms. Table 10 presents the results in terms of f-measure top 10 to the text terms and in terms of recall top 50 to the not-in-text terms of the method proposed in this work with Inspec, SemEval-2010, Krapivin, and NUS corpora, and those obtained by state-of-the-art systems by Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) and Silveira et al. (Reference Silveira, Furtado and Pinheiro2015). The values highlighted in bold represent the best f-measure results in each corpus.

Table 10.

Results in terms of f-measure top 10 to the text terms and in terms of recall top 50 to the not-in-text terms for the Inspec, SemEval-2010, Krapivin, and NUS corpora comparing with the state-of-the-art systems

With the Inspec corpus, TopicCoRank obtained f-measures top 10 of 32.3% and recall top 50 of 4.1% for keyphrases present in the text and for keyphrases absent from the text, respectively. Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) obtained f-measure top 10 of 34.2% for keyphrases in the text and recall top 50 of 10.0% for keyphrases absent from the text. Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) obtained 32.7% for keyphrases in the text and 9.1% for keyphrases absent from the text. In our approach with the Inspec corpus, we obtained 34.3% for keyphrases in the text and 13.2% for keyphrases absent from the text. With the SemEval-2010 corpus, TopicCoRank obtained f-measures top 10 of 25.8% and recall top 50 of 2.3% for keyphrases that are present in the text and absent from the text, respectively. Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) obtained an f-measure top 10 of 30.4% for keyphrases in the text and recall top 50 of 6.7% for keyphrases absent from the text, Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) obtained 28.3% for keyphrases in the text and 4.3% for keyphrases absent from the text, whereas our approach obtained 30.5% for keyphrases in the text and 7.7% for keyphrases absent from the text. In the Krapivin corpus, we obtained f-measure top 10 of 31.3% for keyphrases in the text and recall top 50 of 21.5% for keyphrases not-in-text. Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) obtained 31.2% for keyphrases in the text and 20.1% for keyphrases not-in-text. Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) obtained 26.6% for keyphrases in the text and 20.2% for keyphrases not-in-text, and TopicCoRank obtained f-measures top 10 of 27.2% and recall top 50 of 14.2% for keyphrases present in the text and for keyphrases absent from the text, respectively. For the NUS corpus, TopicCoRank obtained f-measures top 10 of 27.8% and recall top 50 of 5.0% for keyphrases present in the text and for keyphrases absent from the text, respectively. Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) obtained an f-measure top 10 of 32.6% for keyphrases in the text and recall top 50 of 11.6% for keyphrases absent from the text. Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) obtained 33.5% for keyphrases in the text and 11.9% for keyphrases absent from the text. In our approach with the Inspec corpus, we obtained 33.5% for keyphrases in the text and 12.0% for keyphrases absent from the text.

A way to deal with the limitation of the keyphrase extraction methods of classifying only textual terms is to improve the training set with examples of not-in-text terms. The approach of TopicCoRank (Bougouin, Boudin, and Daille Reference Bougouin, Boudin and Daille2016) and Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017) only considers as not-in-text candidates the terms presented in documents of the corpus. Consequently, this approach is not able to recall any not-in-text keyphrases that are not contained in the corpus. Not all the not-in-text keyphrases are contained in the documents of the corpus, for instance, in the CiteULike corpus, 65% of the not-in-text keyphrases are not in the corpus. Silveira et al. (Reference Silveira, Furtado and Pinheiro2015) only deduct the not-in-text candidates from a symbolic model of knowledge base and define a new feature to characterize these terms. Our approach is more generic, allowing us to extract the not-in-text candidate terms from external and internal resources, considering the symbolic model which uses an external knowledge base and the distributional semantic model which uses an LSA matrix performed in its own corpus. Thus, we can extract as candidates not-in-text keyphrases that are present as well as absent from the corpus, augmenting the training set with examples of not-in-text terms that are in fact keyphrases. Not only this, but there are features defined in this paper that capture the relevance of not-in-text candidates, enabling the classification algorithm to learn the not-in-text keyphrases.

Our method outperforms the results of TopicCoRank (Bougouin, Boudin, and Daille Reference Bougouin, Boudin and Daille2016), Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017), and Silveira et al. (Reference Silveira, Furtado and Pinheiro2015). Our claim is that this improvement is due to two aspects: (i) the knowledge representation models used by our approach allow us to identify a set of not-in-text terms that have a semantic relationship with the text, which are potential candidates for keyphrases and that can be present or absent from the corpus; (ii) although the not-in-text terms have a semantic relationship with the textual terms, not all terms are relevant for keyphrases. Therefore, the features created and used to characterize the not-in-text terms allow the classification algorithm to learn with more quality the not-in-text terms that are relevant to keyphrases.