3.1 Definition

The terms ‘keyword’ and ‘keyphrase’ do not refer to any theory. An element is considered as a ‘key’ element with respect to a document, when it is an important descriptor of the document content. The opposition word versus phrase simply refers to the mono- or multi-lexical structure of the textual units, which can be composed of one or several tokens. However, the formal word/phrase distinction is often blurred.

In this paper, the terms ‘keyphrase’ and ‘keyword’ are used interchangeably and they do not refer to any linguistic or semantic theory. In other words, ‘keyword’ stands for any key textual unit that can be composed of one or more words and may work as either a common or proper noun.

Keywords can be different lengths. In the literature, ‘length’ is defined either as the number of the constituent words or the number of the characters in a keyword. A wide variety of length can be observed in the literature on keyword extraction: single token and two tokens, respectively in the approaches proposed by Liu et al. (Reference Liu, Pennell, Liu and Liu2009) and Muñoz (Reference Muñoz1997), up to three tokens in the approaches of Frank et al. (Reference Frank, Paynter, Witten, Gutwin and Nevill-Manning1999) or Hulth (Reference Hulth2003), and four tokens in the work of Matsuo and Ishizuka (Reference Matsuo and Ishizuka2003). Statistics on the number of keyword tokens in two reference benchmarks have been reported by Bougouin (Reference Bougouin2015) (Table 1): unsurprisingly, the majority of keywords consist of two tokens. The statistic reported in SemEval 2017 (Augenstein et al. Reference Augenstein, Das, Riedel, Vikraman and McCallum2017), however, shows that 51% of the assigned keywords have more than three tokens.

Table 1.

Statistics on the length of the keywords in SemEval 2010 and Duc

We note that in this paper, we do not focus specifically on term extraction nor on named entity recognition, although there has been a long tradition of work on the former topic (Jacquemin and Bourigault Reference Jacquemin, Bourigault and Mitkov2003) and much effort put into the latter (Nadeau and Sekine Reference Nadeau and Sekine2007).

In computer science and IR, ‘term’ is a polysemous word: it refers both to the linguistic unit of terminologies, which labels a concept, and to the ‘index term’ or ‘descriptor’, which are expected to describe a document content and are part of a controlled or indexing vocabulary. As stated by Lossio-Ventura et al. (Reference Lossio-Ventura, Jonquet, Roche and Teisseire2013), there are some fundamental differences between term extraction and keyword extraction approaches. One major difference is that extracting terms requires a big collection of texts, which is not a necessary requirement in keyword extraction, which can take only a single document as an input. To highlight another difference, the former approach aims at extracting term-like units and filtering out those that may not be terms, syntactically or terminologically. On the other hand, the latter one extracts the ‘key’ elements of a document, which are not limited to terms. Thus, while keyword extraction approach can be domain independent, term extraction applies only to specialized fields or professional domains. In addition, the main goal of terminology extraction is to build domain terminologies, that is conceptualization of domains, and not to retrieve documents in a heterogeneous collection of documents.

In this paper, we consider domain and language dependent as well as independent methods of keyword extraction. Whenever the applicability and the performance of the reviewed methods depend on these criteria, we try to make it explicit.

3.2 Keyness properties

In keyword extraction, the goal is always to extract important or ‘key’ elements, but ‘keyness’ is an elusive concept and its interpretation depends on the target user and application. It can be associated with various properties, even if they usually are not equally important in all contexts. We distinguish three main types of keyness properties: informational, linguistic, and domain-based.

3.2.1 Informational properties

United Nations Educational, Science and Cultural Organization (UNESCO 1975) established several principles to be followed by keywords, independent of any application and of the way keywords are provided, manually or automatically. Two principles, exhaustivity and specificity, are proposed that are respectively related to a set of keywords and a single keyword. To these two principles, we add three more: minimality, impartiality, and representativity. Minimality, which applies to a set of keywords, is orthogonal to exhaustivity, whereas representativity, which applies to a single keyword, is orthogonal to specificity. In the following, we explain each of the mentioned principles in more detail.

Exhaustivity The set of keywords should cover all the subjects of the studied document, which have potential information value. This principle tends to add implicit keywords to the set of keywords that do not occur in the studied document but target a subject of it.

Specificity Keywords are considered as key elements of a document by contrast with other documents that could belong to other domains. To represent the content of a document, keyword should be ‘as specific as possible’: it is important to select only those linguistic units that are really characteristic of that document and to leave out common textual units that one can find in any similar document. For instance, in legal documents, common law and disclosure statement are rather common terms and give little information on the content of the documents beyond categorizing them as legal ones. Similarly, user guide might be considered as a cross-domain, rather than as a domain-specific term.

Minimality Keywords should differ from each other. The set of keywords has to include unique keywords with different meanings. In Figure 1, quality of service and service quality in the author-assigned keywords refer to the same meaning and so the set is not minimal.

Figure 1.

Example of an annotated text in SemEval 2010 data set. Underlined keywords are those that occur in the text.

Impartiality Keywords should be as objective as possible by reflecting the informational content of the document without involving personal sentiment or opinion. This, however, is not the case for quality of service and service quality (Figure 1), as the author has been biased by this subject and this repetition of the subject may also bias the readers to the author’s point of view.

Representativity Keywords are considered as key elements of a document by contrast with other words or phrases, which reflect minor aspects. While keywords such as end-to-end quality of service and distributed real-time embedded systems in Figure 1 are representative, phrases like experiment, researchers, and resource availability would not represent the subject of the text.

While there is no arbitrary limit to the number of keywords, the right trade-off between the number of the extracted keywords, which must often be minimized, and the exhaustivity of the document description that they give must be found. Similarly, it is preferable that a keyword is not too specific nor too generic.

4.1 Evaluation methods

Extraction methods can be evaluated extrinsically or intrinsically. In the former case, keywords are directly used in a given application and their quality is measured indirectly through their impact on the performance of that application. As an example, Hammouda, Matute, and Kamel (Reference Hammouda, Matute, Kamel, Perner and Imiya2005) evaluated keywords in a text summarization application. In intrinsic evaluation, however, keywords are evaluated directly.

Intrinsic evaluation could be done a posteriori, where experts are asked to label the extracted units as ‘keyword’ or ‘non-keyword’. Since this approach of evaluation is costly, keywords are mostly evaluated by measuring the match between the human-assigned and the extracted keywords.Footnote ^b It should be noted that this approach of evaluation is mostly suitable for low inflectional languages, such as English, and less performant for languages with high inflections, such as Romance languages.

Human judgements must be used carefully, however. People with different degrees of expertise are often asked to distinguish keywords from non-keywords but keyness is not a binary notion and human judgements include an element of arbitrariness and subjectivity (see Section 4.3). It is therefore important to involve different human judges in evaluation and to calculate the degree of agreement among them (Kappa statistics (Viera and Garrett Reference Viera and Garrett2005) is widely used for measuring the inter-evaluator agreement).

4.2 Evaluation measures

In the state of the art of keyword extraction, different measures and protocols are used for evaluating the extracted keywords. The most traditional measures are Precision, Recall, and F-measure. Precision is the frequency with which retrieved keywords are relevant (Equation 1) and Recall is the frequency with which relevant keywords are retrieved by the evaluated approach (Equation 2). To have a trade-off between precision and recall, F-measure is calculated (Equation 3).Footnote ^c

(1)\begin{equation} Precision = \frac{Retrieved \cap Relevant}{Retrieved}\end{equation}

(2)\begin{equation} Recall = \frac{Retrieved \cap Relevant}{Relevant}\end{equation}

(3)\begin{equation} F_{\beta}= (1+\beta^2).\frac{precision.recall}{(\beta^2.precision)+recall}\end{equation}

While a posteriori evaluation of keywords lets us compute the precision of the extracted keywords, evaluating them through the match with the human-assigned keywords gives both the precision and recall over the extracted keywords.

In spite of the fact that these measures are traditional and basic, they are widely used in well-known challenges, such as SemEval (Kim et al. Reference Kim, Medelyan, Kan and Baldwin2010; Augenstein et al. Reference Augenstein, Das, Riedel, Vikraman and McCallum2017) and DÉfi Fouille de Textes (DEFT).Footnote ^d Hasan and Ng (Reference Hasan and Ng2014), however, consider the improvement of the evaluation scheme as a remaining challenge in the field. In fact, the above traditional measures are questionable. They presume the existence of a ‘gold’ standard but one often has to deal with a long list of less or more relevant keywords which cannot be considered either as a ‘gold’ reference or as a standard.

In IR, it is common to evaluate the methods according to the quality of the keywords’ ranking: one approach performs better than another one if it can give a higher rank to more important keywords. The above-mentioned traditional measures, however, do not take this into account and so in the case of having a list of ranked keywords, other evaluation measures are required. Precision@K is one of the measures that can be used for this purpose. This measure ignores keywords ranked lower than K and computes the precision value over the top-K keywords on the list. Singhal et al. (Reference Singhal, Kasturi, Sharma and Srivastava2017) use the precision@K measure to compare the different approaches of keyword extraction for different values of K.

Mean average precision (MAP) is another information retrieval-based measure that can be used for evaluating a list of ranked keywords. MAP is the average of precision values at K, where K takes on the ranks at which correct keywords are returned. Both precision@K and MAP have been exploited by Jiang, Hu, and Li (Reference Jiang, Hu and Li2009) for evaluating their ranking approach of keyword extraction.

Mean reciprocal rank (MRR) (Voorhees Reference Voorhees1999) and binary preference measure (Bpref) (Buckley and Voorhees Reference Buckley and Voorhees2004) are two other measures that can be exploited for evaluating ranked keywords (Liu et al. Reference Liu, Huang, Zheng and Sun2010). MRR evaluates how the first correct keyword of a document is ranked in the list of the keywords extracted by a method. This measure is computed using Equation (4), where D is the set of documents and rank _d denotes the rank of the first correct keyword of d.

(4)\begin{equation} MRR=\frac{1}{|D|}\sum_{d \in D} \frac{1}{rank_d}\end{equation}

Bpref evaluates the portion of the bad keywords, which are ranked higher than the good ones. More specifically, it is calculated using Equation (5), where M is the number of all the extracted keywords, in which r is a correct keyword and n is an incorrect one, and R shows the list of all the correct keywords within M.

(5)\begin{equation} Bpref = \frac{1}{|R|} \sum_{r \in R} 1 - \frac{|{{n}\; {\rm ranked}\; {\rm higher}\; {\rm than} {r}}|}{M}\end{equation}

Considering the ‘exact match’ between the gold standard keywords and the extracted ones has limitations as it fails to detect two similar but morphologically variant keywords. In order to tackle this problem, Zesch and Gurevych (Reference Zesch and Gurevych2009) proposed an approximate matching strategy, which deals with such variations between keywords. These authors believe that this way, keyword extraction approaches can be evaluated in a more precise way.

4.3 Benchmarks

The cost of collecting training or evaluation data motivates the publication of data sets as public benchmarks, which can be used for comparing approaches or evaluating them against state-of-the-art methods. For instance, in 2010 and 2017, SemEvalFootnote ^e proposes tracksFootnote ^f for keyword extraction. A specific experimental setting is defined. At the end, the participating systems are ranked based on their performance.

There are important features related to how keywords are assigned to documents including the number of annotators involved in the selection of the keywords, their degree of expertise, the guidelines they have to follow regarding the number of keywords to extract, and their length. These features have an impact on the reliability and homogeneity of the resulting keyword lists. In general, type of the assigned keywords can be divided into three main categories depending on the profile of people who provide them. Examples of different annotations have been shown in Figures 1 and 2. Although annotations assigned by authors and readers have been differentiated in Figure 1, we did not find such a distinction between the keywords assigned by readers and professional indexers in the SemEval 2017 data set.

• Author-assigned keywords are provided by the author(s) of documents that are aware of their domains and take them into account while annotating the documents. However, Caragea et al. (Reference Caragea, Bulgarov, Godea and Gollapalli2014) point out that in spite of the expertise, because they want to raise awareness on their paper, the authors may assign it several morphological variants of a keyword, even synonyms. These keywords could be subjective too: authors could be biased by the content of their paper while annotating it.

• Reader-assigned keywords are provided by the readers of documents, who can be of various ages and backgrounds. Some have more general knowledge about the domain of the studied document, whereas others may not be very familiar with the domain. Readers are mostly given some guidelines for annotating documents but the reader-assigned keywords could be nevertheless subjective: a reader might consider a keyword as representative of a document, while another reader may not see it as representative enough. In designing Nus benchmark, Nguyen and Kan (Reference Nguyen and Kan2007) reported the Kappa statistic inter-agreement between their two employed readers as 0.70, which is rather low.

  Comparing the assigned keywords in Figure 1, the reader-assigned ones are more likely to be selected from the content, while authors would probably choose a keyword that is morphologically different than the existing keywords in the text. Due to the knowledge that authors have about their documents, they can also assign more abstract keywords, which target the main points of the documents. Readers, on the other hand, assign more keywords that reflect different detailed points in the documents.

• Professional indexer-assigned keywords are provided by expert indexers, who mainly make use of a large controlled vocabulary for assigning keywords to documents. Here, the quality of the indexation depends on the knowledge of the thesaurus or the reference vocabulary. Comparing to the other kinds of assigned keywords, the ones by professional indexers are more objective, even if the subjectivity is never absent.

Figure 2.

Example of an annotated text in SemEval 2017 data set. Underlined keywords are those that occur in the text.

Annotation guidelines are different according to the target application and the requirements and they are mainly designed for annotators who are ‘readers’ of documents. As an example, in SemEval 2017 (Augenstein et al. Reference Augenstein, Das, Riedel, Vikraman and McCallum2017) some guidelines are given to the readers, which define the procedure and the rules of the annotation.Footnote ^g Figure 3 also shows the reader guideline provided by Sterckx et al. (Reference Sterckx, Demeester, Deleu and Develder2017). Clearly, the more detailed a guideline is, the more precise the annotations will be. Additionally, by having such guidelines, the level of disagreement between annotators would decrease.

Figure 3.

Annotation guideline designed by Sterckx et al. (Reference Sterckx, Demeester, Deleu and Develder2017).

Unlike reader-assigned keywords, commonly, there are no guidelines for authors of documents and they assign keywords to their texts liberally. Examples are the SemEval 2010 (Kim et al. Reference Kim, Medelyan, Kan and Baldwin2010) benchmark and also the one generated by Caragea et al. (Reference Caragea, Bulgarov, Godea and Gollapalli2014), in which all the scientific papers selected for the benchmarks already had some author-assigned keywords. Similarly, professional indexers, who are usually asked to perform very controlled tasks, are not provided with any general guidelines but they have to comply with many specific criteria while annotating documents.

Due to the difficulties in assigning reliable keywords to documents, in some works, more than one way of annotating the gold standard set is used. As an example, Kim et al. (Reference Kim, Medelyan, Kan and Baldwin2010) made use of both author- and reader-assigned keywords in generating the SemEval 2010 data set. Nevertheless, they found a degree of overlap between the two sets of keywords.Footnote ^h Augenstein et al. (Reference Augenstein, Das, Riedel, Vikraman and McCallum2017) also recruited both students (readers) and professional indexers for the annotation task. One paragraph was extracted from each article of the data set and two annotations were assigned to it: one by a reader and one by a professional indexer. In case of disagreement between the two annotators, the expert-assigned annotation was taken into consideration. The authors reported the inter-annotator agreement between the readers and the professional indexers in terms of Cohen’s kappa statistics, which ranges between 0.45 and 0.85. This shows that some readers are not qualified enough for annotating documents and their annotations are less reliable.

There are two categories of benchmarks for keyword extraction, depending on whether the keywords assigned to the documents are freely chosen or belong to a controlled vocabulary. Tables 2 and 3, respectively, give examples of widely used public benchmarks in each category. Among the presented benchmarks, Inspec (Hulth Reference Hulth2003) contains both uncontrolled and controlled keywords, where any suitable keyword and thesaurus unit can be assigned to the documents. The other vocabulary-based benchmarks presented in Table 3 and the CiteULike-180 benchmark in Table 2 have been developed by Medelyan (Reference Medelyan2009).

Table 2.

Examples of public free-text benchmarks

^a size of the training set: 144

^b size of the training set: 350

Table 3.

Examples of public vocabulary-based benchmarks

DEFT, which is a French scientific evaluation campaign, provides French benchmarks for keyword extraction task. More specifically, DEFT 2012 and DEFT 2016 focus on the task of evaluating keywords, extracted by different participant systems. In 2012, the training and the test corpora consisted of 234 full scientific papers published in journals of Humanities. To evaluate the different methods, their extracted keywords were compared with the author-assigned keywords. In 2016, the keywords extracted by participant systems were compared with the ones assigned by professional indexers. As in DEFT 2012, the documents of the corpus are all from the same type, that is scientific papers.Footnote ⁱ

It should be mentioned that benchmarks could also be generated from free available data. As an example, Bougouin, Boudin, and Daille (Reference Bougouin, Boudin and Daille2013) exploited the French version of WikiNews to generate their own reference corpus.

One should choose a benchmark carefully. The type of documents that compose the benchmark and their length are important features to take into account, as extractors are often designed for a specific type and length of document. Some benchmarks like Inspec are made of abstracts of scientific papers, while others are composed of longer documents or web pages.

Length of the extracted keywords mainly depends on the domain under study. According to Medelyan (Reference Medelyan2009), physics and medical terms are mostly longer than agricultural ones. This property, that is the length of keywords, is less affected by the type of annotators. As reported by Wan and Xiao (Reference Wan and Xiao2008), the average length of the keywords assigned by readers is 2.09. Caragea et al. (Reference Caragea, Bulgarov, Godea and Gollapalli2014) have also reported that almost half of their author-assigned keywords are bi-grams and they rarely appear as tri-grams or longer N-Grams.

In some benchmarks, the assigned keywords are found in the documents. Some other benchmarks, however, may contain keywords out of vocabulary of the studied documents and so the recall value of different approaches to keyword extraction can never reach 100% on these benchmarks. In this case, people mostly report the reachable recall value, computed over the keywords which are found in the studied documents. As an example, in Inspec, both controlled and uncontrolled keywords may or may not be in the studied abstracts. In SemEval 2010, the readers were asked to assign keywords only from the content of the documents. Analysing the test set keywords, assigned for 100 documents, showed that 15% of the keywords were not found in the texts. This value was, however, less than the one for author-assigned keywords (19%).

In real applications, one may aim to evaluate robustness of an approach on different types of documents within different domains. In this case, multiple benchmarks should be used to meet the requirements. New benchmarks should also be generated in order to cover more properties and languages. The existing ones are all a bit the same: most of them are in English, which is a problem in evaluating approaches on non-English documents; most of the public ones contain scientific papers and cannot be used for evaluating keywords extracted from web pages or commercial documents; most of the benchmarks also contain a single type of document all from the same domain. The large corpus proposed by Sterckx et al. (Reference Sterckx, Demeester, Deleu and Develder2017) is a noticeable exception as it combines different types of documents targeted for a ‘diverse and layman audience’, but it has nevertheless been designed for a specific application (metadata enrichment). Benchmarks provided in DEFT 2016 and SemEval 2017 are also examples of multi-domain benchmarks, which can be used to measure the robustness of keyword extraction approaches along different domains.Footnote ^j

5.1 Morpho-syntactic features

Extraction tools exploit morphological and syntactical features of textual units. All types of words and word sequences do not have the same probability to be selected as keywords. For instance, some parts of speech, such as nouns and adjectives, are more likely to appear in keywords than others, such as adverbs and determiners, as they provide more information about the text under study. As an example, Mihalcea and Tarau (Reference Mihalcea and Tarau2004) showed that nouns and adjectives are two important parts of speech tags commonly used when extracting keywords from abstracts of scientific papers.

Extraction methods rely not only on plain words (tokens) but also on their lemmatized forms, their parts of speech (POS tag), and some of their morphological features, such as gender or number (singular, plural). See Table 4 for an example. When the extracted keywords are to be presented to human users, they are mainly retrained without any inflection so as to satisfy the citationess property.

Table 4.

Morpho-syntactic feature values associated with the word ‘Cities’

It is also important to take into account the syntactic structure of word sequences, as the sequences that do not correspond to well-formed syntactic phrases are usually discarded from the keyword candidate lists. Hulth (Reference Hulth2003) was the first who studied the impact of linguistic information in keyword extraction task. This study showed that having noun phrase chunks as candidate keywords favours precision, while extracting words or phrases with some other pre-defined POS patterns improves the recall value of the extracted keywords. Analysing the annotations on the SemEval 2017 benchmark also shows that 93% of the assigned keywords by readers and professional indexers are noun phrases. Examples of the sequences of POS tags used by Hulth (Reference Hulth2003) are as follows:

• Adjective Noun (singular or mass)

• Noun Noun (both singular or mass)

• Adjective Noun (plural)

• Noun (singular or mass) Noun (plural).

Morpho-syntactic features could also be used as a type of filtering at the end of keyword extraction to verify if the extracted keywords have valid linguistic structures.

Although morpho-syntactic features are language-dependent, they are available for a large family of languages for which reliable morpho-syntactic taggers exist. Advanced syntactic features, which require the parsing of the source text, are less widely available as the performance of parsers varies greatly from one language to another. The performance of those tools often decline on texts which contain many technical or out-of-vocabulary tokens or on non-standard language.

5.2 Statistical features

Statistical features were introduced to IR in the 1970s (Salton, Yang, and Yu Reference Salton, Yang and Yu1975). These features are widely used on large corpora, even if rarely in isolation: they are mainly language and domain independent and most of them are easy to compute, even on big data.

5.2.1 Frequency-based features

Term frequency (TF) (Luhn Reference Luhn1957) is a very low-level statistical feature, which is widely used in keyword extraction task (Ohsawa, Benson, and Yachida Reference Ohsawa, Benson and Yachida1998; Turney Reference Turney2000; Hulth Reference Hulth2003; Yih et al. Reference Yih, Goodman and Carvalho2006; Rose et al. Reference Rose, Engel, Cramer, Cowley, Berry and Kogan2010; Sterckx et al. Reference Sterckx, Caragea, Demeester and Develder2016). The assumption behind this feature is that the more representative a term is in a text, the more frequent it appears in it. Of course, this feature is both more reliable and more useful when processing long documents than short ones.

Since TF strongly correlates with the size of documents, one usually considers a normalized frequency (NF). Equations (2) and (3) show two traditional formulae of NF for a given text. One limitation of TF feature is that it does not distinguish grammatical or common words from content ones as the former are usually highly frequent.

(6)\begin{equation*}NF_{w_i}= \frac{\# \: of \: occ. \: of \: w_i}{Total \: \# \: of \: word \: occ.} \end{equation*}

(7)\begin{equation*} NF_{w_i}= \frac{\# \: of \: occ. \: of \: w_i}{\# \: of \: occ. \: of \: the \: most \: frequent \: word} \end{equation*}

Inverse document frequency (IDF) (Sparck Jones Reference Sparck Jones1972) is a quantity borrowed from IR. It is defined by Equation (8), where the document frequency (DF) of a term corresponds to the number of documents in a target collection in which it occurs. The IDF is lower for the common terms that appear in many documents of the collection and higher for those which have a low DF. This measure provides a valuable indication of the specificity of a term in relation to a document but using IDF requires a collection to which the document can be confronted.

(8)\begin{equation} IDF_{w_i}= log\bigg(\frac{1}{DF_{w_i}}\bigg)=log\bigg(\frac{Number \: of \: documents \: in \: the \: collection}{Number \: of \: documents \: in \:which \: w_i \: occurs}\bigg)\end{equation}

This statistical measure has been adapted to different types of documents. Examples are inverse tweet frequency (De Maio et al. Reference De Maio, Fenza, Loia and Parente2016) and inverse webpage frequency (Chung, Chen, and Nunamaker Reference Chung, Chen and Nunamaker2003), which respectively compute IDF on tweets and web pages.

Term and document frequencies are traditionally combined in the TF-IDF feature (Equation (9)), which is widely used in IR and keyword extraction (Salton et al. Reference Salton, Yang and Yu1975; Medelyan and Witten Reference Medelyan and Witten2006; Wan and Xiao Reference Wan and Xiao2008; Zhang et al. Reference Zhang, Wang, Liu, Wu, Liao and Wang2008; Lopez and Romary Reference Lopez and Romary2010; Caragea et al. Reference Caragea, Bulgarov, Godea and Gollapalli2014). The IDF factor tends to counterbalance the high TF value of common terms present in most documents and to increase the weight of words that appear rarely in the rest of the collection. This way, it favours both representativity and specificity properties of the extracted keywords.

(9)\begin{equation} TF-IDF_{w_i}= TF_{w_i}*IDF_{w_i}\end{equation}

In the literature, there are more variations of the TF-IDF score. Term frequency-inverse sentence frequency (Martins et al. Reference Martins, Pardo, Espina and Rino2001) is one example, which is computed by replacing ‘document’ with ‘sentence’ in Equations (8) and (9). The goal is to compute the score of a term in a document according to its frequency and its distribution in the sentences of that document. Another well-known variation is Okapi BM25 (Robertson et al. Reference Robertson, Walker, Jones, Hancock-Beaulieu and Gatford1996), which improves TF-IDF by taking into account the length of documents.

Specificity and conformity of keywords within a target domain can also be captured through the domain relevance (DR) score proposed by Navigli and Velardi (Reference Navigli and Velardi2002). The DR of a term is high if it appears frequently in the target domain and rarely in other domains. The formula for computing the DR of term t is presented in Equation (10), where D _i is the target domain, containing a set of documents, and D ₁, …, D _i−1, D _i+1, …, D _N represent other domains.

(10)\begin{equation} DR_{D_i}(t) = \frac{freq(t,D_i)}{\displaystyle\max_{j}( freq(t,D_j))}\end{equation}

5.2.2 Co-occurrence-based features

Word co-occurrence is a statistical feature designed to catch keywords representativity and exhaustivity. The basic idea behind using this feature is to capture words which tend to appear together within a given type of context. The basic formula is given by Equation (11),Footnote ^k where c is a type of context.

(11)\begin{equation} Cooc_c(w_i, w_j)= \# \: of \: contexts \: c \: in \: which \: w_i \: and \: w_j \: co-occur\end{equation}

Co-occurrence features mainly differ by the type of context which is considered for the co-occurrences, that is the context in which two words are considered as co-occurrent. The computing complexity of the extraction process increases with the size of the context. Momtazi, Khudanpur, and Klakow (Reference Momtazi, Khudanpur and Klakow2010) list four categories of term co-occurrence features that we present below.

In sentence-wise co-occurrence, such as the one used by Matsuo and Ishizuka (Reference Matsuo and Ishizuka2003) and Palshikar (Reference Palshikar2007), only words occurring in the same sentence are said to be co-occurrent (c = sentence) but it is often less costly to take a smaller and a fixed context into account. In window-wise co-occurrence, a sliding window with a fixed size is set as an input parameter (c = window of n words). Any time two words appear in the same window, one co-occurrence is counted. Of course, it is also possible to restrict to well-formed syntactical phrases to avoid accidental co-occurrences of unrelated terms (c = well-formed phrase) but syntax-wise co-occurrence requires the initial chunking or parsing of the text. TextRank (Mihalcea and Tarau Reference Mihalcea and Tarau2004) is one of the state-of-the-art approaches, which makes use of window-wise co-occurrence. The authors performed experiments on the size of windows being from 2 to 10 words. They, however, reported a higher precision and F-measure values for 2-word window, indicating that long windows may fail to capture the words relation effectively. In the same way, when the whole document is taken into account as the context, the extracted pairs of words are not as strongly associated as in the previous cases. Document-wise co-occurrence gives weak semantic association of words. The feature is nonetheless useful as an indication of the semantic similarity of words.

5.2.3 Similarity-based features

Redundant keywords are not interesting for users in real applications and approaches must ensure that their extracted keywords are diverse enough. Computing similarity between the extracted candidate keywords helps to detect the redundant ones and to satisfy the minimality property of the extracted keywords.

The proposed approaches for measuring the similarity of words are often based on the comparison of word vectors, which show the distributions of words within a corpus (Harris Reference Harris1954; Hindle Reference Hindle1990). Cosine similarity, Dice coefficient, and Jaccard index (Manning and Schütze Reference Manning and Schütze1990) are examples of these measures. Considering A and B as two word vectors, Equations (12), (13), and (14), respectively, show the formulae for computing these measures.

(12)\begin{equation} Cosine(A,B) = \frac{A.B}{||A||_2 ||B||_2}\end{equation}

(13)\begin{equation} Dice(A,B) = \frac{2|A\cap B|}{|A|+|B|}\end{equation}

(14)\begin{equation} Jaccard(A,B)=\frac{|A \cap B|}{|A \cup B|}\end{equation}

Nowadays, similarity between two words or phrases is also computed using a more advanced statistical measure: word embedding (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013). As an example, Bennani-Smires et al. (Reference Bennani-Smires, Musat, Hossmann, Baeriswyl and Jaggi2018) exploit word embeddings to remove the redundant keywords extracted by their approach.

It should be noted that some approaches overcome the redundancy problem by grouping keywords into different clusters, where each cluster represents highly similar (overlapped) keywords (Boudin Reference Boudin2013; Bougouin et al. Reference Bougouin, Boudin and Daille2013). The highest ranked keyword in each cluster is then selected as its representative, whereas the other keywords are considered as the redundant ones.

In addition to the redundancy issue, similarity-based features can be exploited in keyword extraction approaches to capture the relation between words and to detect the representative ones. Turney (Reference Turney2003) captures the words association using the pointwise mutual information computed on Web data. Topic modelling techniques are also widely used for capturing semantic relationship using a corpus of documents. As an example, Liu et al. (Reference Liu, Huang, Zheng and Sun2010) use the latent Dirichlet allocation (LDA) model (Blei, Boudin, and Daille 2003) to find the topical relationship between words, using Wikipedia as the source of information.

5.3 Informational features

Informational or textual features are language and domain independent. They exploit the information clues that authors use to bring attention to important points in their texts. In fact, these features assume that words or phrases with such author-assigned information are more likely to be representative of a studied document. We distinguish four types of informational features, respectively, based on typography, document structure, keyword position within the source documents, and keyword length.

Examples of typographical features are underlined, bold, italicized, and capitalized elements but words and keywords can also be highlighted using quotation marks. Any type of typographical emphasis can be exploited to spot the most relevant keywords. Even if the productivity of these features depends on the type and typographical convention of the source text, they are quite reliable when available. As an example, Yih et al. (Reference Yih, Goodman and Carvalho2006) considered capitalized words to be important ones and took advantage of this assumption to extract advertising keywords from web pages.

If the source text is structured, one can exploit the fact that some specific text areas are more informative than others. This is obviously the case not only for titles and abstract sections but also for link anchors and some captions. In web pages, more important words are usually put in titles and meta descriptions (SEOmoz 2012). Lopez and Romary (Reference Lopez and Romary2010) also consider title, abstract, introduction, section title, conclusion, and reference sections of a scientific paper as its informative parts, which are likely to contain keywords.

The position of the keyword occurrences within the source text – ‘spatial use of the words’ for Herrera and Pury (Reference Herrera and Pury2008) – can also be analysed as an indication of the importance of the studied terms. In academic documents, terms that appear at the beginning or at the end of a chapter are expected to be more informative than the others.

Informational features can be modelled as Boolean values (presence/absence of the term in a given area) or as scalar ones (position of the first occurrence of the term or the average position of the term occurrences), possibly normalized with respect to the length of the text. In any case, informational features are used to measure the representativity of the terms.

The length of keywords may also give information about them. In some approaches, keywords with less than three characters are assumed uninformative and so are filtered out from the list of candidate terms. An example is the pre-processing performed by Turney (Reference Turney2000), where words with less than three characters are removed. On the other hand, some approaches may assume that longer keywords contain more specific information.

Examples of approaches which exploit these informational features are listed in Table 5. The ‘positional’ property in the table indicates the position of the first occurrence of the keywords in the studied document. According to the provided examples, this property is the most widely used among the informational features. Unlike structural property, positional property does not depend on the type of the studied document. Hence, it can be used in a more generic way for targeting various types of input documents.

Table 5.

Examples of approaches exploiting informational features (F)

^a Capitalization

^b Occurrence in ‘anchor text’, ‘Meta’ tags, ‘title’ tag, ‘URL’

^c Occurrence in ‘title’, ‘abstract’, ‘body (full-text)’, ‘heading’, ‘first paragraph’, ‘last paragraph’, ‘references’

^d Occurrence in ‘title’, ‘abstract’, ‘introduction’, ‘at least one section titles’, ‘conclusion’, ‘at least one reference or book title’

5.4 Resource-based features

The quality of the extracted keywords can also be measured with respect to an external semantic resource, such as a dictionary or a thesaurus, which provides additional information on the studied words. Due to the dependency on external resources, these features are considered as domain and language-dependent features.

One can exploit an existing gold standard set to spot the word and phrase occurrences in the source text. This amounts to keyword identification rather than extraction but such a dictionary-based validation can be exploited to assess representativity and univocity of the candidate keywords within the studied domain. In addition, these features can evaluate the conformity of the candidate keywords with respect to the domain-specific terms. In the case of document annotation for the evaluation task, exploiting such resources reduces the judgement or subjectivity of the annotators and as a result satisfies the impartiality property of the assigned keywords.

Structured semantic resources also help to identify semantic relationships, such as groups of synonyms or topic-based clusters, assuming that related terms are more likely to be important than isolated ones. As an example, Wang, Liu, and Wang (Reference Wang, Liu and Wang2007) exploit the WordNet database (Miller et al. Reference Miller, Beckwith and Ch1990) for capturing the semantic relationship between words. Budanitsky and Hirst (Reference Budanitsky and Hirst2001) give a review on the semantic-based measures, which compute the word relationship using the WordNet semantic network.

Resource-based features are also exploited to assess the specificity of the extracted keywords. Yih et al. (Reference Yih, Goodman and Carvalho2006) use query logs as a resource and assess the relevance of keywords depending on how frequent they are in queries. A domain-dependent thesaurus was also exploited by Medelyan and Witten (Reference Medelyan and Witten2006) for keyword indexing task.

5.5 Conclusion on extraction features

A long list of features has been tested for keyword extraction and only the main categories are presented here. Table 6 presents the categories of features used by some of the approaches introduced in this survey, taking all their steps, including the pre-processing, into consideration.

Table 6.

Categories of features exploited in example approaches

According to Table 6, statistical features can be considered as the elementary features in extraction approaches. Any new approach to keyword extraction exploits these features in order to take advantage of the basic properties of words within the studied document. Among the statistical features, the frequency-based ones are the most exploited and can be considered as a ‘must’ feature for developing a new approach to keyword extraction. More specifically, in the case of having a corpus of documents, TF-IDF can effectively capture representativity and specificity. However, if such a corpus is not available, TF has been shown to perform effectively enough for extracting representative candidate keywords.

Morpho-syntactic features are mainly to eliminate ill-formed candidate keywords and linguistically uninformative ones. Although they are not used as frequently as the statistical features, many works exploit them as a basic feature to reduce the number of candidate keywords in the extraction task. Moreover, for many types of input documents, such as scientific papers and web pages, the positional feature has been shown to be effective for the keyword extraction task. On the contrary, resource-based features are not frequently exploited. As a matter of fact, finding a relevant resource can be challenging and for some domains such a resource may not be available. In addition, domain-independent approaches are often preferred and the resource-based feature does not meet that requirement.

As shown in Table 6, the presented features are mainly used in combination and not always on an explicit basis. In fact, most of these features have been introduced on an empirical basis to improve the quality of the extracted keywords and it is often afterwards that they have been related to semantic properties and justified.

6.1 Supervised methods

Keyword extraction can be regarded as a classification problem, each candidate keyword being labelled as either a keyword or a non-keyword. In the case of using a probabilistic model, probability of a candidate keyword being a keyword is returned. Supervised methods take training data as input and rely on training features for training a classifier. These features are the ones which capture the keyness properties of words and phrases (see Section 5). The trained classifier is then applied to documents for which keywords are aimed to be extracted. As will be seen in the following, the supervised approaches differ in the features used for training classifiers and the types of their classifier(s).

Supervised approach to keyword extraction was first proposed by Turney, who tried two classifiers : (1) C4.5 decision tree (Quinlan Reference Quinlan1993), with 12 statistical and morpho-syntactic features, and (2) GenEx algorithm, itself based on Extractor (Turney Reference Turney2000), a keyphrase extraction algorithm that also exploits a combination of 12 statistical and morpho-syntactic features. Results show that GenEx outperforms C4.5. One drawback of the approach is that words with less than three characters are dropped as uninteresting words, which rules out most of the units and abbreviations.

Various improvements have been proposed on Turney’s approach. The Keyphrase Extraction Algorithm (KEA),Footnote ^l proposed by Frank et al. (Reference Frank, Paynter, Witten, Gutwin and Nevill-Manning1999), is based on a Naïve Bayes classifier and uses a smaller set of features, both statistical and informational. More specifically, this algorithm performs a pre-processing on the studied document: splitting the text into phrases based on a set of delimiters and removing non-alphanumeric characters and also all numbers. It then takes phrases up to three tokens as candidate keywords. Ill-formed or uninformative phrases, which either start or end with a stop word or consist only of a proper noun, are discarded. KEA then trains a Naïve Bayes classifier using two features: TF-IDF and the position of the first occurrence of the phrase within the studied document. On a new document, the trained classifier returns the probability of each extracted phrase being a keyword, based on its features.

Turney (Reference Turney2003) improved KEA by increasing the coherence of the extracted keywords using statistical associations between them; using a rule induction approach, Hulth (Reference Hulth2003) showed that exploiting morpho-syntactic features improves the performance of the previously proposed supervised machine learning approaches; KEAWeb (Kelleher and Luz Reference Kelleher, Luz, Kaelbling and Saffiotti2005) exploits hyperlink structure to improve the keywords extracted by KEA; and KEA++ (Medelyan and Witten Reference Medelyan and Witten2006) uses the semantic information of a domain-specific thesaurus to overcome the limitation due to word synonymy.

Many different machine learning approaches have been tested for extracting keywords: Zhang et al. (Reference Zhang, Xu, Tang and Li2006) applied support vector machine (SVM); Yih et al. (Reference Yih, Goodman and Carvalho2006) trained a classifier using logistic regression algorithm, while Caragea et al. (Reference Caragea, Bulgarov, Godea and Gollapalli2014) and also Bulgarov and Caragea (Reference Bulgarov and Caragea2015) used a Naïve Base classifier; and Sarkar, Nasipur, and Ghose (Reference Sarkar, Nasipuri and Ghose2010) extracted keywords from scientific articles using multi-layer perceptron neural network. According to the results, these approaches outperform KEA algorithm. Augenstein et al. (Reference Augenstein, Das, Riedel, Vikraman and McCallum2017) also confirm these trends.

Zhang et al. (Reference Zhang, Wang, Liu, Wu, Liao and Wang2008) were the first to consider keyword extraction as a string labelling problem and used the conditional random fields (CRFs) model to extract keywords. These authors exploited statistical and morpho-syntactic features and showed that the CRF model outperforms other machine learning methods such as SVM and multiple linear regression model.

Recently, recurrent neural network (RNN) models are exploited in keyword extraction domain. As an example, Zhang et al. (Reference Zhang, Wang, Gong and Huang2016) proposed a joint-layer RNN model to overcome the limitation of short content in tweets. Tsujimura, Miwa, and Sasaki (Reference Tsujimura, Miwa and Sasaki2017) also exploited a neural network-based system for extracting entities and their relations. The authors showed the positive impact of word embeddings on the performance of their neural model. The word embeddings have also been exploited by Meng et al. (Reference Meng, Zhao, Han, He, Brusilovsky and Chi2017), who aim at inferring ‘absent keywords’ rather than only extracting the existing ones in the studied document. According to the analysis in their work, absent keywords, which do not exist as such in the source document, appear often in common benchmarks.Footnote ^m Hence, predictive approaches of keyword extraction could positively affect the experimental results on these benchmarks.

Supervised machine learning approaches have promising performance in extracting keywords but the training data requirement is a limitation. When no ‘naturally’ annotated data are available, it must be generated manually for evaluation, and this is not a trivial task. It is not even possible to generate a training set for all types of sources: according to Chen et al. (Reference Chen, Li, Liu, Chen, Bouguettaya and Lin2009), it is impossible to collect a large enough training data set for all types of web content, which hinders the ability to use supervised approaches for extracting keywords from web documents. Sterckx et al. (Reference Sterckx, Caragea, Demeester and Develder2016) also discuss the difficulty of having reliable training data and its consequences on supervised approaches. The authors overcome this limitation by treating the supervised keyword extraction task as positive unlabelled learning, where non-selected phrases are considered as unlabelled and not as negative examples in the training set in order to model uncertainty in the set.

Among the approaches presented in Table 6, the supervised ones widely use statistical and informational features and this can be an indication that both features perform effectively in this category of approaches. Some approaches also make use of morpho-syntactic features by giving a higher importance to nouns. As mentioned before, using external resources can limit the applicability of the approaches. Consequently, resource-based features are not used very often in the supervised methods.

6.2 Unsupervised methods

As explained before, having a reliable and a rich training data set is challenging in supervised approaches. To circumvent the use of training data, unsupervised approaches have also been tested, considering keyword extraction as a ranking problem. In these approaches, candidate keywords are scored using different kinds of techniques.

6.2.3 Graph-based methods

A wide range of unsupervised keyword extraction approaches is graph-based. In these approaches, the goal is to take into account the connectivity of words and to capture the ‘centrality’ of keywords. As stated by Beliga, Mes̆trović, and Martins̆ić-Ips̆ić (Reference Beliga, Meštrović and Martins̆ić-Ips̆ić2015), in the graph theory, ‘centrality’ indicates the importance of vertices within a graph.Footnote ^o Hence, it reflects the representativeness of the vertices and thus the representativity of the keywords.

The overall approach consists of generating a graph of elements and using it to cluster or to rank those elements.

Graph generation. Depending on the type of analysis and the goal of extraction, the generated graphs can be directed or undirected, and weighted or unweighted.

Connectivity can be measured locally, at the document level, or globally on a collection of documents. In the first case, the idea is to exploit the relation and the connectivity of the words within a single document as proposed by Ohsawa et al. (Reference Ohsawa, Benson and Yachida1998). However, more distant cross-document relationships can be exploited. Some works make use of both local and global context to extract keywords of a studied document: the results of Wan and Xiao (Reference Wan and Xiao2008) show that adding global context increases the performance of the proposed approach.

Graph-based approaches also differ in the selection of vertices and edges. Approaches with different goals use different units as vertices of the graph and also various relations and metrics for generating the edges.

Most often, vertices correspond to candidate single or compound keywords of a given document or collection of documents. The edges reflect the semantic relation of the candidates, which is often measured through co-occurrence (two words are connected in a document graph if they co-occur within a certain window in that document). This type of graphs has been popularized by TextRank (Mihalcea and Tarau Reference Mihalcea and Tarau2004) but various algorithms have taken that idea since then. Alternative metrics can also be used to capture the semantic relationship. For instance, Grineva et al. (Reference Grineva, Grinev and Lizorkin2009) rely on a semantic relationship derived from Wikipedia, and Huang et al. (Reference Huang, Tian, Zhou, Ling and Huang2006) exploit syntactic relations.

A marginal approach considers documents as vertices and relies on the connectivity between documents (Kelleher and Luz Reference Kelleher, Luz, Kaelbling and Saffiotti2005). The goal is to capture the intertextuality expressed through the hyperlinks of web documents, in legal document networks for Mimouni, Nazarenko, and Salotti (Reference Mimouni, Nazarenko and Salotti2015) or in chats for Abilhoa and de Castro (Reference Abilhoa and de Castro2014). The underlying idea is that documents related to a document d within a corpus provide additional information for identifying relevant keywords for d.

A more recent approach considers graphs of topics rather than graphs of words or of documents: TopicRank algorithm (Bougouin et al. Reference Bougouin, Boudin and Daille2013) has been proposed as an improvement over TextRank.

Graph-based analysis. Different unsupervised analyses can be performed on the generated graph to get a list of keywords. They are mainly based on clustering and/or ranking algorithms.

Clustering algorithms are used to cluster the nodes of the graph, each cluster corresponding to a set of variant keywords or a group of semantically related ones. For instance, Ohsawa et al. (Reference Ohsawa, Benson and Yachida1998) show that clustering a co-occurrence graph outperforms both TF-IDF and N-Gram approaches for indexing. Grineva et al. (Reference Grineva, Grinev and Lizorkin2009) apply community detection techniques on a weighted semantic graph to identify topically related terms and to rule out unimportant ones.

The most common approach relies on ranking algorithms and aims at ranking the vertices of keyword graph using the global information recursively computed from the entire graph. This approach was initially proposed for TextRank (Mihalcea and Tarau Reference Mihalcea and Tarau2004) and was shown to outperform that of Hulth (Reference Hulth2003) in terms of precision and F-measure. More specifically, TextRank starts by generating a word graph, where words are restricted to nouns and adjectives within the source document. It then uses PageRank algorithm (Brin and Page Reference Brin and Page1998) for ranking the nodes of the graph, but the authors claimed that other algorithms, such as HITS (Kleinberg Reference Kleinberg1999), can also be applied. After selecting the highest ranked nodes, the ones that appear adjacent in the source document are merged to generate compound keywords.

Many variant approaches have been proposed as extensions of TextRank: TimedTextRank (Wan Reference Wan2007) uses a temporal dimension in order to deal with evolving topics. Wan and Xiao (Reference Wan and Xiao2008) also extended TextRank by proposing SingleRank and ExpandRank. Unlike the unweighted graph in TextRank, in these extensions, the degree of co-occurrence between any two nodes of the graph is represented as the weight of the corresponding edge. While in SingleRank nodes of the graph are scored based on the local information in the studied document, ExpandRank uses a collection of nearest neighbour documents to provide a global knowledge and to improve the effectiveness of the keyword extraction task.

A group of unsupervised approaches extended TextRank by incorporating topical information of the documents under study. Liu et al. (Reference Liu, Huang, Zheng and Sun2010) proposed Topical PageRank (TPR), which first detects the topics of the studied document and returns its word topic distributions using the LDA model (Blei et al. 2003). Using this topical information, TPR then detects keywords of different topics and so achieves the maximum coverage over the studied document by taking all its topics into consideration.

Bougouin et al. (Reference Bougouin, Boudin and Daille2013) also proposed TopicRank: a graph of topics,Footnote ^p which aims at ranking topics rather than words. In their approach, keywords are selected from the top ranked topics so as to better target the main topics of the studied document. A comparison over the effectiveness of TopicRank with respect to the state-of-the-art approaches has been presented by Bougouin et al. (Reference Bougouin, Boudin and Daille2013) (Table 7). On all the benchmarks, except for Inspec, TopicRank outperforms the other two graph-based approaches. It, however, performs comparable to the statistical TF-IDF approach.

Table 7.

Comparison over graph-based and statistical approaches on the existing benchmarks in terms of F-measure values (Bougouin et al. Reference Bougouin, Boudin and Daille2013)

Zhang, Huang, and Peng (Reference Zhang, Huang and Peng2013) also proposed a topical graph-based approach, where words are assumed to have different relations in different topics. Similar to the approach of Li et al. (2010), the latent topics of a document are detected using the LDA model. These authors then proposed a measure for capturing word relatedness, which relies on their relation in different topics.

Structural properties of graphs are also exploited for clustering and/or ranking. The notions of connectedness and betweenness centrality were used by Huang et al. (Reference Huang, Tian, Zhou, Ling and Huang2006) and several authors consider that co-occurrence graphs are similar to small-world graphs (Matsuo, Ohsawa, and Ishizuka Reference Matsuo, Ohsawa and Ishizuka2001). Boudin (Reference Boudin2013) compared the efficiency of the PageRank measure with other centrality measures on an undirected and weighted word co-occurrence graph. Results showed that the degree centrality performs comparable to the PageRank but the closeness centrality has the best performance on short documents. Lahiri, Choudhury, and Caragea (Reference Lahiri, Choudhury and Caragea2014) extended this analysis to directed word collocation and noun phrase collocation networks. These authors showed that some other centrality measures, such as degree and strength, perform very similarly, or slightly better, when compared with PageRank and are much less computationally expensive. Centrality was, for instance, used by Abilhoa and de Castro (Reference Abilhoa and de Castro2014), who generated undirected and weighted/unweighted graphs from tweet messages in order to extract their keywords.

6.3 Conclusion on extraction methods

Most of the methods proposed for extracting keywords fall into one of the above categories but even in a given category, they often differ in the features they rely on and in the type and number of keywords that they aim at extracting.

Each method has its own strengths and domain of application but recent works show that combining different types of methods and features leads to more generic and performant keyword extraction approaches. Danesh, Sumner, and Martin (Reference Danesh, Sumner, Martin, Palmer, Boleda and Rosso2015), for instance, report promising results with an unsupervised method that combines N-Grams candidate generation approach with various ranking steps, respectively, based on traditional statistical features, the position of the first occurrences and a co-occurrence graph. Nevertheless, obtaining a high value of F-measure in the task of keyword extraction remains challenging. According to the results reported in SemEval 2010 (Kim et al. Reference Kim, Medelyan, Kan and Baldwin2010), the state-of-the-art performance in terms of F-measure value is mostly 20–30%. F-measure score reported by Hasan and Ng (Reference Hasan and Ng2014) is also another indication of the difficulty of the keyword extraction task. In that study, the best F-measure score was achieved on a dataset of abstracts (45.7%), while on a dataset of papers, the authors reported a score of 27.5%. More recent results, obtained in SemEval 2017 (Augenstein et al. Reference Augenstein, Das, Riedel, Vikraman and McCallum2017), also confirm the difficulty of keyword identification, reporting the highest achieved F-measure to be 56%.

Supervised methods show a good performance in state-of-the-art approaches. The combination of different extraction features is an effective way of capturing the importance of words in documents and supervised methods make these combinations easy to test. While choosing a supervised approach, however, one needs to ensure the availability of rich training data, which sometimes needs to be purposefully generated.

The complexity of the supervised approaches depends on the chosen classifier. For instance, the Naïve Bayes classifier used by Frank et al. (Reference Frank, Paynter, Witten, Gutwin and Nevill-Manning1999) has been shown to be faster than the one proposed by Turney (Reference Turney2000). The choice of the extraction features is also another important factor in determining the effectiveness of the approaches. As an example, the approach of Yih et al. (Reference Yih, Goodman and Carvalho2006) performed better than KEA (Frank et al. Reference Frank, Paynter, Witten, Gutwin and Nevill-Manning1999) thanks to the exploitation of more diverse extraction features. However, a trade-off has to be found between performance and complexity, when using a wide range of features.

According to the results reported in state-of-the-art approaches, when a corpus of annotated data is available, supervised approaches can be a good choice. However, since such data may not be available for a specific domain or language and generating it could be costly, unsupervised approaches are becoming popular. As an example, Litvak and Last (Reference Litvak and Last2008) showed that their supervised graph-based approach outperforms the unsupervised one in terms of precision if a large enough training data set is available. These authors, however, concluded that in the case of a lack of such training data, the unsupervised graph-based approach is a better choice for extracting keywords.

It should be noted that although unsupervised approaches do not have the limitations of the supervised ones, they mostly do not perform as effectively as them and there are still opportunities for further enhancements. In the category of unsupervised methods, the purely statistical ones are quite old and among them, TF-IDF has been widely used as an evaluation baseline. Recent unsupervised methods mainly combine the statistical approaches with other types of approaches, such as the graph-based ones.

Graph-based methods have been widely used since the mid-2000s due to the advantages of ‘unsupervised’ approaches. With certain combinations of extraction features, their performance is comparable to those of supervised approaches. As an example, Mihalcea and Tarau (Reference Mihalcea and Tarau2004) obtained higher values of precision and F-measure compared to the approach proposed by Hulth (Reference Hulth2003). The recall value was, however, higher in the supervised approach.

According to the reported results in this paper, the TF-IDF approach is comparable to some graph-based approaches on specific benchmarks. We note that in spite of the performance, graph-based approaches capture different keyness properties than the TF-IDF. While TF-IDF focuses on the specificity and representativity of each extracted keyword, graph-based approaches also take the exhaustivity of a set of keywords into consideration. Hence, the latter approaches could be more effective in some applications, such as text summarization, where all the topics of a document must be covered by the extracted keywords.

Our review of the state-of-the-art approaches shows that entropic methods are not as frequently exploited as the others, neither as an extraction method nor as a baseline.