3.1.1 Unsupervised lexical statistics techniques

Considered to be one of the earliest LBD techniques, the unsupervised lexical statistics techniques (Gordon & Lindsay, Reference Gordon and Lindsay1996; Lindsay & Gordon, Reference Lindsay and Gordon1999; Gordon et al., Reference Gordon, Lindsay and Fan2002) rely on word count statistics, as illustrated in Figure 4. To perform the ABC discovery model, the steps on the gray plate are repeated N times, where N denotes the number of intermediate term B connecting the source term A to target term C. The effectiveness of this technique is determined by the extent to which the desired intermediate terms and target terms (assumed to be known in advance) rank highly in the list of candidate terms.

Figure 4

The general methodology of the unsupervised lexical statistics technique proposed by Gordon and Lindsay (Reference Gordon and Lindsay1996) and Lindsay and Gordon (Reference Lindsay and Gordon1999). The dotted line marks the system’s boundary.

To replicate Swanson’s magnesium–migraine (MM) discovery (Swanson, Reference Swanson1988), the technique first downloaded bibliographic records containing the source term ‘migraine’. The records included titles, abstracts, and subject descriptors. N-gramsFootnote ³ were subsequently extracted from these records, following prior word-stemming and stopwords removalFootnote ⁴ . Note that these N-grams should not be words that typically characterize the source literature (e.g. the word ‘migraine’ or other closely associated terms), nor should they include very general terms. Finally, for each extracted N-gram, a number of lexical statistics scores were calculated in order to determine and rank its importance in bridging A and C. The used scores included token frequency (tf), document frequency, relative frequency, and frequency×inverse document frequency (tf×idf) (Gordon & Lindsay, Reference Gordon and Lindsay1996; Lindsay & Gordon, Reference Lindsay and Gordon1999).

The unsupervised lexical statistic techniques offer some benefits in terms of their high intuitiveness and ease of computation. However, given their reliance on using lexical statistical measures that tend to favor frequently co-occurring terms, they may miss other interesting associations originating from less frequent terms (Kostoff et al., Reference Kostoff, Block, Solka, Briggs, Rushenberg, Stump, Johnson, Lyons and Wyatt2009; Petrič et al., Reference Petrič, Cestnik, Lavrač and Urbančič2010). Not only that, the success of this technique greatly depends on the user’s prior knowledge and inherent bias. For example, in the MM discovery, Lindsay & Gordon (Reference Lindsay and Gordon1999) removed a number of highly ranked intermediate terms given the authors’ foreknowledge that the terms would not eventually lead to the desired target term ‘magnesium’.

3.1.2 Supervised ensemble statistics technique

The second category of the statistical approach is the supervised ensemble statistics technique. This technique derives a single score by learning a number of different weighted features from a text corpus. As shown in Figure 5, this learning step is facilitated with the help of logistics regression algorithm, after which the learned score is used for deciding the relevance and the interestingness of various intermediate terms that connect source term A and target term C (Torvik & Smalheiser, Reference Torvik and Smalheiser2007).

Figure 5

The procedure for learning feature weights using the supervised ensemble statistics technique (Torvik & Smalheiser, Reference Torvik and Smalheiser2007). LBD=literature-based discovery

In contrast to the previous unsupervised lexical statistics approach, this technique utilizes a machine learning technique to determine the most relevant intermediate terms. Hence, it requires less human intervention and therefore less prone to the user bias. There are limitations, however. Since the choice of the learned features was arbitrary, there is also the possibility that other useful yet unexplored features might have been overlooked (Torvik & Smalheiser, Reference Torvik and Smalheiser2007). More importantly, this technique requires that a large number of high-quality gold standards be available to adequately train the model, which is still a challenging research problem (Smalheiser, Reference Smalheiser2012).

3.1.3 Distributional semantics techniques

Unlike the previous two approaches, distributional semantic technique uses scalable algorithms to build representation of terms based on the patterns of their occurrence in natural language texts (Symonds et al., Reference Symonds, Bruza and Sitbon2014). At the heart of this technique is the assumption that two terms are semantically related if they appear in a similar context (Salton & McGill, Reference Salton and McGill1986; Symonds et al., Reference Symonds, Bruza and Sitbon2014). The context of a term is defined as a vector representation of other terms that co-occur with it. As a result, two semantically related terms are expected to exhibit similar vector representations. Figure 6 depicts the overall methodology of this technique.

Figure 6

The methodology of the distributional semantics technique (Gordon & Dumais, Reference Gordon and Dumais1998)

Gordon and Dumais (Reference Gordon and Dumais1998) applied the Latent Semantic Indexing (LSI) algorithm (Deerwester et al., Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990) that is capable of directly computing the semantic similarity between a source term and a target term even if both terms never appear together. The technique also eliminates the need to generate and evaluate large numbers of intermediate terms. Related to Gordon and Dumais (Reference Gordon and Dumais1998), Cohen et al. (Reference Cohen, Schvaneveldt and Widdows2010) introduced a distributional semantics technique called the Reflective Random Indexing (RRI), which scaled better than LSI for processing very large corpora.

Besides its ability to eliminate the need to generate and evaluate a large number of intermediate terms from the ABC discovery model, the distributional semantics technique has better scalability, up to a million MEDLINE records (Cohen et al., Reference Cohen, Schvaneveldt and Widdows2010). Furthermore, these techniques can be applied to perform LBD on non-English documents (Symonds et al., Reference Symonds, Bruza and Sitbon2014).

There are limitations. LSI experienced difficulties in identifying the desired target terms when evaluated on Swanson’s DFORS hypothesis (Gordon & Dumais, Reference Gordon and Dumais1998). Although the performance reportedly improved when the RRI model was used (Cohen et al., Reference Cohen, Schvaneveldt and Widdows2010), it required that certain intermediate terms (e.g. ‘platelet’) be used to guide the search for the correct target term. Moreover, when used to analyze MEDLINE records, the techniques’ specificity and precision only improved under the condition that the Medical Subject Headings (MeSH) terms were supplied to the algorithm (Cohen et al., Reference Cohen, Schvaneveldt and Widdows2010, Reference Cohen, Widdows, Schvaneveldt, Davies and Rindflesch2012). Lastly, since documents are modeled as bags-of-words, this technique is oblivious to the positional and sectional information of terms, making it difficult to interpret the LBD results (Cohen & Hersh, Reference Cohen and Hersh2005).

In addition to the three subcategories of statistical LBD approach discussed above, other types of statistical measures have been explored. They include term frequency-based measures (Gordon & Lindsay, Reference Gordon and Lindsay1996; Lindsay & Gordon, Reference Lindsay and Gordon1999), association rules (Hristovski et al., Reference Hristovski, Džeroski, Peterlin and Rožić2000; Pratt & Yetisgen-Yildiz, Reference Pratt and Yetisgen-Yildiz2003), fuzzy binary relations (Perez-Iratxeta et al., Reference Perez-Iratxeta, Bork and Andrade2002), mutual information measure (Wren, Reference Wren2004), z-score (Yetisgen-Yildiz, Reference Yetisgen-Yildiz2006; Yetisgen-Yildiz & Pratt, Reference Yetisgen-Yildiz and Pratt2006), and R-scaled score (Frijters et al., Reference Frijters, Van Vugt, Smeets, Van Schaik, De Vlieg and Alkema2010).

3.2.1 Semantic filtering technique

The semantic filtering technique maps natural language text to specific biomedical concepts using NLP tools, such as MetaMap Footnote ⁵ . The Figure 7 illustrates the steps in this technique. The aim is to obtain intermediate terms which satisfy a set of user’s predefined semantic types, thus reducing the total number of intermediate concepts that need to be evaluated by the user.

Figure 7

The steps involved in the semantic filtering technique (Weeber et al., Reference Weeber, Klein, de Jong-van den Berg and Vos2001). The dotted line marks the system boundary

Unlike previously described lexical statistics technique, this technique relies on semantic type filtering to provide its accuracy. Users significantly influence the discovery process through the formulation of appropriate semantic filters. But the technique has a drawback in that it requires non-trivial amount of domain knowledge to select the most effective semantic filters. Furthermore, it assumes that the actual target terms are known in advance. Unfortunately, for many real world scenarios it may not be possible to determine these target terms from the outset of a discovery process (Weeber et al., Reference Weeber, Klein, de Jong-van den Berg and Vos2001).

3.2.2 Semantic profiling technique

Srinivasan (Reference Srinivasan2004) introduced topic profile (TP), which is a vector of semantic type vectors. Figure 8 describes in detail the technique’s algorithm for solving the open discovery problem.

Figure 8

The steps used by the profiling technique (Srinivasan, Reference Srinivasan2004). MeSH=Medical Subject Headings; tf=token frequency; idf=inverse document frequency

A semantic type vector is an unordered set of weighted MeSH terms that belong to a specific semantic type. Since there are 134 semantic types in the UMLS (Unified Medical Language System) ontology, a TP may be made up of up to 134 semantic type vectors. Equation (1) illustrates how a TP _i may look like. In this formula, w _i ,134,1 corresponds to the weight of the MeSH term m. The MeSH term m _134,1, in turn, belongs to the 134th semantic type (Srinivasan, Reference Srinivasan2004):

$$TP_{i} \,{\equals}\,\left\{ {\left\{ {w_{{i,1,1}} m_{{1,1}} ,w_{{i,1,2}} m_{{1,2}} ,\!\ldots\!\!} \right\},\!\ldots\!,\left\{ {w_{{i,134,1}} m_{{134,1}} ,w_{{i,134,2}} m_{{134,2}} ,\!\ldots}\!\! \right\}} \right\}$$

In this technique, the MeSH weight is calculated as a tf−idf score (Salton & McGill, Reference Salton and McGill1986), such that a TP represents the relative importance of various semantic types in the text based on the frequency of occurrence of their component MeSH terms. Similar to the TP technique, van Haagen et al. (Reference van Haagen, AC’t Hoen, Bovo, de Morrée, van Mulligen, Chichester, Kors, den Dunnen, van Ommen, van der Maarel and Kern2009) later proposed concept profile, which was capable of measuring the similarity between two types protein concepts in the literature.

In contrast to the semantic filtering technique (Weeber et al., Reference Weeber, Klein, de Jong-van den Berg and Vos2001), the advantage of semantic profiling technique is obvious: it provides a way to assign the relative importance of a semantic type based on the accumulative weights of its component MeSH terms (Srinivasan, Reference Srinivasan2004). Depending on the nature of an LBD task, this allows one to prioritize certain semantic types over the others. Unfortunately, as suggested in Figure 8, this technique is computationally expensive as it requires generating and evaluating every single intermediate term in order to arrive at the correct target terms. Like many other knowledge-based techniques, the success of this technique also relies on incorporating the information from various domain-specific knowledge bases, for example, MeSH biomedical vocabularies.

3.2.3 Semantic pattern techniques

These techniques use semantic predications extracted from natural language texts. Hristovski et al. (Reference Hristovski, Friedman, Rindflesch and Peterlin2006) introduced discovery pattern, a set of conditions in the form of a subject–predicate–object-like structure known as predication. These pattern-like conditions guide the evaluation of candidate novel associations generated by the LBD system. Figure 9 illustrates the process.

Figure 9

The steps involved in the semantic pattern technique (Hristovski et al., Reference Hristovski, Friedman, Rindflesch and Peterlin2006)

To replicate Swanson’s DFORS hypothesis, Hristovski et al. searched for complementary predications in MEDLINE texts that conform to a certain discovery pattern. For example, the sentence ‘local increase of blood viscosity during cold-induced Raynaud’s phenomenon’ was parsed to produce predication ASSOCIATED_WITH (Raynaud’s, blood viscosity, increase). In the same manner, the sentence ‘a statistically significant reduction in whole blood viscosity was observed at seven weeks in those patients receiving the eicosapentaenoic acid rich oil’ was parsed to obtain predication ASSOCIATED_WITH (eicosapentaenoic acid, blood viscosity, decrease). By assembling these two complementary predications, the user may hypothesize that, since eicosapentaenoic acid (i.e. dietary fish oil) decreases blood viscosity and high blood viscosity is observed among most Raynaud’s patients, it can then be inferred that regularly consuming eicosapentaenoic acid may alleviate Raynaud’s syndrome.

Developing from Hristovski et al.’s discovery pattern model, the Predication-Based Semantic Indexing model represented concepts and their relationships as vectors in a hyperdimensional space (Cohen et al., Reference Cohen, Widdows, Schvaneveldt, Davies and Rindflesch2012). In this case, finding a discovery pattern is viewed as a geometrical function in a hyperdimensional space that is solved by tracing certain predication pathways.

The main strength of the semantic pattern techniques is their ability to better interpret the nature of the associations among concepts in the literature (Hristovski et al., Reference Hristovski, Friedman, Rindflesch and Peterlin2006). As a result, they may be used to figure out more complex hidden associations better than the lexical statistics approach (Kraines et al., Reference Kraines, Guo, Hoshiyama, Makino, Mizutani, Okuda, Shidahara and Takagi2010). And given that the associations among semantic types are explicitly represented in the form of predications, it is easy for the user to verify the plausibility of associations between a source and a target concept.

There are drawbacks, though. First, scalability remains an issue. The technique requires two distinct stages in order to extract the semantic predications: an initial stage where the semantic predications are extracted for the source concept and each of the intermediate concepts, and the second stage where semantic predications must be extracted for each intermediate concept and the target concept. Since the number of intermediate concepts tends to grow exponentially (Wren, Reference Wren2008), these two-staged procedures almost certainly become a bottleneck in the algorithm (Cameron et al., Reference Cameron, Bodenreider, Yalamanchili, Danh, Vallabhaneni, Thirunarayan, Sheth and Rindflesch2013). Other limitations include the technique’s requirement that the user be able to select the most promising intermediate and target concepts, as well as its reliance on the availability of third-party NLP software to enable the extraction of predications from natural language texts (Hristovski et al., Reference Hristovski, Friedman, Rindflesch and Peterlin2006).

3.3.1 Discovery browsing techniques

The discovery browsing technique allows users to navigate through the complex relationships among biomedical concepts, using a combination of semantic predications and graph-based methods (Wilkowski et al., Reference Wilkowski, Fiszman, Miller, Hristovski, Arabandi, Rosemblat and Rindflesch2011; Goodwin et al., Reference Goodwin, Cohen and Rindflesch2012). It extended the discovery pattern technique proposed earlier by Hristovski et al. (Reference Hristovski, Friedman, Rindflesch and Peterlin2006) by allowing a serial chain of multiple intermediate B terms between a source concept A to a target concept C, such that A↔(B ₁ ↔B ₂ ↔B ₃ ↔ … B_n )↔C. This contrasts with the simplistic ABC discovery model where only a single intermediate concept B is assumed between A and C.

At the initial stage, the discovery browsing technique generated a semantic predication graph. Concepts were modeled as nodes and the semantic relations between them were represented as the edges between the nodes. The graph was then built iteratively, using a user-specified concept as seed. From here, the algorithm searched a biomedical predication database for all predications containing the given seed concept as their subject and/or object. Once found, these predications were subsequently used to further grow the graph until the user noticed interesting patterns in the graph. Figure 10 explains this methodology.

Figure 10

The steps involved in the discovery browsing technique (Wilkowski et al., Reference Wilkowski, Fiszman, Miller, Hristovski, Arabandi, Rosemblat and Rindflesch2011)

We provide an example of paths extracted from a semantic predication graph in Figure 11. An edge between two nodes may represent one or more semantic relations between two concepts, denoted by the number label appearing on each edge. Each path is ranked according to its degree centrality score, which is the sum of degree centrality scores of all nodes that make up the path. Paths with high scores are considered interesting on the assumption that they connect many important nodes. In this example, a top ranking path of length 4 is the [Melatonin]↔[Interleukin-1β]↔[Glutamate]↔[CLOCK]↔[Serotonin] path, which suggests an association between the sleep hormone Melatonin and Serotonin. Serotonin is a neurotransmitter known to regulate human mood (Wilkowski et al., Reference Wilkowski, Fiszman, Miller, Hristovski, Arabandi, Rosemblat and Rindflesch2011).

Figure 11

A partial representation of the discovery browsing graph. The image is adapted from Wilkowski et al. (Reference Wilkowski, Fiszman, Miller, Hristovski, Arabandi, Rosemblat and Rindflesch2011)

The main strength of the discovery browsing technique is its ability to allow users to control the growth of the semantic predication graphs (Wilkowski et al., Reference Wilkowski, Fiszman, Miller, Hristovski, Arabandi, Rosemblat and Rindflesch2011), such as fine-tuning the discovery process to avoid overwhelming computing resources. The technique also leverages the existing graph-based algorithms to automatically measure the interestingness of discovery patterns, exemplified by the usage of a node’s degree centrality (Freeman, Reference Freeman1978). Unfortunately, this technique favored two concepts that co-occur 10 times or more in MEDLINE records during the process of extracting the semantic predication. Although applying such threshold may be useful for controlling the maximum number of predications being generated, it may also result in an LBD system incapable of finding rarer yet valuable associations.

4.1.1 Performance evaluation

The current technique was first applied to replicate Swanson’s DFORS discovery in the closed discovery model (Cameron et al., Reference Cameron, Bodenreider, Yalamanchili, Danh, Vallabhaneni, Thirunarayan, Sheth and Rindflesch2013). Out of a total of 2124 associations in which the concept ‘fish oil’ served as the root of the paths, it found that 14 associations containing the ‘Raynaud’s’ concept as their terminal. On the other hand, among the 17 848 associations where the ‘Eicosapentaenoic acid’ concept served as the root concept, 172 associations had the ‘Raynaud’s’ concept as their terminal. In both scenarios, associations that were relevant to the reconstruction of Swanson’s discovery had to be manually selected by the user.

Improving the previous technique, Cameron et al. (Reference Cameron, Kavuluru, Rindflesch, Sheth, Thirunarayan and Bodenreider2015) introduced an automatic way to extract the relevant paths from a predication graph and construct the corresponding context-based subgraphs from these paths. When used to find relevant intermediate terms connecting fish oil and Raynaud’s Syndrome, the technique successfully recovered blood viscosity and platelet aggregation terms but missed vascular reactivity (Cameron et al., Reference Cameron, Kavuluru, Rindflesch, Sheth, Thirunarayan and Bodenreider2015). This is a limitation because the traditional semantic profiling technique by Srinivasan and Libbus (Reference Srinivasan and Libbus2004) had previously managed to recover all three intermediate terms. Likewise, for reconstructing the migraine and magnesium hypothesis (Swanson, Reference Swanson1988), the technique only recovered seven out of 11 possible latent associations between migraine and magnesium (Cameron et al., Reference Cameron, Kavuluru, Rindflesch, Sheth, Thirunarayan and Bodenreider2015). In contrast, Srinivasan and Libbus (Reference Srinivasan and Libbus2004) managed to recover 10 associations.

4.1.2 Strengths and limitations

The main strength of the current technique is its ability to automatically extract thematic subgraphs. These subgraphs make it possible for users to interpret the meaning of semantic predication paths given a highly specific context. As such, the model affords much greater explanatory power compared to other previous LBD techniques. Although it recovered fewer number of intermediate associations than Srinivasan and Libbus (Reference Srinivasan and Libbus2004), this model requires less prior knowledge than the latter, making less prone to user bias.

There are some limitations. Users are still required to manually set the relatedness threshold values and the semantic predication filters. The technique also relies on the availability of MeSH descriptors in order to define the context of each subgraph. It is unclear how the model would have performed in the absence of these descriptors. Lastly, similar to Hristovski et al. (Reference Hristovski, Friedman, Rindflesch and Peterlin2006), this technique relied on third-party NLP tools to extract the semantic predications. Consequently, any lack of accuracy on the part of these tools may result in the technique’s missing important latent associations (Cameron et al., Reference Cameron, Kavuluru, Rindflesch, Sheth, Thirunarayan and Bodenreider2015).

4.2.1 Performance evaluation

Two performance evaluations were conducted to determine the effectiveness of this technique (Kostoff, Reference Kostoff2014). The first evaluation looks at the ability of the technique to uncover the underlying themes connecting the PD and CD literature. At first, the hierarchical clustering software CLUTOFootnote ⁸ was used to cluster papers that corresponded to the shared references between the PD and CD literatures. As a result, records of the same research theme were clustered together. For each research theme, a factor analysis was applied to identify significant phrases in the titles of the shared references that best represent that theme.

In the second evaluation, common phrases in the title or abstract of PD and CD literatures and in the title of their shared references were identified. Subsequently, the author determines whether there were novel linkages established through the phrases in the shared references, which are not found among in the title or abstract of PD and CD papers.

This technique revealed many promising concepts from the titles of the shared references between PD and CD papers (Kostoff, Reference Kostoff2014). Specifically, it discovered three main underlying themes connecting the PD and CD literature, namely genetics, neuroimmunology, and cell death theme. It also found 1226 phrases in the titles of the shared references, of which seven new associations were considered meaningful: (1) anthocyanins, (2) wogonin, baicalin, baicalein; Scutella rivularis extracts, (3) trichothecenes, (4) pyroptosis, (5) adalimumab, (6) cooked foods, and (7) ippases.

4.2.2 Strengths and limitations

The results above strongly indicate that the shared references between two disjoint literatures could harbor many useful linking terms (Kostoff, Reference Kostoff2014). It also suggests that a structural LBD technique that is based on bibliographic coupling, when combined with content-based analysis, could be useful in the selection of potential discovery links (Kostoff, Reference Kostoff2012). This technique exemplifies how an emerging LBD technique begins to adopt various techniques developed in other research fields, such as computational linguistics and data mining. In this case, it uses an NLP technique to extract phrases from the titles and abstracts of records obtained from the Science Citation Index (SCI) and Social Science Citation Index (SSCI)Footnote ⁹ , with the help of a partitional hierarchical clustering algorithm to find out the thematic structures of all references shared between PD and CD literatures.

The technique’s current limitation is its laborious procedure and lack of automation (Kostoff, Reference Kostoff2014). Manual analysis needs to be performed to identify potential intermediate terms from thousands of candidates. Not only that, substantial domain expertise was needed to assess the merit of each intermediate connection between PD and CD.

4.3.1 Performance evaluation

As many as 1630 clusters were identified from the citation network of sustainability science literature and 151 clusters from the complex network literature (Fujita, Reference Fujita2012). In total, 22 largest clusters were selected from the former and nine largest clusters were likewise selected from the latter. Of these, three of the most textually similar pairs of clusters were chosen. Finally, the top ranking shared terms between these pairs of clusters were inspected by a domain expert to infer the possible hidden associations between the two research fields (Fujita, Reference Fujita2012).

Akin to Fujita, Ittipanuvat et al. (Reference Ittipanuvat, Fujita, Sakata and Kajikawa2014) selected pairs of textually similar clusters from the robotics and gerontology literatures. They found the cosine similarity score of tf−idf vectors to be the best performing lexical statistics for identifying related clusters of papers, in comparison to other measures such as Jaccard index, Dice coefficient, and Inclusion index (Manning et al., Reference Manning, Raghavan and Schütze2008).

4.3.2 Strengths and limitations

Several cluster analytic methods have been previously proposed to discover associations between disconnected literatures. Stegmann and Grohmann (Reference Stegmann and Grohmann2003) and Yamamoto and Takagi (Reference Yamamoto and Takagi2007) used co-word analysis technique (Callon et al., Reference Callon, Courtial, Turner and Bauin1983) to form clusters of biomedical articles in MEDLINE based on the term co-occurrence in their titles, abstracts, publication dates, and MeSH keywords. It was found that terms that linked the disconnected literatures occupied the regions of below-median centrality and density of the clusters (Stegmann & Grohmann, Reference Stegmann and Grohmann2003). These methods, however, built clusters merely from term co-occurrence data and did not consider clusters formed from citation structures.

Fujita’s model is different from the techniques described above. It uses citation analysis coupled with an advanced community detection algorithm instead of term co-occurrence. It also demonstrates that LBD can be successfully performed on non-biomedical research papers. Prior to this, only a few examples of non-biomedical LBD applications exist: water purification technology (Kostoff et al., Reference Kostoff, Solka, Rushenberg and Wyatt2008), computer science (Gordon et al., Reference Gordon, Lindsay and Fan2002), chemistry (Valdés-Pérez, Reference Valdés-Pérez1999), and humanities (Cory, Reference Cory1997).

Having said so, the current technique is limited by the use of only one type of bibliographic link, that is direct citation link. Studies in scientometrics have shown that the relationships between documents can be represented by more than one types of citation links, for example, bibliographic coupling and co-citation links (Janssens et al., Reference Janssens, Glänzel and De Moor2008; Boyack & Klavans, Reference Boyack and Klavans2010; Waltman & Eck, Reference Waltman and Eck2012; Boyack et al., Reference Boyack, Small and Klavans2013). Therefore, it is possible that better discovery capabilities and more accurate cluster link prediction results can be attained through combined applications of diverse types of bibliographic links.

4.4.1 Performance evaluation

The results showed that the features, especially centrality measures, can be used to predict the existing drug–disease interactions for Metformin in the CTD database with certain amount of success (Ding et al., Reference Ding, Song, Han, Yu, Yan, Lin and Chambers2013). Out of 697 diseases ranked in CTD for this drug, 16 matches were recovered using the in-degree centrality feature. Three of the matches were among the top 10 diseases ranked in CTD for Metformin. Other features such as the out-degree centrality features recovered 16 disease matches (lowest rank: 439th), the closeness centrality feature found 13 matches (lowest rank: 439th), and the betweenness centrality feature found 17 matches (lowest rank: 439th).

4.4.2 Strengths and limitations

This model is interesting as it represents a step toward a highly seamless integration of scientometrics techniques, knowledge-based techniques, and network analysis in a single LBD framework. Unlike Fujita’s technique that combines the citation and lexical analysis in two consecutive but separate stages (Fujita, Reference Fujita2012), entitymetrics integrates knowledge-based entities (drugs, diseases, genes) with bibliographic entities (papers) in the same bio-entity citation network. This is advantageous because various network-based features can then be computed from the same network. Furthermore, given the large volume of research papers it was capable of analyzing, this technique suggests the promising scalability of a network-based LBD technique.

The performance evaluation results of this technique seemed to suggest that it suffers from a low recall rate with the ability to only recover less than 20 diseases for Metformin out of nearly a total of 700 diseases ranked for it in the CTD database (Ding et al., Reference Ding, Song, Han, Yu, Yan, Lin and Chambers2013). Due to its strong coupling with domain-specific knowledge bases, such as HUGO, CTD, and DrugBank, the effectiveness of this technique may only be limited to situations where such resources are available.

4.5.1 Performance evaluation

BIOMINE demonstrated a good accuracy in predicting novel disease–gene associations (Eronen & Toivonen, Reference Eronen and Toivonen2012). For task that predicts future protein interactions in the Entrez database, the random walk feature emerged as the best predictor with the area under curve equals to 0.82. For the disease–gene prediction task, the random walk feature also emerged as the most predictive feature.

Further, using the random walk feature as the node proximity measure, the performance of three classifiers were compared in predicting disease–gene associations: a supervised classifier, the K-Nearest Neighbour (KNN) algorithm, and a cluster-based classifier (Eronen & Toivonen, Reference Eronen and Toivonen2012). The results showed that the supervised classifier and the cluster-based classifier outperforming the KNN classifier.

4.5.2 Strengths and limitations

The most salient characteristic of this technique is its utilization of a heterogeneous biological graph. The information richness of this graph allows one to generate a strongly predictive feature, such as the random walk feature. The technique was also scalable to a database with 1.1 million concepts and 8.1 million relations (Eronen & Toivonen, Reference Eronen and Toivonen2012). More importantly, this technique used both supervised and unsupervised machine learning algorithms, where the encouraging performance of the random walk feature could motivate further explorations into the use of machine learning techniques in future LBD systems.

In view of a general purpose LBD system, this technique is limited in that it is specifically tailored for discovery tasks in genomics and proteomics. It also appears to heavily depend on the availability of certain knowledge bases to operate successfully. For instance, the reliability score can only be calculated on if the reliability values are available from the STRING database. For domains where such databases do not exist, this technique may not be directly applicable.

4.6.1 Performance evaluation

The model assumes the existence of a certain discovery paper, such as Swanson (Reference Swanson1986a, Reference Swanson1988). A discovery paper provides the starting point for this model to retrospectively reconstruct a LBD scenario. For example, to reconstruct Swanson’s DFORS discovery, the technique started by downloading a total of 485 core papers along with their abstracts (352 on fish oil; 133 on Raynaud’s syndrome) from Thomson Reuter’s Web of Science Footnote ¹⁶ . These core papers were identified using the same search keywords originally used by Swanson (Reference Swanson1986a) and they were to be published prior to Swanson’s discovery, that is between 1900 and 1985. A corresponding HBIN graph was then constructed using these bibliographic data, where each edge in the HBIN graph was weighted according to a specific weighting score (Sebastian et al., Reference Sebastian, Siew and Orimaye2015). Based on these scores, 87 different metapath features were then computed from the HBIN graph.

This technique views link prediction in LBD as akin to solving a multiclass classification problem. The performance of the learned metapath features was studied using five popular machine learning classifiers: the Sequential Minimal Optimization (SMO) variant of the Support Vector Machine algorithm, Neural Networks, Naive Bayes, Bayesian Network, and C4.5 Decision Tree (Witten & Frank, Reference Witten and Frank2005). To construct the training and test sets, the technique collected 117 370 unique core paper pairs from the 485 core papers retrieved previously. It then applied an exhaustive algorithm to search for all distinct metapaths between all unique core paper pairs. To assign a class label to each instance, the technique retrieved another set of fish oil and Raynaud’s syndrome records from the Web of Science, including papers published in 1986 following Swanson’s discovery. A pair of core papers is labeled as class +1 if it consisted of a pair of fish oil paper and Raynaud’s Syndrome paper that had no co-citation link before 1986 but which became co-cited in 1986. A pair was labeled as −1 to represent papers from the same research area (i.e. either fish oil or Raynaud’s syndrome literature) that became co-cited in 1986. Lastly, a pair was labeled as 0 if they never became co-cited.

In reconstructing the DFORS discovery, the HBIN metapath features performed very well in predicting future inter-cluster co-citation links (+1) between fish oil and Raynaud’s syndrome papers, with 0.851 F-Measure (precision: 0.845, recall: 0.857). This performance was better than the performance of other document similarity features such as bibliographic coupling similarity (0.612 F-Measure), LDA topic similarity (0.608 F-Measure), and TF–IDF similarity (0.457 F-Measure) (Sebastian et al., Reference Sebastian, Siew and Orimaye2015). Further, the authors found that using features constructed from the four-degree author-sharing and term-sharing metapaths alone produced a better predictive performance (71.40 and 70.64% accuracy rate, respectively).

In a subsequent experiment to reconstruct Swanson’s migraine and magnesium discovery (Swanson, Reference Swanson1988), the HBIN model also performed well in predicting co-citation links between migraine and magnesium literatures, with 0.80 F-Measure (precision: 0.80, recall: 0.80) (Sebastian et al., Reference Sebastian, Siew and Orimaye2017). Importantly, the study found that the predictive accuracy of the model improved when it incorporated elements of topic modeling and word sense disambiguation techniques. This outcome is consistent with Preiss and Stevenson (Reference Preiss, Stevenson and Gaizauskas2016), who have observed that using appropriate word sense disambiguation could positively enhance the performance of an LBD system.

4.6.2 Strengths and limitations

Unlike BIOMINE and other LBD techniques, HBIN’s efficacy does not depend on the availability domain-specific knowledge-based sources. In the absence of these resources, it managed to perform considerably well in predicting which papers will form future co-citation linkages. This is probably because the HBIN allows much richer information to be exploited by the LBD algorithm compared to if a homogeneous network is used (Sun & Han, Reference Sun and Han2012). The meta-structure of the HBIN graph (i.e. the interconnections between bibliographic entities) could reveal the latent semantic relationships between papers from two separate literature.

Another distinct feature of this technique is its seamless integration of both lexical and citation information into a single unified graphical representations. This approach overcomes the limitations of a purely citation analysis method, such as bibliographic coupling (Kessler, Reference Kessler1963; Kostoff, Reference Kostoff2014), as well as the limitations of purely traditional lexical analysis techniques, such as latent Dirichlet allocation (Blei et al., Reference Blei, Ng and Jordan2003) and tf−idf (Salton & McGill, Reference Salton and McGill1986). Citation analysis is known to yield high precision but low recall retrieval whereas lexical analysis tends to give high recall and low precision results (Bassecoulard & Zitt, Reference Bassecoulard and Zitt2004). By incorporating both citation information and lexical information in the HBIN metapath features, the more balanced precision and recall rates could be achieved.

There are several limitations of the current model. It has not completely addressed the scalability issue of the algorithm, especially its path enumeration function (Sebastian et al., Reference Sebastian, Siew and Orimaye2017). As such, it may be difficult to extend the current model implementation on a much larger data set. Lastly, the model did not perform author name disambiguation when constructing the HBIN graphs. The extent to which this deficiency affects the model’s overall accuracy remains unknown.

4.7.1 Scalable machine learning-driven literature-based discovery techniques

In the past, most LBD techniques have been developed by information scientists, digital librarians, medical researchers, and biologists (Weeber et al., Reference Weeber, Kors and Mons2005; Bekhuis, Reference Bekhuis2006; Jensen et al., Reference Jensen, Saric and Bork2006; Mostafa et al., Reference Mostafa, Seki and Ke2009; Hahn et al., Reference Hahn, Cohen, Garten and Shah2012). As a consequence, most LBD systems tend to be biased toward domain-oriented solutions, especially biomedical applications. Notably, they required extensive incorporation of domain background knowledge, as evident from the rampant usage of UMLS vocabularies, biological databases, and specially designed NLP tools by the majority of traditional LBD techniques. As a result, their applications are limited to certain fields only (Marsi et al., Reference Marsi, Oztürk, Aamot, Sizov and Ardelan2014).

It is likely that future research trend could see more machine learning-oriented techniques capable of automatically learning useful features from a collection of literature. The learned features would be used to predict the existence of hidden connections between disjoint research areas, even in the absence of available background knowledge. In terms of scale, given recent advances in Big Data technologies, future LBD techniques may also incorporate algorithms that scale well against very large data sets. This trend would also reduce LBD systems’ reliance on heuristics-based rules and domain-specific knowledge bases, making it easier to generalize them to relevant applications in various research fields.

4.7.2 Network-centric algorithms

Since the relationships among papers can naturally be modeled as interconnected nodes in complex networks (Newman, Reference Newman2001), the current developments in community detection and network data mining research may inform the future designs of LBD systems (Newman, Reference Newman2003; Leskovec et al., Reference Leskovec, Kleinberg and Faloutsos2005, Reference Leskovec, Lang and Mahoney2010). As a result of this network-centric view, there will be an increasing number of link prediction-oriented LBD algorithms, such as a recent algorithm for predicting co-occurrence associations in the MeSH co-occurrence network (Kastrin et al., Reference Kastrin, Rindflesch and Hristovski2013). Next generation LBD approaches are also likely to integrate network analytic algorithms in their algorithms (Leskovec et al., Reference Leskovec, Lang and Mahoney2010), especially given the availability of existing open source network analytics packages such as GraphX Footnote ¹⁷ , NetworkX Footnote ¹⁸ , and the Stanford Network Analysis Project Footnote ¹⁹ .

4.7.3 Better visualizations

Future LBD research may also integrate more advanced visualization methods to support LBD activities (van Mulligen et al., Reference van Mulligen, van Der Eijk, Kors, Schijvenaars and Mons2002; Wilkowski et al., Reference Wilkowski, Fiszman, Miller, Hristovski, Arabandi, Rosemblat and Rindflesch2011; Goodwin et al., Reference Goodwin, Cohen and Rindflesch2012; Cameron et al., Reference Cameron, Bodenreider, Yalamanchili, Danh, Vallabhaneni, Thirunarayan, Sheth and Rindflesch2013). Smalheiser (Reference Smalheiser2012) argued that the success of LBD systems should be measured based on how well they support researchers in their daily scientific endeavors. Therefore, there would be needs to create semi-automatic, highly usable, and visually attractive LBD systems that would integrate seamlessly with the users’ workflows.

4.7.4 Automatic identification of evaluation gold standards

In terms of new evaluation gold standards, a possible future trajectory may see new methods capable of automatically collecting instances of scientific discoveries that have exhibited strong inter-cluster linkage properties (Chen et al., Reference Chen, Chen, Horowitz, Hou, Liu and Pellegrino2009; Chen, Reference Chen2012). In addition to minimizing the subjective element in the current gold standard selection process, this would reduce the costs associated with using domain expert evaluation. Furthermore, to address the difficulties in finding good samples of discovery papers, researchers have the option to collect evaluation data sets based on real world discoveries in fields such as medicine or physics. For example, the Journal of American Medical Association has identified 51 landmark medical papers (Meyer & Lundberg, Reference Meyer and Lundberg1985), whereas the Physical Review Letters has recognized 83 milestone physical papersFootnote ²⁰ . Given their lasting contributions of these selected papers to their fields, future studies may focus on understanding the extent to which these papers may serve as good evaluation ground truth for LBD systems.