3.1 Three different representations of a mention’s context

The context of a mention is basically the content that resides at the left- and right-hand side of the mention. These are called left and right contexts, respectively. In this research, we represent this context in three different ways. Figure 4 shows a document at the left-hand side where three entity mentions are underlined. At its right, it shows the three different ways of representing the context specific to the mention “Democratic Party.” For that particular mention, the dimmed sections are not part of the used context.

Figure 4.

A sample document and the different types of context used for the named entity mention of “Democratic Party.’’

Word-based context (WC) is a traditional context in the form of a sequence of words adjacent to the mention at its left- and right-hand side. This context has a local viewpoint, since only the words of the sentence that holds the mention are used. This is shown in the second box of Figure 4, where the words in the same sentence with the mention are kept as the WC. Surface form-based context (SFC) keeps the surface forms of the mentions at the left- and right-hand side of each mention, excluding all other words in between as shown in the third box. Compared to WC, this includes words farther than the mention into the context. They reflect the topic of the document better than the other words. Entity-based context (EC) only consists of entity ids surrounding the mention, excluding all the words as shown in the fourth box. Considering the fact that cooccurring entities are related and consistent with the topic of the document, EC also reflects the topic of the document. Both SFC and EC present a global viewpoint at the document level compared to the local viewpoint of WC that is mostly based on the sentence level. Having said that, cluster-based types generated with WC-type context may act more like traditional named entity types, as the surrounding words might reflect their semantic roles better. On the other hand, cluster-based types based on SFC and EC may act more like topic labels.

3.2 Obtaining entity embeddings

We obtain four different embeddings for each entity in KB by using the Wikipedia articles as training data. Three of those embeddings are based on the three different ways of representing the context of the named entity mentions in text. The fourth embedding is based on the synsets (or types in BaseKB terminology) in the YAGO and BaseKB data sets that are associated with each entity. Figure 5 overviews the process.

Figure 5.

Steps of obtaining entity embeddings by using different types of context.

WC-based entity embeddings ( $E_{WC}$ ) are obtained by using the word2vec tool (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado and Dean2013) on the Wikipedia articles. Word2vec gets the input as a sequence of tokens and calculates an embedding for each token (i.e., target token) in the given input based on its window of surrounding tokens at the left- and right-hand sides. Note that word2vec does not use the sentence boundaries as context boundaries as in the definition of WC. Instead, it uses a window of words around the mention. In order to obtain the embeddings for the entities, we reformat the Wikipedia articles based on the EAD–KB approach (Zwicklbauer et al. Reference Zwicklbauer, Seifert and Granitzer2016b). We replace the hyperlinked mention (i.e., a-href tag and the surface form together) of each named entity with its Wikipedia id, which is a single unique token that corresponds to that entity. As we run the word2vec tool on these reformatted articles, we obtain embeddings for both regular words and entity ids.

SFC-based entity embeddings ( $E_{SFC}$ ) do not rely on the immediate adjacent words of the entity mention as in $E_{WC}$ . It is hard to represent this in a linear bag-of-words context as word2vec expects. Hence, we use the word2vecf tool (Levy and Goldberg 2014), which is an altered version of word2vec. While word2vec assumes a window of tokens around the target token as context, word2vecf allows us to define the context tokens arbitrarily, one-by-one for the target token. It takes the input in two columns, where the first column holds the target token and the second column has a single context token. As exemplified in Figure 6, the input file contains one row for each target and context token pair. We select a window of mentions around each hyperlinked mention in Wikipedia and use the words in their surface forms to create the input data for the word2vecf tool.

Figure 6.

Input samples from the example contexts in Figure 3 to obtain $E_{SFC}$ and $E_{SYNC}$ with the word2vecf tool.

EC-based entity embeddings ( $E_{EC}$ ) are calculated by using the entity ids of the surrounding mentions in the given EC. In this case, we remove all the words in the Wikipedia articles and only keep the entity ids of hyperlinked mentions as tokens. As we run word2vec on these reformatted articles, we get an embedding for each mentioned named entity, which is calculated based on a window of the mentioned named entities around it.

Synset-based entity embeddings ( $E_{SYN}$ ) are the only type of entity embeddings that do not reflect the context-based similarity of entities, but their synset-based similarity. In WordNet, a synset is a set of synonymous words grouped together. WordNet uses the synset records to define a hypernym hierarchy between them to reflect which synset is a type of which other. YAGO v3.1 (Mahdisoltani et al. Reference Mahdisoltani, Biega and Suchanek2015) uses those synsets as category labels to represent certain types of named entities, such as person, musician, and city. Moreover, YAGO extends the WordNet synsets with its own specialized synsets, prefixed “wikicat” (e.g., wordnet_person_100007846 is a hypernym of wikicat_Men in the YAGO synset taxonomy). In YAGO data set, named entities are labeled with one or more synsets. In addition to YAGO, we also use BaseKB Gold Ultimate Edition (Bollacker et al. Reference Bollacker, Evans, Paritosh, Sturge and Taylor2008) data set which is based on the last Freebase dump of April 19, 2015. It contains over 1.3 billion facts about 40 million subjects, including 4 million from Wikipedia. It is similar to YAGO, except it has its own simple type taxonomy, independent of WordNet synsets. In our experiments, we combine the type definitions from both YAGO and BaseKB data sets and call them synsets for the sake of simplicity. By combining them, we aim to have a synset for as many named entities as possible. We then use the associated synsets of named entities as their context tokens, as word2vecf allows us to define custom context. We give the entity ids in the first column and the associated types in the second column as shown in Figure 6. In this process, we do not use all available synsets though. We replace any specialized synset (ones starting with wikicat_^*) with its hypernym WordNet synset. We also filter out some BaseKB types that do not reflect type information (e.g., base.tagit.topic and types with user names).

3.3 Clustering named entities

Now that we have entity embeddings, we can use them to cluster named entities. In order to do that, we primarily use the K-means algorithm due to its simplicity and time complexity. It is a centroid-based clustering algorithm. After setting the number of centroids (i.e., K) manually at the beginning, it assigns every data point to the nearest centroid. In this study, we use the Euclidean distance between the entity embedding and the centroid vectors. After the assignment step, the centroids are recomputed. This process continues till a stopping criterion is met.Footnote ^c In addition to K-means, we use the Brown clustering which is originally introduced to group words that have similar distributions of words to their immediate left and right. Both algorithms have time complexity that is linear in terms of the number of items being clustered (Steinbach, Karypis, and Kumar Reference Steinbach, Karypis and Kumar2000) provided that other factors are fixed. Considering that we have over five million named entities in KB, this property makes them very eligible for our experiments.

As in Figure 7, we give entity embeddings ( $E_X$ ) to K-means as input. In case of Brown, we use the Wikipedia articles in the entity-based context format. After getting clusters ( $C_X$ ), the cluster ids are assigned to entities as cluster-based types.

Figure 7.

Creating different clusterings of named entities with the K-means and Brown clustering algorithms, where $X\in$ {WC,SFC,EC,SynC}.

It is important to note that each clustering of the named entity space breaks that space into groups. In terms of the named entity normalization task, the discriminative power of a clustering depends on how well it distinguishes the correct candidate for a named entity mention from the other candidates. The ideal case occurs when the correct named entity for a mention is placed in a different cluster from the other most likely candidates and the cluster of the mention is also predicted to be the same as its corresponding named entity. Using a high number of clusters makes the clustering more discriminative as each cluster corresponds to a lower number of entities. However, that also makes the typing model task harder as it increases the ambiguity. In Section 5.3, we explore the right number of clusters for our experiments. Moreover, since we have five different clusterings and each breaks the entity space differently, using a combination of them is expected to make the aggregate discriminative power even higher. Hence, using multiple clusterings can be seen as an alternative to using a higher number of clusters. Note that Section 5.3 also explains our heuristic to select the best combination (in terms of number of clusters for each clustering) in order to achieve a better performance at the disambiguation step.

3.4 Preparing training data for the typing model

So far, we have described our approach for clustering named entities and assigning the cluster ids to entities as types. In order to train a typing model to predict the type of an entity mention in an input text, we need training data that include mentions labeled with these types. We create the training data from the hyperlinked mentions in Wikipedia articles using distant supervision. We label each hyperlinked named entity mention with its assigned cluster-based type. Since we have five different clusterings, we produce five different training data sets.

As described in Section 3.1, we represent the context of a mention in three different formats and three of our clusterings (namely, $C_{WC}$ , $C_{SFC}$ , and $C_{EC}$ ) are obtained according to the corresponding context formats. Take $C_{WC}$ for example. Clusters in $C_{WC}$ hold entities that are similar to each other in terms of WC. It is compatible to train a typing model for $C_{WC}$ with input in the WC format because labels are created from the characteristics that made up the input in that format. Same is true for $C_{SFC}$ that goes with the SFC and $C_{EC}$ with EC. Figure 8 exemplifies three training instances in WC, SFC, and EC formats, respectively. They are generated based on the example in Figure 4. Each instance consists of three input fields in addition to the label, surface form of the hyperlinked mention, as well as left and right context of that mention. Surface form is the common input. Like surface form, contexts in the first two cases are in the form of words as exemplified in Figure 8. In case of the third case, it is in entity ids.

Figure 8.

Example training instances from Figure 3 in WC, SFC, and EC formats, respectively.

In case of $C_{SynC}$ and $C_{BRO}$ , we use WC and SFC, respectively. In order words, we use the same formatted input for $C_{WC}$ and $C_{SynC}$ pair and $C_{SFC}$ and $C_{BRO}$ pair with the exception of the labels. Note that clusters in the $C_{SynC}$ hold entities that are associated with similar synsets. Synsets are like semantic categories and local context is better suited to infer which synsets the mentioned entity is associated with. Hence, WC is used. On the other hand, using global context is better suited to infer $C_{BRO}$ -based types. We choose to use the SFC over the EC. This is due to the fact that SFC can easily be obtained by gathering the surface forms of the surrounding named entities. However, EC is only available after we get the best predictions from the first stage of our two-stage disambiguation approach described in Section 4. In other words, by using the SFC, we are making the Brown-based mention typing model available at the first stage, which increases the success rate of the first stage.

3.5 Specialized word embeddings for the typing model

When it comes to predicting the cluster of a mention with WC, it is common practice to use word embeddings that are obtained from a data set that is similar to the domain data. In our case, it is Wikipedia, which is a widely used source to obtain word embeddings in the literature. In our experiments, we did not use regular word embeddings calculated with word2vec. Instead, we propose two word embeddings that are more discriminative at predicting the cluster of a given mention. We use these word embeddings together to represent the word-based input for the mention typing model.

Cluster-centric word embeddings ( $W_{CC}$ ): We hypothesize that word embeddings, which are obtained from the data that reflect the characteristics of the problem, may be more effective at solving that problem. In this case, the problem is to predict the cluster-based type of a mention. Hence, we inject a piece of cluster information into the context in hope that word embeddings are being influenced by their presence and become better at predicting the cluster-based type. In order to accomplish that, in Wikipedia articles, we filter out the html tag (i.e., a-href) of the named entity mentions and leave its surface form alone. At the same time, we add a special token to the right-hand side of that surface form. That token corresponds to the assigned cluster id of the mentioned named entity. Embeddings of those words that are close to the specific cluster token are expected to be affected by that and become indicators of that cluster. We use the word2vec tool on this modified data set. Since the clustering of named entities is done at training time and does not change based on the given test input, the same embeddings that are obtained at training time are used at the test time. We obtain different word embeddings for each of the five named entity clusterings, since entities are assigned to different clusters in each case.

Surface form-based word embeddings ( $W_{SF}$ ): While word embeddings in $W_{CC}$ are only based on the words surrounding the named entity mention, another type of word embedding can be obtained with the words inside the surface forms exclusively. In order to do that, we use the Anchor Text data setFootnote ^d of DBpedia (Auer et al. Reference Auer, Bizer, Kobilarov, Lehmann, Cyganiak and Ives2007). It contains over 150 million (entity_id, surface_form) records, one for each mention occurrence in Wikipedia. We extract around fifteen million unique such records for almost all named entities in our KB, along with their frequencies. We name this data set as the surface form data set. Since this data set only includes the surface forms of the entities, but not their surrounding sentences, we use the associated cluster ids of entities as context tokens with the word2vecf tool. In other words, for each word in the surface form of a named entity, the cluster id of that entity is given as a context word, which results in (surface_form_word, cluster_id) pairs. We also take the frequencies of the surface form records into account. The more frequent one surface form is, the more important it is. Therefore, while creating the training data set for $W_{SF}$ from the (surface_form_word, cluster_id) pairs, the number of instances included for each surface_form_word is proportional to the logarithm of its frequency in the surface form data set.

3.6 Model to predict cluster-based types

Now that we generate training data that include labeled mentions with their corresponding named entities’ cluster-based type, we are ready to train a mention typing model to predict those types. At the end, we end up having five typing models: Word model based on $C_{WC}$ , Surface model on $C_{SFC}$ , Entity model on $C_{EC}$ , Brown model on $C_{BRO}$ , and Entity model on $C_{EC}$ .

We experiment with two different typing model architectures: LSTM and CNN models. CNNs are widely used for text classification, whereas LSTMs are effective at modeling sequences of items. Both models receive the same three-part input described in the previous section. However, since we have two sets of word embeddings for surface words as described in Section 3.5, we can represent the surface part as two inputs for both models, one with $W_{CC}$ and the other with $W_{SF}$ . In our experiments, we seek to compare both models and pick the best performing one. Details on these models are given below.

3.6.1 LSTM-based typing model

In this model, each input part is given to a separate LSTM layer and their output is concatenated and fed into the final softmax layer as illustrated in Figure 9. $Surface_1$ gets the cluster-centric word embeddings ( $W_{CC}$ ), which are better at discriminating cluster-specific words. $Surface_2$ gets the surface form word embeddings ( $W_{SF}$ ), which are optimized for discriminating cluster-specific surface words seen in large set of surface forms. We use dropout layers (Srivastava et al. Reference Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov2014) before and after the LSTM layers in order to prevent overfitting. Our preliminary experiments led us to using bi-directional LSTM (BiLSTM) for the $Surface_x$ and uni-directional one for the left and right ones. Moreover, reversing the order of words in the right context produces better results. We argue that words that are closer to the mention are more related to the mention and play a more important role at the prediction.

Figure 9.

LSTM-based model for mention typing.

3.6.2 CNN-based typing model

CNN-based model is very similar to the LSTM-based model, except that the input parts are given to the convolution layers. As pictured in Figure 10, each input part is fed into the six convolution layers with kernel sizes 1 through 6. These layers model the sequence of words in different lengths, like n-grams in a language model. We use hyperbolic tangent as an activation function over the convolution layers and apply max pooling layer prior to merging all into one layer before feeding to the softmax layer. Like in the LSTM-based model, we use dropout after the word embeddings and before the softmax layer.

Figure 10.

CNN-based model for mention typing.

4.1 Candidate generation

In the literature, most of the studies do not mention how to index and search entities to generate candidates based on the surface form of a mention. Some of the ones (Hachey et al. Reference Hachey, Radford, Nothman, Honnibal and Curran2013; Hakimov et al. Reference Hakimov, ter Horst, Jebbara, Hartung and Cimiano2016) providing this information report that they use Apache Lucene.Footnote ^e Lucene provides fuzzy search property which tolerates a certain percentage of mismatch at character level with the Damerau–Levenshtein distance. However, such off-the-shelf tools consider each mention individually and do not use contextual clues from other mentions in the same document. Moreover, they do not provide additional information, such as the type of the matched surface form, which we use as a feature (i.e., surface-form-type-in-binary in Table 1) at the disambiguation step. Hence, we implemented our own candidate generation tool.

Table 1.

Variables, sets, and functions used at defining ranking features

Table 2.

List of features used by the ranking model and their descriptions

For the indexing, as in Hoffart et al. (Reference Hoffart, Yosef, Bordino, Furstenau, Pinkal, Spaniol, Taneva, Thater, Weikum, Guo and Barbosa2011), we use all available surface forms of named entities in DBpedia and Wikipedia (including anchor texts, redirect and disambiguation records). In other studies, while some (Ling, Singh, and Weld Reference Ling, Singh and Weld2015; Ganea and Hofmann Reference Ganea and Hofmann2017) used additional web corpora, others (Ratinov et al. Reference Ratinov, Roth, Downey and Anderson2011; Phan et al. Reference Phan, Sun, Tay, Han and Li2018) used only Wikipedia. We index surface forms based on character tri-grams. In case of searching, the pseudocode of our search procedure is in Algorithm 1.

Algorithm 1

Pseudocode of the candidate generation algorithm

Our algorithm works in two stages. In the first stage, it starts with searching named entities matching with the mention surface form (Lines 17–25). It first retrieves indexed surface forms that have at least a certain amount of character tri-gram overlap ( $T=60$ %) with the search query (i.e., mention surface form). Then, it picks the ones that have the ratio of the edit distance at the character level to the query length less than certain value ( $E=25$ %). If not, then it checks for the word overlap. It can tolerate up to one word mismatch ( $D=1$ ) if the query contains more than one word ( $W=2$ ). After this selection, all matched candidates are returned. This first stage is similar to what off-the-shelf third party indexing and search tools provide.

In the second stage of our algorithm, we expand the candidates of each mention with the candidates from other mentions in the same document, since the same entity can be mentioned more than once. We do this in two steps. The first step looks at if the surface form of one mention is completely seen in the other’s (Lines 6-8). For example, “Barack Obama” can be first mentioned in full name and then as “Obama” in the same document. The second expansion step uses the most frequently cooccurring named entities (Lines 9–14) observed in Wikipedia articles. We only include the ones that have a surface form that includes the mention surface form in itself. At the end, we order the final set of candidates by scoring each candidate based on a specific formulaFootnote ^f (Line 15) and return the highest scored 100 candidates as the final output.

4.2 Ranking features for disambiguation

Each candidate is scored according to certain properties. These properties are named as features and their scalar values are turned into a vector which is given to the ranking model as input. In our experiments, we use a total of 12 features. In order to understand how they are calculated, we first give the definitions of the relevant variables, sets of values, and functions in Table 1.

Variables in Table 1 represent either strings, scalar values, or vectors. $SF_m$ and $D_t$ are the only variables that are not related to the candidate. Whereas, $D_c$ , $E_c$ , and $T_c^e$ are candidate-specific and calculated offline, hence they do not depend on the mention being considered. The rest of the variables depends on the mention. Sets are variables that represent a set of values. For example, $S_c$ corresponds to all the surface forms of the candidate entity c. The rest includes a set of candidate entities that are determined based on a specific position. Functions are applied on these variables as well as sets in order to define more detailed features.

We group the ranking features into four main categories as shown in Table 2. In the literature, the features used for disambiguation are basically divided into two main categories, namely local context-independent and global context-based features. In order to make it more explicit, we further break down the global features into three sub-categories. Those are mention-level, document-level, and second-stage features.

Local features are those that consider the individual properties of the candidate without taking into account any other candidate. For example, the feature surface-form-edit-distance considers the edit distance between the most similarly matched surface form ( $SF_c^{best}$ ) of the candidate in our surface form data set and the actual surface form ( $SF_m$ ) of the mention. The more distant they are, the less likely the candidate is the actual one. In order to take into account the popularity of the candidate, the feature entity-log-freq uses the total number of times that named entity is seen in the surface form data set. It uses the logarithm to scale down and smooth the values. The more popular the candidate is, the more likely it might be mentioned. Another local feature doc-similarity is the cosine similarity between the doc2vecFootnote ^g embeddings of the test document and the Wikipedia article of the candidate. The similarity between the two corresponds to the similarity between the context of the test document and the context in which the candidate is expected to be mentioned.

The two features entity-type-in-binary and surface-form-type-in-binary are binary vectors that encode the type of the candidate entity and the surface form, respectively. These types are based on the categorization of the entities and surface forms defined in Table 3. We assign a type to each named entity in our KB based on whether it is labeled with one of the predefined YAGO synsets or BaseKB types. We define five main types. While Person, Organization, Location, and Misc types are common in the named entity recognition literature, we added the SportsTeam type since many mentioned entities in AIDA data sets are of this type. In case of the surface form types, we assign one or more applicable types to the $SF_c^{best}$ . Other than Redirect and Disambiguation types in Table 3, the rest is based on the similarity between the surface form and the Main Title of the entity, which is obtained from its Wikipedia ID by replacing any underscore character with the blank space and discarding any phrase given in parenthesis. Certain combinations of these two types can give clues about the likelihood of the candidate. For example, it is more likely that mention of Person type entity is made in surface form of type FirstWord or Organization type entity in OrgAcronym type surface form.

Table 3.

Entity types and surface form (SF) types in SFDB

The final local feature $id^t$ -typing-prob is the probability ( $P_c^n$ ) of labeling the mention with the pre-assigned cluster-based type of the candidate by the typing model n. More precisely, when the typing model is applied to the input text containing the mention, it outputs the probability of the mention belonging to each of the cluster-based types. As we know the pre-assigned cluster-based type of the candidate, the probability of that type is set as $P_c^n$ . Since we have five different typing models, this feature contributes with five separate values. Note that each feature that includes $P_c^n$ contributes similarly.

Mention-level features consider the relative value of an individual candidate’s feature with respect to the other candidates’ for the same mention ( $C^m$ ). The first feature avg-surface-form-edit-distance takes the average of all surface-form-edit-distance feature values of $C^m$ . Averaging helps us represent the edit distance of an average candidate. We can argue that the higher it is, the more likely that none of the candidates are the actual entity being mentioned. The features prefixed with max-diff- compare the candidate’s corresponding feature value with respect to the best value seen for the mention. The difference between the two values is used as the feature value. If the value is already the best, then it uses the second best value. The higher positive value it is, the more likely that the candidate is the actual one. Or, the lower negative value it is, the less likely that the candidate is the one. max-diff- converts a context-independent value to a context dependent by representing it with respect to the highest (or the next highest) seen value.

Document-level features consider all occurrences ( $C_c^t$ ) of the same candidate in the same test document. We have one such feature and it uses the highest seen $P_c^n$ value for the consideredcandidate. When the same entity is seen as a candidate in multiple mentions, each has its own $P_c^n$ value depending on the position of the mention in the document. The highest of them increases the chance of the other occurrences of the same candidate with the lower $P_c^n$ in the same document.

Second-stage features are only available after the ranking model with the previously described features is trained and applied on the candidates once. Second-stage features use the ranking probability of each candidate ( $R_c$ ). The first feature max-ranking-score keeps the highest $R_c$ among the previous occurrences of the same candidate ( $C_c^{prev}$ ) in the same document. If there is no previous occurrence of the candidate before the considered mention, then it looks at the future occurrences ( $C_c^{next}$ ) only if $T_{c^{\prime}}^{SF}$ of that candidate is WikiID. This means that having an occurrence of the candidate with high $R_c$ increases the chance of the next occurrences of the same candidate in the same document. The same effect for the early occurrences can only happen if the future occurrence is mentioned in its full title. The second feature max-cos-sim-context uses the cosine similarity between the entity embeddings ( $E_c$ ) of the candidates.Footnote ^h This feature is set to the highest cosine similarity value between the candidate and the highest ranked N candidates in the surrounding window of M mentions.

4.3 Neural network model for disambiguation

Our neural network for ranking candidate named entities is a two-layer feedforward neural network and one softmax layerFootnote ⁱ at the top. We use a dropout layer after each feedforward layer. We turn the ranking task into a binary classification task, where we classify each candidate as true or false candidate at the training time and then use the output probability of the true class to rank the candidates at the test time. For each candidate, we create its own input vector, which includes all numeric values for the features described in Section 4.2. Since we do re-ranking, we have two disambiguation models in our experiments, the first one does the initial ranking and the second one does the final ranking based on additional ranking insight obtained from the first model. In both cases, we use the same network architecture, except the fact that the first model does not include the second-stage features. The values for those features are calculated from the output of the first model. Hence, in case of the test sets, we first need to run the first model on them and get the ranking scores for each candidate. After that, we can run the second model to get the final ranking results.

5.1 Training and test data sets

5.1.1 Data sets for the mention typing

We derived our data sets from Wikipedia which includes over five million well-written documents, each describing one named entity. Most of the mentions of named entities and concepts in the documents are manually annotated with an HTML anchor tag. However, Wikipedia Editing Guidelines suggest authors to annotate only the first occurrence of a named entity in the Wikipedia articles, which means that most of the mentions are not annotated in the articles. Hence, we try to auto-annotate the remaining occurrences in the documents. To do this, we look for the word sequences (with greedy match) that are previously annotated in the same document and then auto-annotate them at the rest of the document. This process increases the number of mentions considerably with an acceptable error rate. However, we did not try to annotate mentions with the partial names.

After this pre-processing step, we create a separate data set for each context representation. Note that we represent the context in three different levels as described in Section 3.2. In case of WC, we break the Wikipedia articles into sentences and collect only those sentences that have at least one named entity mention and the length is between 10 and 50 words. We use the sentence boundary detector tool and tokenizer in the Stanford CoreNLP (Manning et al. Reference Manning, Surdeanu, Bauer, Finkel, Bethard and McClosky2014). We ended up having 46.8M such sentences. Out of those, we randomly selected sentences and created WC-^* data sets. In case of SFC, we start with the same sentences and then for each mention, we get the mention surface forms of the previous and next 10 mentions in the same document. Again we randomly selected instances and created SFC-^* data sets. For the EC, around each mention we collected the previous and next 10 named entities in the same document. This is called EC-^* data sets. Sample training instances are already exemplified in Figure 8 in Section 3.4. For each type, we created small and large training sets and a test set as given in Table 4. The number of instances and the average number of tokens in the context (left and right combined) are also provided.

Table 4.

Statistics on the data sets for the mention typing models

5.1.2 NED data sets

There are a number of publicly available data sets with different characteristics for the NED task. For training, development, and test purposes, we use AIDA (Hoffart et al. Reference Hoffart, Yosef, Bordino, Furstenau, Pinkal, Spaniol, Taneva, Thater, Weikum, Guo and Barbosa2011), which is derived from Reuters news articles of the CoNLL 2003 NER task. Being the widely used and largest NED data set, it comes in three pieces: AIDA.train (train), AIDA.testa (dev), and AIDA.testb (test). We report the results on AIDA-testb as in-domain evaluation results, since all our training is done on AIDA-train. In order to see how the model that is trained on AIDA.train achieves on the data sets that exhibit different characteristics, we test that model on a number of other test sets. This is called cross-domain evaluation. For this purpose, we use MSNBC (Cucerzan Reference Cucerzan2007), AQUAINT (Milne and Witten Reference Milne and Witten2008), and ACE2004 (Ratinov et al. Reference Ratinov, Roth, Downey and Anderson2011), which are the next three most frequently used test sets. ACE2004 is a subset of ACE2004 Coreference documents, while AQUAINT contains news articles from the Xinhua, New York Times, and Associated Press. Since they are also used for wikification, they include Wikipedia concepts apart from named entities. As the recent studies used the revised versions of these data sets prepared by Guo and Barbosa (Reference Guo and Barbosa2016), we report our results on these revised versions as well. Apart from these, we also consider three more test sets in order to observe how our system performs on shorter documents. KORE50 (Hoffart et al. Reference Hoffart, Seufert, Nguyen, Theobald and Weikum2012) includes 50 short sentences on various topics such as celebrities and music. Most of the mentions are single-word such as first names, which makes the deduction of the actual mentioned named entity very difficult. RSS-500 (Roder et al. Reference Roder, Usbeck, Hellmann, Gerber and Both2014) includes short formal text collected from a data set containing RSS feeds of the newspapers compiled in Goldhahn, Eckart, and Quasthoff (Reference Goldhahn, Eckart and Quasthoff2012). The texts are on a wide range of topics such as world, business, and science. Reuters-128 (Roder et al. Reference Roder, Usbeck, Hellmann, Gerber and Both2014) is a small subset of the well-known Reuters-21587 corpus containing short news articles about the economy.

Details about these data sets are given in Table 5. Note that since we use Wikipedia as our reference KB, we map the DBpedia-based annotations in KORE50, RSS-500, and Reuters-128 to the corresponding Wikipedia-based IDs. The AIDA sets have the most annotations among all, which makes them suitable for training and development. KORE50, RSS-500, ACE2004, and Reuters-128 are the sets that have the smallest average number of mentions per document (Avg#M/D). In terms of sentence-based context size, the KORE50 and AIDA sets have the smallest average number of words per sentence (Avg#W/S). Some of these data sets include NIL annotations, which have no corresponding named entity at the reference KB. As a design decision, we exclude such annotations and report the results accordingly in our experiments.

Table 5.

Statistics on the disambiguation data sets

5.2 Evaluation metrics

In this study, we report the standard micro-average F1-score for the evaluation of the cluster prediction model and the disambiguation model. F1-score is the harmonic mean of the precision and recall measures. Precision measures the percentage of machine’s predictions that are correct compared to the gold (human) annotations. On the other hand, recall measures the percentage of the gold annotations that are predicted correctly by the machine. In case of the disambiguation results, a number of studies reported their results in bag-of-title (BoT) F1-score (Milne and Witten Reference Milne and Witten2008; Ratinov et al. Reference Ratinov, Roth, Downey and Anderson2011), which is designed for Wikification systems. It is used to evaluate a NED system for indexing purposes. Hence, it discards the duplicate gold annotations and predictions in a document and uses the same F1-score metric on the filtered out numbers. Moreover, we report our results in InKB accuracy in case of the AIDA test set due to the convention in the literature. On the other test sets, we use a threshold on the ranking probability and do not assign an entity to a mention if our system is not confident with the assignment (i.e., the ranking probability is below the threshold). In our experiments, we use 0.03 as the threshold.

5.3 Optimizing clustering for better disambiguation

Important thing before getting into the experimental results is tuning of the clustering for better disambiguation. As described in Section 3.3, how we cluster named entities determines the effect of the discriminative power of clustering on the disambiguation step. The more clusters we use, the fewer entities each cluster has, hence the more discriminative the clustering becomes. For example, assume that we have a list of candidates for a mention and this set of candidates are grouped together based on their assigned cluster-based types. Think of one particular group that contains the gold standard entity. All candidates in that group are given the same chance of being the actual referred entity as they all have the same cluster-based type. Therefore, when we have fewer entities in that group, we expect it to be relatively easier to disambiguate the correct entity among all candidates compared to when we have more entities in that group On the other hand, when we have more clusters, this makes the typing model difficult to solve due to the increase in the number of classes (i.e., types). Having said all these, we look for an optimization to deal with this trade-off.

There are two main factors we considered for optimization. The first is the window size chosen while calculating the entity embeddings on which we run the K-means algorithm. The second and more important factor is the number of clusters to be generated by the clustering algorithms. However, it is not feasible to try all possible combinations and measure which one achieves the best performance at the disambiguation step. That is because for each combination, we need to get the clusterings, train their typing models, train the ranking model on their predictions, and then finally measure the disambiguation success. Instead, we introduce the following three-step approach.

The first step is to produce a wide range of clusterings for the named entity space with different window sizes for embeddings and different numbers of clusters. The second step includes selecting a good sample of clusterings and training a typing model with each selected clustering. Since we use multiple mention typing models for disambiguation, the third step is to select one clustering for each typing model so as to achieve high at the disambiguation step. In order to choose this combination, we also propose a heuristic that minimizes the number of times we train the ranking model. Below, we explain each step in more detail.

The first step involves creating various clusterings of named entities for each of the five clustering approaches. The first and primary parameter is the number of clusters. In case of the Brown-based approach, we run the tool with a number of clusters from 1000 to 1500 in 100 incremental steps. For the other four clustering approaches, we run the K-means algorithm with number of clusters from 600 to 3000 in 200 incremental steps. The second parameter is related to the calculation of the entity embeddings which are only used by K-means. In case of Synset-based approach, since there is no sequential context, we use word2vecf and the only parameter we change is the iteration. For the other three approaches, we use word2vec and experiment with different values of the window parameter. We use window sizes of 2 and 3. Using other values does not provide any better results. We set the iteration parameter for the Synset-based approach to 20. At the end of the first step, we get a total of 70 clusterings.

Figure 11.

Change of average gold candidate cluster size (AGCCS) as the number of clusters is increased for each clustering approach.

At the second step, we pick one or more parameter combinations for each clustering approach. However, instead of selecting randomly, we proposed a metric that helps us evaluate each clustering based on how well it might help discriminate the candidates for the benefit of the disambiguation model. This metric is called average gold candidate cluster size (AGCCS). When we group the candidates of the mention based on the clustering at hand, we end up having clusters of candidates, or candidate clusters. The one that holds the gold (true) candidate is called the gold candidate cluster. The smaller the size of the gold candidate cluster is, the less number of candidates are being favored by the cluster prediction model at the disambiguation step. In other words, it becomes more discriminative. For each parameter combination, we calculated AGCCS value on the AIDA.train data set, which includes over 18,000 mentions with an average of 84 candidates for each mention.

Figure 11 shows how AGCCS changes, as we increase the number of clusters. Generally, it drops, because using more clusters leads to less number of entities inside the clusters, hence the smaller size for the gold candidate cluster. Note that AGCCS generally decreases fast at first but does not drop below 2.5. The dots in Figure 11 show the chosen cases that we train a typing model for. We particularly pick the ones with the lowest AGCCS value. Moreover, we also consider some other local minimums, since they are obtained with less number of clusters. At the end, we have 19 different typing models calculated,Footnote ^j hence 19 dots in Figure 11.

The third step is to pick the best clustering combination. Note that based on 19 calculated typing models, we ended up having 540 different combinations.Footnote ^k However, it is not feasible to train a disambiguation model for each combination and pick the best performing one. Instead, we proposed a heuristic that scores the typing model combinations according to Equation (1). We omit the Entity typing model in order not to involve the second-stage ranking into this procedure. After getting the first-stage fixed, we pick the best performing Entity model which is obtained with 2200 clusters.

(1) \begin{equation}\begin{aligned}{\textit{argmin}{\phantom{\frac{A}{B}}}_{w \in Word, s \in Surface, y \in Synset, b \in Brown}} P^w + P^s + P^y + P^b \\[9pt] \textrm{where} \quad P^t = \sum_{m \in M} \sum_{c_{ng} \in \{C^m - c_g\} } P_{c_{ng}}^t - P_{c_g}^t, \quad \textrm{if} \quad P_{c_{ng}}^t > P_{c_g}^t\end{aligned}\end{equation}

Note that a typing model t outputs the probabilities of the cluster-based types for a given mention. Since we know the pre-assigned cluster-based type of each candidate $c_{type}$ , we also know $P(c_{type}|mention)$ , or $P_c^t$ in short. The ideal case from the disambiguation point of view is to have all typing models assign the highest probability to the gold candidate $c_g$ so that it can be easily distinguishable from the rest of the candidates (i.e., $C^m - c_g$ ), which are called non-gold candidates, or $c_{ng}$ . However, in a real-world scenario, typing models can make a classification mistake which leads to $c_{ng}$ having higher probability than $c_g$ . We call them competing non-gold candidates. Having higher typing model probability falsely favors them in the disambiguation step. Having said that, Equation (1) chooses the model combination such that it minimizes the aggregate probability difference between gold candidate and competing non-gold candidates calculated over all mentions M. For these calculations, we use the AIDA.train data set.

At the third step, instead of picking the typing model combination that has the lowest value according to Equation (1), we select the lowest scored 10 combinations and train a ranking model for each. Then, we score them on the AIDA.testa development set and pick the one that achieves the highest disambiguation score. The selected combination is Word model with 1000 clusters, Synset model with 1000 clusters, Brown model with 1300 clusters, and Surface model with 2400 clusters. For the Entity model in the second stage of the ranking, we use the model with 2200 clusters.

6.1 Evaluation of the candidate generator

In order to evaluate the candidate generator, we use the gold recall measure, which is defined as the percentage of the annotated (gold) named entities in the data set that have been suggested by the candidate generator. Table 6 gives the gold recall values for our candidate generator and two other candidate generators in the literature. Note that most of the NED studies use a smaller number of candidates in order to discard the least possible cases before doing the ranking. Hence, Ratinov (Ratinov et al. Reference Ratinov, Roth, Downey and Anderson2011) and Ganea (Ganea and Hofmann Reference Ganea and Hofmann2017) use the highest scored 20 and 30 candidates per mention, respectively. In our experiments, we use top 100 ( $N=100$ ) candidates in order to increase the recall of the NED system and to have more negative examples during the training of the model. To evaluate our candidate generator, we also calculate gold recall for top 20 and 30 candidates. Moreover, in order to measure the contribution of the second stage of our candidate generation algorithm described in Section 4.1, we also calculate these recalls without applying the second stage. The results of the proposed candidate generator as well as the ones of Ratinov et al. (Reference Ratinov, Roth, Downey and Anderson2011) and Ganea and Hofmann (Reference Ganea and Hofmann2017) are given in Table 6. In addition to these, Hakimov et al. (Reference Hakimov, ter Horst, Jebbara, Hartung and Cimiano2016) reported a recall of 97.7 on AIDA.train when $N=100$ .

Table 6.

Gold recall values for candidate generation on the NED data sets

Table 7.

Statistics on the candidates generated for each NED data set

When we compare our recall values with other studies, in almost all cases our proposed candidate generator achieves better performance except with respect to the performance of Ratinov et al. (Reference Ratinov, Roth, Downey and Anderson2011) on the AQUAINT. The results also show that the second stage in the candidate generator produces a substantial improvement. It means that using candidates from similarly titled surrounding mentions and extending the candidates even further with their cooccurring entities from Wikipedia results in higher recall values. The only exception is with the AQUAINT and AQUAINT^rev.

When we perform error analysis, we observe that in the case of KORE50, small context and mostly one word mentions result in low recall. In case of RSS-500, the annotated mentions only include part of the existing surface form (e.g., only the word “France” is annotated for the available surface form “Tour de France”). Since we do not take into account the immediate surrounding words of the annotated mentions during the search or use an off-the-shelf named entity recognizer (Phan et al. Reference Phan, Sun, Tay, Han and Li2017) to expand the boundaries, we achieve relatively low recall on RSS-500. Table 7 reports the average number of candidates per mention generated for $N=100$ . It also shows the average length of the mentions in characters and the average edit distance between the surface form ( $SF_m$ ) of the mention and the best matched surface form of the candidate ( $SF_c^{best}$ ) per candidate. The highest values for both metrics are seen in Reuters-128, while the average mention length values on the AIDA sets also show their characteristic difference from the other sets, which is important considering that we train our disambiguation model on the AIDA.train and test it on the other sets.

6.2 Contribution of specialized word embeddings in mention typing

Instead of using regular word embeddings (i.e., $W_R$ ) as input for our mention typing models, we introduced two different word embeddings in Section 3.5. Those are the cluster-centric word embeddings ( $W_{CC}$ ) and surface form-based word embeddings ( $W_{SF}$ ). $W_{CC}$ is proposed as an optimized version of $W_R$ , as it is influenced by the applied clustering during the calculation. Both embeddings are obtained with word2vecFootnote ^l on all Wikipedia articles. They are used as an input to Left, Right, and $Surface_1$ components (either LSTM or CNN layers shown in Figures 9 and 10) of the typing model. In case of $W_{SF}$ , it is obtained with word2vecfFootnote ^m from the large surface form data set as described in Section 3.5. Different from the previous two embeddings, $W_{SF}$ is used as an input to the $Surface_2$ component of the typing model. Note that, like $Surface_1$ component, $Surface_2$ also takes the words of mention’s surface form as input but in $W_{SF}$ embeddings instead.

In order to measure the contribution of using $W_{CC}$ over $W_R$ and using additional $W_{SF}$ , we trained both LSTMFootnote ⁿ- and CNNFootnote ^o-based models with different embedding combinations for each typing model. Note that each typing model is trained and tested on its own designated data set described in Section 5.1.1. To consider all combinations in a feasible time frame, the small version of those data sets (the ones named ^*-SmallTrain and ^*-SmallTest) is used. In Table 8, we measured how accurately the typing model predicts the cluster-based type labels assigned to each mention in the test sets. The results are given in F1-score.

Table 8.

Showing the contribution of the $W_{CC}$ (over $W_R$ ) and $W_{SF}$ embeddings when typing models are trained and tested on ^*-SmallTrain and ^*-SmallTest sets

The first rows for the LSTM and CNN-based models in Table 8 consider the case where we do not use any special word embedding except the $W_{R}$ . The following rows show how much the F1-scores change with respect to the first row (i.e., the baseline), first when we replace $W_{R}$ with $W_{CC}$ , then when we use $W_{SF}$ . Shown at the last rows, using both special embeddings together increases the scores considerably, up to 3–7 points. Their contribution is even more visible in case of the CNN-based models. When we do the same analysis on the other NED sets, we also observe very similar contribution levels. Based on these results, we use $W_{CC}$ and $W_{SF}$ together in the rest of our experiments.

Table 9.

F1-scores and average loss per instance values for the mention typing models trained and tested on the ^*-LargeTrain and ^*-LargeTest sets

6.3 Cluster-based mention typing results

In order to evaluate the cluster-based mention typing models, we train and test our five different models on the ^*-LargeTrain and ^*-LargeTest sets defined in Section 5.1.1. The results in F1-score are given in Table 9 for both the LSTM- and CNN-based models. However, since they are tested on different sets, the F1-scores are not comparableFootnote ^p across the table. Hence, the average loss per instance is included in parenthesis. It is calculated by normalizing the total cross entropy loss value given by the model on the test set with the number of instances in that set. Note that average loss per instance is a better evaluation metric than F1-score for assessing the quality of the predictions for the disambiguation step, since the prediction probabilities are used as features in the ranking model. F1-score only measures how accurate the model predicts the true label, whereas average loss per instance indirectly takes the model’s probability of that predicted label into account.

When the context is local as in the Word and Synset models, CNN performs similarly to LSTM. However, in general the LSTM-based models outperform the CNN-based models. This is supported by both F1-score and average loss per instance values. Hence, in the rest of our experiments, we use LSTM for all mention typing models.

When we compare the different models with each other through the average loss per instance value, the Synset model turns out to be the worst performer and the Word model comes after that. This means that these two sentence-based models are outperformed by the document-level typing models, namely Surface, Brown, and Entity. This might be expected due to the larger context at the document level. In case of the worst performer, the synset-based model is based on clustering of the entity embeddings that have not originated from the context but are based only on the similarity of the assigned synsets. In other words, the cluster-based type labels used for training the Synset model are not optimized for the contextual similarity. This might affect the success of the typing model which learns to predict based on the contextual similarities. The Synset model not being optimal for the typing model is also supported by the results in Table 8. Specialized word embeddings do not help the Synset model much compared to the other models.

6.4 Disambiguation results

For the disambiguation step, we trained our disambiguation modelFootnote ^q on the AIDA.train data set and did the feature selection based on the AIDA.testa development set. We train our ranking model in two stages. At the first stage, the model is trained and run on all data sets. The ranking probabilities for the candidates are stored and used in the additional features, which are classified as the second-stage features in Section 4.2. Our final resultsFootnote ^r are obtained after training the ranking model with all the features. The results are reported in Tables 10 (for test sets with large context) and 11 (for test sets with small context).

Table 10.

Results in F1 (the highest value for each data set is shown in bold) and BoT F1 (shown in italic) on the NED test sets with large context

Table 11.

Results in F1 and BoT F1 on the NED test sets with short context

The upper part of the tables provides the results reported by the previous studies in the literature. Note that the results from the two studies at the bottom are given in BoT F1-score. The lower part of the tables presents our results given in traditional micro F1-score and BoT F1-score. The first line in “Ours” part shows the results taken without using any features related to the mention typing models or the second stage. We call it our baseline. As we add the features calculated with the four typing models (namely, Word, Synset, Surface, and Brown), we can observe an increase of 2–8 points on all test sets. Next, we apply the second stage, where we add the second-stage features and the features calculated with the Entity typing model. The results are improved further, especially on AIDA.testb with 3.3 points increase.

In order to compare our results with the SOTA results in Table 10, we perform the randomization test with respect to the studies that achieved close to our results. On AQUAINT, we achieve better than Phan et al. (Reference Phan, Sun, Tay, Han and Li2018) at statistically significant level.Footnote ^s On MSNBC, our higher F1-score turned out to be not statistically significant compared to Phan et al. (Reference Phan, Sun, Tay, Han and Li2017). On the revised ACE2004, our higher result is not statistically significantFootnote ^t compared to Fang et al. (Reference Fang, Cao, Zhang, Li, Zhang and Liu2019). On the revised AQUAINT, our results are 2.2 points higher than Ganea and Hoffman (Reference Ganea and Hofmann2017) and 1.4 points higher in BoT F1 with respect to Yang et al. (Reference Yang, Irsoy and Rahman2018). However, we are unable to perform valid randomization testsFootnote ^u with respect to their results.

In case of test sets with shorter context, Table 11 shows that our best system cannot achieve better performance than its counterparts. The worst performed test sets compared to the SOTA results are KORE50 and RSS-500. KORE50 has the lowest average number of words per sentence and RSS-500 has only one mention per document on average. Our context-centric approach cannot utilize such short context enough. This is in fact the most common problem for all NED systems. On the other hand, Reuters-128 has a relatively larger context based on those metrics; however, its average number of edit distance per candidate is the highest with respect to the other sets. That may cause the lower scores on Reuters-128. This is actually connected to the fact that AIDA.train on which we train our disambiguation model is characteristically different from these three data sets. One particular difference to mention is that the average edit distances given in Table 6 in the AIDA sets are lower than many of the other sets. Whenever our NED model gets candidates with relatively higher edit distances on any of the test sets, it assigns low scores to such candidates as expected. This affects the gold candidates disproportionately, when there are alternative candidates with lower edit distances.

It is interesting to note that the AQUAINT test set contains many concepts (e.g., “power plant,’’ “radioactive waste,’’ etc.) along with named entities. Our success might be related to the fact that we train our typing models on Wikipedia, which also includes the mentions of concepts.

6.5 Analysis of the experiments

6.5.1 Ablation study on the ranking features

In order to understand the contribution of the ranking features, we perform two ablation tests. Moreover, since we do the ranking in two stages, we calculate the contributions for both stages as well as for the baseline, which is the system that does not use any features obtained from the typing models. Before getting into the analysis, one factor has to be noted while evaluating the contributions. The second stage uses additional “second-stage” features and their success depends on the success of the features used in the first stage. Since those first-stage features are still used in the second stage, their contribution drops as they share their success with the second-stage features. Hence, it is more appropriate to evaluate the contribution of the first-stage features based on the results at the first stage.

The first test examines the contribution of each feature by excluding that feature from the model. Table 12 lists all the features used in our experiments. Note that some of them are not available (N/A) for certain stages. The resultsFootnote ^v are shown in the F1-scores taken on the AIDA.testa (dev) set. As each feature is excluded, the drop in the F1-score is given. The first thing to notice is the high-level contribution of edit distance-based features. Surface form is the major factor at the disambiguation task. The second thing is the decreasing contributions towards the second stage.

Even though there are four groups of features shown in Table 12, for the sake of contribution analysis, it is better to group them as surface form-based (numbered 1,3,6,8), candidate-based (2,7), typing model-based (5,9,10), and ranking-based (11,12) features. Each group considers the disambiguation task from different aspects.

Surface form-based features are mention oriented, completely independent from the context. The results in Table 12 show that they keep their contribution high at all stages. This can be attributed to the fact that other features do not consider surface form. In other words, there is no contribution overlap between surface form-based features and other features.

Table 12.

Showing the contribution of each feature in F1-scores by its exclusion at different stages on the AIDA.testa development set

Typing model features are mostly context driven. Their contribution in Table 12 does not look notable at the individual level. However, the change between the baseline and the first stage, which is 11.2, comes from adding the typing model features. The low individual contributions can be explained by the fact that those three features are derived from the same value hence, their contributions overlap. Interestingly, using the highest typing model probability seen in other occurrences of the same candidate in the same document (10) is more powerful than using the typing model probability of the candidate (5) for the considered mention.

Candidate-based features are more candidate specific, with no direct involvement of the surface form. Their contribution drops substantially at the second stage. Even though (2) is widely used in the NED literature, it is calculated offline and independent of the mention or its context. This might explain its low performance. In case of (7), it is based on doc2vec embedding similarity between candidate’s Wikipedia page and the test document. In other words, it is similar to the other typing model features. The results support that their contributions overlap.

Ranking-based features help us take into account the surrounding candidates. The results show that they contribute quite well compared to others. Especially (12) might be the main driver behind the F1-score improvement of the second stage.

In our further analysis, we observe that using the max-diff version of the features performs better than using the feature itself. max-diff compares the value of the feature with respect to the highest value seen among its sibling candidates. For example, using only doc-similarity does not contribute better than max-diff-doc-similarity. The same is true for max-diff-surface-form-log-freq. We argue that this is related to the fact that doc-similarity and surface-form-log-freq are context independent and applying max-diff makes them context dependent.

Our second ablation analysis evaluates the contribution of each typing model. Table 13 shows the amount of drop in F1-score when we exclude each typing model from the first and second stages on the AIDA.testa (dev) set. Note that features (5,9,10) shown in Table 12 have multiple values, one for each used typing model t. Considering that the Synset typing model is the worst one at predicting the types as shown in Table 9, it is surprising to see that it is the best contributing model. Like Synset, the other local-context based model, namely the Word typing model comes second. This indicates that typing models based on local (sentence-based) context contribute to the success of NED more than typing models based on global (document-based) context.

Table 13.

Showing the contribution of each mention typing model in F1-scores by its exclusion at different stages on the AIDA.testa development set

Table 14.

Analyzing the impact of the popularity of the candidate entity and its surface form frequency when our system fails on the AIDA.testa (dev) set