3.1 Semantic representation

Semantic representation considers how the meaning of the specific synset in a lexical source can be represented using semantic features. The main features in WN are lexical relations and glosses (textual definition of concepts). The most lexical relation that has been used in measuring semantic similarity is the ‘is–a’ relation (hypernyms/hyponyms). In the semantic taxonomy using the ‘is–a’ hierarchical relation, a synset inherits several characteristics from the super-concept and adds its own specific features to represent its semantics. Thus, a synset is an accumulation of the propagated information from a distant ancestor to another less distant ancestor by adding specificities to each descendant (Aouicha, Taieb and Ezzeddine Reference Aouicha, Taieb and Ezzeddine2016). Subsequently, a synset depends strongly on its direct hypernyms and ancestors to form its meaning.

In this research, a synset is semantically represented using the super-concepts generated from hierarchical relations in a given lexical knowledge source. The motivation behind selecting this type of feature is that state-of-the-art experiments (Zhang, Gentile and Ciravegna Reference Zhang, Gentile and Ciravegna2013; Saif et al. Reference Saif, Ab Aziz and Omar2017) have shown that this feature achieves promising results compared with other features. This finding is attributed to the fact that the definition of most concepts can be extracted from the taxonomical relations (hypernyms and hyponyms) of the described concept. For example, the definition of the concept atropine is introduced in WN as ‘a poisonous crystalline alkaloid extracted from the nightshade family; used as an antispasmodic and to dilate the eye pupil; also administered in large amounts as an antidote for organophosphate nerve agents or organophosphate insecticides.’ Thus, the words ‘alkaloid,’ ‘antispasmodic,’ ‘antidote,’ and ‘agent’ can be extracted from the ancestors of this concept. Based on the hyponyms, the terms ‘domestic cats’ and ‘wildcats’ can be extracted from the direct hyponyms of the concept cat that has the following gloss in WN: ‘feline mammal usually having thick soft fur and no ability to roar: domestic cats; wildcats.’

For each node (synset), its semantics is represented as a set of concepts that are extracted from its super-concepts in a semantic taxonomy. The super-concepts of a synset are extracted as nodes in the paths from the node that corresponds to this synset to the root of the semantic taxonomy as well as the synset itself. The reason for including the synset in its super-concept is to overcome the indistinguishability limitation. In this limitation, any two pairs of synsets having the same super-concepts will receive the same semantic similarity, which is not necessarily appropriate. For example, in Figure 1, the similarity between antispasmodic#1 and remedy#2 is equal to the similarity between the concept remedy#2 and itself. Formally, the form of the super-concepts of the synset in the semantic taxonomy T is defined as follows:

(4) $$\begin{equation} {\rm {Super}}\left( {syn} \right) = {\rm {\lbrace }}c \in T{\rm {|}}\ c \in {\rm {ancestors}}\left( {syn} \right)\rbrace {\rm {\ }}\mathop \cup \nolimits ^{} \left\lbrace {syn} \right\rbrace \end{equation}$$

where ancestors(syn) is a function that returns the hypernyms of syn in a recursive manner.

Given that a node can have more than one path to the root (multiple hypernym relations), the ancestors are extracted from each path. Therefore, as shown in Figure 1, the ancestors of the synset atropine#1 are poison#1, antidote#1, alkaloid#1, mydriatic#1, antispasmodic#1, remedy#2, medicine#2, drug#1, organic compound#2, compound#1, agent#2, causal agent#1, substance#1, and entity#1, which are extracted from the paths of the node atropine#1 to the root in the semantic taxonomy.

Fig. 1.

Part of the semantic taxonomy of WN representing the super-concepts of Atropine#1.

3.2 Weighting features

Based on the representation step, the features in the semantic representation of a specific synset are assumed to have the same degree of importance in the meaning of this synset. However, the semantic relations between a synset and its ancestors vary according to the distance between the synset and each concept of its ancestors; a nearer node is more semantically related to its nearest ancestors than a farther node. For instance, the concepts Alkaloid#1, Mydriatic#1, Antispasmodic#1, and Antidote#1 that correspond to the direct hypernyms of the synset Atropine#1 are more semantically related to the synset than Entity#1, Substance#1, and Causal agent#1, which have longer distance to the synset.

In other words, the main limitation of this semantic representation is that it does not consider the specificity of concepts in a semantic hierarchy. In this study, the weighting step is introduced to overcome this limitation. In this step, each feature in the semantic representation of a given synset is weighted according to its importance to form the meaning of a synset. Formally, the semantic interpretation of a synset is represented by a row vector of concepts (its super-concepts) paired with their weights that reflect the degree of association of each concept to the synset. For a given synset syn, its semantic representation is defined based on the semantic taxonomy of the lexical knowledge source as follows:

(5) $$\begin{equation} R\left( {syn} \right) = \left\lbrace {\left( {{c_1},{w_1}} \right),\left( {{c_2},{w_2}} \right), \ldots,\left( {{c_m},{w_m}} \right)} \right\rbrace \end{equation}$$

where c ₁, c ₂, . . ., c _m are the super-concepts of syn and w ₁, w ₂, . . ., w _m are the weights of the corresponding super-concepts.

In this step, four weighting mechanisms are introduced to quantify the semantic association between features in the semantic representation of a given concept and the concept itself. These mechanisms depend on the structural (topological) parameters of the semantic taxonomy of a given lexical source. The structural parameters include edge, depth, density, and descendants. These parameters have been used in many studies to address the specificity issue in path-based measures and to compute the IC in IC-based measures.

3.2.1 Edge

The first weighting mechanism depends on the distance between a given synset and each concept in its semantic representation. The distance metric has been exploited by many researchers to measure semantic similarity. In a semantic taxonomy, distance refers to the length of a path and is typically calculated by counting the number of edges along the path. Formally, the weight of concept c in the semantic representation of synset syn is defined as follows:

(6) $$\begin{equation} w\left( c \right) = \frac{1}{{{\rm {length}}\ \left( {c,\ syn} \right) + 1}} \end{equation}$$

where the length returns the shortest path (number of edges) between c and syn.

Fig. 2.

Histogram of depth in the WN semantic taxonomy.

3.2.2 Depth

The second weighting mechanism uses the depth of the concept in a semantic taxonomy as the weight of the features. The depth of synset syn denoted by depth(syn) is usually the number of nodes along the longest path between syn and root in the semantic taxonomy. Depth is significant in taxonomy-based semantic analysis because it indicates the degree of expressiveness/granularity of the hierarchy. The node at the top level is more general than the node at the bottom level. In other words, moving from a high level to another means moving from a narrow topic to the addition of specificities. The depth of concepts in WN ranges between 1 and 20. The histogram of depth in WN is illustrated in Figure 2.

The difference between distance and depth is that the elements in the direct hypernyms (parents) of the specific node may have different depths in the semantic taxonomy. For example, in Figure 1, the node atropine#1 has five direct hypernyms, namely, alkaloid#1, antidote#1, antispasmodic#1, mydriatic#1, and poison#1, which have depths of 10, 9, 8, 7, and 5, respectively. Meanwhile, the distance between atropine#1 and each element in its hypernyms is 1. The weights of the features of the semantic representation of the concept atropine#1 are presented in Table 1.

Table 1.

Weights of the features of the semantic representation of the concept Atropine#1

Although the weighting scores of the concepts in the super-concepts of the synset range linearly between 1 and the depth of this synset, the semantic similarity using these scores is not balanced across the concepts in the semantic representation. For this reason, the weighting scores are normalized to balance the semantic similarity scores across the different concepts in the semantic taxonomy. For the concepts in the representation, the normalization function for each concept is defined using the natural logarithm and the square root as follows:

(7) $$\begin{equation} w\left( c \right) = \log \left( {\sqrt{{\rm {depth}}\left( c \right)} } \right) \end{equation}$$

Figure 3 shows the semantic similarity scores of the synset ‘ox’ and its super-concepts that have depths ranging from 1 to 18 using direct depth weighting and normalized weighting. In contrast to those of direct depth weighting, the similarity scores of using normalized weighting are linearly distributed.

Fig. 3.

Difference between depth and normalized depth in terms of linearity.

3.2.3 Descendants

The third weighting mechanism exploits the number of descendants of a given concept and reflects the important parameters in semantic measures. The number of descendants of a concept is considered to be directly proportional to its degree of generality. A concept with a large number of descendants is considered more general than a concept with a small number of descendants. For example, in Figure 1, the concept ‘physical entity’ that has a descendant number of 18 is more general than the concept ‘alkaloid#1,’ which has a smaller number of descendants (i.e., 1).

In weighting a concept, the TF-IDF technique that has been used in information retrieval is adopted in this work to quantify the strength of the relevance of concepts. In information retrieval, a document is represented as a vector, in which each dimension corresponds to the TF-IDF of the term. Inverse document frequency (IDF) is defined as $(\log \frac{n}{{df}})$, where n is the number of documents in the collection and df is the number of documents in the collection that contain the term. The term is considered general if it appears in many documents. Consequently, the weight of concept c is defined as follows:

(8) $$\begin{equation} w\left( c \right) = \log \left( {\frac{{\left| T \right|}}{{\left| {{\rm {descendant}}\left( c \right)} \right| + 1}}} \right) \end{equation}$$

where |T| is the number of concepts in the semantic taxonomy or the number of descendants of the root of the semantic taxonomy as well as the root itself. Thus, the weight of the root in the semantic taxonomy is equivalent to zero.

3.2.4 Density

In a semantic taxonomy, the local density of a synset node refers to the number of its sibling nodes or the number of its child nodes (Wang and Hirst Reference Wang and Hirst2011; Zhang et al. Reference Zhang, Gentile and Ciravegna2013). The density parameter has been used in a few semantic similarity measures (Sussna Reference Sussna1993; Jiang and Conrath Reference Jiang and Conrath1997) based on the assumption that two concepts are semantically close if they are densely connected locally. In other words, density is used as a parameter to determine the degree of specificity of a concept; the concepts in densely connected regions in the semantic taxonomy are considered highly specific and thus close to one another.

Based on the direct definition of local density, Wang and Hirst (Reference Wang and Hirst2011) redefined the density of a concept to address information loss on the local density that reflects specificity. For example, the concept ‘geological formation’ has a density value of 37, whereas the concept ‘shore,’ which is one of its hyponyms, has a density value of 28 (less than the density of its parent). Intuitively, the specificity should monotonically increase as one descends the taxonomy. Thus, this study defines the accumulated density of a given concept to be inherited by its descendants as follows:

(9) $$\begin{equation} {\rm {densit}}{{\rm {y}}_{{\rm {Accu}}}}\left( c \right) = \left( {\mathop \sum \limits _{c^{\prime } \in {\rm {ancestors}}\left( c \right)} {\rm {density}}\left( {c^{\prime }} \right)} \right) + {\rm {density}}\left( c \right) \end{equation}$$

where density(c) denotes the number of siblings of the concept as well as the concept itself. Figure 4 shows a comparison of the histograms of density and accumulated density in the semantic taxonomy of WN. The accumulated density is more normally distributed than the local density, given that the local density of more than 65 per cent of the nodes in the semantic taxonomy ranges between 1 and 6. Then, the features in the semantic representation of the given concept are weighted using the accumulated density as follows:

(10) $$\begin{equation} w\left( c \right) = {\rm {densit}}{{\rm {y}}_{{\rm {Accu}}}}\left( c \right) \end{equation}$$

Fig. 4.

Histograms of local density and accumulated density in WN.

3.3 Semantic similarity

After the semantic signature representations for a pair of concepts have been obtained, the semantic similarity of two concepts is calculated by comparing their corresponding semantic representation vectors. In comparing the semantic representation vectors, the method that has been used extensively in previous work to compare the vectors and the cosine is used in this research. The cosine metric is defined to measure the similarity between two concepts c ₁ and c ₂ as follows:

(11) $$\begin{equation} {\rm {Sim}}\left( {{c_1},{c_2}} \right) = \frac{{\mathop \sum \nolimits _{c^{\prime } \in R\left( {{c_1}} \right)\mathop \cap \nolimits R\left( {{c_2}} \right)} {w_1}\left( {c^{\prime }} \right) \times {w_2}\left( {c^{\prime }} \right)}}{{\sqrt{\mathop \sum \nolimits _{c^{\prime } \in R\left( {{c_1}} \right)} {w_1}{{\left( {c^{\prime }} \right)}^2}} \sqrt{\mathop \sum \nolimits _{c^{\prime } \in R\left( {{c_2}} \right)} {w_2}{{\left( {c^{\prime }} \right)}^2}} }} \end{equation}$$

where w ₁(c) and w ₂(c) are the weight of the concept c in semantic representation (R) of the concepts c ₁ and c ₂, respectively.

Given the ambiguity issue, polysemy words in WN are represented by several concepts (at least two concepts). Thus, the semantic similarity between two words depends on the similarity between their corresponding concepts. Following Sánchez et al. (Reference Sánchez, Batet, Isern and Valls2012) and Lastra-Díaz and García-Serrano (Reference Lastra-Díaz and García-Serrano2015a), the semantic similarity between two words is computed in this work based on the maximum technique. First, all of the concepts that correspond to the senses of each word are extracted from WN. Second, the semantic similarity is computed for each pair of concepts in the combination of candidates. Finally, the maximum semantic similarity score is selected from the candidates as the final similarity between two words. Formally, given two potentially polysemous words w ₁ and w ₂, the semantic similarity between them is computed as follows:

(12) $$\begin{equation} {\rm {Sim}}\left( {{t_1},{t_2}} \right) = \mathop {\max }\limits _{i,j} \;{\rm {sim}}\ \left( {{c_{1i}},{c_{2j}}} \right) \end{equation}$$

where c _1i and c _2j are the concepts extracted from WN for the words t ₁ and t ₂, respectively.

4.1 Experimental setup

In the experiments, the noun database in WNFootnote ² 3.1 is used to build the semantic taxonomy to model the semantic measures. The database contains 82,192 noun synsets that have 117,947 unique terms (single and multiple words). Table 2 presents the descriptions of the semantic taxonomy using the noun synsets in WN.

Table 2.

Statistics of the semantic taxonomy of noun synsets in WN

In measuring the semantic relatedness between word pairs, three English gold-standard datasets (Mc-30, ReG-65, and WS-203) that have been widely used in several studies as well as the most recent dataset (SimLex-666) are employed to evaluate the semantic measures in the experiments in this work. Table 3 shows the description of these evaluation datasets.

Table 3.

Evaluation datasets of words

The first dataset is RG-65 (Rubenstein and Goodenough Reference Rubenstein and Goodenough1965) that contains sixty-five word pairs ranging from ‘semantically unrelated’ to ‘highly synonymous.’ Semantic similarity judgments have been obtained from fifty-one human subjects in a scale from 0 to 4. Miller and Charles (Reference Miller and Charles1991) selected a subset of thirty word pairs (MC-30) from the RG-65 dataset and obtained human judgments from thiry-one test subjects. The third dataset is WS-203 introduced by Agirre et al. (Reference Agirre, Alfonseca, Hall, Kravalova, Paşca and Soroa2009). WS-203 is made up of 203 word pairs and is a subset of the Fin-353 dataset. Fin-353 dataset (Finkelstein et al. Reference Finkelstein, Gabrilovich, Matias, Rivlin, Solan, Wolfman and Ruppin2001) contains 353 word pairs that have been rated for semantic relatedness by an average of 13–16 human annotators. Hill et al. (Reference Hill, Reichart and Korhonen2015) introduced the largest and most recent word similarity benchmark (SimLex-999) in literature; it consists of 999 word pairs and comprises 666 noun–noun pairs, 222 verb–verb pairs, and 111 adjective–adjective pairs. Therefore, the final dataset is SimLex-666, which is a subset of SimLex-999 containing only noun–noun pairs.

4.2 Evaluation metrics

The in-vitro (direct) evaluation method (Zhang et al. Reference Zhang, Gentile and Ciravegna2013) is utilized in this study to evaluate the performance of the proposed method in measuring semantic similarity. In this method, the semantic similarity value of concepts is compared directly against a gold-standard dataset in terms of correlations. Two correlation coefficients are used extensively in the in vitro evaluation method; these two are Pearson’s product–moment correlation r and Spearman’s rank correlation ρ. These correlation coefficients are defined between the list of human judgments (X) and the list of values for the semantic relatedness measure (Y) and expressed as the following formulae:

(13) $$\begin{equation} r = \frac{{\mathop \sum \nolimits _{i = 1}^n \left( {{x_i} - \bar{x}} \right)\left( {{y_i} - \bar{y}} \right)}}{{\sqrt{\mathop \sum \nolimits _{i = 1}^n {{({x_i} - \bar{x})}^2}} \sqrt{\mathop \sum \nolimits _{i = 1}^n {{({y_i} - \bar{y})}^2}} }} \end{equation}$$

(14) $$\begin{equation} \rho = 1 - \frac{{6\mathop \sum \nolimits _{i = 1}^n d_i^2}}{{n\left( {{n^2} - 1} \right)}} \end{equation}$$

where d _i is the difference between the ranks of x _i and y _i.

In Gabrilovich and Markovitch (Reference Gabrilovich and Markovitch2009), Hassan (Reference Hassan2011), Sánchez and Batet (Reference Sánchez and Batet2012), Taieb et al. (Reference Taieb, Ben Aouicha and Ben Hamadou2013), and Zhang et al. (Reference Zhang, Gentile and Ciravegna2013), evaluation results were reported using Pearson, Spearman, or both. The Pearson coefficient imposes constraints on a correlation by imposing a linear relation between the two sets of normally distributed values being compared. The Spearman (nonparametric) coefficient relaxes this condition and only constrains the consistency of the relative ranking across the two inputs while imposing no distributional requirements. Hassan (Reference Hassan2011) demonstrated that the harmonic mean of Pearson and Spearman is a good metric that can cover the linear relationship with human judgment and can satisfy the relative order imposed by these judgments. Following this researcher, the evaluation results are reported using the individual metric with the harmonic mean that is defined as follows:

(15) $$\begin{equation} \mu = \frac{{2r\rho }}{{r + \rho }} \end{equation}$$

where r and ρ are Pearson’s product–moment correlation and Spearman’s rank correlation, respectively.

4.3 Evaluation results

The first experiment aims to evaluate the proposed measure for measuring semantic similarity between words. Given that the proposed measure depends on the weighting techniques of features in semantic representation, it is evaluated in this study using these techniques in four datasets. Table 4 summarizes the evaluation results for the proposed measure based on four weighting techniques using four topological parameters, namely, edge, depth, density, and descendants.

Table 4.

Evaluation results of the proposed measure

The results rendered in bold indicate the best results in terms of μ correlation in each dataset.

The second experiment is conducted to compare the proposed measure with hierarchical feature-based measures to determine the effect of the weights of features on semantic representation. In hierarchical feature-based measures, the concept is represented as a set containing its super-concepts in the semantic taxonomy. Based on set-based theory, the similarity between two sets s ₁ and s ₂ can be measured using the following functions.

(16) $$\begin{equation} {\rm {Si}}{{\rm {m}}_1}\left( {{s_1},{s_2}} \right) = \left| {{s_1}\mathop \cap {s_2}} \right| \end{equation}$$

(17) $$\begin{equation} {\rm {Si}}{{\rm {m}}_2}\left( {{s_1},{s_2}} \right) = \frac{{\left| {{s_1}\mathop \cap {s_2}} \right|}}{{\left| {{s_1}} \right| + \left| {{s_2}} \right|}} \end{equation}$$

(18) $$\begin{equation} {\rm {Si}}{{\rm {m}}_3}\left( {{s_1},{s_2}} \right) = \frac{{\left| {{s_1}\mathop \cap {s_2}} \right|}}{{\max \left( {\left| {{s_1}} \right|,\left| {{s_2}} \right|} \right)}} \end{equation}$$

(19) $$\begin{equation} {\rm {Si}}{{\rm {m}}_4}\left( {{s_1},{s_2}} \right) = \frac{{\left| {{s_1}\mathop \cap {s_2}} \right|}}{{\left| {{s_1}\mathop \cup {s_2}} \right|}} \end{equation}$$

(20) $$\begin{equation} {\rm {Si}}{{\rm {m}}_{{\rm {Batet}}}}\left( {{c_1},{c_2}} \right) = - {\log _2}\frac{{\left| {{s_1}\mathop \cup {s_2}} \right| - \left| {{s_1}\mathop \cap {s_2}} \right|}}{{\left| {{s_1}\mathop \cup {s_2}} \right|}} \end{equation}$$

In Table 5, the evaluation results of hierarchical feature-based measures are presented for measuring the semantic similarity of words in the four datasets. Sim_Batet(c ₁, c ₂) refers to the semantic measure introduced by Batet et al. (Reference Batet, Sánchez and Valls2011) to determine similarity based on the normalized ratio between the amount of uncommon features and the sum of common and uncommon features. The last column presents the results of the proposed measure using the descendants weighting technique.

Table 5.

Evaluation results of hierarchical feature-based measures

In each row, the best correlation value is shown in bold.

The third experiment aims to compare feature-based measures with hierarchical feature-based measures and weighting-based measures. The synset can be represented as a set of features using direct relations in WN for measuring semantic similarity. In this experiment, the taxonomical relations (i.e., hypernyms and hyponyms) and non-taxonomical relations (e.g., substance meronym and member of domain usage) are used as a set of representative features to measure the semantic similarity between concepts. Similar to the procedure for hierarchical feature-based measures, (16)–(19) are used to measure the semantic similarity between two sets (representative features). The Sim_Batet measure is excluded because it was proposed to estimate the semantic similarity between hierarchical features. Table 6 presents a summary of the overall evaluation results for different representative features (direct hypernyms, direct hyponyms, combination of hypernyms with hyponyms, non-taxonomical relations, and combination of all direct relations in WN) using four similarity measures (referred to as Sim₁–Sim₄). In addition, this table presents the results of algorithms that were introduced in Saif et al. (Reference Saif, Ab Aziz and Omar2017) and Saif et al. (Reference Saif, Ab Aziz and Omar2014). The last two rows of the table show the best results obtained in hierarchical feature-based and weighting-based measures.

Table 6.

Evaluation results of feature-based measures

*First rank. **Second rank.

4.4 Statistical significance of the results

In statistical hypothesis testing, statistical significance plays an important role in determining whether the null hypothesis should be rejected or retained. This significance is measured using the p value, which is a number between 0 and 1, describing the probability of rejecting the null hypothesis when it is true. In most empirical analyses, results yielding a p value of 0.05 are considered on the borderline of statistical significance. Results are considered statistically significant when the p value is less than 0.01. The p values less than 0.005 indicate that the results are considered highly statistically significant. In measuring semantic similarity, two cases of statistical significance are used to compare two groups. The first case involves a comparison between human judgments and machine similarity values. The second case involves a comparison between the proposed measure and the baseline measure. In the first case, the null hypothesis is that the scores of similarity in a gold-standard dataset and the computed similarity values are unrelated. From the correlation coefficient and t test, the p value in this case, as shown in Table 7, is computed using a software introduced by Wessa (Reference Wessa2016).

Table 7.

Statistical significance of the difference between the proposed measure and human judgments

The results marked with an asterisk indicate statistically significant improvements in the weighting-based results at the 0.05 level of significance.

In the second case, Steiger’s technique with z testFootnote ³ (Steiger Reference Steiger1980) that was introduced for dependent correlations is used to measure the statistical significance of the difference between the performance of the proposed method and that of the baseline methods. Given that the proposed measure and each baseline method compute the correlation coefficients against the same test set, Steiger’s technique is more suitable than Fisher’s z transformation that is used for independent correlation coefficients, as presented in Gabrilovich and Markovitch (Reference Gabrilovich and Markovitch2009). We let X be the human scores in the test set, Y be the similarity results obtained by the proposed method, and Z be the results obtained by the baseline method. Steiger’s technique is employed to examine whether the correlation score XY (representing the accuracy of the proposed method) is significantly different from XZ (representing the accuracy of the baseline method) in consideration of the correlation score YZ. The statistical significance (two-tailed p value) of the difference between the Pearson’s correlation of the proposed measure and that of the baseline measures (Sim ₁ and Sim _Batet) in the evaluation test sets is presented in Tables 8 and 9.

Table 8.

Statistical significance of the difference between the proposed measure and the baseline method (Sim ₁)

The results marked with an asterisk indicate statistically significant improvements in the weighting-based results at the 0.05 level of significance.

Table 9.

Statistical significance of the difference between the proposed measure and Sim_Batet

The results marked with an asterisk indicate statistically significant improvements in the weighting-based results at the 0.05 level of significance.

4.6 Comparison with IC-based measures

IC-based measures have been introduced for modeling the semantic similarity between two lexical concepts based on the information they share in common. These measures are classified according to the type of sources which are used for computing the information content of the concepts into two methods: corpus-based and ontology-based (Intrinsic) methods. The corpus-based method requires a tagged corpus for computing the IC of a concept, while the intrinsic method exploits the taxonomic parameters to determine the IC of a concept. In this section, to ensure a fair comparison of the results, the proposed method is compared with only the intrinsic IC-based method which shares some aspects (using the taxonomical parameters) with the proposed method.

The taxonomical parameters are used in path-based measures to address the specificity issue and in IC measures to determine the IC of concepts instead of using the tagged corpus. The results obtained by the proposed measure using four weighting techniques are compared with those obtained by other intrinsic IC-based similarity measures to investigate the difference between using the parameters in IC measures and the proposed measure. Lastra-Díaz and García-Serrano (Reference Lastra-Díaz and García-Serrano2015a) presented a comprehensive evaluation and comparison of several IC measures. In the datasets that contain a large number of word pairs (WS-203 and SimLex-666), Table 10 shows the best result obtained for IC-based similarity measures using different settings that produce the best value for each measure as well as the results obtained by the proposed measure.

Table 10.

Comparison of IC-based measures and the proposed measure

The results rendered in bold indicate the best results in terms of μ correlation in each dataset.

The results achieved in terms of μ correlation by the proposed measure when using the descendant-based weighting technique are the highest (0.706 and 0.611 in WS-203 and SimLex-666, respectively) compared with the best results obtained from the family of IC-based measures. Although these IC-based measures use the same taxonomical parameters to assess similarity, the proposed measure better captures semantic evidence on the semantic association between words, thus enabling a more accurate differentiation of words and a more precise similarity measurement. This finding can be attributed to the fact that the proposed measure considers the weighting of shared and unshared features between two concepts instead of the weighting of the two concepts with their least common subsumer in IC-based measures.

Based on comparative results, it is important to note that the improvement of the proposed method against IC-based measures is less noticeable, especially for the larger dataset. Previous experimental results (Lastra-Díaz and García-Serrano Reference Lastra-Díaz and García-Serrano2015a; Aouicha et al. Reference Aouicha, Taieb and Ezzeddine2016) have shown a similar observation when comparing among taxonomical IC-based measures. For the WS-203 dataset, the improvement ratio (using μ correlation) of the best IC-based measure (Yuan et al. Reference Yuan, Yu and Wang2013) is 0.005 against the earlier taxonomical IC-based measure (Seco et al. Reference Seco, Veale and Hayes2004). This is largely due to that the taxonomical measures depend mainly on structural features which have limitations in terms of completeness and balancing on granularity. For instance, based on statistics of WordNet, around 79 per cent of concepts have the same descendant (equal to zero) which means that these concepts have the same degree of specificity.

In terms of the performance of WN-based semantic measures, it could be observed that the accuracy of these measures on WS-203 and SimLex-666 datasets is limited, since the best accuracy values (in terms of μ correlation) are 0.611 and 0.706 on these datasets, respectively. This means that the semantic similarity scores of human judgments and WN-based measures did not correlate well on the WS-203 and SimLex-666 datasets. This can be interpreted using two reasons which are related to the knowledge source (WordNet) and the datasets used in the evaluation. The first reason is the limitations of the WN in terms of the coverage and completeness of the semantic relations. Although the WN has been widely used for measuring semantic similarity in the literature, it still requires updating to enrich its semantic relations. The second reason is the difference in the granularity of semantics between the evaluation datasets (the coarse-grained annotation) and the WN (the fine-grained annotation). In the coarse-grained annotation, the annotators pay more attention to the general concepts for quantifying the semantic similarity between two words. On the SimLex-666 dataset, there is a number of word pairs which are synonyms (the semantic similarity between each word pair equals one) in WN and have been annotated by different human judgment scores. For example, the word pairs ‘door-doorway,’ ‘molecule-atom,’ ‘mouth-lip,’ ‘couple-pair,’ and ‘teacher-instructor’ have been annotated to have the semantic similarity scores with a [0–1] scale: 0.540, 0.645, 0.710, 0.833, and 0.925, respectively. However, all WN-based semantic measures return the same score (1, the highest score) for all synonyms.