1 Introduction
Technical terms such as biomedical terms are constantly being generated and need to be correctly translated into numerous languages in order to facilitate the flow of knowledge in a technical community. Although comprehensive monolingual lexicons of biomedical terms are created for the English language, only a fraction of those terms are translated into other languages. For example, the Unified Medical Language System Metathesaurus, one of the comprehensive multilingual medical resources covering twenty-one languages, contains 75.1 per cent English terms, 9.99 per cent Spanish terms, 2.22 per cent Japanese terms, 1.82 per cent French terms, and only 10.87 per cent for terms in all other languages.Footnote 1 The contrastingly disproportionate representation of languages other than English in the Unified Medical Language System demonstrates the necessity to invent methods that can automatically detect translations for English biomedical terms in other languages. Considering that technical fields such as biomedicine are continuously growing, it is both labour-intensive and costly to manually translate all the novel biomedical terms into all target languages. Even a more critical factor is the lack of bilingual experts that can participate in such a large-scale manual annotation process. Therefore, it is necessary to invent methods that can automatically detect translations across biomedical lexicons.
Having been provided with two lists of biomedical terms corresponding to a source and a target language, we propose a method to detect pairs of terms that are translations between the two languages. First, we represent a term using two types of features: character n-grams extracted from the term under consideration, and contextual features consisting of words that appear within a contextual window surrounding the term under consideration. Character n-grams can be considered as intrinsic features because they are extracted from the term itself and represent syllables and inflections in a language. On the other hand, contextual features can be considered as extrinsic features as they represent the other words that are frequently associated with the term under consideration. For example, (‘dwarfism’, ‘nanisme’) is pair of English–French terms that has (‘dw’, ‘wa’,. . ., ‘ism’) as English character n-gram features and (‘na’, ‘an’,. . ., ‘sme’) as French character n-gram features. Regarding the contextual features, the most frequent words that appear around dwarfism term are condition, syndrome,. . ., etc and for nanisme the words such as atteindre and origine appear frequently in its context. The two types of features are mutually exclusive by design. Therefore, by combining the two feature spaces we can potentially better represent the properties of a term in a language.
We train a binary classifier using the two feature types described above. Specifically, we use a training dataset of term pairs (wS, wT ) where the two terms wS and wT are selected respectively from the source S and the target T languages, and wT is the translation of wS . However, naive concatenation of feature vectors representing each term is insufficient to represent the training instances because, the character n-gram and contextual features extracted from different languages do not necessarily overlap. Moreover, the exact method of weighting character n-gram and contextual features enabling them to be used within the same classifier is non-obvious.
We empirically compare different feature weighting and concatenation approaches for this purpose. Specifically, we consider four basic feature spaces in our model: source-language n-gram feature space (Sn ), source-language contextual feature space (Sc ), target-language n-gram feature space (Tn ), and target-language contextual feature space (Tc ). We generate both first-order and second-order combinations of those feature spaces resulting in six different cross-lingual feature spaces. The exact procedures for generating feature spaces are detailed in Section 3.1.4. To our knowledge, such exhaustive analysis of cross-lingual feature spaces has not been conducted before, which can be considered as an important novel contribution of this work.
The classifier returns a confidence score indicating the probability that two terms are translations. Given a term wS , we use the trained classifier to compare wS with each of the terms wT in the target biomedical term list and rank wT according to their probability of being a correct translation for wS . This is particularly useful for human annotators when compiling bilingual term dictionaries as it reduces the effort from going through a large list of terms to identify the correct translations to a few top-ranked candidates.
We evaluate our proposed method on the English–French language pair. Experiments have been conducted for character n-grams and contextual features separately and also their incorporation. Our experiments show that using character n-gram features to measure the probability score of a pair of terms to be a translation achieves more accurate results in terms of mean average precision (MAP) than using contextual features. The proposed method achieves MAP scores of respectively eighty-one per cent and sixty-three per cent for character n-grams features and contextual features. Moreover, the combination of character n-grams and contextual features produces a MAP score of eighty-six per cent, which indicates that we can achieve more effective performance by combining the two feature spaces as opposed to when they are used separately.
We first discuss relevant prior related work in Section 2. Next, in Section 3, we explain the proposed methods for learning similarity of terms across languages. We present experimental results in Section 4 and then conclude the paper.
2 Related work
Character n-grams extracted from a technical term provide useful etymological and inflectional information about that term. Prior works in machine translation have used character n-grams as features for detecting translations of words (Mcnamee and Mayfield Reference Mcnamee and Mayfield2004; Namer and Baud Reference Namer and Baud2005; Vilar, Peter and Ney Reference Vilar, Peter and Ney2007; Xi et al. Reference Xi, Tang, Dai, Huang and Chen2012). Vilar et al. (Reference Vilar, Peter and Ney2007) treat a word as a sequence of characters to establish the translation model of words between two related languages that have corresponding structures such as Catalan–Spanish and Spanish–Portuguese. Their experimental results show that character-based translation is possible between languages that use similar alphabets such as English and French. Character-level translation models have been particularly effective for learning translation models between resource poor languages (Tiedemann Reference Tiedemann2012; Tiedemann and Nakov Reference Tiedemann and Nakov2013). Combining word-level and character-level features has shown to further improve the accuracy of these models (Nakov and Tiedemann Reference Nakov and Tiedemann2012).
Earlier work on building bilingual dictionaries using machine-learning approaches with character n-gram features has shown encouraging results (Claveau Reference Claveau2008; Erdmann et al. Reference Erdmann, Nakayama, Hara and Nishio2009; Kontonatsios et al. Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014b). Kontonatsios et al. (Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014b) propose a novel method to recognize semantically equivalent biomedical terms in language pairs using a random forest (RF) classifier (Breiman Reference Breiman2001). They exploit the internal structure of sequences using the character n-grams to align terms across languages. Their method performs robustly on two language pairs: English–French and English–Chinese. Erdmann et al. (Reference Erdmann, Nakayama, Hara and Nishio2009) train a support vector machine classifier using labelled term-translation pairs for the purpose of automatically constructing a bilingual dictionary. They experimentally show that the trained classifier can predict the correctness of unseen term-translation pairs with high accuracy. In bilingual information retrieval, Mcnamee and Mayfield (Reference Mcnamee and Mayfield2004) found that using character n-gram models is highly effective even for not closely related languages.
On the other hand, distributional similarity has been widely used in NLP for several purposes such as discovering the semantic relationships between biomedical terminologies to organize them into groups (Fan and Friedman Reference Fan and Friedman2007; Weeds et al. Reference Weeds, Dowdall, Schneider, Keller and Weir2007). A distributional semantic model has been successfully applied in order to automatically discover pairs of semantically related words in large monolingual text corpora (Rapp Reference Rapp2008; Dias et al. Reference Dias, Moraliyski, Cordeiro, Doucet and Ahonen-Myka2010). However, those works focus on applying distribution similarity models in a single language. In contrast, cross-lingual distributional similarity models compare the distributional similarity for technical terms across pairs of languages using a context vector to find the target words that have the most similar distributions for a given source word (Rapp Reference Rapp1999; Chiao and Zweigenbaum Reference Chiao and Zweigenbaum2002; Saralegi, San Vicente, and Gurrutxaga Reference Saralegi, San Vicente and Gurrutxaga2008; Kontonatsios et al. Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014b). In the methods developed by Rapp (Reference Rapp1999), Chiao and Zweigenbaum (Reference Chiao and Zweigenbaum2002) and Saralegi et al. (Reference Saralegi, San Vicente and Gurrutxaga2008), a context vector is transferred from a source word into target language’s target context vector relying on existing bilingual lexicon in order to make the vectors comparable. Then, a similarity score is computed between translated context vectors with each target word context vector to produce a ranked list of candidate translations. Similarly, Kontonatsios et al. (Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014b) use the contextual model to compile a bilingual dictionary of technical terms from English–Spanish comparable corpora. They observed that the character n-gram model significantly outperformed the contextual model in terms of top-k translation accuracy.
To take advantage of lexical composition and distributional similarity, Kontonatsios et al. Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014a developed a hybrid method that combines compositional and contextual similarity scores as features in a linear classifier. By investigating the performances using top-1 and top-20 translation accuracy, they show that combining those two different scores improves the top-20 translation accuracy, whereas minor improvements are observed for the top-1 accuracy. Unlike our proposed method, they use a bilingual seed dictionary to map the context vector between the two languages in order to make them comparable using similarity measures such as the cosine similarity. Finally, both of the scores obtained by using the character n-grams-based model and the context vectors are combined as features to train a linear classifier.
Translating contextual features with the usage of a dictionary arises several issues. First, a competent bilingual dictionary is required as the seed dictionary. Often a small amount of translations is insufficient to translate a large contextual feature space. Therefore, both the quality and the coverage of the translation dictionary directly affects the performance of the term translation method. Second, not all contextual features (e.g. bigrams) are proper phrases in a language. Therefore, only a fraction of the contextual features could be correctly translated using a bilingual dictionary. On the other hand, the main purpose of our work is to build a wide-coverage bilingual dictionary. Therefore, it would be unrealistic to assume the availability of a dictionary to translate contextual feature spaces. In our proposed method, we train a classifier using contextual features of source and target languages without translating contexts from source language to the target language.
3 Measuring cross-lingual term similarity
In this section, we describe our proposed method for measuring the similarity between source and target language terms for detecting translations. We model the problem of detecting term translation as a binary classification task. First, we describe the features that we use to represent words in a language. Second, we describe the binary classifier we train for detecting whether two words are translations. The confidence that the trained classifier has about two terms being translations is considered as the degree of cross-lingual similarity between those terms.
3.1 Feature engineering
Designing features for a classifier is an integral part of the machine-learning pipeline. We represent a term in a language using two types of features. Our first feature type is character n-grams extracted from a term that we would like to represent. Character n-gram features capture useful properties about a term such as its inflections. For example, the trigram prefix mal as in malnutrition often indicates the deficiency of a substance. Character n-gram features are described in detail in Section 3.1.1. The second feature type we use to represent a term is contextual features – words that appear within a certain contextual window surrounding the term under consideration. According to the distributional hypothesis, words that are semantically similar occur in similar contexts. Therefore, if a source term and a target term co-occur with similar contexts, then it is likely that those two terms are translations. The contextual features we extract for representing terms are described in Section 3.1.2. Character n-gram features and contextual features are complementary in the sense that character n-gram features are extracted from the term under consideration and contextual features are extracted from the contexts in which the term appears. Consequently, we concatenate the two feature types to create a hybrid feature space as described later in Section 3.1.3.
Given a feature vector for a source term and a target term, which are in a translational relationship, we must construct a single feature vector to represent the training instance consisting of the two terms. We can then train a binary classifier using the pairs of terms in which the two terms are translations of each other (i.e. positive training instances) and not (i.e. negative training instances). However, it is non-obvious as how to construct a single feature vector that captures information from the two feature vectors representing the source and the target terms. For this purpose, we compare two methods: first-order feature combinations, and second-order feature combinations, which we detail in Section 3.1.4.
3.1.1 Character n-gram feature space
Character n-grams are consecutive sequences of characters in the order of their appearance in source/target terms, where n corresponds to the number of characters in each sequence. The Examples of English and French character n-gram features are presented in Table 1. Multi-word terms often demonstrate a compositional structure. Therefore, character n-grams are likely to capture semantic units such as inflections or etymological components in large terms. The number of n-grams that can be generated from a set of terms grows exponentially with the length of the n-gram. Consequently, we consider n-grams for n = 2, 3, 4, and 5 in this work, and select the most frequent n-grams to overcome this n-gram explosion problem. We denote the n-gram feature space for the source and target languages respectively by Sn and Tn .
Example of character n-grams features for English biomedical term antibody, and its French translation anticorps
3.1.2 Contextual feature space
The contexts in which a term occurs provide useful hints regarding the semantics of that term (Baroni and Lenci Reference Baroni and Lenci2010). In fact much prior work in lexical semantics such as similarity measurement (Lin Reference Lin1998), or word sense disambiguation (Chan and Ng Reference Chan and Ng2005) exploit this contextual information. We extract unigrams and bigrams as contextual features from the contexts in which a term occurs in a corpus in order to represent that term. We use a window of seven words surrounding the term under consideration as the context of that term for extracting its contextual features. Prior work on vector space models of semantics (Turney and Pantel Reference Turney and Pantel2010) have shown that it is sufficient to consider window sizes in the range from five to ten tokens for capturing the local context of a word. We denote the source and target language contextual feature spaces respectively by Sc and Tc . Tables 2 and 3 respectively show a subset of English (EN) and French (FR) contextual features. Each row in Table 2 has its translation term represented in the corresponding row in Table 3. A prefix that identifies whether it originated from the source or the target language and whether it is a character n-gram or a contextual feature is assigned to each feature.
English Contextual features representing English biomedical terms alopecia, abnormality, and acetaldehyde. The corresponding PPMI values are shown in the table
French Contextual features representing French biomedical terms calvitie, malformation, and acétaldéhyde. The corresponding PPMI values are shown in the table
Weighting co-occurrences between a term and its contextual features is important because contextual features that do not often co-occur with a term are unlikely to be salient features representing that term. We use positive pointwise mutual information, which has shown excellent performance in numerous NLP tasks (Turney and Pantel Reference Turney and Pantel2010) for this purpose. Pointwise mutual information takes into account the frequency of source term and source language contextual features (similarly for target terms). Pointwise mutual information takes values in the range [0, +∞) and is calculated as follows:
$$\begin{equation}
{\rm PPMI}(w_1,w_2)= \max \left( 0, \log \left(\frac{P(w_1,w_2)}{P(w_1)P(w_2)}\right) \right),
\end{equation}$$
where P(w 1, w 2) is the joint probability of the two words w 1 and w 2, P(w 1) and P(w 2) are respectively marginal probabilities of w 1 and w 2. All probabilities are estimated using corpus counts.
3.1.3 Hybrid models
Character n-gram features are extracted directly from a term under consideration, whereas contextual features are extracted from the contexts in which that term occurs. Therefore, the two feature spaces are mutually exclusive and capture different aspects of a term. To incorporate both feature types, we propose a hybrid feature space by concatenating the two feature spaces into a single feature space. The block diagram shown in Figure 1 illustrates the proposed hybrid model, which includes character n-gram and contextual features in the same classifier. During training, each training instance, given as a pair of source and target terms, is represented by the concatenated vector of the four feature vectors: (a) source term’s character n-gram feature vector, (b) source term’s contextual feature vector, (c) target term’s character n-grams feature vector, and (d) target term’s contextual feature vector. We consider further variants for constructing a feature vector to represent a training instance in Section 3.1.4.
Integrating character n-gram and contextual features into the same classification model (hybrid model).
3.1.4 First and second-order feature spaces
We model the problem of detecting whether a given target language term wT is the translation of a source language term wS , as a binary classification task. Specifically, we classify term-pairs (wS, wT ) as positive or negative indicating whether wT is respectively a correct translation or an incorrect translation of wS . We describe the exact learning algorithm later in Section 3.2 and describe the procedure we propose to represent a pair of terms (wS, wT ) using a feature vector f(wS, wT ).
As we already described in Sections 3.1.1 and 3.1.2, we use two types of features, giving rise to four feature vectors representing the two terms wS and wT as follows:
-
• f(wS, Sn ): Source language character n-gram feature vector for the source term;
-
• f(wS, Sc ): Source language contextual feature vector for the source term;
-
• f(wT, Tn ): Target language character n-gram feature vector for the target term;
-
• f(wT, Tc ): Target language contextual feature vector for the target term.
We propose two methods for creating a single feature vector to represent a term-pair (wS, wT ), given the above-mentioned four feature vectors.
First, we consider linear concatenation of character n-gram, contextual, and hybrid feature spaces to construct three flavors of feature vectors as follows:
-
• Sn + Tn : The concatenation of source and target character n-grams feature vectors f(wS, Sn ) and f(wT, Tn );
-
• Sc + Tc : The concatenation of source and target contextual feature vectors f(wS, Sc ) and f(wT, Tc );
-
• Sn + Sc + Tn + Tc : The concatenation of source and target character n-grams and contextual features (hybrid model), which corresponds to the concatenation of the four vectors f(wS, Sn ), f(wS, Sc ), f(wT, Tn ), and f(wT, Tc ).
The linear concatenation of an n-dimensional vector x = (x 1, x 2, . . ., xn )⊤, and an m-dimensional vector y = (y 1, y 2, . . ., ym )⊤ is defined as the (n + m) dimensional vector (x 1, x 2, . . ., xn, y 1, y 2, . . ., ym )⊤. We refer to the feature spaces created by the linear concatenation of feature vectors for wS and wT as first-order feature vectors.
Although first-order feature vectors consider the features for both source and target language terms, they do not consider the correlation between features across languages or feature spaces. To overcome this problem, we propose second-order feature vectors that are constructed considering all pair-wise combinations of features. Specifically, given an n-dimensional vector x = (x 1, x 2, . . ., xn )⊤, and an m-dimensional vector y = (y 1, y 2, . . ., ym )⊤, their second-order combination is given by the nm-dimensional vector (x 1 y 1, x 1 y 2, . . ., xnym )⊤. Numerous different combinations can be considered to create second-order features, however we have extracted and evaluated the following in-combination second-order features:
-
• Sn ⊗Tn : The pairwise combination of the source n-gram feature vector and target n-gram feature vector;
-
• Sc ⊗Tc : The pairwise combination of the source contextual feature vector and target contextual feature vector;
-
• (Sn ⊗Tn ) + (Sc ⊗Tc ): The concatenation of the above two feature vectors.
In all pairwise combinations, we multiply the two corresponding feature values to compute the feature value of the second-order feature. Multiplication operator has been found to be useful in compositional semantics models in vector spaces (Mitchell and Lapata Reference Mitchell and Lapata2008). Indeed, also in our case multiplication is more efficient than addition because unless a feature is present both in the source and the target vectors, there will be a zero second-order feature when using multiplication. We consider multiword terms and hyphenated terms as single lexical units for the purpose of character n-gram extraction and contextual feature extraction.
3.2 Translation detection as a binary classification problem
We model the problem of determining whether a target term wT is the correct translation of a given source term wS as a binary classification problem. Specifically, given a pair of words (wS, wT ), we train a binary classifier that returns + 1 if wT is the correct translation of wS , and − 1 otherwise. For training, we assume the availability of correct target language translations for a set of source language terms. We create positive training instances by coupling each source language term with its correct target language translation. We generate an equal number of negative training instances by randomly pairing a source language term with a target language term. We use the first-order and second-order feature vectors that we described in Section 3.1.4 to represent a pair of terms. During test time, the trained binary classification model can be used to predict the probability of a given target language term wT being the correct translation of a source language term wS . Specifically, the class conditional probability P( + 1|wS, wT ) is used to rank the target language translation candidates wT for a source language term wS .
Having considered two binary classification algorithms that provide class conditional probabilities: logistic regression classifier (LR), and RF (Breiman Reference Breiman2001), we briefly overview each of those classification algorithms. We use the implementations in the scikit-learn Python library for those two algorithms in our experiments.Footnote 2 The RF is an ensemble-learning mechanism in which multiple decision trees are learnt for collectively predicting the class of an instance. Similar to the decision tree algorithms, RF can build complex combinations of features to learn classification rules from training data. Several methods have been proposed in the literature for learning RF with class probability outputs (Boström Reference Boström2007). Moreover, RF has proved to be effective for very high dimensional feature spaces such as the feature spaces used in our experiments, where the number of features exceeds the number of the training instances (Díaz-Uriarte and De Andres Reference Díaz-Uriarte and De Andres2006).
As a log-linear classifier that produces probabilistic outputs, we train a binary LR. We use l 2 regularization, with the regularization coefficient determined by cross-validation, in order to reduce the overfitting to train data. Unlike the RF classifiers that consider combinations of features, the LR classifier learns a weight for each feature in the train instances that indicates the discriminative power of that feature when separating positive training instances from the negative ones.
We use stratified five-fold cross validation method with MAP, described later in Section 4.2.1, to tune classifier’s parameters. Each fold uses twenty per cent of the train data for validation, and the remainder of eighty per cent for training. In stratified cross-validation, the folds are made by preserving the percentage of samples for each class. In the case of the RF classifier, the number of trees in the forest influences the classifier’s performance. In our preliminary experiments, we observed that the optimal number of trees is different for RF depending on the feature space. These differences could be explained by the fact that the first-order feature spaces require more decision trees to capture the association among source and target features compared to second-order bilingual features because, in the second-order feature spaces, we have already generated pairwise combinations of features. Therefore, we conclude that the number of trees required to boost the performance of a classifier depends on the training feature space. For this reason, we automatically adjust this parameter for each experiment before building a model for training.
One of the main parameters to adjust when using the regularized LR classifier is the l 2 regularization coefficient c. In theory, smaller values for c result in a higher regularization, that is, they increase the complexity of the model (i.e. model with non zero features). According to our findings, this parameter affects the models differently depending on the feature space. For example, the most appropriate choice of regularization coefficient for (Sn ⊗Tn )feature space is 1,000, whereas c equal to 10 results in a more accurate model for (Sn ⊗Sn ) and (Sc ⊗Tc ) feature spaces. Because of those differences, we applied the stratified k-fold cross validation to each experiment with the purpose of identifying the best value of a parameter for each feature space.
4 Experiments
4.1 Dataset
For training a classifier, we use a bilingual dictionary of English and French biomedical terms. We use the dataset described in Kontonatsios et al. (Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014a) to train and evaluate our proposed method. First, a set of 9, 090 English biomedical terms with one or more French translations listed is manually selected from the Unified Medical Language System metathesaurus. The selected terms cover disease names, drug names, and chemical substances. The terms are selected without restricting to any particular biomedical subdomain, and can be considered as a broad collection of terms covering different subdomains. We refer to this term translation pairs as the seed dictionary.
Kontonatsios et al. (Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014a) created a comparable biomedical corpora using the Wikipedia medical articles as a source. Having selected 4,000 English biomedical terms from the Unified Medical Language System metathesaurus, they selected the Wikipedia articles whose titles contain one or more terms from the set of selected biomedical terms. Then, they follow the Wikipedia interlingual links to retrieve the semantically related articles in the target language. Although the accuracy of the information collected from cloud sourced encyclopedic resources such as Wikipedia is debatable, the issue is complementary to the translation prediction problem we consider in this paper. The collected comparable corpora is further divided into train and test parts for extracting contextual features. Table 4 summarizes the statistics of the data used in our experiments.
Dataset statistics
We generate positive training instances for training the binary classifiers (LR and RF) by pairing each English term with one of its French translations. We generate negative training instances by pairing an English term with a randomly chosen French term. We verify that the generated negative training instance term-pairs are not correct translations by comparing them against the seed dictionary. The technique of randomly sampling pseudo-negative instances has been successfully used in several tasks in NLP such as noise contrastive estimation for word representation learning (Mikolov, Chen and Dean Reference Mikolov, Chen and Dean2013a; Mikolov, tau Yih and Zweig Reference Mikolov, Tau Yih and Zweig2013b; Bollegala, Maehara and ichi Kawarabayashi Reference Bollegala, Maehara and Kawarabayashi2015), and binary classifier training in semantic similarity measurement (Bollegala, Matsuo and Ishizuka Reference Bollegala, Matsuo and Ishizuka2007). Although the technique is suboptimal compared to manually creating negative instances, it is popularly applied due to the fact that it obviates the manual effort required in data annotation.
In total, our training dataset consists of 9, 090 positive and 9, 090 negative training instances. The train dataset is denoted by {((S 1, T 1), y 1), ((S 2, T 2), y 2), . . ., ((Sn, Tn ), yn )}, where S 1,S 2,. . ., Sn stand for the feature vectors of source terms, T 1,T 2,. . ., Tn stand for the feature vectors of target terms and y 1,y 2,. . ., yn stand for labels. A positive training sample is defined as a pair of source and target terms (Si, Ti ), where Ti is the correct translation of Si . Positive instances are assigned a label of + 1, whereas a negative instance a label of − 1. A feature vector is created to represent a training instance by concatenating the two feature vectors representing the source and target terms. In addition to the seed dictionary we use for training, we select 1, 000 English terms and their French translations as test data. We verified that there is no overlap between the sets of terms selected for train and test purposes.
4.2 Measuring the performance
4.2.1 Mean average precision
In our proposed system, the ranked probabilities of target language candidates are returned in the descending order. An efficient term similarity prediction method across languages must assign a higher probability score to the correct target term translation for a given source term. Therefore, if our learned model ranks the correct target language (French) translation as high as possible, then our method could be considered as effective. This can be evaluated using MAP, which takes into account the order in which the returned target terms are presented. MAP is a strict measure in the sense that it not only considers whether the correct results are ranked among the top-k results but also takes into account whether those correct results are ranked as high as possible. The MAP for a set of source language (English) test terms is the MAP for each test term and is calculated as follows:
$$\begin{equation}
{\rm MAP}= \cfrac{\sum _{i=1}^{N} {\rm AveP}(i)}{N}.
\end{equation}$$
Here, N stands for the total number of source language test terms and AveP(i) is the average precision for ith source test term, which is determined by the following formula:
$$\begin{equation}
{\rm AveP}(i)=\cfrac{\sum _{r=1}^{N} {\rm Precision}(r) \times {\rm Correct}(r)}{\text{No.of correct target translation in top {\it k}}},
\end{equation}$$
where Correct(r) is a binary-valued function that returns 1 if the target term at rank r is a correct translation for a given source term, otherwise it returns 0. Precision(r) is the precision at rank r, which is defined as
$$\begin{equation}
{\rm Precision}(r)=\cfrac{\text{No.of correct target translations in top {\it r}}}{r}.
\end{equation}$$
In our test dataset, most of the English test terms have only one correct translation. However, a small set has two or three correctly corresponding French terms. For example, English test term ‘mucosa’ has the correct French terms ‘muqueux’ and ‘muqueuse’, the system orders the French candidate as in Table 5 (top-5).
An example of calculating average precision for the translation candidates retrieved for the English term mucosa. Two correct French translations are ranked at the first and fourth places
4.2.2 Top-k translation accuracy
In addition to MAP, we use top-k translation accuracy as an evaluation measure. Top-k accuracy is the percentage of source terms for which their correct translations were encountered among the top-k ranked target terms and is defined as follows:
$$\begin{equation}
top-{\it k} accuracy =\cfrac{no. of terms whose correct translation is below or equal to rank {\it k}}{N} \times 100.
\end{equation}$$
Prior works on biomedical term translation have used top-k accuracy as an evaluation measure (Rapp Reference Rapp1999; Chiao and Zweigenbaum Reference Chiao and Zweigenbaum2002; Kontonatsios et al. Reference Kontonatsios, Korkontzelos, Tsujii and Ananiadou2014a). In this evaluation, we calculate the translation accuracy on the top-k of the returned ranked list for the target terms. For example, when translating English biomedical terms into French, top-10 translation accuracy indicates the percentage of English test terms for which at least one of the correct French translations is ranked among the top-10 ranked candidates. Top-k accuracy is a useful measure when the proposed method is used as a suggestion system for human translators because even if all of the correct translations are not ranked at the top, a human translator can benefit from a system that lists at least one correct translation among the top ranks so that he or she can select from that list. First, we determine the rank of the correct translation of each test term and then aggregate them, where N stands for the total number of English test terms. We compute the top-k accuracy for k values in the range from 1 to 20.
High MAP values often indicate high top-k values, although the reverse is not always true. Therefore, MAP can be considered as a more sensitive evaluation measure than the top-k. However, we use top-k translation accuracy in addition to MAP so that we can with higher accuracy compare our results with the results reported in prior work that uses top-k as the only evaluation measure.
In the case of one classifier with specific feature space performing more accurately than the other one, it is important to determine if the two classifiers are statistically significantly different from each other. We use the binomial exact test with Clopper–Pearson confidence intervals (Clopper and Pearson Reference Clopper and Pearson1934) in order to compare the performance obtained by different feature spaces and their combinations proposed in the paper. The Clopper–Perason confidence interval for a confidence level α can be computed as
$$\begin{equation}
\left[B\left(\cfrac{\alpha }{2};x,n-x+1\right),B\left(1-\cfrac{\alpha }{2};x+1,n-x\right)\right].
\end{equation}$$
Here, x stands for the number of correctly classified instances out of a total of n test instances. B(α; x, n) is the cumulative probability density function of the binomial distribution. If the intervals of two different classifiers do not overlap, it can be concluded that their performances are statistically significantly different.
4.3 Results
Figure 2 shows the MAP scores achieved by the RF and LR on first-order feature spaces. The RF shows good performance using first-order character n-gram and contextual features because the decision trees in the RF are able to automatically identify the associations among source and target features. In contrast, the LR performed poorly using the first-order n-gram or contextual features spaces. The LR is a log-linear classifier and does not combine the source and target first-order feature spaces when predicting whether a target term is a translation of a given source term.
MAP of the RF and LR classifiers on the first-order feature spaces.
When the RF is trained on first-order feature spaces, we observe that using character n-gram features alone in ranking the most similar target terms for a given source term results in more effective performance than contextual features and the integration of character n-gram and contextual features. In addition, the incorporation of first-order character n-grams and contextual features to train the RF classifier results in around eighteen per cent improvement compared to using only the contextual model.
Having implemented the RF and LR classifiers using second-order feature spaces (as shown in Figure 3), we observed that the second-order features significantly improved the performance of the (log-linear) LR classifier. The LR outperformed the RF by one per cent, forty per cent, and five per cent respectively in character n-gram, context and combined models. Experimental results demonstrate that the highest performance is achieved with the LR when integrating character n-gram and contextual second-order features (86.12 per cent), which gives a five per cent improvement over character n-gram alone, and approximately twenty-three per cent higher than contextual feature performance. All improvements reported are statistically significant according to the Clopper–Pearson confidence interval under α = 0.05.
MAP of the RF and LR classifiers on the second-order feature spaces.
Due to the fact that the correct target translation of a source term is closely associated with the similar contextual features, the incorporation of contextual features enables the model to detect cross-lingual contextual similarity, thereby improving the classification accuracy. Moreover, for the RF, the highest performance was achieved with the hybrid model as opposed to using each feature space separately. What is also noticeable with the second-order feature space experiments is the fact that the accuracy of the RF classifier with contextual features dropped significantly (by twenty per cent). Table 6 summarizes the performance achieved by the LR and RF classifiers with all different feature spaces using MAP. It is noteworthy that the LR and RF achieved the highest performance with the hybrid model. Although it is difficult to compare the reported results in prior work on biomedical term translation due to of the differences in train/test data, the pre-processing methods used, and evaluation criteria that have been employed, we can consider the superiority of the second-order feature spaces proposed in this paper over the first-order feature spaces (consisting of linear combinations of character n-grams and contextual features) used in much prior work as implied by the superiority of the proposed method over prior work on this task.
MAP scores obtained by the LR and RF classifiers for different feature spaces
Table 7 illustrates an interesting aspect of the probabilities assigned by the LR classifier between source and target test terms using each feature type separately as well as their combination. From Table 7, we see that character n-gram features are more reliable than contextual features. In other words, the LR mostly assigns comparatively higher probabilities to relevant terms when trained on character n-grams features as opposed to when done so using contextual features. This might be because that character n-gram features are useful for closely related languages such as English and French, as in our case. Another significant point is that probabilities show how incorporating those two types of features increases the probability of the relevant source/target term as well as decreases the probability of the irrelevant pairs.
Probabilities given by the LR classifier for pairs of test terms
Figure 4 illustrates the top-k translation accuracy for the LR classifier with character n-grams, contextual and hybrid models. For k values in the range from 1 to 20, the highest top-k accuracy is achieved with LR when a hybrid model has been learnt, with 69.68 per cent of English test terms having their translation listed at rank 1, and that being 88.26 per cent at rank 20. Experimental results for the character n-gram model show that about sixty-five per cent of the English test words have their correct translation as the first ranked word, and the correct translation is found among the top-20 candidates for 82.5 per cent of those terms. Therefore, using a hybrid model improves the accuracy of the LR by approximately four per cent for the top-1 results, and by six per cent for top-20 translation candidates. The lowest top-k translation accuracy of the LR is observed for contextual model in which the correct translation is ranked first for only 25.5 per cent of the test terms, and is found among the top-20 candidates for 76.5 per cent of those terms.
Top-k translation accuracy of the LR trained on the second-order features.
In Figure 5, top-1 to top-20 translation accuracy of the RF classifier is shown with the second-order character n-grams, contexts, and hybrid models. From top-1 to top-8, the translation accuracy of character n-gram and hybrid model do not show significant variation. However, the performance of the hybrid model increased beyond top-8 rank by four per cent compared to the character n-gram model until reaching top-20, where the hybrid model achieved 86.6 per cent and the character n-gram model achieved 82.3 per cent accuracy. Likewise, with the LR, the lowest top-k accuracy is achieved for the context model. As first-order contextual features provide us with better MAP results than the second-order features for the RF, we found it useful to present the top-k translation accuracy for both. Similarly, we obtained higher top-k translation accuracy for first-order contextual features compared to that of second-order features.
Top-k translation accuracy of the RF trained on different feature spaces.
5 Conclusion
We propose a method for measuring the similarity between biomedical terms across languages. For this purpose, we use monolingual biomedical terms for both the source and the target languages provided in the form of a bilingual dictionary. The terms are represented using character n-grams and contextual features. Character n-gram models exploit the internal structure of the term, whereas contextual models use the distributional semantics across languages. We further define two types of character n-grams and contextual features, namely first-order and second-order features. We investigate the performance of each feature type separately and in combination through a series of experiments.
Our proposed method differs from prior methods for cross-lingual biomedical term translation detection in that it incorporates two different types of features namely, character n-grams and contextual features, within the same classifier. We evaluate the two feature types using two different classifiers: RF and LR. The experimental results show that the character n-grams are more accurate when predicting the target term translations than the contextual features. Moreover, both the RF and LR classifiers report improved MAP and top-k translation accuracies with the hybrid second-order feature spaces compared to character n-grams and contextual spaces used in isolation. We believe that our method could be of assistance to human annotators when compiling the translation dictionaries for highly specialized domains such as medicine.
There are several aspects of the performance of the proposed method that can be further improved. First, the ambiguity of term translations can be addressed in the model. A single term in the source language can be translated differently into the target language depending on the context in which it is used. Although the proposed method uses contextual features for representing a term, it does not disambiguate the different senses in which a term is used in the target domain. We could extend the proposed method by using a probabilistic model p(Tm |Sn, c), where we could predict the likelihood of translating a source term Sn to the target term Tm given the context c in which Sn has been used.
Second, multiple synonymous words in a source language can be translated into a single term in the target language, and vice versa. For example, sleeplessness, somnolence, and somnolency in English are synonyms and translate to somnolence in French. On the other hand, sore throat in English is ambiguous and has different translations in French such as angina, pharyngite, or mal de gorge which are not synonyms, because there are slight differences between these terms meaning any of various inflammations of the tonsils, pharynx, or larynx characterized by pain in swallowing. A model that can pre-classify such types of words before they are translated into the target language using the proposed method could potentially solve this issue. We differ these issues to our future work on this topic.
