2.1 Standard approach

The main work in bilingual lexicon extraction from comparable corpora is based on monolingual lexical context analysis and relies on the simple observation that a word and its translation tend to appear in the same lexical contexts. This is close to the idea of the British linguist J. R. Firth (Reference Firth1957, p. 11) who said: ‘You shall know a word by the company it keeps’. (even if the context here is related to collocates). The basis of this observation consists in the identification of ‘first-order affinities’ for each source and target language: ‘First-order affinities describe what other words are likely to be found in the immediate vicinity of a given word’ (Grefenstette, Reference Grefenstette1994a, p. 279). Context vectors can describe these affinities where each vector element represents a word which occurs within the window of the word to be translated (e.g. a seven-word window approximates syntactic dependencies). In order to emphasize significant words in the context vector and to reduce word-frequency effects, the context vectors are normalized according to an association measure. The translation is obtained by comparing the source context vector to each translation candidate vector after having translated each element of the source vector with a general dictionary.

The implementation of the SA can be carried out by applying the following three steps (Rapp Reference Rapp1999; Chiao and Zweigenbaum Reference Chiao and Zweigenbaum2002; Déjean, Gaussier and Sadat Reference Déjean, Gaussier and Sadat2002; Laroche and Langlais Reference Laroche and Langlais2010; Morin et al. Reference Morin, Daille, Takeuchi and Kageura2010):

• Computing context vectors: For each part of the comparable corpus, the context of each word w is extracted by identifying the words that appear in a window of n words around w. For each word w, a context vector w is obtained. In order to identify specific words in the lexical context and to reduce word-frequency effects, we normalize context vectors using an association score such as Mutual Information (MI) (Fano Reference Fano1961), Log-likelihood (LLH) (Dunning Reference Dunning1993) or the discounted log-odds (DLO) (Evert Reference Evert2005) (see equations (1) to (3) and Table 1). After normalization, we associate to each element w_k of the context vector w the association score assoc(w_k).

• Transferring context vectors: The transfer of a word to translate w from the source language to the target language is based on the translation of each element w_k of its context vector w using a bilingual dictionary. If the dictionary provides several translations for an element, we consider all of them.Footnote ² Other techniques have been proposed in particular by Bouamor, Semmar and Zweigenbaum (Reference Bouamor, Semmar and Zweigenbaum2013) but weight the different translations according to the number of occurrences in the target language.Footnote ³ If the element is not found in the dictionary, it will not be used in the transfer process.

• Finding candidate translations: The translated context vector w of the word w is then compared to each context vector t in the target language through similarity measures such as Cosine (COS) (Salton and Lesk Reference Salton and Lesk1968) or weighted Jaccard (JAC) (Grefenstette Reference Grefenstette1994b) (see equations (4) and (5) where assoc(w_k) indicates after transfer the association score of the word w with an element k of the context vector w).

(1) $$\begin{equation} \textsc {MI}{}(\mathbf {w}_k) = log\frac{a}{(a+b)(a+c)} \end{equation}$$

(2) $$\begin{equation} \begin{array}{l@{\quad }l}\textsc {LLH}{}(\mathbf {w}_k) =& a \log (a) + b \log (b) + c \log (c) + d \log (d) \\ & +\, (a+b+c+d) \log (a+b+c+d) - (a+b) \log (a+b)\\ & -\, (a+c) \log (a+c) - (b+d) \log (b+d) - (c+d) \log (c+d) \end{array} \end{equation}$$

(3) $$\begin{equation} \textsc {DLO}{}(\mathbf {w}_k) = \log \frac{ (a+ \frac{1}{2}) \times (d+\frac{1}{2}) }{(b+\frac{1}{2}) \times (c+\frac{1}{2})} \end{equation}$$

(4) $$\begin{equation} \textsc {COS}{}(\overline{\mathbf {w}},\mathbf {t})=\frac{\sum _k assoc(\mathbf {\overline{w}_k})\,assoc(\mathbf {t_k})}{\sqrt{\sum _k {assoc(\mathbf {\overline{w}}_k})^2}\sqrt{\sum _k assoc(\mathbf {t_k})^2}} \end{equation}$$

(5) $$\begin{equation} \textsc {JAC}{}(\overline{\mathbf {w}},\mathbf {t})=\frac{\sum _k min(assoc(\mathbf {\overline{w}_k}),assoc(\mathbf {t_k}))}{\sum _k max(assoc(\mathbf {\overline{w}_k}),assoc(\mathbf {t_k}))} \end{equation}$$

Table 1.

Contingency table (where cooc(i, j) stands for the number of occurrences of words i and j)

This approach is sensitive to the choice of parameters such as the size of the context, the choice of the association and similarity measures. The most complete study to date concerning the influence of these parameters on the quality of word alignment has been carried out by Laroche and Langlais (Reference Laroche and Langlais2010).

Up till now, the SA has been used by most researchers (Fung Reference Fung1995; Rapp Reference Rapp1995; Fung Reference Fung1998; Rapp Reference Rapp1999; Chiao and Zweigenbaum Reference Chiao and Zweigenbaum2002; Déjean et al. Reference Déjean, Gaussier and Sadat2002; Gaussier et al. Reference Gaussier, Renders, Matveeva, Goutte and Déjean2004; Morin et al. Reference Morin, Daille, Takeuchi and Kageura2007; Laroche and Langlais Reference Laroche and Langlais2010; Prochasson and Fung Reference Prochasson and Fung2011; Bouamor et al. Reference Bouamor, Semmar and Zweigenbaum2013) with the implicit hypothesis that comparable corpora are balanced. The seminal work of Fung (Reference Fung1995) and Rapp (Reference Rapp1995), which laid the foundations of the standard method, presented in Boston in 1995 at the Annual Meeting of the Association for Computational Linguistics, used first non-parallel corpora that were implicitly balanced (see Table 2). Thereafter, the main work that followed on bilingual lexicon extraction used relatively balanced comparable corpora where the ratio between the size of the source and the target texts is between 1 and 1.8 (see Table 2). Since these previous works try to find an alternative to the scarcity of parallel corpora by using comparable corpora, it is not surprising that the relationship between parallel and comparable corpora led researchers to work implicitly with balanced corpora in terms of size.

Table 2.

Characteristics of the comparable corpora used for bilingual lexicon extraction

In fact, the assumption that words which have the same meaning in different languages should have the same lexical context distributions does not involve working with balanced comparable corpora. To our knowledge, no attention has been paid to the problem of using unbalanced comparable corpora for bilingual lexicon extraction. We only found mention of this aspect in Diab and Finch (Reference Diab and Finch2000, p. 1501): ‘In principle, we do not have to have the same size corpora in order for the approach to work’. Since the context vectors are computed from each part of the comparable corpus rather than through the parts of the comparable corpora, the SA is relatively insensitive to differences in corpus sizes. The only precaution for using the SA with unbalanced corpora is to normalize the association measure (for instance, this can be done by dividing each entry of a given context vector by the sum of its association scores).

2.2 Related work

In the past few years, several contributions have been proposed to improve each step of the SA.

Prochasson, Morin and Kageura (Reference Prochasson, Morin and Kageura2009) enhance the representativeness of the context vector by strengthening the context words that happen to be transliterated words and scientific compound words in the target language. Ismail and Manandhar (Reference Ismail and Manandhar2010) also suggest that context vectors should be based on the most important contextually relevant words (in-domain terms), and thus propose a method for filtering the noise of the context vectors. In another approach, Rubino and Linarès (Reference Rubino and Linarès2011) improve the context words based on the hypothesis that a word and its candidate translations share thematic similarities. Yu and Tsujii (Reference Yu and Tsujii2009) and Gamallo (Reference Gamallo2007) propose, for their part, to replace the window-based method by a syntax-based method in order to improve the representation of the lexical context. In the same way, Hazem and Morin (Reference Hazem and Morin2014) propose two strategies to combine a window-based and a syntax-based context representation that improve the performance of bilingual lexicon extraction compared to using each of the representations individually.

To improve the transfer context vector step, and increase the number of elements of translated context vectors, Chiao and Zweigenbaum (Reference Chiao, Zweigenbaum, Baud, Fieschi, Le Beux and Ruch2003) and Morin and Prochasson (Reference Morin and Prochasson2011) combine a standard general language dictionary with a specialized dictionary, whereas Déjean et al. (Reference Déjean, Gaussier and Sadat2002) use the hierarchical properties of a specialized thesaurus. Koehn and Knight (Reference Koehn and Knight2002) automatically induce the initial seed bilingual dictionary by using identical spelling features such as cognates and similar contexts. Hazem and Morin (Reference Hazem and Morin2012) propose a simple filtering process based on a relative frequency ratio to discard the non-useful entries of the seed bilingual dictionary. As regards, the problem of word ambiguities, Bouamor et al. (Reference Bouamor, Semmar and Zweigenbaum2013) carried out a word sense disambiguation process only in the target language whereas Gaussier et al. (Reference Gaussier, Renders, Matveeva, Goutte and Déjean2004) solve the problem through the source and target languages by using approaches based on Canonical Correlation Analysis and multilingual Probabilistic Latent Semantic Analysis.

The rank of candidate translations can be improved by integrating different heuristics. For instance, Chiao and Zweigenbaum (Reference Chiao and Zweigenbaum2002) introduce a heuristic based on word distribution symmetry. From the ranked list of candidate translations, the SA is applied in the reverse direction to find the source counterparts of the first target candidate translations. Then, only the target candidate translations that had the initial source word among the first reverse candidate translations are kept. Laroche and Langlais (Reference Laroche and Langlais2010) suggest a heuristic based on the graphic similarity between source and target terms. Here, candidate translations, which are cognates of the word to be translated, are ranked first among the list of translation candidates.

3.1 Smoothing techniques

Smoothing is widely used in many applications of speech and language processing. It describes techniques for adjusting the maximum likelihood estimate of probabilities to reduce overfitting and to reserve probability mass for unseen events. In other words, smoothing tends to make distribution more uniform. In this section, we briefly review the most widely used smoothing techniques.

3.1.1 Additive smoothing (Add-One)

The Laplace estimator or the additive smoothing (Lidstone Reference Lidstone1920; Johnson Reference Johnson1932; Jeffreys Reference Jeffreys1948) is one of the simplest types of smoothing. Its principle is to estimate probabilities P assuming that each unseen word type actually occurred once. If we have N events and V possible words instead of

(6) $$\begin{eqnarray} P(w)=\frac{occ(w)}{N} \end{eqnarray}$$

where occ(w) is the occurrence count of w. We estimate

(7) $$\begin{eqnarray} P_{Laplace}(w)=\frac{occ(w)+1}{N+V} \end{eqnarray}$$

Applying Laplace estimation to word co-occurrence counts supposes that: if two words co-occur n times, they can co-occur (n + 1) times. According to the maximum likelihood estimation,

(8) $$\begin{eqnarray} P(w_{i+1}|w_{i})=\frac{cooc(w_{i},w_{i+1})}{occ(w_{i})} \end{eqnarray}$$

where cooc(w_i, w _{i + 1}) is the co-occcurrence count of w_i and w _{i + 1}.

According to Laplace smoothing on co-occurrence counts,

(9) $$\begin{eqnarray} cooc_{Laplace}^{*}(w_{i},w_{j})=\frac{cooc(w_{i},w_{j})+1}{occ(w_{i})+V} \end{eqnarray}$$

Laplace smoothing performs rather poorly, mainly because this technique underestimates the probability of frequent n-grams and overestimates the probability of rare or unseen n-grams. All the unseen n-grams are smoothed in the same way and too much probability mass is shifted toward unseen n-grams. One improvement as proposed in Lidstone (Reference Lidstone1920) is to use smaller added counts following the equation below:

(10) $$\begin{eqnarray} cooc_{Lidstone}^{*}(w_{i}, w_{j})=\frac{cooc(w_{i},w_{j})+\delta }{occ(w_{i})+ \delta \times V} \end{eqnarray}$$

with δ ∈ ]0, 1].

Even if Add-δ smoothing performs better than Add-one smoothing, it is still considered as a poor smoothing technique.

3.1.2 Good-turing estimator

The Good-turing estimator (Good Reference Good1953) provides another way to smooth probabilities. It states that for any n-gram that occurs r times, we should pretend that it occurs r* times. The Good-turing estimator uses the count of things you have seen once to help estimate the count of things you have never seen. In order to compute the frequency of words, we need to compute N_c, the number of events that occur c times (assuming that all items are binomially distributed). Let N_r be the number of items that occur r times. N_r can be used to provide a better estimate of r, given the binomial distribution. The adjusted frequency r* is then

(11) $$\begin{eqnarray} r^{*} =(r+1) \frac{N_{r+1}}{N_{r}} \end{eqnarray}$$

We adapt the Good-turing estimator to co-occurrence counts as follows:

(12) $$\begin{eqnarray} cooc_{GT}^{*}(w_i,w_j) =(cooc(w_i,w_j)+1)\frac{N_{cooc(w_i,w_j)+1}}{N_{cooc(w_i,w_j)}} \end{eqnarray}$$

Good-turing is not often used by itself, it is usually used in combination with the backoff and interpolation techniques.

3.1.3 Jelinek–Mercer smoothing or linear interpolation technique

As an alternative to missing n-grams, useful information can be provided by the corresponding (n − 1)-gram probability estimate. A simple method for combining the information from lower order n-gram in estimating higher order probabilities is linear interpolation (Mercer Reference Mercer and Jelinek1980). The equation of linear interpolation is given below:

(13) $$\begin{eqnarray} \mathrm{P^{*}_{int}(w_{i+1}| w_{i})= \lambda \times P(w_{i+1}| w_{i}) + (1-\lambda ) \times P(w_{i})} \end{eqnarray}$$

where λ is the confidence weight for the longer n-gram. In general, λ is learned from a held-out corpus (i.e. a subset of the training data).

We adapt the linear interpolation to co-occurrence counts as follows:

(14) $$\begin{eqnarray} \mathrm{cooc^{*}_{int}(w_{i}, w_{j})= \lambda \times cooc(w_{i}, w_{j}) + (1-\lambda ) \times occ(w_{i})} \end{eqnarray}$$

It is useful to interpolate higher order n-gram models with lower order n-gram models, because when there is insufficient data to estimate a probability in the higher order model, the lower order model can often provide useful information. Instead of the co-occurrence counts, we used the Good-turing estimator in the linear interpolation as follows:

(15) $$\begin{eqnarray} cooc^{*}_{int}(w_{i}, w_{j})= \lambda \times cooc_{GT}^{*}(w_{i}, w_{j}) + (1-\lambda ) \times occ_{GT}^{*}(w_{i}) \end{eqnarray}$$

3.1.4 Katz smoothing

Katz (Reference Katz1987) extends the intuition of the Good-turing estimate by adding the combination of higher order models with lower order models. For a bigram wⁱ _{i − 1} with count r = c(wⁱ _{i − 1}), its corrected count is given by the equation:

(16) $$\begin{equation} cooc^{*}_{katz}(w_{i},{w_{j}})=\left\lbrace \begin{array}{@{} l@{\quad }c @{}}r^{*} & \text{if } r > 0\\ \alpha (w_{i})occ_{GT}^{*}(w_{i}) & \text{if } r = 0 \end{array}\right. \end{equation}$$

and

(17) $$\begin{equation} \alpha (w_{i}) = \frac{1-\sum _{w_{i}:cooc(w_{i},w_{j})>0}{cooc_{katz}(w_{i},w_{j})}}{1- \sum _{w_{i}:cooc(w_{i},w_{j})>0}{ occ(w_{i})}} \end{equation}$$

According to Katz (Reference Katz1987), the general discounted estimate c* of Good-turing is not used for all counts c. Large counts where c > k for some threshold k are assumed to be reliable. Katz (Reference Katz1987) suggests k = 5. Thus, we define c* = c for c > k, and

(18) $$\begin{eqnarray} r^{*}=\frac{ (r+1) \frac{N_{r+1}}{N_{r}} - r \frac{(k+1)N_{k+1}}{N_{1}} }{1-\frac{(k+1)N_{k+1}}{N_{1}}} \end{eqnarray}$$

for r ⩽ k.

3.1.5 Kneser–Ney smoothing

Kneser and Ney (Reference Kneser and Ney1995) have introduced an extension of absolute discounting. The estimate of the higher order distribution is created by subtracting a fixed discount D from each non-zero count. The difference with the absolute discounting smoothing resides in the estimate of the lower order distribution as shown in this equation:

(19) $$\begin{equation} cooc^*_{kney}(w_{i},{w_{j}})= \left\lbrace \begin{array}{@{} l@{\quad }c @{}}\frac{Max(cooc(w_{i},w_{j})-D,0)}{\sum _{w_{i}}{cooc(w_{i},w_{j})}} & \text{if } cooc(w_{i},w_{j})>0 \\ & \\ \alpha (w_{i},w_{j}) occ_{GT}^*(w_{i}) & \text{if } cooc(w_{i},w_{j}) = 0 \\ \end{array}\right. \end{equation}$$

where r = P_kn(w_i|w^{i − 1}_{i − n + 1}) and α(w ^{i − 1}_{i − n + 1}) is chosen to make the distribution sum to 1 (Chen and Goodman Reference Chen and Goodman1999).

3.2 Co-occurrence counts estimation by prediction

Since comparable corpora are usually small in specialized domains (see Table 2), the discriminative power of context vectors (i.e. the observations of word co-occurrences) is reduced. One way to deal with this problem is to re-estimate co-occurrence counts by a prediction function (Hazem and Morin Reference Hazem and Morin2013). This consists in assigning to each observed co-occurrence count of a small comparable corpora, a new value learned beforehand from a large training corpus. We start from the assumption that words that co-occur more often than by chance in a small corpus, should have the same behavior in a large corpus with higher co-occurrence values. Our aim is to estimate these new values in order to encounter the lack of large data in specialized domains. Given a set of observed co-occurrence counts in a small corpus E^S = {o^S ₁, o^S ₂, . . ., o^S_N}, the question that motivates our procedure is what should be the expected co-occurrence counts of words in a large corpus E^L = {o^L ₁, o^L ₂, . . ., o^L_N}?

In the following section, we propose different techniques to estimate the set E^L:

• • a first technique based on the mean increase of the co-occurrence counts of words for each co-occurrence value;

• • a second technique based on a simple linear regression model;

• • two techniques based on the maximum and the mean of the observed co-occurrence counts in a held-out large corpus.

3.2.1 Mean average increase (MAI)

We consider the use of the mean as an intuitive solution that gives information about the mean tendency of the co-occurrence count increase. To estimate the set E^L, we use a training corpus that has been divided into two samples. The first sample is a newspaper corpus of 500,000 words (the small corpus), and the second sample is a newspaper corpus of 10 million words (the large corpus). Hence, we compute the mean increase of each co-occurrence value k of the small corpus thanks to its corresponding value in the large corpus.

We denote by E^S_k = {(w_i, w_j) | cooc^S_k(w_i, w_j) = k and i ∈ [1; N]~and j ∈ [1; M]} the set of words pairs that co-occur k times in the small corpus (each value of each pair is represented by cooc^S_k(w_i, w_j)). We also denote by E^L_k = {(w_i, w_j) | cooc^L_k(w_i, w_j) = k and i ∈ [1; N]~and j ∈ [1; M]} the set of words pairs that co-occur k times in the large corpus (each value of each pair is represented by cooc^L_k(w_i, w_j)).

The mean average increase μ_k of the co-occurrence count k is computed as follows:

(20) $$\begin{eqnarray} \mu _{k} = \frac{1}{|E_{k}^{S}|} \sum _{i=1}^{N}\sum _{j=1}^{M} (cooc_{k}^{L}(w_{i},w_{j})-cooc_{k}^{S}(w_{i},w_{j})) \end{eqnarray}$$

Once the values μ₁, μ₂, . . .μ_l are obtained and for a given test corpus, we increase each co-occurrence count value (cooc(w_i, w_j)) by its corresponding μ_k as follows:

(21) $$\begin{eqnarray} cooc_{MAI}^{*}(w_i,w_j) = cooc(w_i,w_j) + \mu _{k} \end{eqnarray}$$

with MAI corresponding to the mean average increase and μ_k the corresponding mean value if cooc(w_i, w_j) = k.

3.2.2 Linear regression (REG)

In order to make co-occurrence counts more discriminant and in line with Hazem and Morin (Reference Hazem and Morin2013), one strategy consists in addressing this problem through regression: given training corpora which are small and large (abundant in the general domain), we predict word co-occurrence counts in order to make them more reliable. We then apply the resulting regression function to each word co-occurrence count as a pre-processing step of the SA. Our work differs from Hazem and Morin (Reference Hazem and Morin2013) in two ways. First, while they used the linear regression model, we propose to contrast different regression models. Second, we apply regression to unbalanced comparable corpora and study the impact of prediction when applied to the source texts, the target texts and both source and target texts of the used comparable corpora. We use regression analysis to describe the relationship between word co-occurrence counts in a large corpus (the response variable) and word co-occurrence counts in a small corpus (the predictor variable). As most regression models have already been described in great detail (Christensen Reference Christensen1997; Agresti Reference Agresti2007), the derivation of most models is only briefly introduced in this work.

As we cannot claim that the prediction of word co-occurrence counts is a linear problem, we consider in addition to the simple linear regression model (Lin), a generalized linear model which is the logistic regression model (Logit) and non-linear regression models such as polynomial regression model (Polyⁿ) of order n.

(22) $$\begin{equation} {\hat{y}}_{Lin}= \beta _{0} + \beta _{1} x \end{equation}$$

(23) $$\begin{equation} \hat{y}_{Logit}= \frac{1}{1 + \exp (-(\beta _{0}+\beta _{1}x))} \end{equation}$$

(24) $$\begin{equation} \hat{y}_{Poly^n}= \beta _{0} + \beta _{1} x + \beta _{2} x^{2}+\cdots +\beta _{n} x^{n} \end{equation}$$

where β_i are the parameters to estimate.

Let us denote by f the regression function and by cooc(w_i, w_j) the co-occurrence count of the words w_i and w_j. The resulting predicted value of cooc(w_i, w_j), noted $\hat{cooc}(w_{i},w_{j})$ is given by the following equation:

(25) $$\begin{equation} \hat{cooc}(w_{i},w_{j})= f(cooc(w_{i},w_{j})) \end{equation}$$

3.2.3 MEAN and MAX prediction techniques

Unlike the MAI technique presented above, and in order to estimate the new co-occurrence counts values, the MEAN and MAX prediction techniques are exclusively based on the observed values of the newspaper corpus of 10 million words.

The prediction based on the MEAN technique is presented as follows:

(26) $$\begin{eqnarray} MEAN_{k} = \frac{1}{N} \sum _{i=1}^{N}count(k,C_i) \end{eqnarray}$$

where k represents a given observed co-occurrence count in the small training corpus and count(k, C_i) represents the co-occurrence count of the word pair C_i in the large training corpus that co-occur k times in the small training corpus. N represents the number of word pairs that co-occur k times in the small training corpus. The new estimation of a given word pair w_i and w_j in a test corpus is

(27) $$\begin{eqnarray} cooc_{MEAN}^{*}(w_i,w_j) = MEAN_{k} \end{eqnarray}$$

if cooc(w_i, w_j) = k.

In the same way and by replacing the mean by the maximum observed co-occurrence counts in the large training corpus (MAX) we obtain

(28) $$\begin{eqnarray} MAX_{k} = \max _{i=1}^{N}count(k,C_i) \end{eqnarray}$$

The new estimation of a given word pair w_i and w_j in a test corpus is

(29) $$\begin{eqnarray} cooc_{MAX}^{*}(w_i,w_j) = MAX_{k} \end{eqnarray}$$

if cooc(w_i, w_j) = k.

4.1 Specialized comparable corpora

In this study, specialized comparable corpora were built in terms of bilingual terminology access of technical domains. For this purpose, our comparable corpora gather texts sharing common features such as domain, topic, genre, discourse and period without having a source text–target text relationship which guarantees access to the original vocabulary. For instance, the documents making up the Breast cancer corpus share features such as domain: medical, topic: breast cancer, genre: scientific publications, discourse: scientific discourse, period: between 2001 and 2008.

For our experiments, we used two specialized French/English comparable corpora:

• Breast cancer corpus: This comparable corpus is composed of documents collected from the Elsevier website.Footnote ⁴ The documents were taken from the medical domain within the sub-domain of ‘breast cancer’. We have automatically selected the documents published between 2001 and 2008 where the title or the keywords contain the term cancer du sein in French and breast cancer in English. We collected 130 French documents (about 530,000 words) and 1,640 English documents (about 7.4 million words). We split the English documents into fourteen parts each containing about 530,000 words.

• Diabetes corpus: The documents making up the French part of the comparable corpus have been crawled from the web using three keywords: diabète (diabetes), alimentation (food) and obésité (obesity). After a manual selection, we only kept the documents which were relative to the medical domain. In order to increase the French part, some documents were manually selected from the Elsevier website where the keywords contain the French term diabète and obésité. As a result, 104 French documents were extracted (about 350,000 words). The English part was extracted from the medical website PubMedFootnote ⁵ using the keywords: diabetes, nutrition and feeding. We only kept the free fulltext available documents. As a result, 2,339 English documents were extracted (about 3.5 million words). We also split the English documents into fourteen parts each containing about 350,000 words.

In addition to these two specialized comparable corpora, we use a comparable corpus-based general language to build our learning models.

• Newspaper corpus: This comparable corpus was built from the electronic publications of the French newspaper Le Monde and the American newspaper Los Angeles Times of 1994 for the two corpora. We extracted two versions of this corpus, a first version of about 500,000 words and a second version of about 10 million words.

The French and English documents were then normalized through the following linguistic pre-processing steps: tokenization, part-of-speech tagging and lemmatization. These steps were carried out using the TTC TermSuiteFootnote ⁶ that applies the same method to several languages including French and English. Finally, the function words were removed and the words occurring less than twice in the French part and in each English part were discarded. Table 3 shows the number of distinct words (# words) after these steps. It also indicates the comparability degree in percentage (comp.) between the French part and each English part of each comparable corpus. The comparability measure (Li and Gaussier Reference Li and Gaussier2010) is based on the expectation of finding the translation for each word in the corpus and gives a good idea about how two corpora are comparable in terms of amount of data. We can notice that all the comparable corpora have a high degree of comparability with a better comparability of the breast cancer corpora as opposed to the diabetes corpora. In the remainder of this paper, [breast cancer corpus i] for instance, stands for the breast cancer comparable corpus composed of the unique French part and the English part i (i ∈ [1, 14]).

Table 3.

Number of distinct words (# words) and degree of comparability (comp.) for each comparable corpus

4.2 Bilingual dictionary

The bilingual dictionary used in our experiments is the French/English dictionary ELRA-M0033 available from the ELRA catalogue.Footnote ⁷ This resource is a general language dictionary which contains only a few terms related to the medical domain.

4.3 Terminology reference lists

To evaluate the quality of terminology extraction, we built a bilingual terminology reference list for each comparable corpus. We selected all French/English single words from the UMLSFootnote ⁸ meta-thesaurus. We kept only (i) the French single words which occur more than four times in the French part and (ii) the English single words which occur more than four times in each English part i.Footnote ⁹ As a result of filtering, 169 French/English single words were extracted for the breast cancer corpus and 244 French/English single words were extracted for the diabetes corpus. It should be noted that the evaluation of terminology extraction using specialized comparable corpora often relies on lists of a small size: ninety-five single words in Chiao and Zweigenbaum (Reference Chiao and Zweigenbaum2002), hundred in Morin et al. (Reference Morin, Daille, Takeuchi and Kageura2007), 125 and seventy-nine in Bouamor et al. (Reference Bouamor, Semmar and Zweigenbaum2013).

5.1 Standard approach evaluation

In order to evaluate the influence of corpus size on the bilingual terminology extraction task, two experiments have been carried out using the SA. We first performed an experiment using each comparable corpus independently of the others (we refer to these corpora as balanced corpora). We then conducted a second experiment where we varied the size of the English part of the comparable corpus, from 530,000 to 7.4 million words for the breast cancer corpus in 530,000 word steps, and from 350,000 to 5 million words for the diabetes corpus in 350,000 word steps (we refer to these corpora as unbalanced corpora). In the experiments reported here, the size of the context window w was set to 3 (i.e. a seven-word window), the retained association and similarity measures were the DLO associated with the Cosine (DLO-COS), the MI also associated with the Cosine (MI-COS) and the LLH associated with the Jaccard (LLH-JAC) (see Section 2.1). The results shown were those that give the best performance for the comparable corpora used individually.

Tables 4 and 5 show the results of the SA on the balanced and the unbalanced breast cancer and diabetes comparable corpora. Each line corresponds to the English part i (i ∈ [1, 14]) of a given comparable corpus. For each measure (MI-COS, DLO-COS and LLH-JAC), we represent the MAP scores of each individual comparable corpus (Balanced) and of the cumulative comparable corpus (Unbalanced). For instance, line 3 of Table 4 indicates the MAP obtained by using a comparable corpus that is composed (i) only of [breast cancer corpus 3] (MAP of 21.0 per cent) and (ii) of [breast cancer corpus 1, 2 and 3] (MAP of 34.7 per cent) for DLO-COS measures.

Table 4.

Results (MAP per cent) of the standard approach using the balanced and unbalanced comparable corpora for the breast cancer corpus

Table 5.

Results (MAP per cent) of the standard approach using the balanced and unbalanced comparable corpora for the diabetes corpus

As a preliminary remark, we can notice that the results differ noticeably according to the comparable corpus used individually (MAP variation between 21.0 per cent and 29.6 per cent for breast cancer corpora and between 13.9 per cent and 18.6 per cent the diabetes corpora according to DLO-COS configuration). We can also note that the MAP of all the unbalanced comparable corpora is always higher than any individual comparable corpus except for the MI-COS measures where the results significantly drop.

We can also notice that the results significantly differ according to a given association and similarity measure. For the balanced comparable corpora, LLH-JAC shows the best results (MAP of 36.9 per cent for the balanced breast cancer corpus 8 and 28.3 per cent for the balanced diabetes corpus 10). On the contrary, for the unbalanced breast cancer comparable corpora, it is DLO-COS that shows the best scores (MAP of 42.3 per cent for the unbalanced corpus 12) closely followed by LLH-JAC (MAP of 41.1 per cent for the unbalanced corpus 9). The lower results of MI-COS can be explained by its tendency to underestimate high frequencies and to overestimate low frequencies. This drawback seems to be increased by the use of unbalanced corpora (when the size of the English part increases, the amount of low frequency words increases).

Overall, starting with a MAP of 26.1 per cent as provided by the balanced [breast cancer corpus 1], we are able to increase it to 42.3 per cent with the unbalanced [breast cancer corpus 12] (DLO-COS) and starting with a MAP of 31.6 per cent, we are able to increase it to 41.1 per cent with the unbalanced [breast cancer corpus 9] (LLH-JAC). For the diabetes corpus, we are able to increase the MAP scores from 23.7 per cent [diabetes corpus 1] to 31.4 per cent with the unbalanced [diabetes corpus 5] LLH-JAC and from 16.2 per cent to 27.8 per cent [diabetes corpus 8] (DLO-COS).

The variation observed for some unbalanced corpora such as [breast cancer corpus 12, 13 and 14] or [diabetes corpus 12, 13 and 14] can be explained by the fact that adding more data to the source language increases the error rate of the translation phase of the SA, which leads to the introduction of additional noise in the translated context vectors.

5.2 Evaluation of re-estimation techniques

In this section, we evaluate the usefulness of different smoothing and prediction techniques as a preliminary step in the SA. For the smoothing techniques, we refer to: the additive smoothing by Add1, the Good-turing estimation by GT, the Jelinek and Mercer technique by JM, the Katz Backoff technique by Katz and the Kneser–Ney smoothing technique by KN. For the prediction techniques, we refer to the linear regression by Reg, the mean average increase by MAI and the mean and max techniques by Mean and Max. The results of the experiments conducted on breast cancer and diabetes corpora are respectively presented in Tables 6 and 7.

Table 6.

Results (MAP per cent) of different re-estimation techniques on balanced and unbalanced breast cancer comparable corpora (corpus 14)

Table 7.

Results (MAP per cent) of different re-estimation techniques on balanced and unbalanced diabetes comparable corpora (corpus 14)

Concerning MI-COS, three out of five smoothing techniques (GT, JM and Katz) and two out of four prediction techniques (Mean and Max) increase the MAP score of SA on the balanced breast cancer corpus. The best scores are obtained using JM (MAP of 29.3 per cent) and Max (MAP of 29.2 per cent). For the diabetes corpus, three out of five smoothing techniques (JM, Katz and KN) and only one prediction technique (Reg) increase the MAP score of SA. Using the unbalanced corpora, the majority of the results drastically drop, even with the prediction techniques. That said, it is difficult to conclude about the improvements obtained on the balanced diabetes corpora since the results are too low.

Using DLO-COS on the breast cancer corpus, three out of five smoothing techniques improve SA, that is: GT (MAP of 26.9 per cent), JM and Katz (MAP of 27.4 per cent). Even if there is significant improvement moving from the balanced to the unbalanced comparable corpora, using smoothing techniques does not improve SA (MAP of 41.6 per cent). Concerning the prediction techniques, the best results are obtained using the linear regression technique (Reg) with a MAP of 29.0 per cent for the balanced corpora and a MAP of forty-seven per cent for the unbalanced corpora. It should be noted that none of the other prediction techniques improve SA when using the balanced corpora. On the contrary for the unbalanced corpora, MAI reached 44.9 per cent of MAP which is better than SA (MAP of 41.6 per cent). Concerning the diabetes corpus, only the prediction technique Reg improves the MAP scores (MAP of 15.6 per cent for the balanced corpus and 29.7 per cent for the unbalanced corpus). However, unlike the results in Table 6, the best results are obtained using LLH-JAC configuration with SA for the balanced corpus (MAP of 29.9 per cent) and Reg for the unbalanced corpus (MAP of 32.7 per cent).

Finally, using LLH-JAC on the breast cancer corpus, only the additive smoothing (Add1) improves the MAP score of the balanced corpora (MAP increases from 31.7 per cent to 34.1 per cent). Concerning the unbalanced corpora, Reg slightly increased the MAP score from 40.9 per cent to 41.4 per cent. Concerning the diabetes corpus, GT slightly improves the baseline obtaining a MAP of 25.3 per cent. For the unbalanced corpus, Reg increases the baseline from 29.9 per cent to 33.6 per cent.

Overall, if some re-estimation techniques can improve the results of the SA using balanced comparable corpora, the majority failed when using unbalanced corpora. This may be due to an over estimation of frequencies on the source side of the comparable corpus, frequencies which are already reliable.

We retain from these experiments the linear regression technique (Reg) with the DLO-COS and LLH-JAC measures as they have shown the best results.

5.2.1 Regression models comparison

We contrast the prediction models presented in Section 3.2.2 to find out which is the most appropriate model to use as a pre-processing step of the SA. We chose the balanced corpora where the SA has shown the best results in the previous experiment, namely [breast cancer corpus 14] and [diabetes corpus 14].

Table 8 shows a comparison between the SA without prediction noted No prediction and the SA with prediction models. We contrast the simple linear regression model (Lin) with the second- and the third-order polynomial regressions (Poly ² and Poly ³) and the logistic regression model (Logit). We can notice that except for the Logit model, all the regression models outperform the baseline (No prediction). Also, as we can see, the results obtained with the linear and polynomial regressions are very close. This suggests that both linear and polynomial regressions are suitable as a pre-processing step of the standard approach, while the logistic regression seems to be inappropriate according to the results shown in Table 8.

Table 8.

Results (MAP per cent) of the standard approach using different regression models on balanced breast cancer and diabetes corpora

That said, the increase of regression models is not significant. This may be due to the regression parameters that have been learned from a general domain training corpus. Another reason that could explain these results is the prediction process. We applied the same regression function to all co-occurrence counts whereas learning models for low and high frequencies would have been more appropriate. In the light of the above results, we believe that prediction can be beneficial to our task.

5.3 Predicting unbalanced corpora

In this last experiment, we contrast the SA applied to the balanced and unbalanced corpora noted Balanced and Unbalanced with the SA combined with the prediction model noted Balanced + Prediction and Unbalanced + Prediction.

Figure 1(a) illustrates the results of the experiments conducted on the breast cancer corpus. We can see that the Unbalanced approach significantly outperforms the baseline (Balanced). The main difference between the Balanced and the Unbalanced approaches would indicate that the latter is optimal. We can also notice that the prediction model applied to the balanced corpus (Balanced + Prediction) slightly outperforms the baseline while the Unbalanced + Prediction approach significantly outperforms the other three approaches (moreover, the variations observed with the Unbalanced approach are lower than the Unbalanced + Prediction approach). Overall, the prediction increases the performance of the SA especially for unbalanced corpora.

Fig. 1.

Results (MAP per cent) of the standard approach using its best configurations LLH-JAC, LLH-JAC Reg, DLO-COS and DLO-COS Reg on the balanced and the unbalanced corpora. We present the results according to several intervals of frequencies on the diabetes corpora.

The results of the experiments conducted on the diabetes corpus are shown in Figure 1(b). As for the previous experiment, we can see that the Unbalanced approach significantly outperforms the Balanced approach. This confirms the unbalanced hypothesis and would motivate the use of unbalanced corpora when they are available. We can also notice that the Balanced + Prediction approach slightly outperforms the baseline while the Unbalanced + Prediction approach gives the best results. Here also, the prediction increases the performance of the SA especially for unbalanced corpora. It is clear that in addition to the benefit of using unbalanced comparable corpora, prediction shows a positive impact on the performance of the SA.

5.4 Error analysis

In this section, we analyze the number of correct matches of the SA according to different word frequency ranges. We divided the reference lists into five frequency intervals according to frequency terms in the target language. It could be expected that matching frequent words is easier than infrequent words. This can be true if the translation has enough contexts in the target language. In general, we can classify the word pairs (x,y) (where y is the corresponding translation of x in the target language) into four frequency categories as follows:

1. (1) x frequent and y frequent;

2. (2) x frequent and y infrequent;

3. (3) x infrequent and y frequent;

4. (4) x infrequent and y infrequent.

Figures 2(a), (b) and 3(a) and (b) show a comparison between the best configurations of the SA (LLH-JAC, LLH-JAC-Reg, DLO-COS and DLO-COS Reg) on the balanced and the unbalanced breast and diabetes corpora according to the usefulness of frequency categories.

Fig. 2.

Results (MAP per cent) of the standard approach using the best configurations of the prediction models (Lin for Balanced + Prediction and Poly ² for Unbalanced + Prediction) on breast cancer and diabetes corpora.

Fig. 3.

Results (MAP per cent) of the standard approach using its best configurations LLH-JAC, LLH-JAC Reg, DLO-COS and DLO-COS Reg on the balanced and the unbalanced corpora. We present the results according to several intervals of frequencies on the breast cancer corpora.

In the low range frequencies (#freq < 50), we can clearly notice that LLH-JAC, LLH-JAC-Reg outperform DLO-COS and DLO-COS Reg (except for the diabetes [10, 50] where DLO-COS Reg performs the best). In the middle range frequencies (#freq ∈ [50,100]) DLO-COS and DLO-COS Reg show the best results (except for the balanced diabetes corpus). Concerning the high frequency ranges (#freq > 100), it is clear that LLH-JAC, LLH-JAC-Reg outperform DLO-COS and DLO-COS Reg on the balanced corpora, while for the unbalanced corpora the results are almost always better using DLO-COS and DLO-COS Reg. According to this analysis, it is preferable to use LLH-JAC-Reg for low frequency terms, and DLO-COS Reg for middle and high frequency terms.

Overall, and although the improvements are given by the best configurations of the SA, we notice that some translations cannot be matched even if they are part of the middle and high frequency ranges. This leads us to state that high frequency is not a reliable criterion to guarantee a correct translation match. The context of a source word x and its translation y are inter-dependent and some words that are part of the context of x or y may not useful. As the cosine similarity is sensitive to vector size, one possible improvement can be to prune the vectors (use vectors of the same size) while computing the vector similarity.