2.1.1 Context-free measures

A natural way to estimate the number of translation pairs in a bilingual corpus is to rely on the mathematical expectation of finding such pairs. Let us assume that we have a French–English comparable corpus ${\mathcal {C}}$ consisting of a French part ${\mathcal {C}}_f$ and an English part ${\mathcal {C}}_e$. If we consider the translation process from the English part to the French part, the comparability measure M _ef can be defined as the expectation of finding, for each English word w _e in the vocabulary ${\mathcal {C}}_e^v$ of ${\mathcal {C}}_e$, its translation in the vocabulary ${\mathcal {C}}_f^v$ of ${\mathcal {C}}_f$. The definition for M _ef directly reflects our intuition. As one can note, a general English–French bilingual dictionary $\mathcal {D}$, independent from the corpus ${\mathcal {C}}$, is required to judge if two words are the translation of each other. Let σ be a function that indicates whether a translation from the translation set $\mathcal {T}_w$ of a word w is found in the vocabulary ${\mathcal {C}}^v$ of a corpus ${\mathcal {C}}$, i.e.,:

(1) $$\begin{equation} \sigma (w, {\mathcal {C}}^v) = \left\lbrace \begin{array}{ll}1 & \mbox{ iff } \mathcal {T}_w \cap {\mathcal {C}}^v \ne \emptyset \\ 0 & \mbox{ else } \end{array} \right. \end{equation}$$

M _ef is then defined as

(2) $$\begin{equation} M_{ef}({\mathcal {C}}_e, {\mathcal {C}}_f) = {\mathbb {E}} (\sigma (w,{\mathcal {C}}_f^v)| w \in {\mathcal {C}}_e^v) = \sum _{w \in {\mathcal {C}}_e^v} \sigma (w,{\mathcal {C}}_f^v) \cdot \mbox{Pr}(w) \end{equation}$$

As assumed above, the comparable corpus and the general bilingual dictionary are independent from one another. It is thus natural to assume that the dictionary $\mathcal {D}$ covers a substantial part of ${\mathcal {C}}_e^v$ and this substantial part can well represent the whole vocabulary. It means the expectation of finding the translation in ${\mathcal {C}}_f^v$ of a word w in ${\mathcal {C}}_e^v$ can be approximated by that in ${\mathcal {C}}_e^v \cap \mathcal {D}_e^v$. This assumption leads to equation (3):

(3) $$\begin{equation} M_{ef}({\mathcal {C}}_e, {\mathcal {C}}_f) = \sum _{w \in {\mathcal {C}}_e^v \cap \mathcal {D}_e^v} \sigma (w,{\mathcal {C}}_f^v) \cdot \mbox{Pr}(w) \end{equation}$$

where $\mathcal {D}_e^v$ is the English vocabulary of the given bilingual dictionary $\mathcal {D}$. The derivation from equations (2)–(3) seems trivial. It does help us bring up the notion of dictionary coverage and the robustness measure in Section 3.2.3.

There are several possibilities to estimate Pr(w) in equation (3). However, the presence of common words suggests that one should not solely rely on the number of occurrences (i.e., term frequency), which is a broadly used approach in other fields like unigram language models, since the high-frequency words will dominate the final results. For example, in the Europarl corpus, the English word Europe and the French word Europe are very common words. It means that even if one piece of English text and one piece of French text are randomly picked from the Europarl corpus, we can still expect to find many translation pairs Europe–Europe. To avoid the bias common words can introduce in the comparability measure, one can weight each word w as ρ_w through TF-IDF or through the simple Presence/Absence (P/A) criterion. In this case, the part Pr(w) can be estimated as

(4) $$\begin{equation} \mbox{Pr}(w)= \frac{\rho _w}{\sum _{w \in {\mathcal {C}}_e^v \cap \mathcal {D}_e^v} \rho _w} \end{equation}$$

With the P/A criterion, the weight ρ_w is 1 if and only if $w \in {\mathcal {C}}_e^v \cap \mathcal {D}_e^v$, otherwise the value is 0. Alternatively, considering the TF-IDF weight for each word w, the weighting function ρ_w can be defined in a standard TF-IDF style (Salton, Wong and Yang Reference Salton, Wong and Yang1975).

Similarly, when considering the translation process from the French part to the English part, the counterpart of M _ef, M _fe, can be written as

(5) $$\begin{equation} M_{fe}({\mathcal {C}}_e, {\mathcal {C}}_f) = \sum _{w \in {\mathcal {C}}_f^v \cap \mathcal {D}_f^v} \sigma (w, {\mathcal {C}}_e^v) \cdot \mbox{Pr} (w) \end{equation}$$

where Pr(w) is defined in the same way as in equation (4).

The two asymmetric measures M _ef and M _fe above reflect the degree of comparability when considering the translation process in only one direction. They can be combined to form a comprehensive measure M by reviewing the two directions as a whole. The difference between M _ef in equation 3 and M _fe in equation (5) comes from the part which estimates the probability of a word in the corpus vocabulary. We denote ${\mathcal {C}}^v$ as the whole vocabulary of ${\mathcal {C}}$ and $\mathcal {D}^v$ as the whole vocabulary of $\mathcal {D}$. Following the same idea, considering the English and French vocabularies as a whole, one can directly obtain Pr(w) for each w in ${\mathcal {C}}^v \cap \mathcal {D}^v$ by replacing ${\mathcal {C}}_e^v \cap \mathcal {D}_e^v$ with ${\mathcal {C}}^v \cap \mathcal {D}^v$ in equation (4). The combined measure M, considering the translation in both directions, can then be written as

(6) $$\begin{equation} M ({\mathcal {C}}_e, {\mathcal {C}}_f) = \sum _{w \in {\mathcal {C}}^v \cap \mathcal {D}^v} \sigma (w,{\mathcal {C}}^v) \cdot \mbox{Pr}(w) \end{equation}$$

We give here additional comments on the P/A criterion. With this criterion, Pr(w) for each w in ${\mathcal {C}}^v \cap \mathcal {D}^v$ is directly $\frac{1}{|{\mathcal {C}}^v \cap \mathcal {D}^v|}$. One can then obtain the combined measure M as

(7) $$\begin{eqnarray} && M({\mathcal {C}}_e, {\mathcal {C}}_f) = \frac{\sum _{w \in {\mathcal {C}}_e^v \cap \mathcal {D}_e^v} \sigma (w,{\mathcal {C}}_f^v) + \sum _{w \in {\mathcal {C}}_f^v \cap \mathcal {D}_f^v} \sigma (w,{\mathcal {C}}_e^v)}{|{\mathcal {C}}_e^v \cap \mathcal {D}_e^v|+|{\mathcal {C}}_f^v \cap \mathcal {D}_f^v|} \end{eqnarray}$$ $$

which corresponds to the overall proportion of words for which a translation can be found in the comparable corpus. One can notice from equation (7) that M is a symmetric measure.

2.1.2 Context-based measures

The previous development has left aside an important problem, namely the one of polysemy. Due to polysemy, two words can be treated as the translation of each other according to the dictionary but might hold different senses in the corpus. We can make use of the simple assumption that words in parallel translation usually appear in similar contexts to disambiguate the translation candidates in the dictionary. This same assumption has been broadly exploited in the bilingual lexicon extraction tasks. We will embed the assumption in the function σ in equation (1) to build a context-sensitive version of the previous measures. Let us assume that the English word w _e (resp. French word w _f) appears in the context word set $\mathcal {S}_e$ (resp. $\mathcal {S}_f$) consisting of the words surrounding w _e (resp. w _f) in a certain window in the corpora. Then the similarity of the two context sets is measured by the overlap of the two sets which is directly the proportion of words of which the translation can be found in the counterpart set. Formally, the similarity of w _e and w _f, in terms of their context similarity, can be written as

$$\begin{equation*} \mbox{sim}(w_e, w_f) = \frac{\sum _{w \in \mathcal {S}_e \cap \mathcal {D}_e^v} \sigma (w,\mathcal {S}_f) + \sum _{w \in \mathcal {S}_f \cap \mathcal {D}_f^v} \sigma (w, \mathcal {S}_e)}{|\mathcal {S}_e \cap \mathcal {D}_e^v| + |\mathcal {S}_f \cap \mathcal {D}_f^v|} \end{equation*}$$

The enhanced version of the function σ in equation 1 is then defined as

$$\begin{equation*} \sigma _c(w,{\mathcal {C}}^v) = \left\lbrace \begin{array}{ll}1 & \mbox{iff } \exists w^{\prime } \in \mathcal {T}_w \cap {\mathcal {C}}^v, \,\,\, \mbox{sim}(w,w^{\prime }) > \delta \\ 0 & \mbox{else} \end{array} \right. \end{equation*}$$

where δ, empirically set to 0.3 in our experiments, is the threshold for the similarity. A word w is deemed to be translated, according to the function σ_c, if at least one of its translations w′ identified by the function σ in the corpus, is similar to w based on the context similarity measure sim(w, w′). Replacing σ with σ_c in equation (1) will lead to the context versions of the comparability measures above, which are respectively denoted as M^c_ef, M^c_fe and M ^c.

Besides the simple idea used in above context-based measures, there are definitely several other possibilities to improve it. First, the window-based co-occurrence context can be improved by using surface-distance model proposed in Washtell (Reference Washtell2009). Second, such techniques as LSA and LDA (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1900; Blei and Jordan Reference Blei and Jordan2003) are usually considered to be more efficient than the simple context vector approaches according to the experiences. However, as we will show, the simple measure M ^c or even its context-free version M have already shown very satisfactory performance and are enough for practical usage. Moreover, the simple ideas are computationally much cheaper. The possible enhancements are thus not the prior direction of this paper and will not be investigated further.

The above measures rely on the intersection between bilingual corpora and dictionary. One can tell from the intuition that the corpus vocabulary tends to cover all the words in the dictionary while the corpora are long enough, since the dictionary vocabulary is generally smaller compared to that of the corpus. It is to say, those measures could be sensitive to corpus length. Taking the measure M _ef in equation (3) as an example, once the French part ${\mathcal {C}}_f$ is long enough, it might be able to cover all the French dictionary vocabulary, meaning that M _ef could be high even if the French part and the English part are not really comparable. We will review this problem in Section 3.2.2 and give a formal analysis on dictionary coverage in Section 3.2.3.

3.2.1 Constructing test corpora

In order to test the comparability measures introduced before, one needs to have some corpora of known comparability levels, which does not exist according to our knowledge. One possibility is to construct such corpora with gold-standard comparability levels by human annotators, which is a costly task. Moreover, it is much harder for people to assign a precise score on corpus level than on sentence/document level as done in the Semantic Textual Similarity shared task of SemEval. Due to the lack of annotated corpora, existing works (e.g., Upadhyay et al. (Reference Upadhyay, Faruqui, Dyer and Roth2016)) have to employ such indirect tasks as dictionary extraction, document-classification and syntactic dependency parsing in order to test its semantic similarity models.

Without human annotators, the feasible choice is to build such test corpora automatically. The idea is to introduce noise into high-quality comparable corpora so as to obtain artificial corpora with decreasing comparability levels, which might not be as accurate as human annotation. We construct three such groups of corpora from the parallel Europarl corpus and the monolingual AP corpus, as follows:

• • G _a: All the comparable corpora in G _a are built from the parallel corpus Europarl. One starts from the parallel corpus Europarl, considered as having the highest comparability level, and gradually decreases its quality by replacing larger parts with some non-parallel parts also present in Europarl. In other words, noise is added to the parallel corpus, however covering topics similar with the ones in the original parallel corpus;

• • G _b: As before, one starts with the parallel corpus Europarl. The difference here is that one exchanges parallel parts of Europarl with content extracted from the AP corpus. That is to say, the noise brought to the parallel corpus covers different topics from the original parallel corpus.

• • G _c: The comparable corpora obtained in G _a correspond to non-parallel corpora covering similar topics. In G _c, one starts from corpora with the lowest comparability levels in G _a, i.e., the ones containing no parallel parts, and exchanges some of their parts containing similar topics with content from the AP corpus, meaning that parts covering different topics are introduced, leading to comparable corpora belonging to the class non-parallel corpora covering different topics.

We now give more details on this construction process. The first group G _a is built from the Europarl corpus through the following two steps:

1. (1) The English (and its corresponding French) part of the Europarl corpus is split into ten equal parts in terms of number of sentences, leading to ten parallel corpora denoted as $\mathcal {P}_1$, $\mathcal {P}_2$, . . ., $\mathcal {P}_{10}$. The comparability level of these ten parallel corpora are arbitrarily set to one (i.e., the highest level);

2. (2) For each parallel corpus, e.g., $\mathcal {P}_i$ (i = 1, 2, . . ., 10), we replace a certain proportion p of the English part of $\mathcal {P}_i$ with content of the same size, again in terms of the number of sentences, from another parallel corpus $\mathcal {P}_j (j \ne i)$, producing a new corpus $\mathcal {P}_i^{\prime }$ likely to contain less translation pairs and thus be of lower comparability. For each $\mathcal {P}_i$, as p increases one obtains a series of comparable corpora with decreasing comparability scores. In our experiments, p is increased from 0 to 1 with a step of 0.01. All the $\mathcal {P}_i$ and their respective descendant corpora, according to different values of p, constitute the group G _a, the comparability score of each corpora being set to 1 − p. As a result, we have 1,000 comparable corpora in G _a. This process is illustrated in Figure 1.

   Fig. 1.

   Constructing the test corpus group G _a with gold-standard comparability levels.

   

The difference between building the corpora in G _b and in G _a is that, in G _b, the replacement in $\mathcal {P}_i$ is done with documents from the AP corpus and not from another parallel corpus $\mathcal {P}_j$ from Europarl. Compared with the corpora in G _a, we further degrade the parallel corpus $\mathcal {P}_i$ in G _b since the AP corpus covers different topics from Europarl.

In G _c, we start with ten comparable corpora $\mathcal {P}_i^{\prime }$ from G _a having the lowest comparability score (i.e., 0). They thus contain documents from Europarl that are not the translation of each other. Each $\mathcal {P}_i^{\prime }$ is further altered by replacing certain portions, according to the same proportion p used before, with documents from the AP corpus. Although $\mathcal {P}_i^{\prime }$ itself is comparable and not parallel, its English and French parts are likely to cover similar topics embedded in the Europarl corpus. Replacing certain parts of $\mathcal {P}_i^{\prime }$ with the content from AP will thus further degrade the comparability levels of $\mathcal {P}_i^{\prime }$.

From the process of building the comparable corpora in G _a, G _b and G _c, one can note that the gold-standard comparability scores in different groups, e.g., G _a and G _c, cannot be compared with each other directly, since the comparability scores are normalized between 0 and 1 in each group of corpora. These comparability scores do not represent absolute judgements on the comparability of the corpora considered, but rather relative scores within each group of corpora.

3.2.2 Correlation with gold-standard comparability levels

The goal here is to assess whether the comparability measures we have introduced can capture the differences in comparability levels in the three different groups G _a, G _b and G _c. As the comparability levels are set arbitrarily in these groups, we are interested here in assessing whether the measures yield the correct ranking (in terms of comparability scores) or not. In order to quantify this, we use Pearson Correlation Coefficient, defined as

$$\begin{equation*} r= \frac{\sum _{i} (X_i-\overline{X})(Y_i-\overline{Y})}{\sqrt{\sum _{i} (X_i-\overline{X})^2}\sqrt{\sum _{i} (Y_i-\overline{Y})^2}} \end{equation*}$$

where X _i denotes the comparability score provided by one measure on a given bilingual corpus and Y _i is the arbitrary comparability score (i.e., gold standard) assigned to this corpus in the construction process. $\overline{X}$ represents the average of X _is over all the bilingual corpora considered in G _a, G _b or G _c (and similarly for $\overline{Y}$).

Let us first recall the measures we have proposed in Section 2:

• • Measures based on vocabulary overlapping. These measures are defined as the mathematical expectation of finding the translation for each word in the corpus vocabulary. Corresponding to the English/French/whole vocabulary of the corpus, we have the measures M _ef, M _fe, M and their contextual versions M^c_ef, M^c_fe and M ^c. For the six measures, we will consider both the versions with the P/A weighting criterion and the versions with TF-IDF weighting schema, amounting to twelve measures in total.

• • Baseline measures. These measures correspond to the vector spaces mapping and MT approaches discussed before: the measure M ^v is based on the bilingual vector space model and the measures M ^g1 and M ^g2 are based on a MT system and n-grams representation. In addition, we use as baselines the recently proposed bilingual skip-gram model BiSkip (denoted as M ^s) in Luong, Pham and Manning (Reference Luong, Pham and Manning2015).

We first make comparisons between measures based on vocabulary overlapping. The correlation scores are listed in Table 2. Each column in the table, e.g., M _ef, corresponds to the correlation scores between the specific comparability measure M _ef and the gold-standard comparability levels on different corpus groups, namely G _a, G _b and G _c. Let us first pay attention to the measures without context, i.e., M _ef, M _fe and M. One can find that M together with the P/A weighting schema performs the best and correlates very well with the gold standard on all the three groups of corpora, as the Pearson coefficient is close to 1. M _fe performs worst among the three measures with one exception: using TF-IDF weighting, M _fe performs better then M _ef. One can also conclude, for the best measure M, that using IDF to reduce the effects of frequent words does not help to solve the problem of too frequent words, and the P/A weighting schema works better with M than the TF-IDF one. Although the TF-IDF weighting schema seems to be efficient for M _fe on G _b and G _c, the measure M _fe together with TF-IDF still performs far from the other two measures.

Table 2.

Correlation scores of the vocabulary overlapping measures with the gold standard

The rows TF-IDF correspond to the TF-IDF weighting schema, and the rows P/A correspond to the Presence/Absence weighting method.

We then turn to the measures with contextual information, i.e., M^c_ef, M^c_fe and M ^c. One can find from Table 2 that all the context-based measures, weighted by either TF-IDF or P/A, perform very well, and are slightly better than the best performing measure M in the context-free family. Among the three measures with context, M ^c performs slightly better than M^c_ef and M^c_fe. The measures M^c_ef and M^c_fe, with contextual information, are better than their corresponding context-free versions M _ef and M _fe. These findings are in agreement with the assumption that using contextual information helps to disambiguate translation candidates and thus leads to better performance of the measures.

Deeper analysis is given here in order to explain the different performance of the different measures. We will only consider here the P/A weighting scheme so as to simplify the discussion. Figure 2 plots the measures M, M _ef, M _fe, M ^c, M^c_ef, M^c_fe on ten comparable corpora and their descendants in G _c with respect to their gold-standard comparability scores. We first compare here the three context-free measures M _ef, M _fe and M. One can notice from Figure 2(c) that the comparability scores from M _fe decrease at a certain point even though the gold standard scores increase. The reason for the different performances is that asymmetric measures M _ef and M _fe are sensitive to the relative length of the corpus. Given a single English document and a large French document collection, it is very likely that one can find the translations for most of the English words, even though the two corpora are lowly comparable, because the English vocabulary in consideration is very small and it is easy to find all the translations even if the French corpora are not really comparable to the English content. In our case, since the average sentence length in AP is larger than that of Europarl, we increase the length of the English part of the test corpora remarkably when degrading the corpora in G _b and G _c, leading to poor performance of M _fe. In order to further support our judgement, we have also tried to manually control the sentence length in the AP corpus, namely that we intend to choose in AP those sentences of which the length is similar to that in the original corpus. The results then show that M _fe works well under the new settings without a biased length. The length related problem can be overcome by M that considers the translation in both directions.

Fig. 2.

Evolution of the measures M _ef, M _fe, M, M^v_ef, M^v_fe and M ^v w.r.t. gold standard on the corpus group G _c (x-axis: gold-standard comparability scores, y-axis: comparability scores from the measures). (a) M _ef. (b) M^c_ef. (c) M _fe. (d) M^c_fe. (e) M. (f) M ^c.

We now turn to the contextual versions M^c_ef, M^c_fe and M ^c. All of the three measures perform very well on the three groups of corpora. Let us pay attention to the measure M _fe and its context version M^c_fe. The former one is sensitive to the corpus length, whereas the latter one is far less sensitive. We conjecture here that contextual information helps identify the correct translations in cases where they are a lot of possible translations (as is the case with unbalanced, in terms of size, collections), and thus can still capture different comparability levels. Furthermore, as the results show, M and all the context-based measures are able to capture all the differences in comparability artificially introduced in the degradation process we have considered in Section 3.2.1. Last, one can conclude from the results that it is easier to capture the different comparability levels in G _b than in G _a and G _c. This is due to the fact that the differences in G _a (based on the same corpus) are less marked, and thus more difficult to identify. On the opposite, G _c comprises corpora with low levels of comparability, also more difficult to separate out. G _b based on a parallel corpus with additions from a different corpus, displays comparability levels easier to identify.

We finally list in Table 3 the results from the baseline measures. The results shown here are obtained in the same way as the ones for the vocabulary overlapping measures in Table 2. From the results one can find that the recently introduced bilingual skip-gram measure M ^s performs the best on all the three groups of corpora, the measures M ^g2 and M ^v performing worse. All these baseline measures do not perform as well as the measures M (weighed by P/A) or the three context-based measures only relying on the vocabulary overlapping approach. The standard vector space is broadly used to represent textual elements in previous studies aiming at capturing similarity at the sentence or document levels. What our experiments reveal here is that this approach does not yield interesting performance (compared to other approaches) for measuring comparability on the corpus level. The reason is probably that noise involved in dictionary-based mapping in equation (8) destroys the capability of monolingual vector space model. This finding is also partially supported by a recent report of the ACCURAT projectFootnote ¹³ that vector representation is an appropriate method for capturing document-level similarity in a cross-language environment. The measure based on cross-lingual word embeddings is inferior to the measure M, which is probably due to the same reason that it is weak at measuring similarity on corpus level. We will give more discussion on these measures in Section 4.

Table 3.

Correlation scores of the baseline comparability measures with the gold standard

Artificial test corpora built using the above methodology may not fully reflect characteristics of all comparable corpora, although it is the most direct approach by which one could obtain quantitative levels. In addition to those experiments, we make use here of the comparability measure on real world comparable corpora without quantitative levels. The pairs of comparable corpora we will consider are listed in Table 4, which are in accordance with one’s direct intuition. In comparison to artificial corpora constructed in Section 3.2.1, gold-standard comparability levels in Table 4 are qualitative rather than quantitative.

Table 4.

Real world comparable corpora and experimental results

One could conjecture that the first corpus should be more comparable than the second one in each group according to the intuition, which constitutes the gold standard. The last column corresponds to the experiment results where Y denotes a coherence with gold standard.

We use the best-performing measure M to test if the differences between each pair of comparable corpora in Table 4 can be differentiated. The results are listed in the last column of Table 4 where Y denotes experimental results agree with one’s intuition (i.e., the gold standard). One can find from the results that all the comparability levels can be captured correctly by the measure M, which further proves the reliability of M in real world cases.

3.2.3 Robustness of comparability measures

Since the two measures M and M ^c perform the best in their respective classes of measures, i.e., measures without context and measures with context, we choose to compare them in terms of the robustness w.r.t. the change of the dictionary that is the only resource used to cross the language barrier. To simplify the discussion here, we follow the same consideration as in Section 3.2.2 and only use the P/A weighting schema. It is important that the comparability measure one retains remains consistent when the dictionary coverage of the corpus changes slightly, as this is a necessary condition to distinguish between different comparability levels. Indeed, if a slight change in the dictionary coverage entails an important change in the comparability score, then it becomes impossible in practice to compare different corpora, as they will likely have a different coverage with respect to the dictionary. We say, informally, that a comparability measure is robust at certain dictionary coverage range if the measure can distinguish between different comparability levels when the dictionary changes in this range. The experiments and analysis below try to assess the robustness of the retained measures.

To do so, several dictionaries of different sizes, corresponding to different coverages on the corpus vocabulary, are built by randomly choosing the subparts from the original dictionary. The coverage here is simply defined as the proportion of unique words in the corpus vocabulary that are covered by the dictionary. This definition is coincident with the definition of M and M ^c, which corresponds to the proportion of words translated in the part of vocabulary covered by the dictionary in the whole vocabulary. In order to bridge the language barriers, a dictionary that is sufficient large is necessary. We thus choose to randomly pick certain proportions, from 50% to 99% with a step of 1%, of our original dictionary. For each proportion, 30 different dictionaries are built by randomly sampling the original dictionary 30 times at this same proportion. These 1,500 dictionaries (i.e., 50 × 30) are then used to compute M and M ^c on different corpora with decreasing comparability scores in G _a, G _b and G _c.

As we have discussed in the corpus construction process in Section 3.2.1, in each of G _a, G _b and G _c, one tries to obtain a series of decreasing comparability levels starting from ten parallel corpora (in G _a and G _b) or ten high quality comparable corpora (in G _c). In order to estimate the robustness, it is natural to perform several times the experiments in Section 3.2.2 together with various dictionaries. Then one can judge the robustness directly from correlation scores with different dictionaries. The problem exists in the fact that the dictionary coverage might be different on various test corpora, which prevents us from drawing a clear conclusion based on a uniform coverage level.

We will design here another set of experiments to clarify the impact of dictionary coverage on measure performance. For the clarity of analysis, we only take here the first parallel corpus $\mathcal {P}_1$ from ($\mathcal {P}_1$, $\mathcal {P}_2$, . . ., $\mathcal {P}_{10}$) together with its ten descendant corpora that are built by setting the proportion p to 0.1, 0.2, . . ., 1.0 in G _a. That is to say, we exchange 10%, 20%, . . ., 100% of the content in the high quality corpora with noise from other corpora. In this case, we obtain eleven comparable corpora $\mathcal {P}_1$, $\mathcal {P}_1^{0.9}$, $\mathcal {P}_1^{0.8}$, . . ., $\mathcal {P}_1^0$ with the gold comparability scores from 1 to 0, with a step of 0.1. Last, for readability reasons, we only plot in Figure 3 the comparability scores for some of the eleven comparable corpora, i.e., $\mathcal {P}_1$, $\mathcal {P}_1^{0.7}$, $\mathcal {P}_1^{0.4}$ and $\mathcal {P}_1^{0.1}$, w.r.t. the different coverages (but the results are the same for the other corpora).

Fig. 3.

Evolution of M and M ^c w.r.t. different dictionary coverages on comparable corpora $\mathcal {P}_1$, $\mathcal {P}_1^{0.7}$, $\mathcal {P}_1^{0.4}$ and $\mathcal {P}_1^{0.1}$ in G _a (x-axis: different dictionary coverages; y-axis: comparability scores from M or M ^c). (a) M. (b) M ^c.

From Figure 3(a), one can find that when the dictionary coverage lies above a certain threshold (inspected from the figure, roughly set to 0.62), the differences between the four different comparability levelsFootnote ¹⁴ can be captured very well, as the different data points are well separated. In other words, the different comparability levels can be captured very well by the measure M when the dictionary coverage is roughly above 0.62. The same conclusion can be drawn from the inspection of Figure 3(b). One can thus conclude from this qualitative analysis that both M and M ^v are robust to the changes of the dictionary after a certain point.

We have drawn for comparability measures the intuitive conclusion regarding robustness from Figure 3. In order to analyze the results quantitively, we will first try to define the degree of robustness of a comparability measure.

Definition 1 Let us assume that we have different comparable corpora ${\mathcal {C}}_1$, ${\mathcal {C}}_2$, . . ., ${\mathcal {C}}_k$, with gold-standard comparability levels that are increasing, which can be written as: ${\mathcal {C}}_1 \prec {\mathcal {C}}_2 \prec \ldots \prec {\mathcal {C}}_k$ (the symbol ≺ is used here to denote the relation lessin the gold-standard comparability levels). We further assume to have a bilingual dictionary $\mathcal {D}$ such that the coverage of $\mathcal {D}$ on all the k corpora ${\mathcal {C}}_i (i=1,2,\ldots ,k)$ belongs to a range [r, r + ε], with ε being a small fixed value. Then, we define the degree of robustness of a comparability measure M w.r.t the dictionary coverage r as

$$\begin{equation*} \chi (M,r) = \underset{i \in {1,2,\ldots ,k-1}}{\mbox{avg}} \mbox{Pr}(M({\mathcal {C}}_{i+1}) > M({\mathcal {C}}_i)) \end{equation*}$$

Through the above definition, one measures to which extent a certain measure, associated with a certain coverage value r, can well separate different comparability levels in the average case.

We now turn to the problem of estimating the degree of robustness of the comparability measures M and M ^v. Let us first focus on the measure M. We notice from Figure 3 that, for each of the test corpora, e.g., $\mathcal {P}_1^{0.1}$ in Figure 3(a), the comparability scores corresponding to a certain coverage range (e.g., from 0.56 to 0.58, identified by the circle in the figure) follow a normal distribution, according to the Shapiro–Wilk test (Shapiro and Wilk Reference Shapiro and Wilk1965) at the significance level 0.05. This fact is also illustrated by the frequency distribution histogram in Figure 4. Thus, in a coverage range of size 0.02 (i.e., the value ε in the Definition 1), the comparability scores of M for a specific corpus can be modeled as a normally distributed variable Z. Hence, on each span, the scores of M on two bilingual corpora, say $\mathcal {P}_1^{0.1}$ and $\mathcal {P}_1^{0.2}$, can be described as two normally distributed variables denoted as Z _0.1 and Z _0.2 of which the parameters (i.e., the mean μ and variance σ²) can be estimated from the samples, i.e., all the comparability scores on $\mathcal {P}_1^{0.1}$ and $\mathcal {P}_1^{0.2}$ in the specific coverage range. With the estimated parameters, we can write:

$$\begin{equation*} Z_{0.1} \sim N(\mu _{0.1}, \sigma _{0.1}^2), \,\,\, Z_{0.2} \sim N(\mu _{0.2}, \sigma _{0.2}^2) \end{equation*}$$

To compute the degree of robustness in Definition 1, one needs to obtain all the probability $\mbox{Pr}(M({\mathcal {C}}_{i+1}) > M({\mathcal {C}}_i))$. Taking here the corpora $\mathcal {P}_1^{0.2}$ and $\mathcal {P}_1^{0.1}$ as an example, one needs to estimate Pr(Z _0.2 > Z _0.1). With the independence assumption between variables Z _0.1 and Z _0.2, the new variable Z _0.2 − Z _0.1 satisfies:

$$\begin{equation*} Z_{0.2}-Z_{0.1} \sim N(\mu _{0.2} - \mu _{0.1}, \sigma _{0.1}^2 + \sigma _{0.2}^2) \end{equation*}$$

and computing Pr(Z _0.2 > Z _0.1) amounts to the computation of Pr(Z _0.2 − Z _0.1 > 0), which can be done through tabulations of the normal distribution. In Table 5, we list, for different coverage ranges, the robustness values for both comparability measures M and M ^c on all the eleven comparability levels. Let us first pay attention to robustness of the measure M. One can find from the results that the higher the dictionary coverage is, the more reliable M is to distinguish between different comparability levels. Furthermore, when the dictionary coverage is above a certain threshold (e.g., 0.60), we have a high confidence (⩾0.85) that the different comparability levels between the corpus pairs can be reliably captured by M, so that the measure is robust in a given coverage range, here set to 0.02. The same conclusions can be drawn for the other comparable corpus pairs we have constructed (from G _b and G _c).

Fig. 4.

The frequency histogram of the comparability scores (from M) on $\mathcal {P}_1^{0.1}$ between the coverage 0.56 and 0.58 (x-axis: comparability scores; y-axis: frequency), with the associated normal approximation.

Table 5.

The degree of robustness with two measures M, M ^c and different dictionary coverages on all the eleven comparability levels

The coverage range is set to [r, r + ε] as in Definition 1 with ε being fixed to 0.02.

One can further conclude from Table 5 that, compared to M, it is easier to achieve high confidence, and thus higher robustness, with M ^c. Even when the dictionary coverage is only 0.50, the confidence is larger than 0.86 that M ^c captures the different comparability levels. However, the computation complexity of the context-based measures is usually very expensive. For this reason, we will only make use, in the remainder of this study, of the measure M weighted by the P/A criterion.