2.1 Our approach to bilingual lexicon induction

We frame bilingual lexicon induction as a binary classification problem: for a pair of source and target language words, we predict whether the two are translations of one another or not. Since binary classification does not inherently give us a list of the best translations, we need to take an additional step. For a given source language word, we find its best translation or its n-best translations by first using our classifier on all target language words. We then rank them based on how confident the classifier is that each target word is a translation of the source word. The features used by our classifier include a variety of signals of translation equivalence that are drawn from past work in bilingual lexicon induction, notably by Rapp (Reference Rapp1995), Fung (Reference Fung1995), Schafer and Yarowsky (Reference Schafer and Yarowsky2002), Klementiev and Roth (Reference Klementiev and Roth2006), Klementiev et al. (Reference Klementiev, Irvine, Callison-Burch and Yarowsky2012) and others. The features that we use in our model are:

• • Contextual similarity – In a similar fashion to how vector space models can be used to compute the similarity between two words in one language by creating vectors that representing their co-occurrence patterns with other words (Turney and Pantel Reference Turney and Pantel2010), context vector representations can also be used to compare the similarity of words across two languages. The earliest work in bilingual lexicon induction by Rapp (Reference Rapp1995) and Fung (Reference Fung1995) used the surrounding context of a given word as a clue to its translation. Fung and Yee (Reference Fung and Yee1998) and Rapp (Reference Rapp1999), used small seed dictionaries to project word-based context vectors from the vector space of one language into the vector space of the other language. We use the vector space approach of Rapp (Reference Rapp1999) to compute similarity between word in the source and target languages.

  More formally, assume that (s ₁, s ₂, . . .s_N) and (t ₁, t ₂, . . .t_M) are (arbitrarily indexed) source and target vocabularies, respectively. A source word f is represented with an N-dimensional vector and a target word e is represented with an M-dimensional vector (see Figure 2). The component values of the vector representing a word correspond to how often each of the words in that vocabulary appear within a two word window on either side of the given word. These counts are collected using monolingual corpora. After the values have been computed, a contextual vector for f is projected onto the English vector space using translations in a given bilingual dictionary to map the component values into their appropriate English vector positions. This sparse projected vector is compared to the vectors representing all English words, e. Each word pair is assigned a contextual similarity score based on the similarity between e and the projection of f.

  Various means of computing the component values and vector similarity measures have been proposed in literature (e.g. Fung and Yee (Reference Fung and Yee1998); Rapp (Reference Rapp1999)). Following Fung and Yee (Reference Fung and Yee1998), we compute the value of the kth component of f’s contextual vector, f_k, as follows:

  (1) $$\begin{equation} f_{k} = n_{{f},k} * (log( {n / n_{k}}) + 1) \end{equation}$$
  where n _{f, k} and n_k are the number of times s_k appears in the context of f and in the entire corpus, and n is the maximum number of occurrences of any word in the data. Intuitively, the more frequently s_k appears with f_i and the less common it is in the corpus in general, the higher its component value. After projecting each component of the source language contextual vectors into the English vector space, we are left with M-dimensional source word contextual vectors, F_context, and correspondingly ordered M-dimensional target word contextual vectors, E_context, for all words in the vocabulary of each language. We use cosine similarity to measure the similarity between each pair of contextual vectors:
  (2) $$\begin{equation} sim_{context}(F_{context},E_{context}) = \frac{F_{context} \cdot E_{context}}{||F_{context}|| ||E_{context}||} \end{equation}$$

• • Temporal similarity – Usage of words over time may be another signal of translation equivalence. The intuition that is that news stories in different languages will tend to discuss the same world events on the same day and, correspondingly, we expect that source and target language words which are translations of one another will appear with similar frequencies over time in monolingual data. For instance, if the English word tsunami is used frequently during a particular time span, the Spanish translation maremoto is likely to also be used frequently during that time. To calculate temporal similarity, we collected online monolingual newswire over a multi-year period and associate each article with a time stamps. We gather temporal signatures for each source and target language unigram from our time-stamped web crawl data in order to measure temporal similarity, in a similar fashion to Schafer and Yarowsky (Reference Schafer and Yarowsky2002), Klementiev and Roth (Reference Klementiev and Roth2006) and Alfonseca, Ciaramita and Hall (Reference Alfonseca, Ciaramita and Hall2009). We calculate the temporal similarity between a pair of words, using the method defined by Klementiev and Roth (Reference Klementiev and Roth2006).

• • Orthographic similarity – Words that are spelled similarly are sometimes good translations, since they may be etymologically related, or borrowed words, or the names of people and places. We compute the orthographic similarity between a pair of words using Levenshtein edit distance,Footnote ² normalized by the average of the lengths of the two words. This is straightforward for languages which use the same character set, but it is more complicated for languages that are written using different scripts. For non-Roman script languages, we transliterate words into the Roman script before measuring orthographic similarity with their candidate English translations (Virga and Khudanpur Reference Virga and Khudanpur2003; Irvine et al. Reference Irvine, Callison-Burch and Klementiev2010). More details of our transliteration method are given in Section 3.

• • Topic similarity – Articles that are written about the same topic in two languages, are likely to contain words and their translations, even if the articles themselves are written independently and are not translations of one another. We use Wikipedia’s interlingual links to identify comparable articles across languages. These links define a number of topics, and we construct a topic vector. We compute cosine distance between topic signatures.

  (3) $$\begin{equation} sim_{topic}(F_{topic},E_{topic}) = \frac{F_{topic} \cdot E_{topic}}{||F_{topic}|| ||E_{topic}||} \end{equation}$$
  The length of a word’s topic vector is the number of interlingually linked article pairs. Each component f_k of F_topic is the count of the word f in the foreign article from the kth linked article pair, normalized by the total occurrences of k. The dimensionality of the topic signatures varies depending on the language pair. The number of linked articles in Wikipedia range from 84 (between Kashmiri and English) to over 500 thousand (between French and English). Figure 3 illustrates this signal. More details on our topic similarity are in Irvine (Reference Irvine2014).

• • Frequency similarity – Words that are translations of one another are likely to have similar relative frequencies in monolingual corpora. We measure the frequency similarity of two words, sim_freq, as the absolute value of the difference between the log of their relative corpus frequencies, or

  (4) $$\begin{equation} sim_{freq}(e,f) = \left| log\left(\frac{freq(e)}{\sum _{i} freq(e_i)}\right) - log\left(\frac{freq(f)}{\sum _{i} freq(f_i)}\right) \right| \end{equation}$$
  This helps prevent high frequency closed class words from being considered viable translations of less frequent open class words.

• • Burstiness similarity – Burstiness is a measure of how peaked a word’s usage is over a particular corpus of documents (Pierrehumbert Reference Pierrehumbert, Santos, Lindén and Nganga2012). Bursty words are topical words that tend to appear when some topic is discussed in a document. For example, earthquake and election are considered bursty. In contrast, non-bursty words are those that appear more consistently throughout documents discussing different topics, use and they, for example. Church and Gale (Reference Church and Gale1995, Reference Church, Gale, Armstrong, Church, Isabelle, Manzi, Tzoukermann and Yarowsky1999) provide an overview of several ways to measure burstiness empirically. Following Schafer and Yarowsky (Reference Schafer and Yarowsky2002), we measure the burstiness of a given word based on Inverse Document Frequency:

  (5) $$\begin{equation} IDF_{w} = - log \frac{df_w}{|D|} \end{equation}$$
  where df_w is the number of documents that w appears in, and |D| is the total number of documents in the collection. We have also experimented with a second burstiness measure, similar to that defined by Church and Gale (Reference Church and Gale1995), as the average frequency of w divided by the per cent of documents in which w appears. We make one modification to the definition provided by Church and Gale (Reference Church and Gale1995) and use relative frequencies rather than absolute frequencies to account for varying document lengths:
  (6) $$\begin{equation} B_{w} = \frac{\sum _{d_i \in D}{rf_{w{_{d_i}}}}}{df_w} \end{equation}$$
  where as before, df_w is the number of documents in which w appears and rf _{w di} is the relative frequency of w in document d_i. Relative frequencies are raw frequencies normalized by document length.

• • We also compute a number of variations on the above using word prefixes and suffixes instead of fully inflected words, and based on two different sources of data (web crawls and Wikipedia). In total, our model uses eighteen such features in order to rank English words as potential translations of the input foreign word.

Fig. 2.

Example of projecting contextual vectors over a seed bilingual lexicon. The Spanish word crecer appears in the context of the words empleo, extranjero, etc. in monolingual texts. We use this co-occurence information to build a context vector. Each position in the context vector for corresponds to a word in the Spanish vocabulary. The vector for crecer is projected into the English vector space using a small seed dictionary. Context vectors for all English words (policy, expand, etc.) are collected and then compared against the projected context vector for Spanish crecer. Finally, contextual similarities are calculated by comparing the projected vector with the context vector of each target word using cosine similarity. Word pairs with high cosine similarity are likely to be translations of one another.

Fig. 3.

Illustration of how we compute the topical similarity between troops and three Russian candidate translations. We first collect the topical signatures for each word (e.g. troops appears in the page about Barack Obama fifteen times and in the page about Virginia thirty-two times.) based on the interlingually linked pages. We can then directly compare each pair of topical signatures.

Table 1 shows some examples of the highest ranking English translations of five Spanish words for several of our signals of translation equivalence. Each signal produces different types of errors. For instance, using topic similarity, montana, miley and hannah are ranked highly as candidate translations of the Spanish word montana. The TV character Hannah Montana is played by actress Miley Cyrus, so the topic similarity between these words makes sense.

Table 1.

Examples of translation candidates ranked using contextual similarity, temporal similarity, orthographic similarity and topic similarity. The correct English translations, when found, are bolded

A significant research challenge is how best to combine these signals. Previous approaches have combined signals in an unsupervised fashion. One method of combining the ranked lists of translations that are independently generated by each of the signals of translation equivalence is using mean reciprocal rank (MRR), which is a measure typically used in information retrieval. It is defined as the average of the reciprocal ranks of results for a sample of queries Q:Footnote ³

(7) $$\begin{equation} \textnormal {MRR} = \frac{1}{|Q|} \sum _{i=1}^{|Q|} \frac{1}{\textnormal {rank}_i} \end{equation}$$

In the case of bilingual lexicon induction, we query each signal of translation equivalence with a source word, the value |Q| corresponds to the number of signals, and rank_i corresponds to the rank of a target language translation under the ith signal. The translation with the highest MRR value is output as the best translation. The disparate of signals of translation equivalence all provide an equal contribution in MRR, regardless of how good they are at picking out good translations.

Instead of weighting each signal equally, we use a discriminative model that is trained using entries in the seed bilingual dictionary as positive examples of translations, and random word pairs as negative examples (we use a 1:3 ratio of positive to negative examples). Discriminative models have an advantage over MRR in that they are able to weight the contribution of each feature based on how well it predicts the translations of words in a development set. When feature weights are discriminatively set, these signals produce dramatically higher translation quality than MRR. In Irvine and Callison-Burch (Reference Irvine and Callison-Burch2015), we present experimental results showing consistent improvements in translation accuracy for twenty-five languages. The absolute accuracy increases over the MRR baseline ranges from five per cent–thirty-one per cent, which correspond to 36 per cent–216 per cent relative improvements. Our discriminative approach requires a small number of translations to use as a development set. This requirement is not a major imposition, since bilingual lexicon induction already typically requires a small seed bilingual dictionary.

2.2 Experiments with bilingual lexicon induction

We excerpt a number of experiments from Irvine and Callison-Burch (Reference Irvine and Callison-Burch2015) that show our method’s performance on four of the Indian languages that we examine in the end-to-end machine translation experiments (Section 5).

2.2.1 Data

We created bilingual dictionaries using native-language informants on Amazon Mechanical Turk. In Pavlick et al. (Reference Pavlick, Post, Irvine, Kachaev and Callison-Burch2014), we describe a study of the languages demographics of workers on Amazon Mechanical Turk. In that work, we focused on the hundred languages which have the largest number of Wikipedia articles and posted Human Intelligence Tasks asking workers to translate the 10, 000 most frequent words in the 1, 000 most viewed pages for each source language. For the experiments in this article, we filter the dictionaries to include only high quality translations. Specifically, we limit ourselves to words that occurred at least ten times in our monolingual data sets, and we only use translations that have a quality score of at least 0.6 under the worker quality metric defined by Pavlick et al. (Reference Pavlick, Post, Irvine, Kachaev and Callison-Burch2014). Workers provided between one and thirty two reference translations for each word (with an average of 1.4 translations per word).

We gathered monolingual data sets by scraping online newspapers in each language, and by downloading the content of each language version of Wikipedia. For all languages, we use Wikipedia’s January 2014 data snapshots. Table 2 gives statistics about the monolingual data sets.

Table 2.

Statistics about the data used in our bilingual lexicon induction experiments

2.2.2 Measuring accuracy

We measure performance using accuracy in the top-k ranked translations. We define top-k accuracy over some set of ranked lists L as follows:

(8) $$\begin{equation} acc_k = \frac{\sum _{l \in L}{I_{lk}}}{|L|} \end{equation}$$

where I_lk is an indicator function that is 1 if and only if a correct item is included in the top-k elements of list l. That is, top-k accuracy is the proportion of ranked lists in a set of ranked lists for which a correct item is included anywhere in the highest k ranked elements. The denominator |L| is the number of words in a test set for a language. The numerator indicates how many of the words had at least one correct translation in the top-k translations posited for the word. Top-k accuracy increases as k increases.

A translation counts as correct if it appears in our bilingual dictionary for the language. We split our dictionaries into separate training and test sets. The test sets consist of 1, 000 randomly selected source language words and their translations. The training sets consist of the remaining words. We use the training set to project vectors for contextual similarity, and to train the weights of our discriminative model.

2.2.3 Experimental results

We answer the following research questions:

• • How often does our discriminative model for bilingual induction produce a correct translation within its top-10 guesses? Table 3 gives the top-10 accuracy for our model on Bengali, Tamil, Telugu and Hindi, and shows its improvements over the standard unsupervised approach for combining multiple signals of translation equivalence.

• • How much bilingual training data do we need in order to reach stable performance? We analyzed how accuracy changed as a function of the number of bilingual dictionary entries used to train the discriminative model. Figure 4 shows learning curves that hold steady after approximately 300 training words.

  Fig. 4.

  Learning curves varying the number of dictionary entries used as positive training instances to our discriminative models, up to 1,000. For all languages, performance is fairly stable after about 300 positive training instances. The x-axis shows the number of dictionary entries used in training, and the y-axis gives the top-k accuracy of the model.

  

• • How much monolingual data would we need? Figure 5 shows a learning curve function of the size of the monolingual corpora used to estimate the similarity scores that are used as features in the model. The accuracy continues to increase, even beyond 10 million words. More monolingual data is better, but it is sometimes difficult to acquire even monolingual data in huge volumes for low resource languages.

• • How well can our models translate rare words versus frequent words? Figure 6 shows that words that appear with higher frequency in our monolingual corpora tend to be translated better. Pekar et al. (Reference Pekar, Mitkov, Blagoev and Mulloni2006) also investigated the effects of frequency on finding translations from comparable copper. This makes sense since we have more robust statistics when constructing their vector representations. The performance drops slightly for the highest frequency words, which are likely function words.

Fig. 5.

Bilingual lexicon induction learning curves over varying monolingual corpora sizes for Tamil. The x-axis is shown on a log scale.

Fig. 6.

Bilingual lexicon induction accuracy as a function of source word frequency in Wikipedia monolingual data. Frequency is plotted along the x-axis. Top-k accuracy for the model is given in the y-axis.

Table 3.

Top-10 Accuracy for bilingual lexicon induction on a test set. The accuracy increases significantly moving from the unsupervised MRR baseline to our discriminative model

The effect of frequency has largely been ignored in past work on bilingual lexicon induction – most past work tried to discover translations only for the 1,000 most frequent words in a language.Footnote ⁴ The fact that low frequency words do not translate as well as high frequency words has significant implications for the application of bilingual lexicon induction to SMT. The most obvious use of learned translations would be as a way of augmenting what a SMT model learned from bitexts by applying bilingual lexicon induction to the OOV words. Unfortunately, the OOVs are lower frequency than the words that occurred in the bilingual training data. Therefore, the translations are of mixed quality. Table 4 shows some induced translations of Bengali words which were OOV with respect to a small bilingual training set.

Table 4.

Examples of OOV Bengali words, our top-3 ranked induced translations, and their correct translations. Correct induced translations are bolded

3.1 Our approach to transliteration

Following Virga and Khudanpur (Reference Virga and Khudanpur2003), we treat transliteration as a monotone character translation task. Rather than using a noisy channel model, our transliteration models is based on the log-linear formulation of SMT described in Och and Ney (Reference Och and Ney2002). Whereas SMT systems are trained on parallel sentences and use word-based n-gram language models, we use pairs of transliterated words along with character-based n-gram language models. We apply the word alignment algorithms from SMT to automatically align characters in pairs of transliterations. In fact, transliteration is simpler than translation, since phrases are often reordered in translation, but characters sequence are monotonic in transliteration. Our feature functions include a character sequence mapping probability (similar to the phrase translation probability), a character substitution probability (similar to the lexical probability), and a character-based language model probability. Table 5 shows some example transliteration rules that are learned using the SMT machinery.

Table 5.

Examples of automatically learned transliteration rules from Russian to English and from Greek to English, along with their associated log probabilities for a character sequence mapping probability, a character substitution probability, and a character-based language model probability

3.2 Transliteration experiments

3.2.1 Data

We can use the standard SMT pipeline to learn transliteration rules, and we can produce transliterations of previously unseen words using an SMT decoder. The key is simply to find appropriate parallel data that shows transliterated pairs across different character sets (like between English’s Roman alphabet and the Devanagari script used by Hindi). In Irvine et al. (Reference Irvine, Callison-Burch and Klementiev2010), we detailed how we mined transliteration training data from Wikipedia page titles for 150 languages. Wikipedia’s interlanguage links can be used as a source for example transliterations. We use the titles of non-Roman script languages that are paired with English pages that correspond to names. Wikipedia categorizes articles and maintains lists of all of the pages within each category. In mining transliteration data, we took advantage of a particular set of categories that list people born in a given year. For example, the Wikipedia category page ‘1961 births’ includes links to the ‘Barack Obama’ and ‘Michael J. Fox’ pages. We iterated through birth years and the links to pages about people born in each year and then followed interlingual links from each English page about a person, compiling a large list of person names (Wikipedia page titles) in many languages. We found a total of 8,26,508 English Wikipedia pages about people. A similar process could be done to scrape other types of NEs, for instance by iterating over Wikipedia page categories for things like ‘Countries in Africa’ or ‘Cities in Europe’, but the expected yield would be lower than the number of person names. Table 6 gives the number of pairs of names between the English articles and the Indian languages that we examine in our end-to-end SMT experiments.

Table 6.

The number of Wikipedia articles with interlanguage links to English Wikipedia articles that describe people. These name pairs are used as training data to our SMT-inspired transliteration system

3.2.2 Experimental results

Here, we reproduce some of the experimental results from Irvine et al. (Reference Irvine, Callison-Burch and Klementiev2010) that demonstrate the quality of our transliteration system. We evaluated our transliteration system on the ACL 2009 NEs Workshop, which featured a shared task on transliteration (Li et al. Reference Li, Kumaran, Pervouchine and Zhang2009). The shared task evaluated systems trained to transliterate from English to several other languages using a variety of metrics. We used the workshop data to build a English–Hindi transliteration system, and compared our results against the other entries to the shared task. Table 7 shows our system’s performance on the NEWS task – it is competitive with other systems entered into the shared task. Table 8 shows some example transliterations produced by our system paired with reference transliterations. Sometimes the system produces near-misses that could still be useful.

Table 7.

A comparison of our performance (Irvine et al. Reference Irvine, Callison-Burch and Klementiev2010) against the systems submitted to the Hindi transliteration shared task at the ACL 2009 Named Entities Workshop. There were 4,840 training pairs for English → Hindi in the NEWS shared task

Table 8.

Example transliterations. Sometimes the errors are near-misses where the system’s proposed transliterations are only a few letters off from the reference transliteration. In these cases, the system does not receive any credit under metrics like the Bleu score, even though they may still be useful for human readers. Normalized edit distance is the number of edits divided by the length of the reference

In our end-to-end translation experiments, we output the single best transliteration of each OOV word using our transliteration model. This transliteration was placed alongside the top-k translations proposed by the bilingual lexicon induction module. Hermjakob, Knight and Daumé III (Reference Hermjakob, Knight and Daumé2008) trained a system so that it was able to learn when to transliterate versus translate. In our simpler setup, the SMT decoder had access to both transliterations and translations, and it used its model scores to select between the different options.

4.1 Data

Post, Callison-Burch and Osborne (Reference Post, Callison-Burch and Osborne2012) used Amazon Mechanical Turk to collect small parallel corpora for the following Indian languages and English: Tamil, Telugu, Bengali, Malayalam, Hindi and Urdu. They collected both parallel sentence pairs and a dictionary of word translations. We use all six datasets, which provide real low resource data conditions for six truly low resource language pairs. Tables 9 and 10 show statistics about the datasets.

Table 9.

Information about datasets released by Post et al. (Reference Post, Callison-Burch and Osborne2012): words in the source language parallel sentences and dictionaries, and per cent of development set word types and tokens that are OOV (do not appear in either section of the training data). Post et al. (Reference Post, Callison-Burch and Osborne2012) did not provide a dictionary for Hindi, so we exclude it from the baseline SMT system

Table 10.

The number of sentence pairs in the training/dev/test set splits for the Indian-language bilingual parallel corpora released by (Post et al. Reference Post, Callison-Burch and Osborne2012)

As usual, we use both our web crawls and our Wikipedia comparable corpora for each language pair. Dataset sizes are given in Table 2 for Bengali, Hindi, Tamil and Telugu. For Malayalam, we had 4 million words in our web crawled data, and 5 million words in our Wikipedia data (with 17,000 interlanguage links). For Urdu, we had 285 million words in our web crawled data, and 3 million words in our Wikipedia data (with 15,000 interlanguage links).

4.2 Experimental setup

We use the training/development/test data splits given by Post et al. (Reference Post, Callison-Burch and Osborne2012) and, following that work, include the dictionaries in the training data and report results on the devtest set using case-insensitive BLEU and four references. We use the Moses phrase-based MT framework (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007). For each language, we extract a phrase table with a phrase limit of seven. In order to make our results comparable to those presented in Post et al. (Reference Post, Callison-Burch and Osborne2012), we follow that work and use the English side of the training data to train a language model. Using a language model trained on a larger corpus (e.g. the English side of our comparable corpora) may yield better results, but such an improvement is orthogonal to the focus of this work. Throughout our experiments, we use the batch version of MIRA (Cherry and Foster Reference Cherry and Foster2012) for tuning the feature set. We rerun tuning for all experimental conditions and report results averaged over three tuning runs (Clark et al. Reference Clark, Dyer, Lavie and Smith2011).

Our baseline uses the bilingually extracted phrase pairs and standard translation probability features. We augment it with the single top-ranked translation for each OOV to improve coverage (+ OOV Trans) and with additional features to improve accuracy (+Features). We make each modification separately and then together. Then, we present additional experiments where we induce translations for low frequency words, in addition to OOVs (4.2.3), append top-k translations (4.2.4), vary the amount of training data used to induce the baseline model (4.2.5), and vary the amount of comparable corpora used to estimate features and induce translations (4.2.6).

4.2.1 Results: bilingual lexicon induction

Before presenting end-to-end MT results, we examine the performance of the supervised bilingual lexicon induction technique that we use for translating OOVs. In Table 11, top-1 accuracy is the per cent of source language words in a held out portion of the training dataFootnote ⁵ for which the highest ranked English candidate is a correct translation. Post et al. (Reference Post, Callison-Burch and Osborne2012) gathered up to six translations for each source word, so some have multiple correct translations. Performance is lowest for Tamil and highest for Hindi. For all languages, top-10 accuracy is much higher than the top-1 accuracy. In Section 4.2.4, we explore appending the top-k translations for OOV words to our model instead of just the top-1.

Table 11.

Per cent of word types in a held out portion of the training data which are translated correctly by our bilingual lexicon induction technique. Evaluation is over the top-1 and top-10 outputs in the ranked lists for each source word

4.2.2 Improving coverage and accuracy in end-to-end SMT

Table 12 shows our results adding OOV translations, adding features and then both. Simply adding monolingually estimated features functions to the phrase table improves our models’ accuracy, increasing BLEU scores between 0.18 (Bengali) and 0.60 (Malayalam).

Table 12.

BLEU scores improve for all six low resource languages when we add translations of OOV using bilingual lexicon induction (+OOV Trans.), and when we add monolingually derived features to the standard phrase table features (+Features). The greatest gains come from incorporating both OOV translations and new features (+Features & Trans)

Adding OOV translations makes a big difference for some languages, such as Bengali and Urdu, and almost no difference for others, like Malayalam and Tamil. The OOV rate (Table 9) is low in the Malayalam dataset and high in the Tamil dataset. However, as Table 11 shows, the translation induction accuracy is low for both. Since few of the supplemental translations are correct, we don’t observe BLEU gains. In contrast, induction accuracies for the other languages are higher, OOV rates are substantial, and we do observe moderate BLEU improvements by supplementing phrase tables with OOV translations.

Combining the two methods results in translations that are better than applying either technique alone for five of the six languages. BLEU gains range from 0.5 (Bengali) to 1.4 (Urdu). We attribute the particularly good Urdu performance to the relatively large monolingual corpora (Table 2). In Section 4.2.6, we present results varying the amount of Urdu–English comparable corpora used to induce translations and estimate additional features.

4.2.3 Translations of low frequency words

Beyond adding translations just for strictly OOV words, we wanted to evaluate whether bilingual lexicon induction could also be useful for low frequency words. Strictly speaking, adding translations of OOV words will never decrease the BLEU score, since even adding in a random translation is no worse (under BLEU) than outputting a foreign word written in a non-Roman script.

For source words which only appear a few times in the parallel training text, the bilingually extracted translations in the standard phrase table are likely to be inaccurate and incomplete. Augmenting a model with additional translations for low frequency words may fix some other types of errors, for instance, a source word was observed in training with a translation that is not the correct sense for the test set.

We perform additional experiments varying the minimum source word training data frequency for which we induce additional translations. That is, if freq(w_src) ⩽ M, we induce a new translation for it and include that translation in our phrase table. Note that in the results presented in Table 12, M = 0, meaning that it only adds induced translations for OOVs and not for low frequency words that occurred once or more in the training data. In these experiments, we include our additional phrase table features estimated over comparable corpora and hope that these scores will assist the model in choosing among multiple translation options for low frequency words, one or more of which is extracted bilingually and one of which is induced using comparable corpora. Table 13 shows the results when we vary M. As before, we average BLEU scores over three tuning runs.

Table 13.

Varying minimum parallel training data frequency of source words for which new translations are induced and included in the phrase-based model. In all cases, the top-1 induced translation is added to the phrase table and features estimated over comparable corpora are included (i.e. +Feats & Trans model)

In general, modest BLEU score gains are made as we augment our phrase-based models with induced translations of low frequency words. The highest performance is achieved when M is between 5 and 50, depending on language. The largest gains are 0.5 and 0.3 BLEU points for Bengali and Urdu, respectively, at M = 25. This is not surprising; we also saw the largest relative gains for those two languages when we added OOV translations to our baseline model. With the addition of low frequency translations, our highest performing Urdu model achieves a BLEU score that is 1.7 points higher than the baseline.

In different data conditions, inducing translations for low frequency words may result in better or worse performance. For example, the size of the training set impacts the quality of automatic word alignments, which in turn impacts the reliability of translations of low frequency words. However, the experiments detailed here suggest that including induced translations of low frequency words will not hurt performance and may improve it.

4.2.4 Appending top-k translations

So far we have only added the top-1 induced translation for OOV and low frequency source words to our phrase-based model. However, the bilingual lexicon induction results in Table 11 show that accuracies in the top-10 ranked translations are, on average, nearly twice the top-1 accuracies. Here, we explore adding the top-k induced translations. We hope that our additional phrase table features estimated over comparable corpora will enable the decoder to correctly choose between the k translation options. We induce translations for OOV words only (M = 0) and include all comparable corpora features.

Table 14 shows performance as we append the top-k ranked translations for each OOV word and vary k. With the exception of Bengali, using a k greater than 1 does not increase performance. In the case of Bengali, and additional 0.2 BLEU is observed when the top-25 translations are appended. In contrast, we see performance decrease substantially for other languages (0.7 BLEU for Telugu and 0.2 for Urdu) when the top-25 translations are used. Therefore, we conclude that, in general, the models do not sufficiently distinguish good from bad translations when we append more than just the top-1. Although using a k greater than 1 means that more correct translations are in the phrase table, it also increases the number of possible outputs over which the decoder must search.

Table 14.

Adding top-k induced translations for source language OOV words, varying k. Features estimated over comparable corpora are included (i.e. +Feats & Trans model). The highest BLEU score for each language is highlighted. In many cases, differences are less than 0.1 BLEU

4.2.5 Learning curves over parallel data

In the experiments above, we only evaluated our methods for improving the accuracy and coverage of models trained on small amounts of bitext using the full parallel training corpora released by Post et al. (Reference Post, Callison-Burch and Osborne2012). Here, we apply the same techniques but vary the amount of parallel data in order to generate learning curves. Figure 7 shows learning cures for all six languages. In all cases, results are averaged over three tuning runs. We sample both parallel sentences and dictionary entries.

Fig. 7.

Comparison of learning curves over lines of parallel training data for four SMT systems: our baseline phrase-based model (baseline), model that supplements the baseline with translations of OOV words induced using our supervised bilingual lexicon induction framework (+Trans), model that supplements the baseline with additional phrase table features estimated over comparable corpora (+Feats) and a system that supplements the baseline with both OOV translations and additional features (+Trans & Feats).

All six learning curves show similar trends. In all experimental conditions, BLEU performance increases approximately linearly with the log of the amount of training data. Additionally, supplementing the baseline with OOV translations improves performance more than supplementing the baseline with additional phrase table scores based on comparable corpora. However, in most cases, supplementing the baseline with both translations and features improves performance more than either alone. Performance gains are greatest when very little training data is used. The Urdu learning curve shows the most gains as well as the cleanest trends across training data amounts. As before, we attribute this to the relatively large comparable corpora available for Urdu.

4.2.6 Learning curves over comparable corpora

In our final experiment, we consider the effect of the amount of bilingual comparable corpora that we use to estimate features and induce translations. We present learning curves for Urdu–English because we have the largest amount of monolingual corpora for that pair. We use the full amount of parallel data to train a baseline model, and then we randomly sample varying amounts of our Urdu–English monolingual corpora. Sampling is done separately for the web crawl and Wikipedia comparable corpora. Figure 8 shows the results. As before, results are averaged over three tuning runs.

Fig. 8.

Urdu to English translation results using varying amounts of monolingual corpora to estimate features and induce translations.

The phrase table features estimated over comparable corpora improve end-to-end MT performance more with increasing amounts of comparable corpora. In contrast, the amount of comparable corpora used to induce OOV translations does not impact the performance of the resulting MT system as much. The difference may be due to the fact that data sparsity is always more of an issue when estimating features over phrase pairs than when estimating features over word pairs because phrases appear less frequently than words in monolingual corpora. Our comparable corpora features are estimated over phrase pairs while translations are only induced for OOV words, not phrases. So, it makes sense that the former would benefit more from larger monolingual corpora.