1 Motivation
The need for machine translation (MT) is continuously growing due to the evolution of information society. Today translation technology is trying to keep pace with the demand for high-quality translations.
MT systems can be classified into rule-based and corpus-based approaches, in terms of their core technology. Recently, the statistical systems, which fall in the corpus-based category, are popular given that the knowledge is automatically extracted by analyzing translation examples from a parallel corpus built by human experts. The advantage is that, once the required techniques have been developed for a given language pair, statistical machine translation (SMT) systems can be quickly developed for new language pairs using provided training data. For further information on SMT systems please refer to surveys such as López (Reference López2008) and Koehn (Reference Koehn2010).
One of the main limitations of SMT as in many machine learning systems is domain adaptation. The task of domain adaptation consists of building models for some specific target domain, given that training data is generally extracted from various domains (including or not the target domain). For example, the Workshop on Statistical Machine Translation (WMT)Footnote 1 organizes an international evaluation campaign, with different shared tasks. Generally, it provides training data from different sources including: the European Parliament Plenary Speeches (EPPS), news and United Nations (UN). However, the test data is only from the news domain. Since EPPS and UN use different vocabulary words and a different language style than news, it may seem reasonable that several techniques have to be applied to give more importance to the news training data in order to obtain a better translation output.
The main challenges of performing out-of-domain translations are the out-of-vocabulary words, the unknown expressions and the syntactic constructions. These challenges have mainly been addressed either with limited or noFootnote 2 in-domain data by using morphology/linguistic rules or exploiting different kinds of in-domain material. The nature of in-domain material in SMT varies from bilingual or monolingual corpus, translation memories (TMs) to lexicons.
This paper presents an overview on overcoming the challenge of non-existent in-domain data or exploiting limited, large in-domain data to improve domain adaptation in SMT. Also, we dedicate a section to mention how domain adaptation techniques are exploited on online scenarios. Table 1 shows a summary of this classification.
Classification of domain adaptation approaches
2 Domain adaptation with limited or non-existent in-domain data
SMT is a corpus-based approach. Usually, a large quantity of parallel sentences is required in order to build a high-quality translation system. The more in-domain material, the better the translation. However, usually, in-domain material is limited or even non-existent. Under these circumstances, to solve out-of-domain problems, researchers have to either overcome the challenge of non-existent in-domain data or exploit limited resources by making use of linguistic knowledge or statistical techniques.
2.1 Adaptation with non-existent in-domain data
There are several techniques that somehow focus on adapting data without resources. We are going to describe most relevant techniques in this area classified into linguistically motivated and data driven. The former is when techniques use some kind of linguistics inherent in the languages dealt with. The latter does not take into account any characteristic of the languages used.
2.1.1 Linguistically motivated techniques
The use of linguistic knowledge with language dependencies helps in domain adaptation. For example, in Farrús et al. (Reference Farrús, Costa-jussà, Hernández, Hneríquez, Mariño and Fonollosa2009), authors develop linguistic rules to further improve the Catalan–Spanish translation. The main advantage of linguistic rules is that they generalize for any type of domain. Examples of linguistic rules in this work includes codifying time expressions, solving homonymy by attaching the part-of-speech to the surface word and post-processing verbs and pronouns either to be attached with a hyphen (Catalan) or without (Spanish).
Other works such as Formiga et al. (Reference Formiga, Hernández, Mariño and Monte2012) try to deal with the morphology challenge. Usually, a training set does not contain all morphology alternatives of a single word, specially when working with highly inflected languages. In this approach, authors propose considering Spanish morphology generation as a standard classification task. The idea is that one characteristic of verb forms like number is generalized. So, the translation system learns words without the number information. A posterior classifier is in charge of deciding whether the word is singular or plural. Therefore, after defining a set of simple relevant features, they apply machine learning techniques to classify this independent morphology generation task. The main drawback of this linguistic exploitation is that it is not directly generalizable to other pairs of languages. However, the linguistic rules approach is still investigated, specially working in hybrid MT approaches, see Costa-jussà et al. (Reference Costa-jussà, Banchs, Rapp, Lambert, Eberle and Babych2013).
2.1.2 Data-driven techniques
Authors in Hildebrand et al. (Reference Hildebrand, Eck, Vogel and Waibel2005) try to exploit information retrieval techniques to retrieve sentence pairs from the training corpus that are relevant to the test sentences. Then, both the language and the translation models are retrained on the extracted data.
Other works try to exploit source similarities such as España-Bonet et al. (Reference España-Bonet, Giménez and Màrquez2010). Authors built classifiers inspired by those used in word-sense disambiguation (WSD) in order to select translation phrases. In Carpuat and Wu (Reference Carpuat and Wu2007), authors embed context-rich approaches from WSD methods into SMT systems. In Ueffing et al. (Reference Ueffing, Haffari and Sarkar2008), authors use semi-supervised model adaptation in the source text. In Haque (Reference Haque2011), the author adds source-side contextual features by incorporating context-dependent phrasal translation probabilities learned using decision trees.
A recent approach by Henríquez et al. (Reference Henríquez, Mariño and Banchs2011) interpolates the phrase and lexical re-ordering tables with a new adapted translation model with derived units. Authors obtain the units identifying the mismatching parts between the non-adapted translation and the actual reference. This approach has successfully been tested in the context of the WMT evaluation (Formiga et al., Reference Formiga, Costa-jussà, Mariño, Fonollosa, Barrón-Cedeño and Màrquez2013).
There are several works that use different techniques to extract monolingual or parallel in-domain corpora, that is, Moore and Lewis (Reference Moore and Lewis2010), Axelrod et al. (Reference Axelrod, He and Gao2011), by using perplexity or cross-entropy methods. Pecina et al. (Reference Pecina, Toral, Way, Papavassiliou, Prokopidis and Giagkou2011) use web-crawled data.
2.2 Limited resources exploitation
Actually, in SMT, there are many works that try to exploit limited resources to their great advantage like using big target language models or factored language models. One interesting approach is using techniques of neural language modeling, see Schwenk et al. (Reference Schwenk, Costa-jussà and Fonollosa2007). The basic idea of continuous space language models (CSLM), also called neural network language models, is to project the word indices onto a continuous space and to use a probability estimator operating on this space. Since the resulting probability functions are smooth functions of the word representation, better generalization to unknown n-grams can be expected. This is believed to be particularly important for tasks with limited resources. A neural network can be used to simultaneously learn the projection of the words onto the continuous space and to estimate the n-gram probabilities. This is still an n-gram approach, but the Language Model (LM) conditional probabilities are interpolated for any possible context of length n−1, instead of backing-off to shorter contexts. The main disadvantage of this approach is the high computational cost.
Another interesting approach when having limited resources is smoothing the translation probabilities. Zens and Ney (Reference Zens and Ney2004) suggested to use IBM-1 probabilities or other lexical translation probabilities. Another method of smoothing is the one presented in Foster and Kuhn (Reference Foster, Goutte and Kuhn2007). Two types of phrase-table smoothing are compared: black-box and glass-box methods. Black-box methods do not look inside phrases but instead treat them as atomic objects and can use all the methods developed for language modeling.
3 Techniques for exploiting larger quantities of different types of in-domain data
SMT can be further developed by using available in-domain material, which can be in the form of bilingual or monolingual corpus; TMs and lexicons.
3.1 Exploiting in-domain parallel corpora
Exploiting in-domain parallel corpora has been recently addressed in the scientific community by the annual WMT international evaluation campaign.
The straightforward way to use in-domain parallel corpora is to simply concatenate it with the out-of-domain corpora available. Then, use the combined data for both translation model and language model training. However, in case in-domain data is much smaller than out-of-domain data, the gain expected through a simply concatenation is not much. The result can be worse than using only in-domain data. This normally happens when the in-domain data has a very particular style, as shown by Khalilov et al. (Reference Khalilov, Costa-Jussà, Henríquez, Fonollosa, Hernández, Mariño, Banchs, Chen, Zhang, Aw and Li2008).
Another proposal is to implement a translation model interpolation strategy. Authors in Koehn and Schroeder (Reference Koehn and Schroeder2007) show that the linear interpolation of translation models achieve gains in BLEU. Alternatively, a log-linear interpolation also improves translation, see Khalilov et al. (Reference Khalilov, Costa-Jussà, Henríquez, Fonollosa, Hernández, Mariño, Banchs, Chen, Zhang, Aw and Li2008). Similarly, Ceausu et al. (Reference Ceausu, Tinsley, Zhang and Way2011) show that in-domain translation models and general language models perform better for patent translation than other combinations of in-domain and general data.
The technique of mixture models is explored in works such as Foster et al. (Reference Foster and Kuhn2006), Finch and Sumita (Reference Finch and Sumita2008) and Senrich (Reference Sennrich2012). Mixture modeling is a standard technique for density estimation that is capable of learning specific probability distributions that better fit subsets of the training data set. Extensions of these works can be found in Foster et al. (Reference Foster, Kuhn and Johnson2010) and Civera and Juan (Reference Civera and Juan2007).
In Rogati (Reference Rogati2009), the author adds in-domain corpora by automatically weighting and combining multiple translation resources, according to several criteria, in order to better match a target corpus or a specific domain sample. The criteria include lexical-level domain match, translation quality estimates, size and taxonomy representation. The main drawback of these approaches is that they may not generalize for different types of corpus. Usually, it depends on the size of the in-domain corpora. In Niehues and Waibel (Reference Niehues and Waibel2010), authors use the corpus id as a target word factor in order to optimize a factored-based translation.
3.2 Exploiting comparable corpora
Comparable corpora are those texts that talk about the same topic in different languages, but do not have corresponding translations as parallel texts. There is plenty of work in the exploitation of comparable corpora from different perspectives. Most of them focus on extracting parallel sentences from comparable corpora. If we focus on domain adaptation, we found the work of Daumand Jagarlamudi (Reference Daum and Jagarlamudi2011) that use the specific setting in which they have plentiful parallel (labeled) data in a source domain and plentiful comparable (unlabeled) data in a target domain. They can use the unlabeled data in the target domain to build a good language model. Finally, they assume access to a very small amount of parallel (labeled) target data, but only enough to evaluate or run weight tuning. All knowledge about unseen words is extracted from the comparable data. As a related project, we found ACCURATFootnote 3 that particularly targets a number of under-resourced languages and evaluates applicability of comparable corpora for adapting MT to specific narrow domains. The techniques that have been used to improve MT with this additional corpora include a recommendation system toward better translations, see Sakadina et al. (Reference Skadiņa, Aker, Mastropavlos, Su, Tufiş, Verlič, Vasiļjevs, Babych, Clough, Gaizauskas, Glaros, Paramita and L., Pinnis2012).
3.3 Exploiting in-domain monolingual corpora
The exploitation of in-domain monolingual corpora have been mainly limited to the target language model, see Eck et al. (Reference Eck, Vogel and Waibel2004). In Bulyko et al. (Reference Bulyko, Matsourkas, Schwartz, Nguyen and Makhoul2007), they explored discriminative estimation of language model weights by directly optimizing MT performances. In Wu et al. (Reference Wu, Wang and Zong2008), they investigate linear and log-linear interpolation of in-domain and out-of-domain language models, which is still highly used in state-of-the-art systems given its simplicity and usefulness.
In Schwenk and Estève (Reference Schwenk and Estève2008), authors present how the CSLM technique, seen in Section 2.2, scales in larger sets. Bertoldi and Federico (Reference Bertoldi and Federico2009) have used in-domain monolingual corpora to synthesize a bilingual corpus by translating the monolingual adaptation data into the counterpart language. Translation, re-ordering and language models were estimated after translating in-domain texts with the baseline system.
3.4 Exploiting lexicons
The WMT evaluation, which evaluates domain adaptation, does not provide lexicons. In fact, from the SMT perspective, there are not many works in this direction. We can summarize the related studies as follows.
There are several straightforward approaches. One of them is to introduce the lexicon as phrases in the phrase-based table as in Farrús et al. (Reference Farrús, Costa-jussà, Hernández, Hneríquez, Mariño and Fonollosa2009). Unfortunately, the word has no context information. Another approach is to introduce them to substitute the unknown words in the translation, which have the same problem as before. Okuma et al. (Reference Okuma, Yamamoto and Sumita2008) present a more sophisticated approach where the lexicon words are introduced in the training corpus to obtain their corpus. The criteria that they use is basically a Name Entity Recognition classification. Notice that their lexicon is based on person and places Name Entities. Wu et al. (Reference Wu, Wang and Zong2008) uses domain dictionaries as a translation table. To solve the challenge of assigning a probability to each dictionary entry, they used alternatively uniform, constant or monolingual corpus probabilities.
3.5 Exploiting translation memories
TMs are widely used to enhance human translation efforts as in Abekawa and Kageura (Reference Abekawa and Kageura2007). A typical TM system has a database of translated sentence pairs, called translation units, and exploits a well-designed matching algorithm to retrieve the most relevant translation units as translation references for a given source sentence to be translated.
Up to this point, works mentioned above do not use SMT methods together with TMs. As far as we know, there are not many works that use in-domain TMs to improve SMT. Despite the fact that TM is widely used to improve human translation, it has yet to be much applied to improve SMT. Indeed, there appears to be just one paper on this topic by Marcu (Reference Marcu2001). In this paper, the authors extract a TM from a parallel corpus. This automatic TM is used together with a phrase-based system to improve the translation output. Basically, they modify the decoder so that it attempts to find good translation starting from two distinct points in the space of possible translations: one point corresponds to a phrase-table; the other point corresponds to a translation that resembles most closely translations stored in the TM.
More recently, Google translator tools provide the possibility to translate texts by using user’s translation materials such as TM’s and glossariesFootnote 4. The tool searches all available translation databases for previous human translations of each segment. If any previous human translations of the segment exist, the tool picks the highest ranked search result and ‘pretranslate’ the segment with that translation. If no previous human translation of the segment exists, the tool uses SMT.
3.6 Online scenarios
Finally, this subsection reviews the emerging topic of incremental (or online) adaptation. Incremental adaptation fits naturally in those translations scenarios where human translators supervise the output of MT systems and relevant works are cited as follows. Hardt and Elming (Reference Hardt and Elming2010) show an application of online learning techniques to process training data in a sentence-wise manner. Ortíz-Martnez et al. (Reference Ortíz-Martnez, García-Varea and Casacuberta2010) show an application similar to the previous work, but removing the heuristics that they introduce to estimate the phrase-based model. This is achieved by means of the application of the incremental Expectation-Maximization (EM) algorithm. Levenberg et al. (Reference Levenberg, Callison-Burch and Osborne2010) show an application of online learning techniques to process large chunks of training data using stepwise EM algorithm.
4 Remarks
One relevant challenge in SMT is domain adaptation, because the more similar is the training material to the test material, the better the translation quality. The increase in the amount of available data has pushed research in domain adaptation. In addition to allowing for a better translation quality, research in domain adaptation may allow to develop SMT systems of better quality to offer to professional translators. Domain adaptation may be really helpful in the recent initiative of interactive machine translation (IMT), where the system aids the translator by interactively making suggestions for completing the translation. The IMT idea have been developed in the TransType and TransType2 European projects, see Barrachina et al. (Reference Barrachina, Bender, Casacuberta, Civera, Cubel, Khadivi, Lagarda, Ney, Toms and Vidal2009) and Esteban et al. (Reference Esteban, Lorenzo, Valderrábanos and Lapalme2004). The continuation of the mentioned projects is the Casmacat European ProjectFootnote 5. Out of these projects, a relevant online IMT (i.e. Caitra) system is now availableFootnote 6.
Acknowledgments
This work has been funded by the Seventh Framework Program of the European Commission through the International Outgoing Fellowship Marie Curie Action (IMTraP-2011-29951), the Spanish Ministerio de Economía y Competitividad, contract TEC2012-38939-C03-02 and the European Regional Development Fund (ERDF/FEDER).
