1 Introduction
Statistical machine translation (SMT) systems require a parallel corpus to train the translation model and monolingual data to build the target language model. A parallel corpus, also called bitext, consists in bilingual/multilingual texts aligned at the sentence level. Unfortunately, parallel texts are a sparse resource for many language pairs with exception of English, French, Spanish, Arabic, Chinese and some European languages included in EuroparlFootnote 1 and OPUS.Footnote 2 Furthermore, the existing available corpora do not cover some special domains or sub-domains (Hewavitharana and Vogel Reference Hewavitharana and Vogel2011).
For the field of SMT, this can be problematic, because translation systems trained on data from a specific domain (e.g. parliamentary proceedings) will perform poorly when applied to other domains, e.g. scientific articles.
One way to overcome this lack of data is to exploit comparable corpora which are much more easily available (Munteanu and Marcu Reference Munteanu and Marcu2005). A comparable corpus is a collection of texts composed independently in their respective languages and combined on the basis of similarity of content. These are bi- or multilingual documents that are comparable in content and form in various degrees and dimensions. Potential sources of textual comparable corpora are the multilingual news organizations such as Agence France Presse, Xinhua, Reuters, CNN, BBC, etc. These texts are widely available on the web for many language pairs (Resnik and Smith Reference Resnik and Smith2003). Another example is Euronews, which proposes news text in several languages and clustered by domain (e.g. sports, finance, etc.). This corpus will be detailed in Section 3.1. The ability to detect the parallel sentences or sub-sentences contained on these kinds of comparable corpus enable the automatic creation of large parallel corpora.
However, for some languages, text comparable corpora may not cover all topics in some specific domains. One of the main challenges of our research is to build data and techniques for these under-resourced domains. We propose to explore other sources in other modalities (audio in our case), which are widely available on the web for many languages, to generate parallel texts for each domain.
In this paper, we explore several methods for generating parallel sentences and phrases extracted from multimodal comparable corpora (audio and text). We would expect a useful technique to meet three criteria:
• Feasibility: parallel data can be extracted from the multimodal comparable corpora.
• Good quality: the quality of the parallel data generated from multimodal corpora should be equivalent to the quality of bitext extracted from (monomodal) text comparable corpora.
• Effectiveness: one of our motivations is to adapt an SMT system to a specific domain, consequently, extracted bitext should improve SMT system performance over a baseline system.
The methods for improving translation quality proposed in this work rely upon multimodal comparable corpora, that is, multiple corpora in different modalities that cover the same general topics and events.
This paper describes an experimental framework designed to address two situations. The first one is when we translate data from a new domain, different from the training data. In such a condition, the translation quality is generally rather poor. The second one is when we seek to improve the quality of an SMT system already trained on the same kind of data (same domain and/or style).
We start by giving a brief overview of related work in parallel data extraction and our multimodal comparable corpora developed to test our proposed method. Section 4 details our parallel sentence extraction method and shows results of our experiments on TED data. In Section 6, we present our methods in extracting parallel phrases and the experimental comparison between them using Euronews and TED data. We conclude with a discussion and perspectives of this work.
2 Related work
In the SMT community, there is a long-standing belief that there is no better data than more data. Following this idea, a considerable amount of work have been undertaken for discovering parallel fragments in a comparable corpus in order to improve SMT systems. Thus, there is already an extensive literature related to the problem of exploiting comparable corpora, although from a different perspective than the one taken in this paper.
Typically, comparable corpora do not have any information regarding document pair similarity. They are made of many documents in one language which do not have any corresponding translated document in the other language. Also, when the documents are paired, they are not literal translations one of each other. Thus, extracting parallel data from such corpora requires special algorithms.
Many works use the web as a comparable corpus. An adaptive approach, proposed by Zhao and Vogel (Reference Zhao and Vogel2002), aims at mining parallel sentences from a bilingual comparable news collection collected from the web. A maximum likelihood criterion was used by combining sentence length models and lexicon-based models. The translation lexicon is iteratively updated using the mined parallel data to get better vocabulary coverage and translation probability estimation. Resnik and Smith (Reference Resnik and Smith2003) propose a web-mining-based system called STRAND and show that their approach is able to find large numbers of similar document pairs. In Yang and Li (Reference Yang and Li2003), an alignment method at different levels (title, word and character) based on dynamic programming is presented. The goal is to identify the one-to-one title pairs in an English/Chinese corpus collected from the web, They apply longest common sub-sequence to find the most reliable Chinese translation of an English word.
One of the main method relies on cross-lingual IR, with different techniques for transferring the request into the target language (using a bilingual dictionary or an full SMT system. Utiyama and Isahara (Reference Utiyama and Isahara2003) uses cross-language information retrieval (IR) techniques and dynamic programming to extract sentences from an English/Japanese comparable corpus. They identify similar article pairs, and then, considering them as parallel texts, they align their sentences using a sentence pair similarity score and use Dynamic Programming to find the least-cost alignment over the document pair. Munteanu and Marcu (Reference Munteanu and Marcu2005) uses a bilingual lexicon to translate some of the words of the source sentence. These translations are then used to query the database to find matching translations using IR techniques. AbdulRauf and Schwenk (Reference Abdul-Rauf and Schwenk2011) bypass the need of the bilingual dictionary by using proper SMT translations. They also use simple measures like word error rate or translation edit rate (TER) in place of a maximum entropy classifier.
Also, other modalities can be considered, e.g. Paulik and Waibel (Reference Paulik and Waibel2009) demonstrated that statistical translation models can be trained in a fully automatic manner from audio recordings of human interpretation scenarios.
In this paper, we are interested in generating a parallel text from a comparable corpora composed by an audio part in one language and a text part in another language.
3 Building multimodal comparable corpora
In this work, data is extracted from the available news (video and text modalities) on the Euronews web site. We also use TED-LIUM (Rousseau, Deléglise and Estève Reference Rousseau, Deléglise and Estève2012) corpus to build our TED multimodal comparable corpus and test our extraction methods.
3.1 Euronews
Figure 1 shows an example of multimodal comparable data coming from the Euronews web site. An audio source of a political news and its text version, both in English, are available along with the equivalent news in French (audio and text modalities). The audio content in the videos are not exactly the same for each language, but are dealing with the same subject. Then, audio in one language and the text content in the other language can be considered as comparable data. This corpus can be used to extract parallel data, at the sentence and the sub-sentential level. In other words, in comparable corpora, it is sometimes possible to find some fully parallel sentences, but in most of the cases, only parallel fragments can be extracted. This is more detailed in Section 6, and illustrated in Figure 12.
Example of multimodal comparable data from the Euronews web site.
Euronews web site clusters news into several categories or sub-domains (e.g. Sport, Politics, etc.). These categories are preserved in the raw version of the provided corpus (but not in extracted versions). Table 1 shows the statistics of our English/French Euronews-LIUM corpus created from FrenchFootnote 3 and English news data from 2010–2012 period.
Size of the Euronews transcribed English audio corpus and English–French texts
This corpus is composed of a comparable corpus, made of transcriptions (performed with the automatic speech recognition (ASR) system described in Section 6.1) and article content (text found on the webpage). The extracted data obtained with the extraction system described in Section 6 are also provided.
3.2 TED
TED-LIUM corpus has been created within the context of the IWSLT’11 evaluation campaign. It has been built from some video talks crawled on the TED (Technology, Entertainment, Design) web site. The corpus is made of 773 talks representing 118 hours of speech as we can see in Table 2. We used the English audio part of this corpus and the French text part of the WIT3 parallel corpus,Footnote 4 to create the TED multimodal comparable corpus, further called TED-LIUM.
TED English audio corpus statistics
Figure 2 shows an example of such multimodal comparable data.
Example of multimodal comparable data from the TED web site.
This corpus completes the already available TED-LIUM corpus,Footnote 5 with the extracted parallel data.
4 Extracting parallel sentences from multimodal comparable corpora
One of our main goals is to address the situation when the data to translate is from a different domain than the training data. In such a condition, the translation quality is generally rather poor.
In this work, we seek to improve SMT systems in domains that suffer from resource deficiency by automatically extracting bitexts from a comparable corpus which includes audio in one side and text in the other side. We propose an extension of the method described in AbdulRauf and Schwenk (Reference Abdul-Rauf and Schwenk2011). The basic system architecture is described in Figure 3. We can distinguish three steps: ASR, SMT and IR. The ASR system accepts audio data in language L1 and generates an automatic transcription. This transcription is then translated by a baseline SMT system into language L2. Then, we use these translations as queries for an IR system to retrieve the most similar sentences in the text part of our multimodal comparable corpus. The transcribed text in L1 and the IR result in L2 form the final bitext. We hope that the errors made by the ASR and SMT systems will not impact too severely the quality of the IR queries, and that the extracted bitext will improve the SMT system.
Extracting parallel texts from multimodal comparable corpora.
4.1 Task description
4.1.1 Feasibility
This framework raises several issues. Each step in the system can introduce a certain number of errors. It is important to highlight the feasibility of the approach and the impact of each module on the generated data. Thus, we conducted three different types of experiments, described in Figure 4.
Different experiments to analyze the impact of the errors of each module.
In the first experiment (Exp 1), we use the reference translations as queries for the IR system. This is the most favorable condition, it simulates the case where the ASR and the SMT systems do not commit any error. In the second experiment (Exp 2), we use the reference transcription as input to the SMT system. In this case, the errors only come from the SMT system since no ASR is involved. Finally, the third experiment (Exp 3) represents the complete proposed framework, described in Section 4. It corresponds to a real scenario.
4.1.2 Parallelism
Another issue is the importance of the degree of similarity between the two parts of the comparable corpora. In a real life comparable corpus, we can only expect to find matching sentences for a fraction of the sentences. Therefore, we artificially created four comparable corpora with different degrees of similarity. The source part of our comparable corpus is always the TED corpus (see next section). The target language part of the comparable corpus consists of a large generic corpus plus twenty-five per cent, fifty per cent, seventy-five per cent and hundred per cent, respectively of the reference translations of the TED corpus.
For each candidate sentence pair, we need to decide whether the two sentences in the pair are mutual translations. Thus, we classify the IR result with TER (Snover et al. Reference Snover, Dorr, Schwartz, Micciulla and Makhoul2006) calculated between the query, i.e. the automatic translation, and the sentence selected by IR.
4.1.3 Evaluation
In all cases, an evaluation of the approach is necessary. Thus, the final parallel data extracted are re-injected into the baseline system. The various SMT systems are evaluated using the BLEU score (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002). This is the most commonly used metric in the domain of automatic machine translation, but the choice of the best metric is actually still an open research issue.
5 Experimental setup
5.1 Data description
In these experiments, we used TED comparable corpus described in Section 3.2. For MT training, we considered the following corpora among those available: News–Commentary (nc7) and Europarl (eparl7) corpus, the TED corpus provided by IWSLT’11 (TEDbi) and a subset of the French–English Gigaword corpus (ccb2). The Gigaword corpus was filtered with the same techniques described in Rousseau et al. (Reference Rousseau, Bougares, Deléglise, Schwenk and Estève2011). We transcribed all the TED audio data with the ASR system described in Section 5.2 and name it TEDasr. Table 3 summarizes the characteristics of those different corpora. Each corpus is labeled whether it is in- or out-of-domain with respect to our task.
MT training data
The development corpus (dev) consists of nineteen talks and represents a total of 4 hours and 13 minutes of speech. We use the same test data as provided by IWSLT’11 organizers for the speech translation task. dev.outASR and test.outASR are the automatic transcriptions of the development and test corpus respectively. The reference translations are named dev.refSMT and tst.refSMT. Table 3 and 4 summarize the characteristics of the different corpora used in our experiments.
MT development and test data
5.2 ASR system description
Our ASR system is a five-pass system based on the open-source CMU Sphinx toolkit (version 3 and 4), similar to the LIUM’08 French ASR system described in Deléglise et al. (Reference Deléglise, Estève, Meignier and Merlin2009). The acoustic models were trained in the same manner, except that a multi-layer perceptron is added using the Bottle-Neck feature extraction as described in Grézl and Fousek (Reference Grézl and Fousek2008). Table 5 shows performances of ASR system on the dev and test corpora.
Performances of the ASR system on dev and test data (Per cent WER)
5.3 SMT system description
Our system is a phrase-based system (Koehn, Och and Marcu Reference Koehn, Och and Marcu2003) which uses fourteen features functions, namely phrase and lexical translation probabilities in both directions, seven features for the lexicalized distortion model, a word and a phrase penalty and a target language model. It is based on the Moses SMT toolkit (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007) and is constructed as follows. First, word alignments in both directions are calculated. We used the multi-threaded version of the GIZA++ tool (Gao and Vogel Reference Gao and Vogel2008). Phrases and lexical reordering are extracted using the default settings of the Moses toolkit. The parameters of our system were tuned on dev.outASR, using the MERT tool. The language model was trained with the SRI LM toolkit (Stolcke Reference Stolcke2002), on all the French data distributed in IWSLT 2011 evaluation campaign without the TED data. The baseline system is trained with eparl7 and nc7 bitexts.
5.4 IR system
We use the Lemur IR toolkit (Ogilvie and Callan Reference Ogilvie and Callan2001) for the sentence extraction procedure. We first index all French text data into a database using Indri Index. This feature enable us to index our text documents in such a way that using the specialized Indri Query Language, we can use the translated sentences as queries to run TF-IDF retrieval in the database. By using these means, we can retrieve the best matching sentences from the French side of the comparable corpus. The index data consist of the French part of ccb2 (described in Table 3) and different percentage of the French side of TEDbi as described in Section 4.1.1.
5.5 Experimental results
As mentioned in Section 4, the TER score is used as a metric for filtering the result of IR. We only keep the sentences which have a TER score below a certain threshold determined empirically. Thus, in each condition, the retrieved sentences are filtered with a different TER thresholds ranging from 0 to 100. The extracted bitexts are then added to our generic training data in order to adapt the baseline system. Figures 5–8 present the BLEU score obtained for these different experimental conditions.
BLEU score on dev using SMT systems adapted with bitexts extracted from ccb2 + 100 per cent TEDbi index corpus.
BLEU score on dev using SMT systems adapted with bitexts extracted from ccb2 + 75 per cent TEDbi index corpus.
BLEU score on dev using SMT systems adapted with bitexts extracted from ccb2 + 50 per cent TEDbi index corpus.
BLEU score on dev using SMT systems adapted with bitexts extracted from ccb2 + 25 per cent TEDbi index corpus.
In Exp2, we use automatic translations for the IR queries. One can hope that IR itself is not too much affected by the translation errors, but this will be of course the fact for the filtering based on the TER score. AbdulRauf and Schwenk (Reference Abdul-Rauf and Schwenk2011) propose to vary the TER threshold between 0 and 100 and to keep the threshold value that maximizes the BLEU score once the corresponding extracted bitexts were injected into the generic system. We did not observe such a clear maximum in our experiments and the BLEU score increases almost continuously. Nevertheless, in order to limit the impact of noisy sentences, we decided to only keep the sentences with a TER score below the threshold of 80. One can observe that the BLEU score of the adapted system matches the one of Exp1 in most of the cases.
In fact, as we can see in the Figures 5–8, there is no big difference in term of BLEU score between the curves of Exp2 and Exp3 comparing to the one of Exp1. Therefore, we conclude that the errors induced by the SMT system have no major impact on the performance of the parallel sentence extraction algorithm. These findings are in line with those of AbdulRauf and Schwenk (Reference Abdul-Rauf and Schwenk2011).
These results show that the choice of the appropriate TER threshold depends on the type of data. Our baseline SMT system trained with generic bitext only achieves a BLEU score of 22.93. In Exp1, we use the reference translations as query and the IR should in theory find all the sentences in the large corpus with a TER of zero. It can happen that our generic corpus also contains some similar sentences which are ‘accidentally’ retrieved. The four figures show that the IR does indeed work as expected: the observed improvement in the BLEU score does not depend on the TER threshold (with the exception of some noise) since all the sentences have a TER of zero. The achieved improvement depends of course on the amount of TED bitexts that are injected in our comparable corpus: the BLEU score increases from 22.93 to 24.14 when hundred per cent is injected while we only obtain a BLEU score of 23.62 when twenty per cent is injected. These results give us the upper bound that we could expect to get when extracting parallel sentences from this particular multimodal comparable corpus.
Finally, in Exp3, we use ASR on the source side of the comparable corpus. Our ASR system has a word error rate of about eighteen per cent. These errors on the source side can obviously lead to wrong translations and have a negative impact on the IR process. It is important to note that these automatic transcriptions represent the source side of our extracted parallel corpus. By these means, eventual transcription errors should less affect the translation system since it is unlikely that wrong source phrases will be used to translate other texts. We observed in our experiments that the extracted sentences do improve the SMT system. The performance in this stage is only 0.5 BLEU points below those obtained in Exp1 or Exp2.
Table 6 lists the adaptation results of the baseline system in different conditions. It shows that starting with a BLEU score of 23.96 per cent on the test set for the baseline system, adaptation with automatically extracted in-domain bitext resulted in better results in all conditions (between 1.18 in Exp1 and 0.73 BLEU points in Exp3).
BLEU scores on dev and test after adaptation of a baseline system with bitexts extracted in conditions Exp1, Exp2 and Exp3 (hundred per cent TEDbi)
Table 7 provides an analysis of the performance in function of the degree of parallelism of the comparable corpus as explained in Section 4.1.2. Remember that the whole TED corpus (in text version) amounts to about 1.8 M words. We were able to automatically extract about 400 k words of new bitexts, i.e. a little more than twenty per cent. If less data is injected, the amount of extracted data decreases linearly.
BLEU scores for different degrees of parallelism of the comparable corpus. The more the comparable corpus contains in-domain data (TEDbi), the better the results
We argue that this is an encouraging result since we automatically aligned source audio in one language with texts in another language, without the need of human intervention to transcribe and translate the data. The TED corpus contains only 118 hours of speech. There are many domains for which much larger amounts of untranscribed audio in one language and related texts in another language are available, for instance news. However, the quantity of extracted sentences is still insufficient in comparison with the rejected data (by TER filtering).
Figure 9 illustrates the selection process as a binary selection process where the non-selected sentences are simply discarded. In order to make a better use of the rejected sentences, we proposed to use them as unsupervised data. Unsupervised adaptation method (Schwenk Reference Schwenk2008) is described in Figure 10.
Principle of parallel sentence extraction with TER filtering.
Principle of the unsupervised training method (Unsup).
A baseline (generic) SMT is used to translate some sentences, which are then filtered using the translation score (provided by moses decoder in our case). The data is then added to the generic SMT training data in order to train the adapted system.
In our case, all the sentences rejected by the TER filtering are then considered as candidate for unsupervised training. If their translation score is above a certain threshold (determined empirically), then they are kept. The data (called Unsup) is added to the generic SMT training data and a new system is trained. The comparative results are presented in Figure 11.
BLEU score using the unsupervised training method Unsup compared to the baseline system and the proposed extraction method SentExtract.
It shows an analysis of the performance of the Unsup and SentExtract methods measuring by the BLEU score for different TER threshold. We can observe that adding the Unsup data does not provide additional improvement compared to using data extracted with our SentExtract method only. An explanation of this is that the segments are too long for IR to retrieve good parallel ones. As for the Unsup method, the baseline system can not provide good translations for specialized sentences, which result in no performance increase.
This clearly suggest to look at the sub-sentential level in order to make the most of the available in-domain data. This is the purpose of the next section.
6 Extracting parallel phrases
Most of existing studies dealing with comparable corpora look for parallel data at the sentence level (Zhao and Vogel Reference Zhao and Vogel2002; Utiyama and Isahara Reference Utiyama and Isahara2003; Munteanu and Marcu Reference Munteanu and Marcu2005; AbdulRauf and Schwenk Reference Abdul-Rauf and Schwenk2011). However, the degree of parallelism can vary considerably, from noisy parallel texts, to quasi parallel texts (Fung and Cheung Reference Fung and Cheung2004). Corpora from the last category contain none or few good parallel sentence pairs. There could have parallel phrases in comparable sentences that can prove to be helpful for SMT (Munteanu and Marcu Reference Munteanu and Marcu2006). As an example, consider Figure 12, which presents two paragraphs from news articles from the English and French editions of the Euronews web site.Footnote 6 The paragraphs report on the same event with different sentences that contain some parallel translations at the phrase level. These two documents contain in particular no exact sentence pairs, so techniques for extracting parallel sentences will not give good results. We need a method to extract parallel phrases which exist at the sub-sentential level.
Examples of two comparable paragraphs with highlight on bilingual segments.
There has been considerable amount of work on comparable corpora for discovering parallel phrases.
In Munteanu and Marcu (Reference Munteanu and Marcu2006), a first attempt to extract parallel sub-sentential fragments (phrases) from comparable corpora is presented. They used a method based on a log-likelihood-ratio (LLR) lexicon and a smoothing filter. They showed the effectiveness of their method to improve an SMT system from a collection of a comparable sentences. The weakness of their method is that they filter source and target fragments separately, which cannot guarantee that the extracted fragments are a good translations of each other.
Hewavitharana and Vogel (Reference Hewavitharana and Vogel2011) show a good result with their method based on a pairwise correlation calculation which suppose that the source fragment has been detected.
The second type of approach consist in extracting parallel phrases with an alignment-based approach (Quirk, Udupa and Menezes Reference Quirk, Udupa and Menezes2007; Riesa and Marcu Reference Riesa and Marcu2012). These methods are promising, because Cettolo, Federico and Bertoldi (Reference Cettolo, Federico and Bertoldi2010) show that mining for parallel fragments is more effective than mining for parallel sentences, and that comparable in-domain texts can be more valuable than parallel out-of-domain texts. But the proposed method in Quirk et al. (Reference Quirk, Udupa and Menezes2007) do not significantly improve MT performance and the model in Riesa and Marcu (Reference Riesa and Marcu2012) is designed for parallel data. So, it’s hard to say whether this approach is actually effective for comparable data.
We propose a method based on a combination of the translation approach and the alignment one for the extraction of parallel phrases.
6.1 System architecture
The system architecture is described in Figure 13. As in the SentExtract method, we can distinguish three steps: ASR, SMT and IR. The ASR system accepts audio data in the source language L1 and generates an automatic transcription. This transcription is then split into phrases and translated by a baseline SMT system into language L2. Then, we use these translations as queries for an IR system to retrieve the most similar phrases in the texts in L2, which were previously split into phrases as well. The transcribed phrases in L1 and the IR result in L2 form the final parallel data. We expect that the errors made by the ASR and SMT systems will not impact too severely the extraction process.
Principle of the PhrExtract parallel phrase extraction system from multimodal comparable corpora.
We report an extension of the work of the Section 4 by splitting the transcribed sentences and the text parts of the multimodal corpus into phrases with a length (empirically chosen) between two to ten tokens. All combinations of two to ten word sequences are extracted from each sentence of the corpus as we can see in Figure 14.
Example of a sentence which has been split into several sequences of words with variable length (from two to ten words).
This does actually increase drastically the size of the indexed corpus, by adding some kind of redundancy. But, this allows to the IR system to closely match the request.
However, the extracted phrases are of different level of quality, and a filtering step is required in order not to degrade the performance of the baseline system. One of the drawbacks of TER filtering method is that it can remove a large number of phrases, which often results in a lower impact on the baseline system.
In order to resolve this problem, we proposed a new parallel phrase extraction system presented in Figure 15. We begin by extracting comparable sentences with the same method as presented in Section 4 called SentExtract. Then, we apply two steps. First, parallel phrase pair candidates are detected using the IBM1 model (Brown et al. Reference Brown, Pietra, Pietra and Mercer1993). Then, the candidates are filtered with probabilistic translation lexicon (learned on the baseline SMT system training data) to produce parallel phrases using LLR method (see Munteanu and Marcu (Reference Munteanu and Marcu2006) for details). This technique is similar to that of the PhrExtract system, but we bypass the need of the TER filtering by using an LLR lexicon. We call this new extended system PhrExtract_LLR.
Principle of the PhrExtract_LLR parallel data extraction system from multimodal comparable corpora.
6.2 Data description
To train, optimize and test our baseline MT system, we used the data presented in Table 3.
For each comparable corpus (Euronews-LIUM and TED-LIUM), we chose the most appropriate development and test corpus. devEuronews and tstEuronews are the news corpora used in the WMT’10 and WMT’11 evaluation campaigns, respectively.
devTED and tstTED are the official development and test corpora from the IWSLT’11 international evaluation campaign. Their statistics are presented in Table 8.
MT development and test data
6.3 Results
For the sake of comparison, we ran several experiments with the two methods on the sub-sentential level. The first one, is PhrExtract_LLR (presented in Section 4), and the second one corresponds to the method described in the Section 6 (called PhrExtract).
Experiments were conducted on English to French TED and Euronews tasks.
PhrExtract uses TER for filtering the result returned by IR, keeping only the phrases which have a TER score below a certain threshold determined empirically. Thus, we filter the selected sentences in each condition with different TER thresholds ranging from 0 to 100 by steps of 10. The various SMT systems are evaluated using the BLEU score.
Tables 9 and 10 show the statistics of the bitexts extracted from Euronews-LIUM and TED-LIUM. These bitexts are injected into our generic training data in order to adapt the baseline MT system.
Number of words and sentences extracted from TED-LIUMcorpus with PhrExtract and PhrExtract_LLR methods
Number of words and sentences extracted from Euronews-LIUM corpus with PhrExtract and PhrExtract_LLR methods
Tables 11 and 12 present the BLEU scores obtained with the best bitext extracted from each multimodal corpus with PhrExtract and PhrExtract_LLR methods. The TER threshold is set to 50 for Euronews-LIUM and 60 for TED-LIUM.
BLEU scores on devTED and tstTED after adaptation of a baseline system with bitexts extracted from TED-LIUMcorpus
BLEU scores on devEuronews and tstEuronews after adaptation of a baseline system with bitexts extracted from Euronews-LIUM corpus
In the experiment with TED data, we seek to adapt our baseline SMT system to a new domain. We can see in Table 11 that our new system obtains similar results as the PhrExtract method. This means that the extracted texts are useful for adaptation purpose.
The same behavior is observed on Euronews task (Table 12). The extracted text can be used to improve an existing SMT system already trained on the same kind of data.
This new extraction method bypass the use of the TER filtering. This avoid the need of many experiments to determine the best threshold for each task. Moreover, looking at the extracted text sizes in Tables 9 and 10, we can observe that the LLR method generate less data while obtaining equivalent performance. This suggests that only the most relevant data is extracted by this technique.
We can see in the example in Table 13, that adding the extracted phrases can have a positive effect on translation quality.
Example of translation quality improvements of the baseline MT system after adding parallel data extracted from Euronews corpus
7 Conclusion
In this paper, we presented a new system to extract parallel sentences and phrases from a multimodal comparable corpus. We have proposed to extend the exploitation of the comparable corpora to multimodal comparable corpora, i.e. the source side is available as audio and the target side as text. This is achieved by combining a large vocabulary speech recognition system, an SMT system and IR.
We validate the feasibility of our approach by a set of experiments to analyze the impact of the errors committed by each module. Experiments conducted on TED and Euronews data showed that our method significantly outperforms the existing approaches and improves MT performance both in two different situations. The first is domain adaptation; for the TED task, the generated data help a baseline system unadapted to this particular domain to improve its performance. For the Euronews task, additional in-domain data are injected in an already adapted system to the news domain.
Our approach can be improved in several aspects. A parallel corpus is used to generate the LLR lexicon used for filtering. An alternative method could be to construct a large bilingual dictionary from comparable corpora, and use it in the filtering module. In this case, the lexicon would benefit from containing words specific to the targeted task (in the case of adaptation).
