3.1 Weather warnings when severe weather threatens

Weather warnings are issued by Environment Canada when severe weather threatens, allowing its clients to protect themselves and their property. Their corresponding bulletins are broadcast in a variety of formats, including e-mails (e-weather), the Internet (http://weatheroffice.gc.ca), and radio (weatherradio). When the trajectory and strength of a potentially dangerous storm system is known with certainty, a warning is issued, explaining the presence or imminence of severe weather. This information is updated regularly so that members of the public can take the necessary precautions. The Official Languages Act of Canada requires that a warning be emitted simultaneously in both official languages (French and English). Consequently, the translation can cause a delay in the emission of warnings.

Figure 1 shows an example of a strong wind warning issued on 22 November 2011, as it appeared on Environment Canada's website. A short header identifies the affected area, followed by a discussion explaining the threat. A French version of every warning is available in the same format. That version is accessed by clicking on the ‘Français’ link in the upper left corner of the page. There are different kinds of warnings, depending on the threat reported. For instance, WU bulletins describe severe thunderstorms, WF bulletins warn about the formation of tornadoes and WW bulletins are ‘omnibus’ bulletins most often compiling warnings currently in effect.

Fig. 1.

A sample weather warning as it appeared on Environment Canada's website in November 2011.

3.2 Lifecycle of warnings

The broadcast of the warning shown in Figure 1 is in fact one of the end results of an intricate dissemination system. First, a warning is composed by a meteorologist based on the available weather data for a specific region. The warning is issued from one of the numerous emitting stations spread over Canada, and is written in English, except when it originates from the French-speaking province of Quebec. The warning is then split in two components. One part contains meta-information describing the affected area, the event identification etc., and is automatically translated using simple rules. The other part contains the text of the discussion, where the meteorologist explains the weather condition. This last file, whose format will henceforth be called MTCN, is illustrated by the example in Figure 2 for the same bulletin displayed in Figure 1. For historical and logistical reasons, the discussion is in capital letters and uses the ASCII character set, precluding the use of accented characters. Apostrophes are also often missing in MTCNs.

Fig. 2.

The MTCN bulletin corresponding to the warning seen in Figure 1.

The original MTCN is then sent to the Canadian translation bureau, where professionals produce a translation of the text. At present, they are already assisted in their work by an MT module, which we will call ECMT in this paper. They post-edit ECMT's output and return a translation in MTCN format. The average turnaround time is 3.5 minutes for a WU bulletin, and 6.5 minutes for a (usually longer) weather warning bulletin.

The two components of the bulletin are then reassembled, for each language, to form two complete bulletins: the original and the translation. Before being sent to Environment Canada's website, the case and the diacritical marks of each bulletin are restored by an automatic process. In this paper, we will employ ‘truecasing’ to mean the process of restoring case and ‘accenting’ the process of adding missing diacritical marks.

3.3 Improvements to the current system

The system currently used by Environment Canada suffers from certain shortcomings, some of which are addressed in this paper.

The delay the translation process imposes on the broadcast of warnings becomes problematic during what meteorologists call ‘short fuse convective situations', weather conditions that typically cause the emission of a large number of bulletins in a short period. The problem is threefold: (1) the volume of the discussion text increases, (2) the frequency of bulletins grows and (3) the number of high-priority translations increases as well. A translation whose priority is high may concern the imminent formation of a tornado, for instance. These problems overwhelm the translation pipeline, and result in a few bulletins without translations, or whose discussion is missing altogether. The need for swift translations is therefore particularly acute.

Another difficulty lies in the complexity of the current pipeline. Part of the problem is the fact that ECMT's output has to go through a separate case- and diacritic-restoration process further down the pipeline. This creates two distinct processing steps: one for the machine translation, and the other for truecasing and accenting, and therefore as many failure points. Furthermore, since the truecasing and accenting steps are not revised by the Translation Bureau (contrarily to the translation), typographical errors are commonly found in warnings published on Environment Canada's website. One could instead think of a single translation module whose output would be proper French or English. The translation professionals would then be post-editing not only the machine translation of the system but also the post-processing steps, ensuring a closer revision.

Finally, discussions with Environment Canada's executives suggested that the update and maintenance of ECMT is difficult in some regards, and that they would welcome a new way of incorporating feedback from meteorologists and translation professionals into the translation system.

3.4 WATT translation system

In light of these needs, the RALI started working in 2009 on an MT system specifically aimed at translating weather warning bulletins issued by Environment Canada. We call this system Watt. Its design is described in this paper. Watt is a fully automated MT component, combining a fuzzy translation memory and an SMT engine. It integrates a truecasing component for English and French and, when it outputs French text, an accenting step as well.

Watt's architecture is shown in Figure 3, along with a translation example. This paper explains in detail the system's inner workings. We outline them here from the outset.

1. (1) A source bulletin in MTCN format is submitted to the system. Word and sentence segmentation are performed on its discussion.

2. (2) A light rewriting module reformulates and corrects some elements of the input, for instance the format of dates in English.

3. (3) Serialization is performed, where all numbers and time expressions are folded into their respective special token, for instance __TIME__ for time expressions.

4. (4) A sentence-based translation memory made of past translations is queried with the source text. If it is found, the corresponding translation is retrieved; otherwise, an SMT system produces a translation.

5. (5) If the output text is in French, its diacritics are restored.

6. (6) For all bulletins, deserialization converts the specialized serialized tokens to their corresponding target language version.

7. (7) The case of mixed cased words is restored.

8. (8) Watt applies a few cosmetic rules to the output text, e.g. the restoration of apostrophes in French text.

In the end, a bulletin in ‘proper’ English or French is produced, conforming to the text format typically seen on Environment Canada's public warning website. If the client needs a file in MTCN format however, it can be trivially derived from Watt's result.

Fig. 3.

Watt's architecture, with a translation example from English to French, adapted from the example in Figure 2. Changes between each step are highlighted in the example boxes.

Watt's design started in 2009 within the Multi-Format Environmental Information Dissemination project outlined earlier. Preliminary discussions with Environment Canada allowed RALI to identify the needs of the government and to propose the principles of the solution presented in this paper. Afterward, we started gathering and preparing the corpora, which would be necessary to build Watt's first prototype. This conceptually simple task proved exceedingly complex and required 80 percent of man hours devoted to the project. The reasons for this are explained in the following section. Suffice it to say that putting together enough textual material for our needs proved challenging because of cryptic file formats and the state of some of the warning archives we had to work with. We then trained a few variants of an SMT engine with this data, and populated a sentence-based translation memory. Five prototypes were successively submitted to our client and two of those were formally tested to validate our design and their performance.

4.1 Tokenization, serialization and ‘deserialization’

To create a bitext out of the material at our disposal, we wrote a sentence and word segmentation program specifically for the task, accounting for the format explained above. Particular care was taken to strip pseudo-formatting marks (e.g. horizontal rules, hard returns) from the source text, and to account for the presence of domain-specific abbreviations, urls and e-mail addresses.

We reduced the vocabulary of the texts through serialization, a form of hard clustering which folds a number of expressions into the same special token. For instance, all times in both French and English are replaced by __TIME__. We serialized times, months, days of the week and all numbers. For instance, the second sentence in Figure 4 is serialized as MONTREAL AT __TIME__ PM EST __DAY__ __NUM__ __MONTH__ __NUM__ FOR __DAY__ __NUM__ __MONTH__ __NUM__. The same serialization and tokenization process is applied to source phrases during the translation process.

Because we work with serialized text, the reverse procedure must be applied when producing a translation, i.e. the serialized tokens obtained must be replaced by the translation of their source language counterpart. For the above example if ‘MONTREAL AT __TIME__’ is translated into the French phrase ‘MONTREAL À __TIME__’, then the source text ‘4.30 PM’ for which __TIME__ stands must be translated and substituted to __TIME__ in the French version, yielding the completed translation ‘MONTREAL A 16H30’. This makes the important assumption that the count of each serialization token is the same in the source sentence and in the translation produced (but see Section 4.4.3).

When sentence alignment was not already provided in the source material, we used the Japa sentence aligner (Langlais Reference Langlais1997). If there was more than one sentence which could not find a corresponding sentence (n-0 alignment), the enclosing document pair was discarded. This proved useful in weeding out pairs of documents which were erroneously deemed a translation of one another.

4.2 Forecast text (2002–2003)

For forecast texts, we used the bitext created by Langlais et al. (Reference Langlais, Gandrabur, Leplus and Lapalme2005) that covers the years 2002 and 2003.Footnote ³ The authors' preparation of the corpus is very similar to the protocol we used here, so importing their data into our corpus was quite easy. Moreover, they chose XML as the file format, which further simplified our task. We wrote tools to make their tokenization identical to ours, and serialized the corpus.

The content of this corpus exhibits some differences from the corpus of weather warnings: its sentences are shorter and its vocabulary is simpler. Moreover, the style is more telegraphic than that of warning discussions. Nonetheless, these forecasts represent a large amount of material (87 k bulletins) due to the high frequency of their emission, so we felt their use was warranted here as preliminary tests showed their usefulness.

4.3 Weather warning text (2000–2011)

Weather warning discussions (see Figure 2) are the main interest of this study. It was therefore important to gather as many of these as possible. Thankfully, we were able to extract more than a decade's worth of text from three sources described below.

4.4 Additional hurdles when preparing the corpora

Aside from the technical problems described in the previous section, a few additional difficulties were encountered when preparing the bitext used in this study.

4.5 Bitext partition and statistics

Table 1 shows the breakdown of all the textual data we gathered into the four sources we described in Section 4.2 (forecasts) and Section 4.3 (weather warnings). The statistics are for the tokenized and serialized texts. We collected 205 k pairs of bilingual documents, corresponding to 4.9 M sentence pairs. The forecasts accounted for 83 percent of sentence pairs, but only 55 percent of the unique sentence pairs were found in the corpus, which shows their level of repetitiveness.

We used the bitext described in Table 1 to create three non-overlapping subcorpora: train, tune and test. The statistics for this partition are shown in Table 2. The training corpus is used to train the SMT engine; its decoding parameters are tuned with the tune corpus, and the SMT is tested on the test slice.

Table 2.

Main statistics for the train, tune and test subcorpora used in this study

The train, tune and test bitexts were designed to reflect as closely as possible the situation the translation engine would be confronted with, i.e. the translation of weather warning discussions. Therefore, the tune and test corpora consist exclusively of sentence pairs drawn from weather warnings. The training corpus is the only subcorpus containing text from forecasts, in addition to weather warning text.

Moreover, to avoid any overlap between, on the one hand, the training corpus, and on the other hand, the tune and test corpora, the training corpus excludes all warnings issued during the winter of 2010 and the summer/fall of 2011. From the warnings issued during these periods were drawn 400 warnings for tuning and 400 others for testing. It is noteworthy that the time periods covered by the test and tune corpora include the summer months during which most of the weather warnings are issued.

We chose to make these two corpora as ‘hard’ to translate as possible by picking warnings that contained at least one word or sentence unknown to the training corpus. We did this for two reasons. First, we wanted our test and tune corpora to reflect the appearance of new words and phrasings over time. Second, the SMT engine is destined to be augmented by a translation memory (Section 4.3), which would take care of translating previously encountered sentences. This bias that we introduced in favor of unseen words and sentences can be indirectly observed through the ratio of unique French sentences over unique English sentences in Table 2. While in the training corpus this ratio is 558,996/523,305 ≈ 1.067, it falls to approximately 1.0 for the test and tune corpora. In other words, the training corpus contains numerous English sentences for which there is more than one possible French equivalent, whereas this is significantly less important in the train and test corpora. An example of this one-to-many mapping is shown in Table 3.

Table 3.

Example of an English sentence associated with multiple French equivalents

The corpus's sentences are quite short: 8.4 tokens on average for English and 10.2 tokens for French. On average, a weather warning bulletin contains a discussion that is 5.8 sentences long. On average, each sentence is found for 9.1 times in the corpus, a figure that shows the highly repetitive nature of this type of text.

The vocabulary is very limited and consists for the training corpus of a mere 14.5 k types for English and 18.7 k types for the morphologically richer French version. As mentioned in Section 4.4.1, a small but non-negligible percentage of those types are typographical errors. The out-of-vocabulary tokens account for 0.2 percent of the French test corpus and 0.3 percent of the English test corpus, a very low figure.

The complete corpus, consisting of all of the forecasts and warnings at our disposal, was used to retrain the MT engine before its deployment in a production environment. This corpus (last line in Table 1) is not merely the concatenation of the train, tune and test corpora: it also includes all warnings from the winter of 2010 and the summer/fall of 2011, which were not among the 800 warnings selected for the tune and test corpora.

5.1 Variants tested

We experimented with different parameters of the translation pipeline for both translation directions. For all the variants we always filter the phrase table first using the procedure described above. These experiments were prompted by various problems we observed in the quality of preliminary translations.

5.1.2 Phrase filtering

Another apparent source of errors in translations was word misalignments, which in turn caused the presence of spurious phrase pairs in the phrase-based model used by Moses. One such phrase pair erroneously associated ‘BALAIE’ (French for ‘sweeps’) with the translation ‘CHIBOUGAMAU TOWARD LAKE ERIE WILL SWEEP’ and caused the two locations (Chibougamau and Lake Erie) to appear out of thin air in a translation. Although its associated scores are very low, the pair's target still showed up in our preliminary tests. This prompted us to attempt to sanitize the phrase-based model by filtering out all phrases for which |l _src - l _trg| > threshold, where l _src is the number of words in the source phrase, and l _trg is the number of words in the target phrase. We tried two thresholds: 4 (configuration labeled maxdelta4) and 5 (maxdelta5). More aggressive filters showed poor preliminary results. An additional configuration did no filtering (configuration labeled integral). More sophisticated filtering techniques exist (see, for instance Johnson et al. Reference Johnson, Martin, Foster and Kuhn2007) and have shown to significantly shrink the phrase-based model while leaving the translation quality unharmed, or even improved. In our case, we opt for a simpler approach, in part because our sole focus is identifying and weeding out nonsensical associations, rather than reducing the size of the translation model. However, further research is necessary to determine the impact of table pruning or smoothing (see Foster et al. Reference Foster, Kuhn and Johnson2006) on translation quality.

When translating from English, the integral phrase-based model contains 7.05 M pairs; configuration maxdelta5 subtracts 30 k (0.4 percent) pairs and maxdelta4, 143 k pairs (2 percent). These figures are almost identical for the other translation direction. A matrix showing the distribution of phrases based on l _src and l _trg is shown in Table 4, with French as the source language. Most of the pairs are concentrated along the diagonal, which is to be expected. The maxdelta configurations trim off the phrases in the upper right and lower left triangles of the matrix, which we suspect are spurious.

Table 4.

Phrase distribution matrix (in percentage) based on source and target length when the source language is French. The total number of phrases = 7.05 M. The underlined cells show the phrase pairs filtered out by the maxdelta4 configuration

5.2 Results for the SMT engine

Three configurations were tested for phrase reordering and three others for filtering for a total of nine different combinations of configurations. This was done for both translation directions. We greatly benefited from the multiprocessor support that the Moses toolkit offers, and were able to take full advantage of a 24-core machine. Training took 5 hours on average, and tuning needed ten MERT iterations on average, for an average time of 24 hours.

MERT tuning relies on a pseudo-random component and often finds different local minima in the parameter space for the same tuning corpus. Although the tuning scores converge for a given configuration, the test scores are different between MERT tuning results. This phenomenon is illustrated in Figure 5, showing five different tuning runs, for the configuration French-to-English, integral, msd. While tuning does converge nicely to reach an average of BLEU = 74.7 percent, the test results (legend of Figure 5) vary more significantly, with a mean BLEU score of 76.9 percent (standard deviation of 0.4 percent). Therefore, the standard deviation (in the form of an error on the measure) must be factored in when comparing configurations.

Fig. 5.

MERT tuning results for five different runs for the configuration French-to-English, integral, msd. The progression of the tune score is shown with each iteration. While the tuning converges to an average of BLEU = 74.7 percent, the BLEU test results shown in parentheses vary more significantly, average BLEU = 76.9 percent, stdev = 0.4 percent. The number of MERT iterations needed here varies between six and seventeen.

To compensate for optimizer instability when comparing experimental results, we took to heart some of the suggestions made by Clark et al. (Reference Clark, Dyer, Lavie and Smith2011) and ran each MERT optimization five times in order to obtain a mean and standard deviation on the final BLEU test results. The comparison between configurations can then be made on the basis of the BLEU score but also according to potential error represented by the standard deviation. The test results measured with BLEU are shown in Figure 6 when English is the source language and in Figure 7 for French.

Fig. 6.

Average BLEU results (n= 5) on the test corpus for nine different configurations of the Moses pipeline when translating from English. The error bars correspond to the standard deviation.

Fig. 7.

Average BLEU results (n= 5) on the test corpus for nine different configurations of the Moses pipeline when translating from French. The error bars correspond to the standard deviation.

For the English-to-French translation direction (Figure 6), the BLEU score hovers around 78.5 when not using the maxdelta4 filtering, and 75 when using it. Clearly, the latter is too heavy-handed a technique and should be discarded. That being said, a BLEU score of 78.5 is very high for an MT task. Our results are comparable to those reported by Langlais et al. (Reference Langlais, Gandrabur, Leplus and Lapalme2005) for a similar task, where they obtained a BLEU score of 77 percent when translating severe storm warnings from English to French. On a more diverse warning corpus, however, this figure dropped to 58 percent in their study.

When French is the source language (Figure 7), the BLEU score is slightly lower, which is a bit surprising given the fact that the target language, i.e. English, is morphologically poorer. The BLEU score now peaks at 77.2 percent when discarding the clearly unusable maxdelta4 configurations. This time, even when taking the error into account, all configurations are not equal. The msd-integral and msd-maxdelta5 settings improve the BLEU score by an average of 0.2 percent.

As a result, when translating from French, we will favor the msd-maxdelta5 configuration. This means that we will indeed be using a lexicalized reordering model. We prefer using the maxdelta5 configuration because the phrase model is slightly smaller as a result of the filtering phase. Moreover, it has the added merit of removing some dubious phrase associations that crept into our preliminary translations.

When translating from English, since all configurations excluding maxdelta4 perform equally well, we opt for the same configuration as the reverse translation direction, i.e. the msd-maxdelta5. Having the same configuration for both translation directions simplifies Watt's implementation without harming its performance. Table 5 shows the complete tuning and test results for this configuration. Out of the five configurations that the tuning process created using our methodology, we picked the one with the best BLEU score to produce the test results shown in Table 5. The word error rate (WER) and sentence error rate (SER) are also reported. A good translation maximizes BLEU and minimizes WER and SER scores. At around 15 percent, the WER figures are especially encouraging, although this relative success does not clearly carry over to the SER. The test BLEU scores resemble those obtained in tuning, and are quite acceptable.

Table 5.

Complete tuning and test results for the configuration selected, msd-maxdelta5. All measures are in percentages

It is noteworthy that for many of our experiment configurations the BLEU test score is higher than the tune score (as seen, for instance, in Table 5 when translating from English). This is unexpected, and we first thought our tune and test corpora were not of equivalent difficulty. However, after switching the test and tune corpora one for another, we observed the same tendencies.

For the sake of comparison, we submitted the same test corpus to Google Translate™. When translating from English, their engine produced a translation with WER = 43.1 percent, SER = 98.2 percent, and BLEU = 45.3 percent. When translating from French, the evaluation yields WER = 44.9 percent, SER = 99 percent and BLEU = 44.3 percent. Compared with Watt, this significantly lower translation quality is to be expected from a generic translation engine not tuned to the specificity of meteorological text.

5.3 Adding a translation memory

Given the fact that even with high BLEU scores, the SER remained quite high (see Table 5), we opted to augment the translation system with a translation memory of previous weather warnings (see Figure 3 for its place in the pipeline). Before a source sentence is submitted to the SMT, it is first looked up in the memory. If the sentence is found (memory hit), then the corresponding human-crafted text is the output. Otherwise (memory miss), the SMT is used. It is noteworthy that the memory is populated with tokenized and serialized sentences. This allows a certain degree of flexibility when matching a source sentence with the memory's content: e.g. both original sentences ‘HIGH NEAR 7.’ and ‘HIGH NEAR -10.’ match the memory's ‘HIGH NEAR __NUM__ .’.

For the purpose of this study, we populated the translation memory with all the weather warnings from 2009 to 2011 drawn from our training corpus (see Table 1). On top of these warnings, we added from the weather forecasts all the sentence pairs whose frequency was more than 5 in the complete corpus, which amounted to 30 k pairs of sentences. This minimum frequency is sensible, but admittedly arbitrary. When more than one translation exists for the same source sentence, only the most frequent is kept. The resulting memory contains 118 k unique sentence pairs. Some minor manual interventions were required to make sure that the most frequent pairs corresponded perfectly to the wording that meteorologists have been using recently so as to avoid producing valid, but ‘outdated’ sentences.

The obvious advantage of this strategy is the production of text originally written by humans, and, in our case, revised professionally. However, the memory's utility is dependent on the quality of the material it stores. We did find two sources of errors in our memory: sentence misalignments and mistakes made when the text was originally prepared. We randomly sampled one hundred sentences from the translation memory and found that 3 percent contained man-made mistakes. Sentence alignment errors were dealt with early on in the project, and none were found in this sample. The percentage of errors rises to 7 percent when considering only sentence pairs that appeared only once in the warning corpus. Table 6 shows examples of such errors, mostly minor typos. Because these singletons represent 74 percent of the memory, eliminating them would be problematic.

Table 6.

Examples of errors found in the translation memory. The erroneous string is underlined

To assess the performance of the translation memory, we compared the BLEU score of the SMT engine alone (SMT), the SMT engine augmented with the translation memory (SMT + MEM) and the SMT with the translation memory stripped of sentence pairs appearing only once (SMT + MEM_small). The complete memory contains 290 k sentence pairs and the smaller memory contains 95 k pairs. This last configuration was created to mitigate the risk of producing sentences containing typos, as mentioned above.

We used the Moses configuration recommended in Section 5.2. The test corpus is slightly different from that of Section 5.1, however. That corpus was created by gathering sentences from winter 2010 and summer/fall 2011 which were not part of the training corpus. Therefore, using it as a test corpus here would strongly bias our results against the memory, which, by definition, could not have seen these sentences. We rather chose to create a more representative corpus: We randomly drew the test sentences from winter 2010 and summer/fall 2011, but this time without systematically excluding the sentences not encountered in the training corpus. This test corpus, called here test_random, is similar in size compared with the test corpus from Section 5.1.

The results are shown in Table 7 along with the hit ratio, i.e. the proportion of source sentences found in the translation memory, which therefore did not need to be machine-translated. The results are remarkably similar for both translation directions. The scores obtained when the SMT is augmented with a memory are 0.4 percent higher overall in BLEU, and therefore confirm our working hypothesis. There is little difference between the scores of SMT + MEM and SMT + MEM_small, probably because the sentence pairs seen only once do not clearly contribute to the translation of a random sample of sentences. Nonetheless, the hit ratio is improved by 0.6 percent when using the full memory, and the SER benefits also. For this reason, it seems reasonable to adopt the SMT + MEM strategy.

Table 7.

Test BLEU scores (in percentage) obtained with the baseline (SMT) and with two variants of the translation memory for both translation directions. The best performances for each score are in boldface

The hit ratio at 45 percent on average proves the repetitiveness of the discussion sentences. It also indicates that a substantial gain in translation time is possible when relying on a memory. Based on the observation that the memory lookup time is almost instantaneous, we could expect a speedup similar to the hit ratio, which would amount to halving the translation time in our case. The overhead of loading the memory in RAM is negligible.

That being said, these results must be put into perspective. First, since both memory and test corpus contain typographical errors, it is possible that our protocol overestimates the performance of the memory. When translating, drawing from the memory may produce typographical errors that are also contained in the test corpus. Had the test corpus been manually revised to remove those errors, it would have been a more accurate evaluation tool, and the memory would have fared a little worse when evaluated.

Second, this evaluation assumes that the memory is populated only once and does not evolve over time. However, in a production environment, this would be a very unfortunate design. More realistically, the memory will receive a feedback from the end result of the translation pipeline in the form of new sentence pairs, revised by a human. Therefore, its performance when confronted with repeated occurrences of the same source sentence should improve.

5.4 Error analysis

The BLEU metric primarily used in this study to assess the translation quality is imperfect. It most notably penalizes translations that are not identical to the reference, even if the candidate conveys the same meaning. To get a clearer idea of the quality of the translations, we examined 100 translations produced by the SMT engine coupled with a translation memory. The 100 sentences were randomly selected from the corpus test_random described in the previous section. Each translation was manually compared to the reference. We checked whether the sentence was grammatically irreproachable and also if no semantic elements were lost in translation.

When translating from French, 8 percent of sentences had translation problems, 15 percent had grammatical issues and 16 percent had either one or the other. For English, these figures were 10 percent, 18 percent and 24 percent respectively. Most of the time, when a translation candidate was not a word-for-word match of the reference, it was because of differences in wording which did not alter the meaning of the original sentence. Table 8 shows examples of errors of varying types found when translating from French.

Table 8.

Examples of mismatches between Watt's output and the reference translation, illustrating different types of errors. Grammatical errors are underlined

The first example in Table 8 shows that a different wording may be produced and still conveys the same meaning as the reference. This is penalized by BLEU, but in this case we see that this is too harsh an evaluation. The other examples do show that Watt produces gibberish when confronted with very unfamiliar source sequences, which is to be expected. For instance, while the source sentence in the third example does not contain out-of-vocabulary French words, its wording is unusual. It reads ‘IL N Y A EU QUE __NUM__ MM A ST. JOHN S EN PARTIE EN RAISON DE L AIR SEC QUI S EST ENGOUFFRE DANS LE CENTRE DE L OURAGAN ALORS QU ILEFFECTUAIT SA TRANSITION EN TEMPETE POST-TROPICALE .’. The phrase ‘IL N Y A EU QUE’ (translation: a mere) is encountered only twice in the training corpus, and the words ‘ENGOUFFRE’ (translation: engulfed) and ‘EFFECTUAIT’ (translation: made) have a frequency of 2 in the training corpus.

The fourth example shows the effect of an unexpected error in the source sentence. The latter reads ‘ILS DEVRAIENT TRAVERSER LE SECTEUR DE LA MINERVE - RIVIERE ROUNGE DANS LA PROCHAINE HEURE .’, which contains an error in the location name ‘RIVIERE ROUNGE’, namely an extra ‘N’. Errors in the source confound the translation engine, and are therefore a non-negligible source of erroneous translations (see Section 4.4.1 for a description of the source material's quality).

We were very harsh during our evaluation to provide a lower bound on quality. This can prove useful when using such a subjective evaluation protocol. Further evaluations, employing more rigorous methods, are described in Section 8. At this stage, the errors we detected show no discernible recurrent pattern, and are within the bounds of the limitations expected from typical SMT output. When producing French, we did observe however some rare but systematic errors in agreement between distant words. We are still working on possible solutions.

Ultimately, with a sentence error rate of 16 percent when translating from French and 24 percent from English (at least in our random sample), the translation of a complete bulletin containing on average 5.8 sentences (Section 4.5) will contain on average about one error. This precludes the use of Watt as a fully automated translation system. Human intervention is required to post-edit the translations and make sure that the complete pipeline is up to the task.

6.1 Accenting

Accenting was framed as a disambiguation problem. More precisely, given the raw stream of French tokens, our goal is to produce a stream of accented tokens according to a table, mapping each source token to one or more target tokens. When an ambiguity arises in producing a target token, a language model trained on accented text is used to eliminate the ambiguity by selecting the more probable target production. As with the translation engine, this is a corpus-driven approach, and therefore an accented text is needed to build (1) the mapping of raw tokens to accented ones, which we call disambiguation map and (2) a language model of accented text. Our two data sources are listed below.

reaccoutput on weather forecasts RALI had already developed in the past a generic accenting program called reacc (Simard and Deslauriers Reference Simard and Deslauriers2001) that we used to restore diacritics on all French weather forecasts described in Section 4.2 (see also Table 1 for statistics). reacc's output provided us with a rather large accented corpus to feed our accenting models.

MPCN weather warnings Since 2009, RALI automatically receives a copy of the most accented and truecased weather warnings produced by the current system deployed at Environment Canada. This way we were able to extract the text of 40 k French bulletins corresponding to 3.6 M words.

Both these corpora were used to create a language model using the SRILM toolkit (Stolcke Reference Stolcke2002). We were forced to clean up some systematic errors in the corpus beforehand with a set of handcrafted rules based on our observations of preliminary translations. We then created an interpolated 5-gram language model, trained using Chen and Goodman's (Reference Chen and Goodman1999) modified Kneser–Ney discounting.

The disambiguation map between raw French words and accented tokens was derived trivially from the same corpus, but was found to be lacking in terms of coverage. We therefore completed the map with a comprehensive vocabulary of French words and their flexions using an in-house resource.

The disambiguation map contains 4,139 entries. We use it with SRILM's disambig utility to perform accenting.

6.2 Truecasing

6.2.1 Principle

An example of the end result of the truecasing step is shown in Figure 1, whose text is reproduced in Figure 8 for convenience, in which the tokens with a mixed case are underlined. On the one hand, we observe that case restoration involves conceptually simple rule-based transformations, such as the capitalization of the first word of a sentence or that of a weekday or month. We readily implemented these rules. On the other hand, truecasing the numerous names of places and other named entities is trickier. It actually consists of two problems: performing named entity recognition (NER), and then properly truecasing these named entities. Named entity recognition is made difficult by the lack of case information.

Fig. 8.

An excerpt of a discussion as it appeared on Environment Canada's public weather warning website. We underlined mixed case phrases.

For these reasons, and for the sake of simplicity, we chose a gazetteer-based approach to truecasing. We started by compiling a list as exhaustive as possible of all mixed case expressions susceptible to appear in Watt's translations. We will describe this preliminary step in detail in Section 6.2.2.

We compiled two lists, one for each output language, and made sure these lists spelled each place name with its correct case. We added to these lists other capitalized words (e.g. ‘Environment Canada’) extracted from sources described in Section 6.2.2. An excerpt of the English entries is shown in Table 9.

Table 9.

An excerpt of the list of mixed case expressions in English

With these lists at our disposal, it is then possible to scan the raw translations in order to find these expressions in their raw form. Whenever a match is found, the raw translation is replaced with the truecased expression. Each list is compiled into a trie to speed up the matching step. We use a greedy algorithm that scans the raw translation from left to right to find the longest matching list entry. For instance, the raw ‘AN AREA NEAR METSO A CHOOT INDIAN RESERVE 23’ will only match ‘Metso A Choot Indian Reserve 23’ entry given in Table 9, and not shorter expressions.

It is noteworthy that our lists hold entries that contain diacritical marks (e.g. ‘Metro Montréal’ in Table 9). This allows a few final additions to the accenting step of French text described earlier (see Section 6.1). As for the English text, this single step is sufficient to restore the few diacritical marks necessary and no further processing was deemed necessary.

7.1 A few preprocessing and post-processing touches

A small number of ad hoc rules were necessary to finalize Watt's pipeline because these rules were not easily captured by the statistical models in place. More than fifty rewriting rules were created to rephrase input sentences in some cases, and were derived when examining Watt's output for errors. Among these rewriting rules are expansions of province abbreviations (‘P.E.I.’ → ‘PRINCE EDWARD ISLAND’) and normalization of dates (‘JAN 1ST’ → ‘1 JANUARY’).

Some of these rewriting rules concern hail sizes. Hail sizes are often described using similes, e.g. ‘QUARTER SIZED HAIL’ for hail whose diameter is 24 millimeters (1 inch). Traditional hail size description charts exist, as documented for instance by the American National Oceanic and Atmospheric Administration (NOAA).Footnote ⁸ We chose to convert these (French and English) similes to millimeters. This simplification of the input sentences allows for the subsequent serialization process to kick in and convert hail sizes to the corresponding ‘__NUM__’ special tokens. Once converted, these sentences are more likely to be found in the translation memory or, failing that, to be faithfully translated and deserialized. This rewriting has the added benefit of removing some awkward similes that Environment Canada itself has recently been trying to avoid in the warnings to make the text more accessible. Creating these conversion rules is an ongoing process.

At the other end of the pipeline, after deserialization and truecasing (see Figure 3), a few beautification rules are applied to the translations. These include most notably the restoration of apostrophes (e.g. French ‘l arrivée’ becomes ‘l'arrivée’ – ‘the onset’), but also some fixes, such as the introduction of dashes in some places.

The end result is a bulletin in ‘proper’ English or French, ready to be revised before publication on any of Environment Canada's media outlet, including its website.

7.2 Technical details

Watt's complete pipeline is shown in Figure 3. All the modules described in the previous sections are called into action from a single Python program, which makes sure that all the different technologies involved work together. They include C++ programs (like the Moses decoder and SRILM utilities) and a few shell scripts. The complete code base of the project, including the Python modules written to prepare our training corpora and the translation script mentioned earlier, weighs in at 10,000 lines of code.

Watt runs on a 64-bit architecture machine using Linux, and requires relatively few resources: 2 Gb of RAM and 20 Gb of disk space to store the different statistical models involved. On a 3-GHz core, translating a single sentence takes 4.5 seconds on average, with Moses' translation accounting for a third of the translation time, on average. Starting the translation server involves loading in RAM the various statistical models and tools, and takes 10 seconds on average. Its design is relatively simple, and allows for simple maintenance and updates. Most of the resource files are in plain text, and can be modified readily. The statistical models however require retraining should the need arise to update them significantly.

We implemented Watt as two translation servers, one for each source language. Each server uses a simple ad hoc communication protocol. This proved very useful in providing a centralized system, capable of translating requests originating from multiple points. Translations can be sent to Watt from a demonstration websiteFootnote ⁹ as well as from a command-line tool (for batch processing). Moreover, the latest prototype available at any time has been translating actual weather warnings in a staging environment, which is almost identical to the one currently used by Environment Canada. Studying the translation logs from these sources was instrumental in fine-tuning the numerous heuristics and rules embedded into Watt, and in adapting them to the evolving turns of phrase and vocabularies employed by meteorologists.