2. Related work

For any individual NLP system, such as an MT system, evaluation can be approached from various points of view with varied implications for the conduct of any evaluation and the lessons to be drawn from it (Jones and Galliers Reference Jones and Galliers1995). While the quality of an MT system can simply be characterized by the errors it makes, MT quality can have different interpretations depending on the purpose of the assessment. In literature, we can see various studies, in which the quality of MT output is assessed for its adequacy and fluency (White Reference White1995; Toury Reference Toury2000; Graham et al. Reference Graham, Baldwin, Moffat and Zobel2014; Bojar et al. Reference Bojar2016, Reference Bojar2017). Drawing a distinction between the different properties of machine-translated text has proved fundamental to error analysis. In a bid to study the strengths and weaknesses of MT systems in detail, fine-grained error analysis has generated interest and different taxonomies have been proposed to identify MT errors (Vilar et al. Reference Vilar, Xu, D’haro and Ney2006; Daems, Macken, and Vandepitte Reference Daems, Macken and Vandepitte2014; Lommel et al. Reference Lommel, Uszkoreit and Burchardt2014; Costa et al. Reference Costa, Ling, Luıs, Correia and Coheur2015; Tezcan et al. Reference Tezcan, Hoste, Macken, Pastor and Durán-Mu˜ñoz2017b). With the recent improvements in the field of MT, post-editing, which can simply be defined as correcting errors in the MT output, has become a common practice in translation workflows. Given that MT quality is not uniform and certain machine-translated segments may contain more errors than others, the quality of MT systems has been increasingly measured in terms of the overall post-editing effort that is required to correct machine-translated text (Blatz et al. Reference Blatz2004; Specia et al. Reference Specia, Turchi, Cancedda, Dymetman and Cristianini2009; Koponen et al. Reference Koponen, Aziz, Ramos and Specia2012; Bojar et al. Reference Bojar2016, Reference Bojar2017).

QE is the task of providing a quality indicator for machine-translated text without relying on reference translations (Gandrabur and Foster Reference Gandrabur and Foster2003). Like the different interpretations of MT quality, QE approaches also vary. Early work on QE for MT focused on estimating the quality at the word level with the aim of suggesting translations of the source text within interactive MT scenarios (Ueffing and Ney Reference Ueffing and Ney2005). Word-level QE has regained attention in the recent years thanks to its ability to identify words that require post-editing (Bojar et al. Reference Bojar2015, Reference Bojar2016). Sentence-level QE, on the other hand, aims to estimate an overall quality score, which has often been measured in terms of human quality judgments or post-editing effort (Specia et al. Reference Specia, Turchi, Cancedda, Dymetman and Cristianini2009, Reference Specia, Shah, De Souza, Cohn and Kessler2013; Bojar et al. Reference Bojar2015, Reference Bojar2016).

With the aim of capturing overall quality of SMT output, state-of-the-art QE systems often analyze both the bilingual and monolingual properties of a source text and MT output (Bojar et al. Reference Bojar2016, Reference Bojar2017). This is a natural choice as monolingual information obtained on the target text alone is not enough to capture all types of translation errors, such as errors that correspond to incorrect meaning transfer from source to target text. On the other hand, with the assumption that the quality of a human translation is much less affected by features of the source text than factors such as time pressure and familiarity with the domain, Hokamp et al. (Reference Hokamp2014) demonstrated that target-centric features are highly informative on the task of sentence-level QE of human translation. Moreover, in a recent study, Avramidis (Reference Avramidis2017) found out that grammatical and fluency-related features made a solid contribution to the performance of sentence-level QE systems, which were trained for English–German and German–English language pairs using various MT architectures, excluding NMT.

In recent years, with the advances in NMT, a number of studies have revealed the error profiles of different MT paradigms. Multiple studies showed that, compared to NMT, SMT systems make more fluency errors in the form of grammar and lexical choice (Bentivogli et al. Reference Bentivogli, Bisazza, Cettolo and Federico2016; Klubička, Toral, and Sánchez-Cartagena Reference Klubička, Toral and Sánchez-Cartagena2017). Moreover, in different studies it has been observed that fluency errors make up the majority of translation errors made by SMT systems (Tezcan et al. Reference Tezcan, Hoste, Macken, Pastor and Durán-Mu˜ñoz2017b; Daems et al. Reference Daems, Vandepitte, Hartsuiker and Macken2017). These studies suggest that while the bilingual aspects of MT text cannot be neglected, the monolingual properties of SMT output play an important role in assessing its overall quality.

In the literature, we see a variety of studies that focus on monolingual information to detect grammatical errors in SMT output (Stymne and Ahrenberg Reference Stymne and Ahrenberg2010; Ma and McKeown Reference Ma and McKeown2012; Tezcan, Hoste, and Macken Reference Tezcan, Hoste and Macken2017a). In one of these studies, Tezcan et al. (Reference Tezcan, Hoste and Macken2017a) showed that the morpho-syntactic features used for detecting grammatical errors can also learn meaningful patterns to estimate overall post-editing effort at the word level. However, morpho-syntactic features lack the information about the semantic properties of words, and additional information is needed to detect all fluency errors, including semantic errors, and to predict overall post-editing effort.

One way of capturing both syntactic and semantic properties in MT output is to use word embeddings, which are real-valued vectors whose dimensions represent a latent feature of a given word (Li et al. Reference Li, Li, Fu, Masud and Huang2016). In the last decade, word embeddings have been successfully applied to variety of NLP tasks, including named entity recognition (Turian, Ratinov, and Bengio Reference Turian, Ratinov and Bengio2010), parsing (Socher et al. Reference Socher, Lin, Ng and Manning2011a), sentiment analysis (Socher et al. Reference Socher, Pennington, Huang, Ng and Manning2011b) and MT (Cho et al. Reference Cho, van Merriënboer, Gülçehre, Bahdanau, Bougares, Schwenk and Bengio2014). In recent years, word embeddings have also been used to estimate overall post-editing effort within neural network (NN) architectures (Kreutzer, Schamoni, and Riezler Reference Kreutzer, Schamoni and Riezler2015; Abdelsalam, Bojar and El-Beltagy Reference Abdelsalam, Bojar and El-Beltagy2016; Kim and Lee Reference Kim and Lee2016; Patel and Sasikumar Reference Patel and Sasikumar2016; Martins et al. Reference Martins, Astudillo, Hokamp and Kepler2016; Scarton et al. Reference Scarton, Beck, Shah, Smith and Specia2016; Kim, Lee, and Na Reference Kim, Lee and Na2017; Hokamp Reference Hokamp2017; Martins, Kepler, and Monteiro Reference Martins, Kepler and Monteiro2017).

For word-level QE, Kreutzer et al. (Reference Kreutzer, Schamoni and Riezler2015) proposed a system combination approach which combined a multilayer perceptron trained with the word-embedding representations of the context of the target word and the context around its source alignment, with a linear classifier trained with the baseline features of Workshop on Statistical Machine Translation 2015 (WMT’15) QE task (Bojar et al. Reference Bojar2015), which are based on information such as language model scores, the context words of a given target word and its source alignment, and part-of-speech (PoS) labels. Similarly, Patel and Sasikumar (Reference Patel and Sasikumar2016) used the context information obtained on the target and source words in an RNN-based system for the same task. In a different study, Kim and Lee (Reference Kim and Lee2016) proposed a two-stage RNN-based word-level QE model consisting of a neural word prediction model (i.e. word predictor), which uses both source and target text information from large-scale parallel corpora and a neural QE model (i.e. quality estimator) to predict the quality label for each target word. Kim et al. (Reference Kim, Lee and Na2017) advanced this model by applying a stack propagation method by allowing backpropagation from the quality estimator to the word predictor and achieved improvements in QE performance. Martins, Kepler, and Monteiro (Reference Martins, Kepler and Monteiro2017) proposed a word-level QE system which consists of a neural model stacked into a linear feature-rich classifier. In this system, they use a number of features that depend on the context of a given target word and its aligned source word, as well as syntactic features that provide information about the dependency structure of a given sentence. Moreover, Martins et al. (Reference Martins, Kepler and Monteiro2017) proposed using an APE system for predicting word-level quality labels, which relies on a large amount of artificially generated MT–PE sentence pairs, also referred to as “roundtrip translations.” The idea of using such artificial data was a major contribution toward improving APE systems (Junczys-Dowmunt and Grundkiewicz Reference Junczys-Dowmunt and Grundkiewicz2016). Hokamp (Reference Hokamp2017) achieved the state-of-the-art QE performance by using a similar APE approach proposed by by Martins et al. (Reference Martins, Kepler and Monteiro2017), by building ensembles of specialized NMT systems for the APE and word-level QE tasks while additionally incorporating syntactic features as “factors” in the input to NMT systems. Similar to the work of Martins et al. (Reference Martins, Kepler and Monteiro2017), this system relies on roundtrip translations for generating additional training data.

The proposed NN-based word-level QE system in this work differs from the above-mentioned studies in three ways. The most important difference is that, unlike many of the related work on QE, we propose a QE system that uses only monolingual information extracted from the MT output. We elaborate on the work of Tezcan et al. (Reference Tezcan, Hoste and Macken2017a), which introduced novel morpho-syntactic word representations, which we now combine with word embeddings within surface and syntactic context windows in a single RNN-based QE system and apply this model on two different tasks, namely detecting fluency errors and predicting technical post-editing effort on the word level. The third novelty in this study is that we investigate the impact of combining morpho-syntactic word representations with word embedding models learned from lemmas and the impact of learning word embedding models from in-domain data on word-level QE estimation performance.

3. Word representations: Morphology, syntax, and semantics

As a form of written text, MT output can be studied by the three main subdomains of linguistics, viz. morphology, syntax, and semantics. While morphology studies the rules that govern the way morphemes are used in a language, syntax studies the rules that pertain to the ways in which words can be combined to form grammatical sentences in a language. Finally, semantics studies the meaning of words and combinations of words in a language. Even though features that represent the properties of source text can additionally be beneficial to detect fluency errors, our expectation is that each of these three domains will provide the necessary information obtained on target text alone to identify the fluency errors made by MT systems to a great extent. Based on this idea, we consider two types of word representations relevant for monolingual QE in SMT: morpho-syntactic word representations (Tezcan et al. Reference Tezcan, Hoste and Macken2017a), which include information from both morphology and syntax, and word embeddings, which capture semantic and syntactic similarities between words (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013). Tezcan et al. (Reference Tezcan, Hoste and Macken2017a) defined morpho-syntactic word representations as binary, multi-hot vectors consisting of PoS, morphology, and dependency information about a given word. In this method, each word w is mapped to a binary vector y_w with one dimension for each possible PoS, morphology, and dependency label, where $ y_w^i = 1$ and $ y_w^j = 0, \forall j \ne i $, with i representing the morpho-syntactic features of the given word w. Figure 1 shows an example source sentence (EN), its machine-translated version (NL), and the morpho-syntactic representation for the word “zijn (are)”.

Figure 1.

Binary vector for “zijn (are)” consisting of 1s for its PoS, morphology, and dependency features and 0s for the remaining items in the vocabulary.

One difficulty of using dependency parsing on MT output is that the generated parse trees can be unreliable when the MT output itself contains fluency errors. On the other hand, multiple studies demonstrated that parse trees obtained on MT output nevertheless provide useful information in terms of MT quality. Tezcan, Hoste, and Macken (Reference Tezcan, Hoste and Macken2016) showed that parsers were useful to detect grammatical errors in the MT output due to the unusual dependency structures they produce. The same year, Martins et al. (Reference Martins, Astudillo, Hokamp and Kepler2016) included syntactic features extracted from the parse trees obtained on the SMT output in their system and achieved the best results on the WMT’16 shared task on word-level QE. Dependency parsing on SMT output has also been applied to the task of quality evaluation in the past. Owczarzak, van Genabith, and Way (Reference Owczarzak, van Genabith and Way2007) presented an MT evaluation metric which achieved highest correlation with human evaluation of translation fluency by using labeled dependencies on the MT output.

Unlike morpho-syntactic features, word embedding models capture both the syntactic and the semantic relationships between words (Turian et al. Reference Turian, Ratinov and Bengio2010). As word embeddings also model semantic information, our hypothesis is that morpho-syntactic word representations and word embeddings can complement each other and achieve better estimation performance than when they are used in isolation, on the task of word-level QE of SMT output.

In a recent study, Avraham and Goldberg (Reference Avraham and Goldberg2017) argue that lemmas are responsible for semantic aspects of the similarities between words and have the ability to improve semantic similarity models when used in isolation, while affixes are mostly responsible for morphological similarity. When word embeddings are used in combination with accurate morpho-syntactic features, the question also arises whether the additional morphological information stored in the affixes provides any valuable information. We investigate whether using lemmas for learning word embedding models is a better alternative to using surface forms, when they are combined with accurate morpho-syntactic features.

The positive effect of learning translation and language models from in-domain data on the quality of SMT systems has been shown in various studies in the past (Xu et al. Reference Xu, Deng, Gao and Ney2007; Bertoldi and Federico Reference Bertoldi and Federico2009; Axelrod, He, and Gao Reference Axelrod, He and Gao2011). Even though the ability to train on very large data sets allows word embedding models to learn complex word relationships (Kusner et al. Reference Kusner, Sun, Kolkin, Weinberger, Blei and Bach2015), Anastasakos, Kim, and Deoras (Reference Anastasakos, Kim and Deoras2014) showed that word embeddings learned from domain-specific data performed better over those obtained from general-domain data on the task of spoken language understanding. Furthermore, this positive effect was observed even when the word embedding models were learned from much smaller domain-specific data compared to general-domain data (Anastasakos et al. Reference Anastasakos, Kim and Deoras2014). In a different study, Blain, Scarton, and Specia (Reference Blain, Scarton and Specia2017) used word embeddings that are obtained from in-domain data (for the IT and medical domains) on word-, phrase-, and sentence-level QE tasks. Based on the reasonably good results they achieved on sentence-level QE compared to similar systems in literature, which used out-of-domain word embeddings, the authors argue that in-domain word embeddings encoded useful information for QE tasks. To our knowledge, the impact of the word-embedding domain on the QE-task has never been studied before. In this study, we also analyze the impact on QE performance of learning word embedding models from different domains with different data sizes.

4. NN architecture

We propose an NN architecture which uses both morpho-syntactic word representations and word embeddings as input vectors. Four fixed-sized context windows are created using each type of word representation. The four context windows consist of the following surface and syntactic n-grams, as described by Tezcan et al. (Reference Tezcan, Hoste and Macken2017a)Footnote ^a:

• Surface n-gram (n): Sequence of words as they appear in MT output, centered around the target word (n = 5)

• Syntactic n-grams (sn):

  • - Parents (sn_p): Vertical sequence of parent nodes in a given dependency tree for a given target node (n = 3)

  • - Siblings (sn_s): Horizontal sequence of sibling nodes sharing the same parent in a given dependency tree, centered around the target node (n = 5)

  • - Children (sn_c): Sequence of children nodes for a given target node (depth 1), containing the target node in the center (n = 5)

We include additional placeholder tokens in the vocabulary of the morpho-syntactic features to indicate boundaries (namely “<s>“ to indicate a sentence boundary, “[ROOT]” to indicate the root of the dependency tree, and “[NA]” to indicate horizontal boundaries in the sub-trees). Moreover, as described in Tezcan et al. (Reference Tezcan, Hoste and Macken2017a), we retain the original word order in each syntactic n-gram, with a view to capturing word-ordering errors in machine-translated texts.

The context windows built for the two types of word representations are provided as input to eight Gated Recurrent Unit (GRU) (Cho et al. Reference Cho, van Merriënboer, Gülçehre, Bahdanau, Bougares, Schwenk and Bengio2014) layers, respectively. GRU is a variant of RNNs and aims to capture dependencies of different time scales. Despite their similar performances on different tasks, unlike long short-term memory (LSTM) units, GRUs feature gating units without the separate memory cell. GRUs have outperformed LSTM units on different NLP tasks, including QE, both in terms of convergence in CPU time and parameter updates and generalization (Chung et al. Reference Chung, Gülçehre, Cho and Bengio2014; Patel and Sasikumar Reference Patel and Sasikumar2016). Assuming that each time step t has an input x_t and a hidden state h_t, GRU is defined as

$${r_t} = sigm({W_x}_r{x_t} + {W_h}_r{h_t}_{ - 1} + {b_r}) \quad \quad \quad \quad \quad \quad (reset\,gate),$$ (1)

$${z_t} = sigm({W_x}_z{x_t} + {W_h}_z{h_t}_{ - 1} + {b_z}) \quad \quad \quad \quad \quad \quad(update\,gate),$$ (2)

$$\widetilde{{h_t}} = tanh({W_x}_h{x_t} + {W_h}_h({r_t} \odot{h_t}_{ - 1}) + {b_h}) \quad \quad \quad \quad \quad(memory\,content),$$ (3)

$${h_t} = {z_t} \odot{h_t}_{ - 1} + (1 - {z_t}) \odot\widetilde{{h_t}} \quad \quad \quad \quad \quad \quad \quad(activation),$$ (4)

where ʘ is an element-wise product, W _* the weight matrices, b _* the bias vectors, r the reset gates, z the update gates, and sigm and tanh the sigmoid function and the hyperbolic tangent function, respectively.

The outputs of the eight GRU layers are concatenated before they are connected to the output layer, which consists of two units. The softmax over the activation of these two units is taken as the score for the two classes OK and BAD which represent the correct and erroneous words, respectively. To reduce overfitting, we apply dropout (Srivastava et al. Reference Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov2014) within the GRU layers (for the input gates and the recurrent connections) and after the concatenated hidden units. Figure 2 illustrates the proposed NN architecture.

Figure 2.

The proposed NN architecture.

In all of our experiments, we use Gaussian initialization scaled by fan_in + fan_out (Glorot and Bengio Reference Glorot and Bengio2010) to initialize the weights, binary cross-entropy as loss function, and RMSProp (Tieleman and Hinton Reference Tieleman and Hinton2012) as optimizer. We fix the mini-batch size to 50 and train each model for 50 epochs. We implement all models using the TensorFlow framework (Abadi et al. Reference Abadi2016). We adjust the sizes of the GRU layers according to the sizes of the two different data sets we use in our experiments, which is further explained in Section 5.

5. Experiments

We test the proposed method on two different word-level QE tasks. In the first experiment, the task is to detect fluency errors in the SMT output for the EN–NL language pair. As a binary classification task, we consider each word in a given SMT output either erroneous or correct and we extract the error labels by utilizing a corpus that contains manually annotated errors. Furthermore, we investigate whether the information provided by the morpho-syntactic features and word embeddings complement each other and lead to better estimation performance. Our hypothesis is that, unlike morpho-syntactic features, word embeddings provide additional useful information about the semantic similarities between words and complement the information provided by morpho-syntactic features on the task of QE. In a recent study, Avraham and Goldberg (Reference Avraham and Goldberg2017) showed that lemmas were able to improve semantic similarity models compared to using surface forms of words. Encouraged by their findings, we test whether QE performance can be maximized when lemmas are used to train word embedding models instead of surface forms of words. Moreover, based on the positive results obtained in other tasks (as outlined in Section 2), we study the impact of using smaller but in-domain data to train word embedding models compared to using large, general-domain data.

In the second experiment, we test the proposed method on the task of predicting overall post-editing effort, by detecting the words that require post-editing. In this binary classification task, the error labels are automatically extracted for each word by comparing the SMT output to its post-edited version. As post-editing is the task of correcting all types of translation errors, including accuracy and fluency errors, the extracted error labels in this experiment do not draw a distinction between error types. They only indicate that a post-editing operation has been performed on a given word. This experiment seeks to analyze whether monolingual information alone is effective to estimate the overall post-editing effort for a given SMT output. Similar to the task of detecting fluency errors, we analyze the impact of different NN configurations and word embedding models on estimation performance. To test the applicability of the proposed method across languages, we conduct our experiments on two language pairs (EN–NL and EN–DE).

5.1 Detecting fluency errors

In the first experiment, we evaluate the performance of the proposed method on the detection of fluency errors for the EN–NL language pair. For this experiment, we use the SCATE corpus of SMT errors (Tezcan et al. Reference Tezcan, Hoste, Macken, Pastor and Durán-Mu˜ñoz2017b), which consists of 2967 sentence pairs with manually annotated, fine-grained error annotations. The source sentences in this data set are extracted from three different text types of the Dutch Parallel Corpus (Macken, De Clercq, and Paulussen Reference Macken, De Clercq and Paulussen2011), namely external communication, nonfictional literature, and journalistic texts. The translations in this data set are obtained from Google Translate,Footnote ^b which was based on SMT technology. The SCATE MT error taxonomy, proposed by Tezcan et al. (Reference Tezcan, Hoste, Macken, Pastor and Durán-Mu˜ñoz2017b), is hierarchical in nature and is based upon the familiar notions of accuracy and fluency. In this taxonomy, any error that can be detected in the target text alone is referred to as a fluency error. Fluency errors make up 68% of all error annotations in this corpus. The SCATE error taxonomy is further divided into four main error categories: grammar, lexicon, orthography, and multiple errors. “Grammar” errors include all errors against the grammatical rules of the target language (in our case Dutch) with respect to word order, word form, missing or extra words, and multi-word syntax. “Lexicon” errors are concerned with words that either do not exist in the target lexicon or words that belong to the target lexicon but do not fit in a given context. “Orthography” errors include spelling, capitalization, and punctuation errors. Multiple errors refer to a combination of fluency errors which make these difficult to annotate separately. In the SCATE error taxonomy, any error that can only be detected by analyzing both the source and target text is referred to as accuracy error. For the details on the SCATE MT error taxonomy, we refer the readers to Tezcan et al. (Reference Tezcan, Hoste, Macken, Pastor and Durán-Mu˜ñoz2017b).

Despite the availability of a similar error taxonomy, namely Multidimensional Quality Metrics (MQM) (Lommel et al. Reference Lommel, Uszkoreit and Burchardt2014) and a data set annotated with MT errors using the MQM taxonomy, namely the WMT’14 data set (Bojar et al. Reference Bojar2014), in this study, we opted for the SCATE corpus of MT errors due to the differences in the annotation process. MQM uses a decision tree to guide the error annotation process and prioritizes annotating accuracy errors over fluency errors. This approach results in data sets in which fluency errors can remain invisible due to overlaps with accuracy errors. Unlike MQM, the SCATE error annotation guidelines allow for the annotation of multiple error types on the same text span independently, which allows us to analyze all fluency error annotations in isolation.

As the aim of the proposed NN architecture is to capture all types of fluency errors at once, for our experiments, we use the subset of fluency error annotations from the SCATE corpus of SMT errors. We mark all words that are annotated by any type of fluency error as BAD and the words that are not annotated with fluency errors as OK. The resulting data set consists of 58,002 words, 16,275 of which are annotated as fluency errors, with an OK to BAD ratio of approx. 2.56:1. We provide the number of words annotated with different subtypes of fluency errors in Table 1.Footnote ^c

Table 1.

Number of words annotated as fluency errors in the SCATE corpus of SMT errors

As Logacheva, Lukasik, and Specia (Reference Logacheva, Lukasik and Specia2016b) state, one of the reasons word-level QE is a challenging problem is the fact that OK and BAD labels are not equally important: we are generally more interested in finding incorrect words than in assigning a suitable category to every single word. According to Logacheva et al. (Reference Logacheva, Lukasik and Specia2016b), for evaluating word-level QE tasks, an ideal metric should be oriented toward the recall for the BAD class. However, they add that the case of F1_BAD score, which is calculated as the harmonic average of precision and recall for the BAD class, shows that this is not the only requirement: in order to be useful, the metric should not favor pessimistic labelings, that is, all or most words labeled as BAD. Logacheva et al. (Reference Logacheva, Lukasik and Specia2016b) further argue that the weighted average of F1 scores for the two classes is not suitable as it will be dominated by F1_OK due to label imbalance, with the assumption that any reasonable MT system nowadays will generate texts where most words are correct. Therefore, they suggest F1_MULTI, the multiplication of the F1 scores for the two labels, as an appropriate evaluation metric. In this experiment, we follow the work of Logacheva et al. (Reference Logacheva, Lukasik and Specia2016b) and use F1_MULTI as the primary evaluation metric for evaluating word-level QE systems. F1_MULTI has also been used as the primary evaluation criterion in the WMT shared tasks on QE, in recent years (Bojar et al. Reference Bojar2016, Reference Bojar2017). During our experiments, to avoid any biasing toward choice of initial weights, we train each NN three times with random weight initialization and calculate the average scores for each evaluation criterion.

Considering the relatively small number of samples in this data set, we evaluate the systems using the average 10-fold cross-validation scores. During training, we assign class weights that are inversely proportional to their frequency in each training fold using the following formula:

$${W_{ok}} = 1 - ({C_{ok}}/{C_{total}}),$$ (5)

$${W_{bad}} = 1 - ({C_{bad}}/{C_{total}}),$$ (6)

where W_ok and W_bad refer to the weights and C_ok and C_bad refer to the counts of the words labeled as OK and BAD, respectively. C_total refers to the total count of words in the data set. Setting class weights that are inversely proportional to their frequency aims to handle the imbalance we observe in the data sets we experiment with as it down-weighs the loss for the majority class and up-weighs the loss for the minority class.

We use the Alpino parser (Van Noord Reference Van Noord2006) to obtain the PoS, morphology, and dependency information for each given word in the machine-translated output. The morpho-syntactic word vectors that are built with this information consist of 129 features. Similarly, we build word embeddings with 129 dimensions, using 328M words from the SoNaR corpus (Oostdijk et al. Reference Oostdijk, Reynaert, Monachesi, Noord, Ordelman and Schuurman2008), which consists of texts extracted from 38 text types, including newspapers and e-mails. To build word embeddings, we implement word2Vec (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013) using the PythonFootnote ^d library gensim (Řehuek and Sojka Reference Řehůřek and Sojka2010). For the EN–NL experiments, we train the NN systems with GRU layer sizes of 25.

First, we analyze the impact on QE performance of morpho-syntactic features and word embeddings within different NN configurations. The performance of the different systems we build is included in Table 2.

Table 2.

The impact of the machine learning algorithm and NN architecture

As a first baseline, we build a conditional random field (CRF) system which uses the baseline features used in the WMT’16 shared task on QE (22) with one difference. Instead of using all baseline features, to be able to make a fair comparison with the proposed NN architecture that relies on monolingual information, we only used the features that refer to the target text (14). We used the CRF++ toolkitFootnote ^e to train QE models and to predict quality labels, and the Marmot tool (Logacheva, Hokamp, and Specia Reference Logacheva, Hokamp and Specia2016a) to extract the features, which are further outlined in Bojar et al. (Reference Bojar2016).Footnote ^f

As a second baseline, we build the system proposed by Tezcan et al. (Reference Tezcan, Hoste and Macken2017a), which provides morpho-syntactic features as input to the four context windows, as described in Section 4, and which uses unidirectional GRU layers (UniGRU-MS). The estimation performance of these two systems shows us that the NN system, which is based on morpho-syntactic information alone, is able to outperform the CRF system, which is based on the features that are used to build the baseline system in WMT’16 shared task on QE, with relative improvements of 53% and 48% in F1_BAD and F1_MULTI scores, respectively.

We use the same NN architecture to build two additional systems, a first one which uses only word embeddings (UniGRU-W2V) and a second system using a concatenation of morpho-syntactic features and word embeddings as a single vector (UniGRU-MS/W2V). By their performance, we can observe that morpho-syntactic features outperform word embeddings on this task. The informativeness of morpho-syntactic features over word embeddings can be associated with the high number of grammatical errors present in the data set used in this study, which is shown in Table 1. We also see that when the two types of word vectors are concatenated as input vectors (UniGRU-MS/W2V), the results are slightly better (F1_MULTI = 0.4037).

An alternative approach for combining morpho-syntactic features and word embeddings is to provide two separate vectors as input to a total of eight context windows, as described in Section 4 (UniGRU-MS+W2V). We see that this alternative architecture is able to reflect the added benefit of combining the two types of inputs, as it improves the performance up to a F1_MULTI score of 0.4144. Splitting the morpho-syntactic information and distributed representation of words into separate vectors has a positive effect on estimation performance. Using bidirectional GRU layers (BiGRU-MS+W2V), as proposed in Section 4, further increases the performance and achieves the best results with respect to all evaluation criteria, with an F1_MULTI score of 0.4209.

Secondly, we analyze the impact of the domain of the corpus and the types of information, from which word embeddings are learned. In addition to using the (out-of-domain) SoNaR corpus, as described above, we additionally build word embeddings using 2.9M words from the (in-domain) Dutch Parallel CorpusFootnote ^g and on the merged data set which consists of all sentences from both corpora. Furthermore, we build alternative systems by using base forms of words (lemmas) instead of surface forms. In all cases, we build the proposed architecture as already described in Section 4, which provides separate word vectors as input, consisting of morpho-syntactic features and word embeddings. The performance of the different systems is provided in Table 3.

Table 3.

The impact of word embedding model on estimation performance. Word embedding models learned from surface forms of words are indicated with “(S)” and models learned from lemmas with “(L)”

In the upper section of Table 3, we see that word embedding models learned from relatively small but in-domain data lead to better QE performance on the task of detecting fluency errors (BiGRU-MS+W2V_DPC (S)). Using very large, out-of-domain data in this case does not only achieve slightly inferior performance compared to in-domain data when used alone (BiGRU-MS+W2V_SONAR (S)), but also deteriorates the performance slightly when it is added to the in-domain data set (BiGRU-MS+W2V_DPC+SONAR (S)). Although further research would be necessary to make conclusions, the positive effect of the in-domain data might be explained by its ability to learn specific meanings of certain words in limited context. Even when both data sets are combined, learning additional meanings for words in larger, general-domain data might lead to a decrease in the error detection performance.

In the lower section, we see the performance of the systems using word embeddings models that are learned from lemmas instead of surface word forms, with the exception of the system displayed in the first row (BiGRU-MS+W2V_SONAR.33M (S)). Alpino provides lemmas for each word in the parse tree it builds. As parsing brings additional computational cost, we acquire lemmas only for approx. 33M words, which are obtained from 2M randomly selected sentences from the SoNaR corpus (BiGRU-MS+W2V_SONAR.33M (L)). We also combine the same data set with the in-domain data (BiGRU-MS+W2V_DPC+SONAR.33M (L)). To observe the impact of reducing the training data, we use the same 33M words to learn word embeddings from surface word forms (BiGRU-MS+W2V_SONAR.33M (S)). As a result, the lower section of Table 3 shows similar trends as in the upper section and confirms the positive effect of using in-domain data for building word embedding models when lemmas are used. Moreover, we see that combining morpho-syntactic features with word embeddings learned from lemmas achieves slightly better F1_MULTI scores than combining morpho-syntactic features with word embeddings learned from surface forms (BiGRU-MS+W2V_DPC (L) vs. BiGRU-MS+W2V_DPC (S)).

5.2 Predicting post-editing effort

In the second experiment, we evaluate the proposed method on the task of predicting overall post-editing effort, which consists of detecting the words that require post-editing. In this experiment, we do not have manually annotated errors. Instead, the error labels are extracted using the Translation Error Rate (TER) tool (Snover et al. Reference Snover, Dorr, Schwartz, Micciulla and Makhoul2006), by comparing the SMT output with its post-edited version. The motivation behind this experiment is to investigate the feasibility of using the same NN architecture to predict the technical post-editing effort involved in correcting a given MT output. While detecting all errors in a given MT output, without making a distinction between the error types, can be considered as an alternative to this experimental setup, previous studies show that different post-editors can arrive to different correct translations as a result of post-editing the same MT output and that MT errors are not always good indicators of post-edit operations (de Almeida Reference de Almeida2013; Daems et al. Reference Daems, Macken and Vandepitte2014). Therefore, we consider predicting technical post-editing effort on the word level as a different task than detecting MT errors.

For extracting the error labels with TER, the following settings are adopted from the WMT’16 shared task on QE:

• Tokenized, case-insensitive, exact matching is performed.

• Insertions made during post-editing are not annotated as they cannot be associated with any word in the MT output.

• Shifts are disabled, but rather annotated as edits in the form of deletions and insertions.

As a result, all words that have been subject to post-editing are marked as BAD. All remaining words in the data set are marked as OK. Since post-editing involves correcting all types of translation errors, automatically extracted error labels do not distinguish between the error types but only indicate that a post-editing operation has been performed on a given word.

Tezcan et al. (Reference Tezcan, Hoste, Macken, Pastor and Durán-Mu˜ñoz2017b) showed that morpho-syntactic features, even though they require different parsing tools and use different PoS and morphology tagsets, can successfully be applied to the task of QE for both EN–NL and EN–DE language pairs. To analyze to what extent overall post-editing effort can be predicted using only monolingual information, similarly, we experiment with both language pairs. Word-level prediction of overall post-editing effort on SMT output has been a part of the WMT shared task on QE in recent years (Bojar et al. Reference Bojar2015, Reference Bojar2016). To be able to compare our results with previous studies, we use the WMT’16 data set (Specia, Logacheva, and Scarton Reference Specia, Logacheva and Scarton2016) for the EN–DE and the SCATE corpus of SMT errors for the EN–NL language pairs.

The WMT’16 data set consists of 15k source-target sentence pairs (279,976 words in the target language) in the IT-domain, automatically translated from EN into DE by a phrase-based SMT system. The data are partitioned into 12k, 1k, and 2k sentence pairs as training, tuning, and test sets, respectively. Each word in the given data sets is labeled as OK or BAD, with an OK to BAD ratio of approx. 3.70:1, 4.17:1, and 4.25:1 in the training, tuning, and test sets, respectively. We used two data sets to learn word embedding models in German. As in-domain data, given the small size of the WMT’16 training set for learning word embedding models, we combined it with the post-edited SMT segments from the Autodesk Post-Editing Data corpus.Footnote ^h This additional data set consists of 107,353 sentences and approx. 1.5M words in total.Footnote ⁱ As out-of-domain data, we used a collection of Wikipedia articles (Wolk and Marasek Reference Wolk and Marasek2015),consisting of approx. 2.45M sentences and 43.5M words. To build the morpho-syntactic word vectors, we obtained PoS, morphology, and dependency labels for each German word in the MT output, using the Mate tools (Bohnet and Nivre Reference Bohnet and Nivre2012). The morpho-syntactic word vectors that are built for German consisted of 131 features. Similarly, we built word embeddings learned on German data with 131 dimensions. For the EN–DE experiments, we train the NN system with GRU layer sizes of 50 and increase the complexity of the NN, given the relatively bigger data set compared to EN–NL language pair, for which we set the GRU layer sizes to 25.

For the EN–NL language pair, we obtained the post-edited version of the SMT output by a Master student in translation studies. We applied the same label extraction method, as described above, for marking erroneous words.Footnote ^j The OK to BAD ratio in this data set is approx. 4.81:1. The word embedding models are learned from the same three data sets described in Section 5.1.

When analyzing the SCATE corpus of SMT errors (EN–NL), we see that the fluency errors make up the majority of errors in this data set (68%). As the SCATE error taxonomy allows annotating multiple errors on the same text span, further analysis reveals that 64% of all accuracy errors in this data set overlap with at least one fluency error. This means that only a small portion of erroneous words in the target text (corresponding to 36% of the accuracy errors and 11.5% of all errors) can be detected only by analyzing the source text together with the SMT output. As a result, the erroneous words corresponding to 88.5% of all errors in this data set can be detected using monolingual information alone. These statistics indicate that monolingual information alone can effectively be used, not only to detect fluency errors made by SMT systems but also to estimate overall post-editing effort. However, the question remains whether similar error statistics can be observed in other data sets, consisting of SMT output obtained for other language pairs and domains. In this study, while we can analyze the error statistics in the EN–NL data set, no explicit information is available for the EN–DE data set.

The experiments performed for both language pairs on the task of predicting overall post-editing effort are similar to the ones we did on the task of detecting fluency errors: Firstly, we analyze the impact of combining morpho-syntactic features and word embeddings on this task. Our hypothesis is that word embeddings provide useful information on top of morpho-syntactic features and can capture nongrammatical errors. We expect the combination of the two types of information to improve QE results on the task of predicting overall post-editing effort. Secondly, we analyze the impact of learning word embedding models from smaller in-domain data compared to larger general-domain data. Finally, we test whether learning word embedding models from lemmas, instead of surface forms of words, maximize the estimation performance.

We provide the results obtained from the QE systems we built for the EN–NL and EN–DE language pairs in Tables 4 and 5, respectively. Similarly to the task of detecting fluency errors, we use F1_MULTI as the main evaluation criterion, while we also report F1_BAD and F1_OK scores. We train each NN three times with random weight initialization and calculate the average scores for each evaluation criterion. Tables 4 and 5 consist of three sections: an upper section which provides the results obtained by using either morpho-syntactic features or word embeddings in isolation, and middle and lower sections which show the impact on QE performance of using different word embedding models in combination with morpho-syntactic features.

Table 4.

QE results for EN–NL language pair. Word embedding models learned from surface forms of words are indicated with “(S)” and models learned from lemmas with “(L)”

^a The performance of this system is slightly worse than the results reported by Tezcan et al. (Reference Tezcan, Hoste and Macken2017a) due to using GRU layers of size 25, instead of 50. Setting the layer sizes to 25 reduced the training times considerably with minimal loss in performance.

From the first row in Table 4, we see that using morpho-syntactic features alone within unidirectional GRU layers, as described by Tezcan et al. (Reference Tezcan, Hoste and Macken2017a), results in a strong baseline, with an F1_MULTI score of 0.3741. Applying bidirectional GRU layers instead (BiGRU-MS) improves the results further to 0.3986. In the top section, we also see that word embeddings, when used in isolation, are outperformed by morpho-syntactic features on this task for the EN–NL language pair (BiGRU-W2V_SONAR (S)). In Table 1, we notice that the majority of words annotated with fluency errors in the SCATE corpus are associated with grammatical errors. Even though morpho-syntactic features alone are not able to capture semantic properties of words, the dominance of grammatical errors in this data set can explain the benefit of using these features over word embeddings for this task.

In the middle section, we see that combining word embeddings with morpho-syntactic features (BiGRU-MS+W2V_SONAR (S)) achieves better results than using either type of information alone. Similar to our observations on the task of detecting fluency errors, learning word embedding models from a smaller in-domain data set boosts the QE performance further (BiGRU-MS+W2V_DPC (S)). Finally, the lower section shows us that using lemmas is a better choice than using surface forms of words, when learning word embedding models for this task. Again, the highest estimation performance is achieved by the system which learns word embedding models from the lemmas and the in-domain data set (BiGRU-MS+W2V_DPC(L)), with an F1_score of 0.4116.

Table 5.

QE results for EN–DE language pair

^a The performance of this system is slightly worse than the results reported by Tezcan et al. (Reference Tezcan, Hoste and Macken2017a) due to using GRU layers of size 50, instead of 100.

The QE results obtained on the EN–DE language pair are especially interesting as we do not have statistics about the errors made by the SMT system, which generated the German translations. Similar to the previous results, the upper section in Table 5 shows that using bidirectional GRU layers has a positive impact on performance (BiGRU-MS vs. UniGRU-MS). However, on the German SMT output, unlike the Dutch output, word embeddings outperform morpho-syntactic features for detecting post-editing effort, when both types of information are used in isolation. There are several differences between the two experiments on the EN–NL and EN–DE language pairs, which could explain this difference. The difference in the target languages, domains, and the SMT systems used can all have an impact on the error profiles of the two SMT outputs, and as a result, the amount of specific error types in both data sets can differ. If grammatical errors are not as dominant in the WMT’16 SMT output as in the SCATE corpus of MT errors, it would not be surprising word embeddings to outperform morpho-syntactic features on this data set. On the other hand, given that we extract the morpho-syntactic features on two different languages using different toolkits, the quality of the tags obtained for PoS, morphology, and dependency relations can be different. Nevertheless, combining morpho-syntactic features with word embeddings still provides better results than using either type of information alone on both language pairs, as we can see in the middle section (BiGRU-MS+W2V_WIKI (S)).

Similar to previous results we obtained on the EN–NL language pair, learning word embeddings from the small but in-domain data set boosts the estimation performance (BiGRU-MS+W2_WMT/AD (S)). Combining this data set with the out-of-domain data deteriorates the results (BiGRU-MS+W2_WMT/AD+WIKI (S)). Finally, learning word embedding models from lemmas seem to improve the results slightly as the best system achieves an F1_MULTI score of 0.3979 (BiGRU-MS+W2_WMT/AD (L)).

The performance of the EN–DE system can be better understood when compared to the participating systems in the WMT’16 shared task on word-level QE. In Table 6, we provide the results for a number of systems from the shared task (Bojar et al. 2016) and the ranks they obtained, together with the best system we built using the proposed architecture (MS+W2V_WMT/AD (L)). We also include the results of the system proposed by Tezcan et al. (Reference Tezcan, Hoste and Macken2017a), which utilizes morpho-syntactic features in isolation, as an additional baseline.

Table 6.

Performance of the proposed method in comparison to competing systems in the WMT’16 shared task on word-level QE

By using monolingual information given in the SMT output alone, the proposed system outperforms the baseline system in this shared task, which uses features representing monolingual and bilingual properties of each translated text. In fact, the proposed system achieves the 4th best result in the WMT’16 shared task and outperforms 11 participating systems out of 14. From all the systems that were submitted to WMT’16, the NN architecture used in the CDACM/RNN system shows similarities to the proposed NN architecture proposed in this study. Both systems use GRU layers and provide surface n-grams around the target word as input, in the form of pre-trained word embeddings. On the other hand, while the CDACM/RNN system combines this information with information obtained from the source text, in the form of word embeddings for the context words centered around the source alignment of the target text, the proposed architecture in this study combines this information with syntactic n-grams and morpho-syntactic word representations obtained on the target text. From this perspective, Table 6 confirms the informativeness of monolingual information on predicting post-editing effort of SMT output. More specifically, it shows us that an RNN system that uses only monolingual information, in the form of morpho-syntactic word representations and word embeddings within surface and syntactic n-grams, outperforms a similar RNN architecture which uses bilingual information in the form of word embeddings within surface n-grams, instead (+0.0448 F1_SCORE).

6. Error analysis

In Sections 5.1 and 5.2, we have observed the effectiveness of using monolingual information alone on estimating word-level quality of SMT output. Despite the improvements achieved over different baselines we considered, the results indicate that word-level QE systems still contain errors, which has also been observed in the past on the task of predicting post-editing effort (Bojar et al. Reference Bojar2015 Reference Bojar2016).

A main source of errors in this study is the limited amount of training data we have for the two language pairs, as NNs require large amounts of training data to sufficiently learn the weights between neurons. Although the proposed NN architecture was able to outperform the majority of the submitted systems in the WMT’16 shared task on word-level QE with the provided data sets, the impact of the data size on its performance is yet to be studied. Moreover, the EN–NL data set we used in this study can be considered relatively small, consisting of only 52,202 training samples in each cross-validation fold. In this section, we first analyze the impact of the size of the available training data on estimation performance. For this purpose, we train the best systems for each task and on each language pair by using 25% (every 4th sample) and 50% (every 2nd sample) of the available training data.Footnote ^k We provide the results in Figure 3.

Figure 3.

The impact of data size on estimation performance.

The results in Figure 3 confirm that reducing the amount of samples available in training data to 50% and 25% reduces the estimation performance (F1_MULTI score) each time on all tasks and for each language pair. On the other hand, the trend lines drawn for each system indicate that the expected increase in estimation performance is different in each case. While comparing all systems at once is difficult given the differences in the language pairs and the tasks, we can make a pairwise comparison. The best system obtained for the EN–DE language pair on the task of predicting post-editing effort (Best PE (DE)) shows a trend with fewer gains, compared to the system on the EN–NL language pair (Best PE (NL)), where the data size is doubled. This is not surprising as the EN–DE data size is already approximately four times the size of the EN–NL data set we use (consisting of approx. 210k and 52k samples, respectively) and we can expect the positive impact of doubling the data size to decrease for larger data sets.

For the EN–NL data set, when we draw a comparison based on the type of the QE task, we notice that the trend line for the best system obtained for detecting fluency errors (Best FLU (NL)) shows a constant increase, whereas the estimation performance of the best system obtained for predicting post-editing effort (Best PE (DE)) seems to increase with a decreasing rate. This can be explained with the type of translation errors captured by the two QE tasks: it seems that using monolingual information for detecting errors of monolingual nature (fluency errors) benefits more from an increase in data size than predicting post-editing effort, which represents correction of all error types in the SMT output. Even though it is difficult to make a fair comparison between different tasks, we see that the estimation performance of “Best FLU (NL)” is already better than “Best PE (NL)” when the same data sets are used (F1_MULTI scores of 0.4303 and 0.4116, respectively). The trends in Figure 3 further suggest that the difference between their performance will further increase in favor of “Best PE (NL)”, with increasing data sizes.

When we analyzed the best systems described above, on the task of predicting post-editing effort, we discovered a common type of mistake especially made by the EN–DE system. As mentioned in Section 5.2, while extracting error labels for this task, inserted words during post-editing are not marked erroneous in the MT output simply because they do not exist in the MT output yet.Footnote ^l Therefore, the resulting training set is not able to inform the systems about the missing words in the MT output, that is, the words inserted during post-editing. Interestingly, the EN–DE systemFootnote ^m we built seemed to successfully locate the context of missing words in many cases, even though no information was available regarding this error type in the training data. In fact, we observed that the ratio of the false positive (predicting a label as BAD whereas the correct label is OK) made on the labels corresponding to the left and right context words of the insertions made during post-editing over all other labels was 1.77:1 and 1.38:1 for EN–DE and EN–NL, respectively. We illustrate this phenomenon in Figure 4.

Figure 4.

QE system locating the context of missing words, where insertions are made during post-editing.

In Figure 4, we see a source sentence (SRC), the corresponding SMT output containing the predicted errors by the QE system in bold (MT_PRED), and the post-edited version of the SMT output (MT_PE). In the post-edited version, we can see that the word “verfügbar (available)” has been inserted to the SMT output. In this example, the QE system was able to localize the context of the missing word (by marking the words “monochrome Bitmapbilder”), even though it did not encounter error labels for such errors in the training data.

One possible explanation for this system’s ability to approximate the location of insertions is that the surface and syntactic context windows enables the NN architecture to learn contextual anomalies regarding the fluency of the SMT output to a certain extent, even when no error labels are provided. To see the effect of this phenomenon on QE performance, we retrained the two best systems we built for predicting post-editing effort on EN–NL and EN–DE with a difference: in the SMT output, we additionally marked the words to the left and right of all the inserted words during post-editing as BAD, which is illustrated in Figure 5.

Figure 5.

Marking the context of missing words, which require an “insertion” in the MT output.

In Figure 5, even though they are not associated with errors themselves, the word “Bitmapbilder (bitmap images)” and the punctuation token “.” have been marked as erroneous to indicate the context of a missing word in the SMT output (MT_LABELS). Marking more words as erroneous introduced 5.9% and 6.7% additional BAD labels in the training data for the EN–NL and EN–DE language pairs, respectivelyFootnote ⁿ). In Tables 7 and 8, we compare the estimation performance of the retrained systems with the best systems we obtained for both language pairs in Section 5.2. While Table 7 provides the QE performances on the full test sets, Table 8 provides the QE performances only on the labels for the sentences that contain at least one insertion. To have a better understanding of the systems’ error profiles, in addition to the F1 scores, in both tables, we also provide the precision and recall scores for the BAD label.

Table 7.

The impact of retraining systems with additional labels that mark the context of insertions made during post-editing. The retrained systems are indicated with “(ins.)”. Evaluation is performed on the full test sets

What we see in Table 7 is that marking the context of inserted words during post-editing provides both systems additional useful information as the overall performance of both systems improves with respect to F1_MULTI. However, the level of improvement is different for both systems. The EN–DE system (PE-effort DE) seems to benefit from this additional information more compared to the EN–NL system (PE-effort NL) for all evaluation criteria (except F1_OK, which deteriorates in both cases). The improvement, especially for the Recall_BAD and F1_MULTI scores, is much higher for the EN–DE system (+0.0978 Recall_BAD and +0.0322 F1_MULTI) than for the EN–NL system (+0.0170 Recall_BAD and +0.0079 F1_MULTI). Given that the relative amount of newly introduced BAD labels in both training data is roughly similar (5.9% and 6.7% in EN–NL and EN–DE, respectively, the difference in the performance improvements can be attributed, at least partially, to the different data sets used for both language pairs. It is also possible that the QE systems benefit more from using this additional information with increasing sizes of the training data.

To get a better understanding about how modifying error labels affects the QE performance, in a second experiment, we evaluate only the labels that come from the sentences, in which new words have been inserted during post-editing. As a result, in this experiment, 35,449 labels (from 1435 sentences) from the EN–NL data set and 19,557 labels (from 1066 sentences) from the EN–DE set have been evaluated, which correspond to 61% and 57% of all labels in the test sets for EN–NL and EN–DE, respectively.Footnote ^o Table 8 provides the QE performances obtained on this reduced test set.

Table 8.

The impact of retraining systems with additional labels that mark the context of insertions made during post-editing. The retrained systems are indicated with “(ins.)”. Evaluation is performed only on the labels for the sentences that contain insertions

Similar to Table 7, Table 8 shows that introducing the additional BAD labels to the data sets leads to overall improvements in QE performance for both language pairs. Moreover, for both language pairs, Table 8 shows that the sentences, in which insertions are made, benefit more from the modification of error labels, as the improvements in QE performance become more outspoken for the labels that come from such sentences (+0.0506 and +0.0692 F1_MULTI for EN–NL and EN–DE data sets, respectively).

7. Conclusions and future work

The aim of this study was to investigate the effectiveness of using monolingual information contained in the machine-translated text to estimate its quality. To this end, we presented an RNN architecture which takes morpho-syntactic and distributed word representations obtained from monolingual data alone as input to estimate word-level quality of SMT. Our approach combines PoS, morphology, and dependency information extracted from words with the semantic relationships modeled by word embeddings as separate input vectors to the NN. We tested the proposed method first on word-level detection of fluency errors in SMT output for EN–NL. Our results show that the proposed method is able to outperform alternative NN architectures on detecting all fluency errors at once. We observed that splitting morpho-syntactic information and semantics modeled by word embeddings leads to better neural models and estimation performance. Moreover, we also showed that careful selection of word representations is key to building accurate error detection systems on specific error types.

Secondly, we utilized this method to estimate the overall post-editing effort by detecting words that have been modified during post-editing for the EN–NL and EN–DE language pairs. We demonstrated that by only using monolingual information, successful QE systems can be built to predict overall post-editing effort involved in correcting the SMT output. The proposed method achieved better estimation performance than the majority of the competing systems in the WMT’16 shared task on QE. The high predictive power of monolingual information additionally revealed the importance of fluency errors on defining overall SMT quality.

Other key findings in this study unveiled the impact on QE performance of the type of information and the resources from which word embeddings are learned. On both EN–NL and EN–DE language pairs and for both tasks, learning word embeddings from lemmas had a positive impact on estimation performance, when they were combined with morpho-syntactic features. Furthermore, learning word embeddings from relatively small but in-domain data sets improved the results in all cases, compared to training them from large but general-domain data.

An obvious direction for future work is to add bilingual information to the proposed architecture. This can be done in different ways for both word- and sentence-level QE tasks. On the one hand, the same NN architecture can be applied to the task of predicting post-editing effort by extending it with additional bilingual information. To this end, we would like to analyze the impact of adding different types of source-related information, such as morpho-syntactic word representations and word embeddings, as input to the proposed NN architecture.

On the other hand, the proposed method can be used as a main component in two-step sentence-level QE-systems which consist of detecting specific error types in the first step and use them to estimate a global score on sentence-level quality in a second step. The proposed method can be combined with a second error detection system that focuses on adequacy errors to build such a two-step QE system. Two-step QE systems would not only inform the users on the nature of errors that are made by SMT systems on word level but also would provide reasons why estimated sentence-level quality is better in certain cases than others. This would lead to more informative QE systems compared to existing systems that try to estimate overall sentence-level quality with a single score. Moreover, the informative QE approach is not limited to SMT applications and it can be applied to different MT paradigms, provided that similar data sets with manual error annotations are available in the future.

Finally, the tasks of detecting fluency errors in MT and detecting errors made by non-native speakers can be considered similar, given that they both focus on the monolingual aspects of a single language. From this perspective, the results of this study might also be useful to develop supervised error detection systems which can locate the mistakes in texts that are written by non-natives.