1 Introduction
Annotating linguistic resources consists of adding information of interpretative nature to the original raw data (Garside, Leech and McEnery Reference Garside, Leech and McEnery1997). This information may be terminological, lexical, morphological, syntactic or semantic. Linguistic resources can be lexicons, transcriptions of dialogues, text corpora, etc. (Veronis Reference Veronis2000).
Annotating corpora with linguistic information (part-of-speech tagging, sense tagging, syntactic analysis, named entity identification and semantic role annotation) involves significant human efforts. The availability of parallel corpora has recently led to several research studies on cross-lingual annotation projection. The idea is to explore the use of unsupervised (no-supervision on the targets languages) approaches which project labels from the (resource-rich) source language (such as English which has large amounts of annotated corpora and several analysis tools available) to the (under-resourced) target language with less resources. The goal of cross-language projection is, on the one hand, to provide all languages with linguistic annotations, and on the other hand, to automatically induce text analysis tools for these languages.
In this work, we address the task of automatic building of multilingual linguistic resources and tools. This article presents the use of unsupervised approaches to learn multilingual Natural Language Processing (NLP) tools from parallel corpora with only annotations in the source language.
Contributions. We investigate different architectures of RNNs — unidirectional RNN (URNN) and bidirectional RNN (BRNN) — for multilingual sequence labeling tasks. Two NLP tasks are considered: Part-of-speech (POS) tagging and Super Sense tagging (SST) (Ciaramita and Altun Reference Ciaramita and Altun2006). Our RNN architectures demonstrate very good results on unsupervised training for new target languages. In addition, we show that the integration of POS information in RNN models is useful to build a multilingual coarse-grain semantic (Super Sense) tagger. For this, a simple and efficient way to take into account low-level linguistic information for more complex sequence labeling tasks is proposed. Semi-supervised scenarios (where a small amount of annotated target data is available) are also investigated in this journal article.
Outline. The remainder of the article is organized as follows. Section 2 reviews related work. Section 3 describes our cross-language annotations projection approaches based on RNN. Section 4 (POS tagging) and Section 5 (SST) present our empirical studies and experimental results. We finally conclude the article in Section 6 and present our future research directions.
2 Related work
We present below most approaches for cross-lingual projection based on word alignment or representation learning.
2.1 Cross-lingual projection based on word alignment
The availability of parallel corpora has recently led to several research works exploring the use of unsupervised or semi-supervised cross-lingual annotation projection based on word alignment information. Through word alignments in parallel corpora, the annotations are transferred from the (resource-rich) source language to the (potentially under-resourced) target language.
2.1.1 Unsupervised methods
Cross-lingual projection of linguistic annotations was pioneered by Yarowsky, Ngai and Wicentowski (Reference Yarowsky, Ngai and Wicentowski2001) who proposed to transfer annotations from resource-rich languages onto resource-poor languages, based only on word alignments from a parallel corpus. After alignment, the source language is annotated, and annotations are projected to the target language. The resulting (noisy) annotations are used in conjunction with robust learning algorithms to build cheap unsupervised NLP tools. This approach has been successfully used to transfer several linguistic annotations between languages. Examples include POS (Das and Petrov Reference Das and Petrov2011; Duong, Cook, Bird and Pecina Reference Duong, Cook, Bird and Pecina2013), named entities (Kim, Toutanova and Yu Reference Kim, Toutanova and Yu2012), syntactic constituents (Jiang, Liu and Lü Reference Jiang, Liu and Lü2011), word senses (Bentivogli, Forner and Pianta Reference Bentivogli, Forner and Pianta2004; van der Plas and Apidianaki Reference Van der Plas and Apidianaki2014) and semantic role labels (Pado and Pitel Reference Pado and Pitel2007; Annesi and Basili Reference Annesi and Basili2010).
In recent years, several works have investigated transfer learning (Pan and Yang Reference Pan and Yang2010) in NLP. We can cite the work of Jiang et al. (Reference Jiang, Lü, Huang and Liu2015) who applied transfer learning on annotation adaptation. They implemented several models for transferring annotations of a source corpus to the annotation format of another target corpus. These models are based on a transfer classifier which learns correspondence regularities between annotation guidelines from a parallel annotated corpus, which has two kinds of annotations for the same data. In this specific application, the source corpus and the target corpus are in the same language. In the same vein, Passban, Liu and Way (Reference Passban, Liu and Way2017) implemented statistical and neural machine translation engines that are trained on one language pair but are used to translate another language. They trained a reliable model for a high-resource language, and then they exploited cross-lingual similarities in order to adapt the model to work for a close language with few resources.
In order to build our baseline system, we use direct transfer which is similar to the method described in Yarowsky et al. (Reference Yarowsky, Ngai and Wicentowski2001). First, we tag the source side of the parallel corpus using an available supervised tagger. Next, we align words in the parallel data using GIZA++ (Och and Hermann Reference Och and Ney2000) to find out corresponding source and target words. Then, we project tags in the target language using following projection criteria:
• For 1-to-1 alignments, we project tags directly.
• For N-to-1 mappings, we project the tag of the word at the position N/2 rounded up to the next whole number.
• The unaligned words (target) are tagged with their most frequent associated tag in the corpus.
We finally train a tagger on the target language using Trigrams’n’Tags (TnT) statistical tagger (Brants Reference Brants2000).
2.1.2 Semi-supervised methods
Cross-lingual projection requires a parallel corpus and word alignment between source and target languages. Many automatic word alignment tools are available, such as GIZA++ which implements IBM models (Brown et al. Reference Brown, Pietra, Pietra and Mercer1993). However, the noisy (non-perfect) outputs of these methods are a serious limitation for the annotation projection based on word alignments (Fraser and Marcu Reference Fraser and Marcu2007). To deal with these limitations, recent studies have proposed to combine projected labels with partially supervised monolingual information in order to filter out invalid label sequences. For example, Li, Graça and Taskar (Reference Li, Graça and Taskar2012), Täckström et al. (Reference Täckström, Das, Petrov, McDonald and Nivre2013) and Wisniewski et al. (Reference Wisniewski, Pécheux, Gahbiche-Braham and Yvon2014) have proposed to improve projection performance by using a dictionary of valid tags for each word (coming from WiktionaryFootnote 1).
2.2 Projection based on cross-lingual representation learning
In another vein, recent studies based on cross-lingual representation learning methods have been proposed to avoid using such pre-processed and noisy alignments for label projection. First, these approaches learn language-independent features, across many different languages (Durrett, Pauls and Klein Reference Durrett, Pauls and Klein2012; Al-Rfou, Perozzi and Skiena Reference Al-Rfou, Perozzi and Skiena2013; Täckström, McDonald and Nivre Reference Täckström, McDonald and Nivre2013; Luong, Pham and Manning Reference Luong, Pham and Manning2015; Gouws and Søgaard Reference Gouws and Søgaard2015; Gouws, Bengio and Corrado Reference Gouws, Bengio and Corrado2015). Then, the induced representation space is used to train NLP tools by exploiting labeled data from the source language and applying them in the target language. Cross-lingual representation learning approaches have achieved good results in different NLP applications such as cross-language SST and POS tagging (Gouws and Søgaard Reference Gouws and Søgaard2015), cross-language named entity recognition (Täckström, McDonald and Uszkoreit Reference Täckström, McDonald and Uszkoreit2012), cross-lingual document classification and lexical translation (Gouws et al. Reference Gouws, Bengio and Corrado2015), cross language dependency parsing (Durrett et al. Reference Durrett, Pauls and Klein2012; Täckström et al. Reference Täckström, Das, Petrov, McDonald and Nivre2013) and cross-language semantic role labeling (Titov and Klementiev Reference Titov and Klementiev2012).
Our approach described in the next section, is inspired by these works since we also try to induce a common language-independent feature space (cross-lingual words embeddings). Unlike Durrett et al. (Reference Durrett, Pauls and Klein2012) and Gouws and Søgaard (Reference Gouws and Søgaard2015), who use bilingual lexicons, and unlike Luong et al. (Reference Luong, Pham and Manning2015) who use word alignments between source and target languages,Footnote 2 our common multilingual representation is very agnostic. We use a simple (multilingual) vector representation based on the occurrences of source and target words in a parallel corpus and we let the RNN learn the best internal representations (corresponding to the hidden layers) specific to the task (SST, POS tagging or other tasks).
In this work, we learn a cross-lingual POS tagger (multilingual POS tagger if a multilingual parallel corpus is used) based on a recurrent neural network (RNN) on the source labeled text and apply it to tag target language text. We explore URNN and BRNN architectures, respectively. We also show that the proposed architecture is well suited for lightly supervised training (adaptation). Starting from the intuition that low-level linguistic information is useful to learn more complex taggers, we also introduce three new RNN variants to take into account external (POS) information in a more complex task (multilingual SST).
3 Our approach: Using recurrent neural networks in cross-lingual projection of annotations
To avoid projecting label information from deterministic and error-prone word alignments, we propose to represent the bilingual word information intrinsically in an RNN architecture. Then, this RNN is used as a multilingual sequence labeling model (we investigate POS tagging and SST tasks).
3.1 Proposed method
We propose a method for learning multilingual sequence labeling tools based on RNN, as can be seen in Figure 1. In our approach, a parallel or multi-parallel corpus between a resource-rich language and one or many target (potentially under-resourced) language(s) is used to extract common (multilingual) and agnostic words representation. These representations, which rely on sentence-level alignment only, are used with the source side of the parallel/multi-parallel corpus to learn a neural network tagger in the source language. Since a common representation of source and target words is chosen, this neural network tagger is authentically multilingual and can also be used to tag texts in target language(s).
Overview of the proposed model architecture for inducing multilingual RNN taggers.
In our agnostic representation, we associate to each word (in source and target vocabularies) a common vector representation, namely V w (equation (1)).
$$\begin{equation}
V_{w} = \left\lbrace \begin{array}{r c l}V_{wi} = 1 & if & w \in S_{i} \\
V_{wi} = 0 & else \end{array} \right.
\end{equation}$$
The idea is that, in general, a source word and its target translation appear together in the same bisentences and their vector representations are close. We can then use the RNN tagger, initially trained on source side, to tag the target side (because of our common vector representation). This simple representation does not require multilingual word alignments since the RNN learns the optimal internal representation needed for the annotation task (for instance, the hidden layers of the RNN can be considered as multilingual word embeddings).
3.2 Recurrent neural networks
There are two major architectures of neural networks: Feedforward (Bengio et al. Reference Bengio, Ducharme, Vincent and Jauvin2003) and RNN (Schmidhuber Reference Schmidhuber1992; Mikolov et al. Reference Mikolov, Karafiát, Burget, Cernockỳ and Khudanpur2010). Sundermeyer et al. (Reference Sundermeyer, Oparin, Gauvain, Freiberg, Schluter and Ney2013) showed that language models based on recurrent architecture achieve better performance than language models based on feedforward architecture. This is due to the fact that RNNs do not use a context of limited size. This property led us to use, in our experiments, the Elman recurrent architecture (Elman Reference Elman1990), in which recurrent connections occur at the hidden layer level.
We consider in this work two Elman RNN architectures (see Figure 2): URNN and BRNN. In addition, to be able to include low-level linguistic information in our architecture designed for more complex sequence labeling tasks, we propose three new RNN variants to take into account external information (such as POS tags) for a more complex task: multilingual SST.
High-level schema of RNN used in our work unidirectional (left schema) and bidirectional RNN (right schema).
3.2.1 Unidirectional RNN
In the Elman URNN, the recurrent connection is a loop at the hidden layer level. This connection allows URNN to use at the current time step hidden layer’s states of previous time steps. In other words, the hidden layer of URNN represents all previous history and not just n − 1 previous inputs, thus the model can theoretically represent long context.
The architecture of the URNN considered in this work is shown in Figure 2. In this architecture, we have four layers: input layer, forward (also called recurrent or context layer), compression hidden layer and output layer. All neurons of the input layer are connected to every neuron of forward layer by weight matrix I F and R F, the weight matrix H F connects all neurons of the forward layer to every neuron of compression layer and all neurons of the compression layer are connected to every neuron of output layer by weight matrix O.
The input layer consists of a vector w(t) that represents the current word w t in our common words representation (all input neurons corresponding to current word w t are set to 0 except those that correspond to bisentences containing w t, which are set to 1), and of vector f(t − 1) that represents output values in the forward layer from the previous time step. We name f(t) and c(t) the current time step hidden layers (our preliminary experiments have shown better performance using these two hidden layers instead of one hidden layer), with variable sizes (usually 80–1,024 neurons) and sigmoid activation function. These hidden layers represent our common language-independent feature space and inherently capture word alignment information. The output layer y(t), given the input w(t) and f(t − 1) is computed with the following steps:
$$\begin{equation}
f(t)=\Sigma (w(t) . I_F(t) + f(t-1). R_F(t))
\end{equation}$$
$$\begin{equation}
c(t)=\Sigma (f(t) . H_F(t))
\end{equation}$$
$$\begin{equation}
y(t)=\Gamma (c(t) . O(t))
\end{equation}$$
For many sequence labeling tasks, it is beneficial to have access to future in addition to the past context. So, it can be argued that our URNN is not optimal for sequence labeling, since the network ignores future context and tries to optimize the output prediction given the previous context only. This URNN is thus penalized compared with our baseline projection based on TnT (Brants Reference Brants2000) which considers both left and right contexts. To overcome the limitations of URNN, a simple extension of the URNN architecture — namely, BRNN (Schuster and Paliwal Reference Schuster and Paliwal1997) — is used to ensure that context at previous and future time steps will be considered.
3.2.2 Bidirectional RNN
An unfolded BRNN architecture is given in Figure 2. The basic idea of BRNN is to present each training sequence forwards and backwards to two separate recurrent hidden layers (forward and backward hidden layers) and then somehow merge the results. This structure provides the compression and the output layers with complete past and future context for every point in the input sequence. Note that without the backward layer, this structure simplifies to a URNN.
The inference procedure for the unfolded bidirectional network for one time slice 1 ⩽ t ⩽ T to determine all predicted outputs, was described in the reference paper on BRNN (Schuster and Paliwal Reference Schuster and Paliwal1997), and can be summarized as follows:
• Compute the backward hidden layers states from t = T to t = 1 (equation (5)):
(5)
$$\begin{equation}
b(t)=\Sigma (w(t) . I_B(t) + b(t+1). R_B(t))
\end{equation}$$
• Use the backward hidden layers states to compute the compression layers c(t) (equation (6)) and output layers y(t) (equation (4)) from t = 1 to t = T:
(6)
$$\begin{equation}
c(t)=\Sigma (f(t) . H_F(t) + b(t). R_b(t))
\end{equation}$$
3.2.3 Network training
The first step in our approach is to train the neural network, given a parallel corpus (training corpus) and a validation corpus (different from train data) in the source language. In typical applications, the source language is a resource-rich language (which already has an efficient tagger or manually tagged resources). Our RNN models are trained by stochastic gradient descent using usual back-propagation and back-propagation through time algorithms (Rumelhart, Hinton and Williams Reference Rumelhart, Hinton and Williams1985). We learn our RNN models with an iterative process on the tagged source side of the parallel corpus. After each epoch (iteration) in training, validation data is used to compute per-token accuracy of the model. After that, if the per-token accuracy increases, training continues in the new epoch. Otherwise, the learning rate is halved at the start of the new epoch. Eventually, if the per-token accuracy does not increase anymore, training is stopped to prevent over-fitting. Generally, convergence takes 5–10 epochs, starting with a learning rate α = 0.1.
The second step consists in using the trained model as a target language tagger (using our common vector representation). It is important to note that if we train on a multilingual parallel corpus with N languages (N > 2), the same trained model will be able to tag all the N languages.
Hence, our approach assumes that the word order in both source and target languages are similar. In some languages such as English and French, word order for contexts containing nouns could be reversed most of the time. For example, the compound word the European Commission would be translated into la Commission Européenne. In order to deal with the word order constraints, we also combine the RNN model with the baseline cross-lingual projection model in our experiments. In case of syntactic divergence between source and target languages, another idea would be to pre-process text in target language to better match the text in source language. This could be done using linguistically motivated rules automatically extracted from typological databases such as the World Atlas of Language StructuresFootnote 3. This latter possibility is part of our future work.
3.3 Dealing with out-of-vocabulary words
For the words unseen in the initial parallel corpus (OOV words), their vector representation is a vector of zero values. Consequently, during testing, the RNN model will use only the context information to tag the OOV words found in the test corpus. To deal with these types of OOV words,Footnote 4 we use the CBOW model of Mikolov et al. (Reference Mikolov, Sutskever, Chen, Corrado and Dean2013) to replace each OOV word by its closest known word in the current OOV word context (to achieve this, we compute cosine similarity between word embeddings). Once the closest word is found, its common vector representation is used (instead of the vector of zero values) at the input of the RNN.
3.4 Combining simple cross-lingual projection and RNN models
Since the direct transfer model M1 and RNN model M2 use different strategies for tagging (TnT is based on Markov models, while RNN is a neural network), we assume that these two models can be complementary. To keep the benefits of each approach, we explore how to combine them with linear interpolation. Formally, the probability to tag a given word w is computed as
$$\begin{equation}
P_{M12}(t|w)=(\mu P_{M1}(t|w,C_{M1})+(1-\mu )P_{M2}(t|w,C_{M2}))
\end{equation}$$
$$\begin{equation}
f(w)= \arg \max _{t}(P_{M12}(t|w))
\end{equation}$$
4 Multilingual part-of-speech tagging-based RNN
We applied our method to build RNN POS taggers for four target languages — French, German, Greek and Spanish — with English as the source language. No annotation is available for the target languages (we suppose these languages are under-resourced).
In order to determine the effectiveness of our common words representation described in Section 3.1, we also investigated the use of state-of-the-art bilingual word embeddings (using MultiVec ToolkitFootnote 5 (Bérard et al. Reference Bérard, Servan, Pietquin and Besacier2016)) as input to our RNN tagger.
4.1 Light supervision (adaptation) of RNN model
While the unsupervised RNN (no-supervision on the target languages) model described in the previous section has not seen any annotated data in the target language, we also consider the use of a small amount of adaptation data (manually annotated in target language), relatively to the huge number of sentences used to learn supervised tagger, in order to capture target language specificity. In that case, the RNN model is adapted in a light supervision manner, using a small monolingual target corpus (manually annotated — the exact size used for adaptation is provided in Table 5 –) and the common vector representation of words (extracted from the initial parallel corpus). We plan to experiment with a gradually increasing amount of annotated data in the target language (from 100 to 10,000 utterances). Such an approach is particularly suited for an iterative scenario where a user would post-edit (correct) the unsupervised POS tagger output in order to produce rapidly adaptation data in the target language (light supervision).
4.2 Data
For French as a target language, we used a train set of 10,000 parallel sentences, a validation set of 1,000 English sentences and a test set of 1,000 French sentences, all extracted from the ARCADE II English–French corpus (Veronis et al. Reference Veronis, Hamon, Ayache, Belmouhoub, Kraif, Laurent, Nguyen, Semmar, Stuck and Zaghouani2008). The test set is tagged with the French TreeTagger (Schmid Reference Schmid1995) and then manually checked.
For German, Greek and Spanish as a target language, we used training and validation data extracted from the Europarl corpus (Koehn Reference Koehn2005) which are a subset of the training data used in Das and Petrov (Reference Das and Petrov2011) and Duong et al. (Reference Duong, Cook, Bird and Pecina2013). This choice allows us to compare our results with those of Das and Petrov (Reference Das and Petrov2011), Duong et al. (Reference Duong, Cook, Bird and Pecina2013) and Gouws and Søgaard (Reference Gouws and Søgaard2015). The train data set contains 65,000 bisentences, a validation set of 10,000 bisentences is also available. For testing, we use the same test corpora as (Das and Petrov Reference Das and Petrov2011), (Duong et al. Reference Duong, Cook, Bird and Pecina2013) and (Gouws and Søgaard Reference Gouws and Søgaard2015) (bisentences from CoNLL shared tasks on dependency parsing (Buchholz and Marsi Reference Buchholz and Marsi2006)). The evaluation metric (per-token accuracy) and the universal tagset defined by Petrov, Das and McDonald (Reference Petrov, Das and McDonald2012) (twelve tags common for most languages) are used for evaluation. This universal POS tagset defines the following twelve POS tags, which exist in similar form in most languages: NOUN (nouns), VERB (verbs), ADJ (adjectives), ADV (adverbs), PRON (pronouns), DET (determiners and articles), ADP (prepositions and postpositions), NUM (numerals), CONJ (conjunctions), PRT (particles), ‘.’ (punctuation marks) and X (a catch-all for other categories such as abbreviations or foreign words).
For training, the English (source) sides of the training corpora (ARCADE II and Europarl) and of the validation corpora are tagged with the English TreeTagger toolkit. Using the matching provided by Petrov et al. (Reference Petrov, Das and McDonald2012), we map TreeTagger tagset as well as tagset found in CoNLL shared task data to the common universal tagset.
In order to build our baseline unsupervised tagger (based on a Direct Cross-lingual Projection — see Section 2.2.1), we also tag the tagger side of the training corpus, with tags projected from English side through word-alignments established by GIZA++ (alignment models were trained using IBM Models 1–5). After the tags projection, a target language POS tagger based on TnT approach (Brants Reference Brants2000) is trained.
The combined model is built for each considered language using cross-validation on the test corpus. First, the test corpus is split into two equal parts and on each part, we estimate the interpolation parameter μ (equation (7)) which maximizes the per-token accuracy score. Then each part of the test corpus is tagged using the combined model tuned on the other part, and vice versa (standard cross-validation procedure).
We trained MultiVec bilingual word embeddings on the parallel Europarl corpus between English and each of the target languages considered.
4.3 Qualitative evaluation
As we have previously stated, we propose to represent the bilingual word information intrinsically in our RNN model by letting the RNN learn its internal words representation (bilingual word embeddings) using our common words representation (Section 3.1). We present here a qualitative evaluation of English–French bilingual word embeddings learned from the ARCADE II English–French corpus. We jointly visualized the RNN bilingual words representation of the most frequent words in English and French using the t-SNE (Van der Maaten and Hinton Reference Van der Maaten and Hinton2008) visualization tool. The embeddings are shown in Figure 3. We observe that the visualization of the embeddings of English words and their French translations are close and even in some cases they are overlapped. Therefore, it seems that our RNN model is able to represent bilingual word information.
A t-SNE visualization of RNN vector (word embeddings) representations of the same few frequent English and French words.
4.4 Results and discussion
4.4.1 Results
Table 1 reports the results obtained for the unsupervised POS tagging. We note that the POS tagger based on BRNN has better performance than URNN, which means that both past and future contexts help select the correct tag.
Token-level POS tagging accuracy for two baseline methods –B/L– (simple projection and URNN using MultiVec bilingual word embeddings as input), RNNFootnote 6 –Our–, Projection+RNN –Comb.– and state-of-the-art methods –SOTA– (Das & Petrov (Reference Das and Petrov2011), Duong et al. (Reference Duong, Cook, Bird and Pecina2013) and Gouws & Søgaard (2015))
The OOV rate of the test corpora is around twenty per cent.
Table 1 also shows the performance before and after performing our procedure for handling OOVs in BRNNs. It is shown that after replacing OOVs by the closest words using CBOW, the tagging accuracy significantly increases (McNemar’s Test p < 0.005 on French, German and Greek).
As shown in the same table, our RNN models accuracy is close to that of the simple projection tagger. It achieves comparable results to Das and Petrov (Reference Das and Petrov2011) and Duong et al. (Reference Duong, Cook, Bird and Pecina2013) (who used the full Europarl corpus while we use only a 65,000 subset of it) and to Gouws and Søgaard (Reference Gouws and Søgaard2015) (who used extra resources such as Wiktionary and Wikipedia). Interestingly, RNN models learned using our common words representation (Section 3.1) seem to perform significantly better than RNN models using MultiVec bilingual word embeddings.
It is also important to note that only one single URNN and BRNN tagger applies to German, Greek and Spanish, so this is an authentic multilingual POS tagger. Finally, as for several other NLP tasks such as language modeling (where standard and NN-based models can be combined in order to obtain optimal results), the combination of standard and RNN-based approaches (Projection+_) seems necessary to further optimize POS tagging accuracies.
In order to know in what respect considering right context (bidirectional architecture) improves RNN model accuracy, we analyzed the French test corpus. In the example provided in Table 2, future time steps (tags of the words — de la Commission Européenne –) information helps to resolve the French word Finances tag ambiguity. We hypothesize that the context information is better represented in BRNN.
Effect of bidirectional architecture
In case of word order divergence, we observed that our model can still handle some divergence, notably for the following cases:
• Obviously, if the current tag word is unambiguous (case of ADJ and NOUN order from English to French — see Table 3), then the context (RNN history) information has no effect.
• When the context is erroneous (due to the fact that word order for the target test corpus is different from the source training corpus), the right word tag can be recovered using the combination (RNN+Cross-lingual projection — see Table 4).
Word order divergence — unambiguous tag word –.
Word order divergence — ambiguous tag word -
To deal with the word order divergence limitation, we also propose light supervision (adaptation) of RNN model. In Table 5, we report the results obtained after adaptation with a gradually increasing amount of target language data annotated (from 100 to 10,000 utterances).
Lightly supervised model: effect of German adaptation corpus (manually annotated) size on unsupervised RNN model (unsupervised RNN + DE adaptation trained on EN Europarl and adapted to German)
Contrastive experiments with German supervised POS taggers using same data (RNN, TnT and RNN+TnT). 0 means no German corpus used during training.
We focus on German target language only. It is compared with two supervised approaches based on TnT or RNN. The supervised approaches are trained on the adaptation data only. For supervised RNN, it is important to mention that the input vector representation has a different dimension for each amount of adaptation data (we recall that the vector representation is V wi, i = 1, . . ., N, where N is the number of sentences, and N is growing from 100 to 10,000). The results show that our adaptation, on top of the unsupervised RNN is efficient in very low resource settings (<1, 000 target language utterances). When more data is available (>1, 000 utterances), the supervised approaches start to improve (but RNN and TnT are still complementary since their combination improves the tag accuracy).
Figure 4 details the behavior of the same methods for OOV words. We clearly see the limitation of the unsupervised URNN + adaptation to handle OOV words, since the input vector representation is the same (comes from the initial parallel corpus) and does not evolve as more German adaptation data is available.
Accuracy on OOV according to German training corpus size for unsupervised URNN + DE adaptation, supervised URNN DE and supervised TNT DE.
In order to know the impact of adaptation, we analyzed the German test corpus. The example provided in Table 6, shows that RNN better handles tag ambiguity with adaptation: In the unsupervised RNN model, the word kandidiert is tagged as a noun (NOUN), whereas it is a verb (VERB) in this particular context.
Effect of adaptation on German data
5 Multilingual super sense tagging-based RNN
Our RNN model is applied to a more complex task: multilingual SST. In order to measure the impact of the parallel corpus quality on our method, we also learn our SST models using the multilingual parallel corpus MultiSemCor (MSC) which is the result of manual or automatic translation of SemCor from English into Italian and French.
5.1 Super sense tagset
SST is an NLP task that consists of annotating each significant entity in a text, within a general semantic taxonomy defined by the WordNet (Fellbaum Reference Fellbaum1998) lexicographer classes (called super-senses). In Ciaramita and Altun (Reference Ciaramita and Altun2006), SST was defined as a task half-way between named entity recognition and Word Sense Disambiguation (WSD): It is an extension of NER, since it uses a larger set of semantic categories, and it is an easier and more practical task with respect to WSD, that deals with very specific senses (the complete list of super-senses and a short description is shown in Table 7).
Nouns and verbs super sense labels (WordNet lexicographer classes), and short description (from the Wordnet documentation)
5.2 RNN variants
We propose three new RNN variants to take into account low-level (POS) information in a higher level (SST) annotation task. The question addressed here is at which layer of the RNN should this low-level information be included to improve SST performance? As specified in Figure 5, the POS information can be introduced either at input layer or at forward layer (forward and backward layers for BRNN) or at compression layer. In all these RNN variants, the POS of the current word is also represented with a vector (POS(t)). Its dimension corresponds to the number of POS tags in the tagset (universal tagset of Petrov et al. (Reference Petrov, Das and McDonald2012) is used). We propose one hot vector representation where only one value is set to 1 and corresponds to the index of current tag (all other values are 0).
URNN variants with POS information at three levels: (a) input layer, (b) forward layer, (c) compression layer.
5.3 Data
SemCor. The SemCor (Miller et al. Reference Miller, Leacock, Tengi and Bunker1993)Footnote 7 is a subset of the Brown Corpus (Kucera and Francis Reference Kucera and Francis1979) labeled with the WordNet (Fellbaum Reference Fellbaum1998) senses. In Table 8, we provide an example extracted from SemCor.
An example of SemCor (v.3.0) in SGML format (standard generalized markup language)
MultiSemCor. The English–Italian MultiSemcor (MSC-IT-1) corpus is a manual translation of the English SemCor to Italian (Bentivogli, Forner and Pianta Reference Bentivogli, Forner and Pianta2004)Footnote 8. As we already mentioned, we are also interested in measuring the impact of the parallel corpus quality on our method. For this, we use two translation systems: (a) Google Translate to translate the English SemCor to Italian (MSC-IT-2) and French (MSC-FR-2); (b) LIG machine translation phrase based system, off-the-shelf SMT system based on Moses (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007) taken from LIG (Besacier et al. Reference Besacier, Lecouteux, Azouzi and Luong2012) to translate the English SemCor to French (MSC-FR-1)Footnote 9 (Salah et al. Reference Salah, Blanchon, Zrigui and Schwab2016).
Training corpus. The SemCor was labeled with the WordNet synsets. However, because we train models for SST, we convert SemCor synsets annotations to super senses. We train our models using the four different versions of MSC (MSC-IT-1,2 - MSC-FR-1,2), with modified Semcor on source side.
Test Corpus. To evaluate our models, we used the SemEval 2013 Task 12 (multilingual WSD) (Navigli, Jurgens and Vannella Reference Navigli, Jurgens and Vannella2013) test corpora, which are available in five languages (English, French, German, Spanish and Italian) and labeled with BabelNet (Navigli and Ponzetto Reference Navigli and Ponzetto2012) senses. We map BabelNet senses to WordNet synsets, then WordNet synsets are mapped to super senses (see Tabel 9). We used the sense mapping between Princeton WordNet 3.0 and BabelNet 1.1.1 provided by Nasiruddin et al. (Reference Nasiruddin, Tchechmedjiev, Blanchon and Schwab015).
WordNet synset, BabelNet sense and super sense mapping
5.3.1 SST systems evaluated
The goals of our SST experiments are the following: First, to investigate the effectiveness of using POS information to build multilingual super sense tagger, second to measure the impact of the parallel corpus quality (manual or automatic translation) on our RNN models (URNN, BRNN and our proposed variants). To summarize, we build four super sense taggers based on direct cross-lingual projection (see Section 2.2.1) using four versions of MSC (MSC-IT-1, MSC-IT-2, MSC-FR-1 and MSC-FR-2) described above. Then we use the same four versions to train our multilingual SST models based on URNN and BRNN. For learning our multilingual SST models based on RNN variants proposed in Section 5.1, we also tag SemCor using TreeTagger (POS tagger proposed by Schmid (Reference Schmid1995)).
5.4 Results and discussion
5.4.1 Results
Our models are evaluated on SemEval 2013 Task 12 test corpora. Results are directly comparable with those of systems which participated to this evaluation campaign. We report two SemEval 2013 (unsupervised) system results for comparison:
• MFS Semeval 2013: The most frequent sense is the baseline provided by SemEval 2013 for Task 12, this system is a strong baseline, which is obtained by using an external resource (the WordNet most frequent sense).
• GETALP: A fully unsupervised WSD system proposed by Schwab et al. (Reference Schwab, Goulian, Tchechmedjiev and Blanchon2012) based on Ant-Colony algorithm.
The DAEBAK! (Manion and Sainudiin Reference Manion and Sainudiin2013) and the UMCC-DLSI systems (Gutiérrez Vázquez et al. Reference Gutiérrez Vázquez, Fernández Orquín, Montoyo Guijarro and Vázquez Pérez2011) have also participated to SemEval 2013 Task 12. However, they use a supervised approach.Footnote 10
Table 10 shows the results obtained by our RNN models and by two SemEval 2013 WSD systems. URNN-POS-X and BRNN-POS-X refer to our RNN variants: In means input layer, H1 means first hidden layer and H2 means second hidden layer. We achieve the best performance on Italian using MSC-IT-1 clean corpus, while noisy training corpus degrades SST performance. The best results are obtained with combination of simple projection and RNN which confirms (as for POS tagging) that both approaches are complementary.
Super sense tagging (SST) accuracy for simple Projection, RNN and their combination
We also observe that the RNN approach seems more robust than simple projection on noisy corpora. This is probably due to the fact that no word alignments are required in our cross language RNN. Finally, BRNN-POS-H2-OOV achieves the best performance, which shows that the integration of POS information in RNN models and dealing with OOV words are useful to build efficient multilingual super senses taggers. Finally, it is worth mentioning that integrating low-level (POS) information lately (last hidden layer) seems to be the best option in our case.
The example in Table 11, shows that the integration of POS information in our neural SST tagger (BRNN-POS-H2) helps to resolve the French word conseiller (adviser in English) SST tag ambiguity, tagged by BRNN (model without the integration of POS information) as verb.communication, whereas it is an noun.person
Improved tagged example for French target language
6 Conclusion
In this article, we have described an approach based on RNNs to induce multilingual NLP tools. In particular, we have used URNN and BRNN architectures on two NLP tasks: POS tagging and SST. We have also proposed and experimented new RNN variants which take into account low-level information (POS) in a higher level task (SST). Our approach needs only parallel corpora without using word alignment information and it does not assume any knowledge about target languages (however, one requirement is that the two languages (source and target) are not too syntactically divergent).
We first empirically evaluated the proposed approach on two unsupervised POS taggers: (1) English–French bilingual POS tagger; and (2) English–German–Greek–Spanish multilingual POS tagger. The performance of the second model is close to state-of-the-art with only a subset (65, 000 sentences) of Europarl corpus used (when state-of-the-art approaches use the whole Europarl corpus). In addition, when a small amount of supervised data is available, the experimental results demonstrated the effectiveness of our approach in a weakly supervised context (well adapted to low resource scenarios).
Second, in order to demonstrate the genericity of our approach we have applied our RNN model on a more complex task: multilingual SST. We have investigated two pairs of languages: English–Italian and English–French which allowed us to measure the impact of the parallel corpus quality on the results of our approach. For the English–Italian pair, we have used two MSC parallel corpora: The first results from English–Italian manual translation and the second was constructed using English–Italian automatic translation. Concerning the English–French pair, we have used two MSC parallel corpora generated with two Statistical Machine Translation tools. The obtained results (on SST) showed that our approach seems more robust than state-of-the-art methods (based on word alignment) on noisy corpora. Additionally, we have observed that the integration of POS information in RNN models is useful to build efficient multilingual super senses taggers.
Finally, we have also proposed a method to deal with OOV words which has led to improvements for both tasks (multilingual POS tagging and multilingual SST).
We then conclude that our approach is generic and has the following advantages: (a) it uses a language-independent word representation (based only on word co-occurrences in a parallel corpus), (b) it provides authentic multilingual taggers (one tagger for N languages) and (c) it can be easily adapted to a new target language (when a small amount of supervised data is available).
For future work, we plan to apply multitask learning to build systems that simultaneously perform syntactic and semantic analysis. We also plan, on the one hand, to exploit multiple source languages to improve our RNN taggers with our common (multilingual) vector representation (this is possible with a multi-parallel corpus and our common words representation), and on the other hand, to apply our approach to truly under-resourced languages as defined in Besacier et al. (Reference Besacier, Barnard, Karpov and Schultz2014). In the case of strong differences in word order between source and target languages, and based on Aufrant, Wisniewski and Yvon (Reference Aufrant, Wisniewski and Yvon2016) works, we plan to use pre-processing based on features extracted from large typological databases such as World Atlas of Language StructuresFootnote 11 for instance.
